Title: The Quiet Prompt: Erasing Ineffable Styles from Diffusion via Concept Embedding
Primary area: generative models
Keywords: Text-to-image generation, Generative AI, Diffusion model, Concept erasure, Style removal

Abstract: Recent text-to-image diffusion models (DMs) depend on vast, uncurated datasets that often include copyrighted or personal images, risking the generation of unwanted content. Fully curating these datasets is costly, and retraining the entire DMs from scratch is impractical. To address this, concept erasure methods have emerged to suppress undesirable outputs in pre-trained models without additional training. However, because these methods solely rely on text prompts for erasure, they are largely limited to concepts that can be explicitly described in words. This raises an important question: how can we erase highly personalized or visually nuanced concepts that are difficult to articulate verbally? To tackle this issue, we leverage fewer than ten reference images to derive a unified concept embedding that seamlessly captures even ineffable styles or content. During inference, the concept embedding serves as negative guidance—facilitating both the precise removal of complex visual moods and comprehensive erasure of target concepts without verbose prompts. Our method consists of two stages: (1) Concept embedding generation, which compresses the color, texture, and morphology of an unwanted style into a single latent vector; and (2) Concept‐aware negative guidance, which uses the extracted concept embedding as a semantic anchor to steer the diffusion process away from the undesirable concept without requiring model retraining. Building on these components, we propose Quiet Prompt (QuP), a method that simultaneously delivers consistent style removal—effectively erasing complex styles without verbose text—and comprehensive concept removal, allowing broad specification of target concepts without enumerating multiple synonyms or descriptions. Extensive evaluations on idiosyncratic and historical art styles, as well as object removal tasks, demonstrate that our method achieves superior performance in both unwanted style suppression—measured by a novel style score—and content preservation, outperforming state-of-the-art text-based baselines.

TL;DR: We propose a novel method that effectively removes a target style using only a few reference images.