An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Authors: Rinon Gal
Yuval Alaluf
Yuval Atzmon
Or Patashnik
Amit Haim Bermano
Gal Chechik
Daniel Cohen-Or
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Personalized generation
text-to-image
inversion

Abstract: Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes.
In other words, we ask: how can we use language-guided models to turn *our* cat into a painting, or imagine a new product based on *our* favorite toy? 
Here we present a simple approach that allows such creative freedom. Using only $3$-$5$ images of a user-provided concept, like an object or a style, we learn to represent it through new ``words" in the embedding space of a frozen text-to-image model.
These ``words" can be composed into natural language sentences, guiding *personalized* creation in an intuitive way.
Notably, we find evidence that a *single* word embedding is sufficient for capturing unique and varied concepts. 
We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available.

Published as a conference paper at ICLR 2023
AN IMAGE IS WORTH ONE WORD: PERSONALIZING TEXT-TO-IMAGE GENERATION USING TEXTUAL IN- VERSION
Rinon Gal 122
Yuval Alaluf
Yuval Atzmon?
Or Patashnik'
Amit H. Bermano
Gal Chechik²
Daniel Cohen-Or!
Tel-Aviv University
NVIDIA
A ABSTRACT
Text-to-image models offer unprecedented freedom to guide creation through nat- ural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them use language-
roles and novel scenes. In other words, ask: how guided models to turn our cat into a painting, or imagine a new product based our favorite toy? Here we present simple approach that allows such cre- ative freedom. Using only 3-5 images of a user-provided concept, like an object or style, we learn to represent through new "words" in the embedding space of frozen text-to- text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that single word embedding is sufficient for capturing unique and varied concepts. We compare our approach demonstrate that it can more faithfully portray the concepts across range of ap- plications and tasks. Code, data and new words are available at our project page
wide range baselines, and
Inpul samples
(nvert "S.
oAn painting Su"
"App loo of S."
T3moo Be sancc poi d Su""
"Ondhe S+"
"Sa ncon lights
Input samples Knvert "S."
PPintieg fahing cE boa""
"A Sa. backpeck"
"Baskay ancef S.*
Inndioos
"Rosal RinE S
Figure (left) We find new pseudo-words the embedding space of pre-trained text-to-image models which describe specific concepts. (right) These pseudo-words are composed into new sen- tences, placing our targets in new scenes, changing their style or ingraining them into new products.
INTRODUCTION
Nichol et al., 2021 Yu et al.) 2022; Saharia al. (022) have demonstrated an unprecedented capability to reason over natural language descriptions. They allow users to synthesize novel scenes with unseen com- positions and produce vivid pictures in a myriad of styles. These tools have been used for artistic creation, sources of inspiration, and even to design
Large-scale text-to- eox-to--maae models Rombach et al. 2021 Ramesh et al. 2021 2022
physical products (Yacoubian) 2022). Their use, however, is constrained by the user's ability to describe the desired target through text. One can then ask: How could instruct such models mimic the likeness of specific object? How could we ask them to craft a novel scene containing a cherished childhood toy? Or to pull our child's drawing from its place the fridge, and turn it into artistic showpiece?
Introducing new concepts into large scale models is often difficult. Re-training model with an expanded dataset for each concept is prohibitively expensive, and fine-tuning few exam-

---

Published as a conference paper at ICLR 2023
ples typically leads to catastrophic forgetting (Ding et al. 2022: Li et al. 2022) More measured approaches freeze the model and train transformation modules to adapt its output when faced with new concepts (Zhou et al., 2021 Gao et al.) 2021 Skantze & Willemsen 2022). However, these ap- proaches are still prone to forgetting prior knowledger or face difficulties in accessing it concurrently
with newly learned concepts (Kumar et al. 2022 Cohen et al. 2022).
We propose to overcome these challenges by finding new words in the textual embedding space of pre-trained text-to-image models. We consider the first stage of the text encoding process (Figure 2 Here, an input string is first converted to a set of tokens. Each token is then replaced with its own embedding vector, and these vectors are fed through the downstream model. Our goal is to find new embedding vectors that represent new, specific concepts.
We represent new embedding vector with a new pseudo-word (Rathvon 2004) which we denote by S.. This pseudo-wort is then treated like any other word, and can be used to compose novel textual queries for the generative models. One can therefore ask for "a photograph of S. on the beach", "an oil painting of S. hanging on the wall", or even compose two concepts, such as "a drawing of S! in the style of 5 Importantly, this process leaves the generative model untouched. In doing so, we retain the rich textual understanding and generalization capabilities that are typically lost when fine-tuning vision and language models on new tasks.
To find these aseudo-words. to-image model and a small (3-5) image set depicting the concept. We aim to find a word embedding, so that prompts of the form "A photo of S." will lead to the reconstruction of images from our set. This embedding is found using an optimization process, which we refer to as "Textual Inversion".
frame the task one of inversion. We are given a pre-trained text-
We further investigate series of extensions based tools typically used in Generative Adversarial Network (GAN) inversion. Our analysis reveals that, while some core principles remain, applying the prior art in naïve way may harm performance.
We demonstrate the effectiveness of our approach over a wide range of concepts and prompts, show- ing that it can inject unique objects into new scenes, transform them across different styles, transfer poses, diminish biases, and even imagine new products.
In summary, our contributions are as follows:
We introduce the task of personalized text-to-image generation, where we synthesize novel scenes of user-provided concepts guided by natural language instructions.
We present the idea of "Textual Inversions" in the context of generative models. Here the goal is to find new pseudo-words in the embedding space of a text encoder that can capture both high-level semanties and fine visual details.
We conduct a prelminary analysis of the properties of the textual embedding space.
RELATED WORK
Text-guided synthesis.
Text-guided image synthesis has been widely studied in the context of
GANs (Goodfellow al., 2014). Typically, conditional model is trained reproduce samples
from paired image-caption datasets (Zhu et al. 2019 Tao et al.] 2020 by leveraging attention mech-
anisms (Xu et al. 2018) or cross-modal contrastive approaches (Zhang et al. 2021 Ye al. 2021).
More recently, impressive visual results were achieved using large scale regressgees (Ramesh
ltt al.
2021 Yu al. 2022) or diffusion models (Ramesh al. 2022] Saharia al. 2022] Nichol et al.
2021 Rombach et al.] 2021). Alternatively. test-time optimization can be used to explore
the latent space of pre-trained generators (Crowson et al. 2022, Murdock] 2021 Crowson 202
Typically, by maximizing a text-to-image similarity score derived from CLIP (Radford et al. 202
Moving beyond pure generation, large body of work explores the use of text-based interfaces for image editing (Patashnik et al., 202T Abdal et al., (201]: Avrahami et al. 2022b)) generator domain adaptation (Gal et 2021 Kim et al. 2022) and style transfer (Kwon& & Ye 202 Liu ct al. 2022).
Our approach builds the open-ended, conditional synthesis models. Rather than training new model from scratch, we show that we can expand a frozen model's vocabulary and introduce new pseudo-we pseudo-words that describe specific concepts.
2

---

Published as a conference paper at ICLR 2023
"A photo of S,"
Input Sample
Tokenizer
Pg
73
Pb
508
701
73
(+)
508
Uspe
Embedding Lookup
701
Pasa
Vroi
(+)
%
Text Transformer
Ca (v)
Text Encoder
Generator
Noised Sample
Figure 2: Outline of the text-embedding and inversion process. A string containing our placeholder word is first converted into tokens (i.e. word sub-word indices in dictionary). These tokens are converted to continuous vector representations (the "embeddings", v). Finally, the embedding vectors transformed into conditioning code coly) that guides the generation. We optimize the embedding vector u, associated with our pseudo-word S., using a reconstruction objective.
GAN inversion. Manipulating images with generative networks often requires one to find cor- responding latent representation of the given image, process referred to inversion (Zhu et al, 2016 Xia et al. 2021). the GAN literature, this inversion done through either an optimization-
based technique (Abdal al. 2019 2020; Zhu et al., 2020b Gu al. 2020)
using Tov et al. (021).
coder (Richardson et al. 2020
Zhu et al. 2020a Pidhorskyi et al.) 2020
concepts. En- coders face harsher generalization requirements, and would likely need to be trained on web-scale data to offer the same freedom. We further analyze embedding space in light of the GAN- inversion literature, outlining the core principles that remain and those that do not.
In work, we follow the optimization approach, it can better adapt
Diffusion-basee inversion. Diffusion inversion can be performed naively by adding noise to an image and then de-noising through the network. However, this process tends change the image content. Choi et al. (2021) improve inversion by conditioning the denoising process on low-pas pass filter data from the target image. (Dhariwal & Nichol (02T) demonstrate that the DDIM (Song al., 2020) sampling process can be inverted in closed-form manner, extracting latent noise map that will produce given real image. DALL-E (Ramesh et al. 2022) builds on this method and demonstrates that it can be used to facilitate cross image interpolations semantic editing.
Whereas the above works invert given image into the model's latent space, we invert a user- provided concept. Moreover, represent this concept as new pscudo-word in the model's vocab- ulary, allowing for more general and intuitive editing.
Personalization. Recent work in graphics aims to adapt models to better represent specific indi- viduals or objects. There, it is typical to delicately tune model to better reconstruct specific faces or scenes (Bau et al.
2019 Roich et al. 2021 Alaluf et al. 2021 Nitzan et al. 2022).
PALAVRA Cohen et 2022) identifies pseudo-wordd words in the textual embedding space of CLIP for
personalized retrieval and segmentation. However, their task and losses are discriminative, aiming to separate object from other candidates. As later show (Figure this approach fails to capture details required for plausible reconstructions or synthesis in new scenes.
METHOD
Our goal to enable language-guided generation of new, user-specified concepts. To do so, we aim to encode these concepts into intermediate representation of pre-trained text-to-image model. Ideally, this should be done manner that allows us to leverage the rich semantic and visual prior represented by such model, and use it to guide intuitive visual transformations of the concepts.

---

Published as a conference paper at ICLR 2023
It is natural to search for candidates for such a representation in the word-embedding stage of the text encoders typically employed by text-to-image -mage models. There, the discrete imput text is first converted into continuous vector representation that is amenable to direct optimization.
Prior work has shown that this embedding space is expressive enough to capture basic image seman- ties (Cohen et al. 2002 I'simoouelli al. 2021). However, these approaches leveraged contrastive or language-complen
objectives, neither of which require an in-depth visual understanding of the image. As we demonstrate Section 4 such methods fail to accurately capture the appearance of the concept, and attempting to employ them for synthesis leads to considerable visual corruption. Our goal is find pseudo-words that can guide generation, which is a visual task. As such, we propose to find them through visual reconstruction objective.
Below, we outline the core details of applying our approach specific class of generative models we then analyze a set of extensions
Latent Diffusion Models (Rombach et al. 2021). In Section 5 to this approach, motivated by GAN-inversion literature.
Latent Diffusion Models. We implement (LDMs) (Rombach et al., 2021), a recently introduced class of Denoising Diffusion Probabilistic Models (DDPMs) (Ho al. 2020) that operate in the latent space of an autoencoder.
method over Latent Diffusion Models
LDMs consist of two core components. First, an autoencoder is pre-trained on a large collection of images. An encoder & learns to map images Dr into spatial latent code 2 = Exx), regularized through either a KL-divergence loss or through vector quantization (Van Den Oord et al. 2017). The decoder D learns to map such latents back to images, such that D (E(x)) T.
The second component, a diffusion model, is trained to produce codes within the learned latent space. This diffusion model can be conditioned on class labels, segmentation masks, or even on the output of a jointly trained text-embedding model. Let ca (y) be a model that maps conditioning input y into a conditioning vector. The LDM loss is then given by:
[II c0( col)))
LLDM ==
where t is the time step, 2g is the latent noised to time t, is the unscaled noise sample, and eg is the denoising network. Intuitively, the objective here is to correctly remove the noise added to a latent representation of an image. While training, Co and eg jointly optimized to minimize the LDM loss. At inference time, a random noise tensor is sampled and itcratively denoised to produce new image latent, 20. Finally, this latent code transformed into an image x' D(z)).
We employ the publicly available 1.4 billion parameter text-to-image model of Rombach et al. (202T), which was pre-trained on the LAION-400M dataset (Schuhmann al.] (021). Here, co is realized through a BERT (Devlin et al.] 2018)) text encoder, with y being a text prompt. We next review the early stages of such text encoder, and our choice of inversion space.
Text embeddings. Typically, text encoder models begin with a text processing step (Figure 2 left). First, each word or sub-word in an input string is converted to token, which is an index in some pre-defined dictionary. Each token is then linked unique embedding vector, retrieved through an index-based lookup. These embedding vectors are typically learned as part of the text encoder Cg.
In our work, we choose this embedding space the target for inversion. Specifically, placeholder string, S., to represent the new concept we wish to learn. We intervene in the embedding process that the vector associated with the tokenized string is new, learned embedding v.. in essence "injecting" the concept into our vocabulary. We can then compose new sentences containing S., just as we would with any other word.
designate
Textual inversion. To find these new embeddings, we use a small set of images (typically 3- 5). which depict our target concept across multiple settings such as varied backgrounds or poses. We find v. through direct optimization, by minimizing the LDM loss of Equation (I) over images sampled from the small set. To condition the generation, we randomly sample neutral context texts, derived from the CLIP ImageNet templates (Radford al. (021). These contain prompts of the form "A photo of S," " "A rendition of S. etc. The list of templates is provided in the appendix.
4

---

Published as a conference paper at ICLR 2023
Skull Mug
Тeapot
Input samples
Object variations generated using our method, the CLIP-based reconstruction of DALLE-2 (Ramesh et al, 2022). and human captions of varying lengths.
Figure 3:
Ours
ploto
placo of
Human captions capture high- level semantics, but fail to con- vey fine details or ensure a con- sistent style. DALLE-2's im- age guidance fares better, but still falls short of capturing the unique details the subject. Our method generates varia- tions which are typically more faithful the original subject. These demonstrate an ability to capture and represent an object using a single pseudo-word
DALLE-2
(Image Inputs)
(Long Captions)
LDM
(Long Captions)
Sculpture of
with lears and leace with lovves
LDM (Short Captions)
muiiee standing on dull
colocfa tea luriin leaf priats
Our optimization goal then be defined as:
arg min Ex
[lle cel.
(2)
and is realized by re-using the same training scheme as the original LDM model, while keeping both co and ég fixed. Notably, this is reconstruction task. As such, expect it to motivate the learned embedding to capture fine visual details unique to the concept.
Implementation details, including syper-parameee choices, are provided in the appendix.
QUALITATIVE COMPARISONS AND APPLICATIONS
In the following section, we demonstrate range of applications enabled through Textual Inversions, and provide visual comparisons to the state-of-the of-too--fhtr- art and human-captioning baselines.
Image variations We begin by demonstrating our ability to capture an object using a single wordd In Figure caption and DALLE-2 guided by either a human caption or an image prompt. To gather captions, we provided annotators with four images of concept and asked them describe it in a manner that could allow an artist to recreate it. We collected five short (5. 12 words) and five long (< 30 words) captions per concept. Additional details are provided the appendix. Figure shows results generated with a randomly chosen caption for each setup.
compare method two baselines: LDM guided by human
As our results demonstrate, our method better captures the unique details of the concept. Human captioning typically captures the high-level semantics, but fails to cover finer details like color pat- terns (e.g. the teapot). some cases (e.g. the skull mug) the object itself may be difficult to describe through natural language. The image-guided DALLE-2 creates more appealing samples, but it still struggles with unique details of novel objects. Our method can successfully capture these finer de- tails. However, note that while our creations are more faithful to the source, they are still variations.
Text-guided synthesis Figures I and 4 show ability to compose novel scenes by incorporat- ing the learned pseudo-words into new conditioning texts. The frozen sot-tto-mmae model jointly reasons over both the new concepts and its large body of prior knowledge, bringing them together in a new creation. Importantly, despite the fact that our training goal was visual in nature, our pseudo- words still encapsulate semantic knowledge. For example, observe the bowl's ability (row 4) to
5

---

Published as a conference paper at ICLR 2023
Input samples
"Waereeker paintax
Unisy S bide"
DEATINMETAA
BUYY FLOM
Input samples
feauuitg S.
p hangiax the will"
"Au cntit
S,
Input samples
cadeev male"
boad
Sy mak
served
seEp.
depieting S
Figure 4: Additional text-guided personalized generation results. In each row, we show exemplars from the image set representing the concept, and novel compositions using the learned pseudo- woodd.
Clock
Mug
Input
samples
(VQUANCCLIP)
AVEA
VOGANiCLIP Image
Golded Diffedon (Tet Lows fiil Image)
Ours
"A plaso of S."
"A of S. bradh
"A of
'Elax beiding 3
"A plaso of S
of
"A plottod mout!
"Bhec beblin Sta
Figure 5: Comparisons alternative personalized creation approaches. Our model can more accu- rately preserve the subject, and can reason over both the novel embedding and the rest the caption.
contain other objects, or the ability to preserve the Furby's s bird-like head and crown while adapting his color to better match prompt (row 3). Additional results are provided in the appendix.
We further compare our method to several personalization baselines (Figure 5). We first consider the recent PALAVRA (Cohen et al. 2022) which encodes object sets into the textual embedding space of CLIP, using contrastive learning and cyclic consistency goals. We find a new pseudo- word using their approach and CLIP-Guidee Diffusion (Crowson) 202T). As second baseline, we apply the CLII--gided models of Crowson et al. and jointly minimize the CLIP-based distances both the training images and target text (VQGAN-CLIP) or by initializing the optimization with an image from our training set, following the recommendations Letts et al. (2021) (Guided Diffusion).
synthesize
images with VQGAN-CLIP (Crowson al. 2022) and

---

Published as a conference paper at ICLR 2023
Images produced by PALAVRA (rows 2, 3) contain elements from the target prompt (beach, moon), but fail to capture the concept and display considerable visual corruption. This is unsurprising, as PALAVRA trained with a discriminative goal. There, the model only needs to encode enough information to distinguish between two typical concepts. Moreover, this goal does not constrain the model to embedding vectors that be mapped outputs the natural image manifold. When using text-and-iman guided synthesis methods (rows 4, 5), results appear more natural and closer to the source image, but fail to generalize new texts. Moreover, these models use CLIP for test-time optimization and thus require expensive optimization for every new creation.
Style transfer A typical use case for text-guided synthesis is in artistic circles, where users draw upon the unique style of specific artist and apply it to new creations. We show that our model can also find pseudo-words representing specific, unknown style. To do we provide the model with a small set of images with shared style, and replace the training texts with prompts of the form: "A painting in the style of S,". Results are shown in Figure These demonstrate that our ability to capture concepts extends beyond simple object reconstructions and into more abstract ideas.
Iaput sacolees
TTh owri of Pins B.2
TAderrllee reedi ufle
"Paniing eF's Mlak Jark
TTime scporn Bisttll
Tide perted regats
Figure 6: The rextual-embedding space can represent more abstract concepts, including styles. Im- age credits: @ @QinniArt (top), @ David Revoy (bottom).
Additional applications In Appendix [A] we investigate a range of additional uses textual inver- sion, including: composing multiple learned concepts in single prompt, reducing gender and racial bias, and using our pseudo-words in downstream models built on LDM.
Image curation Results in this section are partially curated. For each prompt, we generated 16 candidates (six for DALLE-2) and manually selected the best result. Similar curation processes with larger batches are often employed in text-conditioned generation work (Avrahami et al. 2022b Ramesh et al. [0221] Yu al. 2022). This process can be largely automated by using CLIP rank images (Ramesh et al. 2021) We provide uncurated galleries of generated results in the appendix.
QUANTITATIVE ANALYSIS
Inversion into an uncharted latent space provides us with a range of possible design choices. Here, examine such choices, and through them we analyze the properties of our embedding space.
5.1 EVALUATION METRICS
To analyze the quality of our embeddings, we consider two fronts: reconstruction and prompt- adherence. First, we wish to gauge our ability to capture a target concept. We do so by considering semantic CLIP-space distances. Specifically, for each concept, we generate 64 images using the prompt: "A photo of S.". Our reconstruction score the average pair-wise CLIP-space cosine- similarity between the generated images and the images of the concept-specific training set.
Second, we evaluate our ability to modify the concepts using textual prompts. We produce a set of images using prompts of varying difficulty and settings. These include background modifications ("A photo of S, on the moon"), style changes ("An oil painting of S. "), and compositional relations ("Elmo holding S.")) ") For each prompt. we synthesize 64 samples using 50 DDIM steps, calculate the average CLIP embedding of the samples, and compute their cosine similarity with the CLIP embedding of the textual prompts, where we omit the placeholder S, (i.e. "A photo of on the moon"). Higher scores indicate more faithfulness to the prompt itself.
7

---

Published as a conference paper at ICLR 2023
5.2 EVALUATION SETUPS
Our evaluations span the following experimental setups:
Extended latent spaces Following Abdal et al. (2019), we consider an extended latent space, where S. is embedded into multiple learned vectors equivalent to using multiple pseudo-words. We consider two and three pseudo-words (denoted "2-word" and "3-word", respectively), aiming to alleviate the potential bottleneck of a single embedding. and enabling more accurate reconstructions.
Progressive extensions We follow Tov et al. (2021) and consider progressive multi-vector setup. We begin training with single embedding vector, introduce second vector following 2, 000 train- ing steps, and a third vector after 4, 000 steps. Here, we expect the network to initially focus on the core details, and then leverage the additional wseudo-words to capture increasingly finer details.
Regularization Tov et al. (202T) observed that codes in the latent space of a GAN have increased editability when they lie closer to codes observed during training. We investigate a similar scenario by introducing loss term to minimize the L2 distance between the learned embeddings, and the embedding of a coarse descriptor of the subject (e.g. "sculpture" and "cat" for images in Figure
Per-image tokens We further investigate a novel scheme where we introduce unique, per-image tokens into process. Let {z,) be the set of input images. We introduce both universal placeholder, S., and an additional placeholder unique to each image, [St associated with a unique embedding U,- We train with prompts of the form "A photo of S. with S..", where each image is matched to prompts containing its own, unique placeholder. We jointly optimize over both S. and {S: H= The intuition here is that the model should prefer encode shared information (i.e. the concept) in the shared code S., while relegating per-image details such as the background to S;.
Human captions In addition to the formance using the captions outlined in Section 4 Here, we simply replace the placeholder strings S. with the human captions, using both the short and long-caption setups.
setups, we compare to human- per-
Reference setups To provide intuition for the scale of the results, we add two reference baselines: (1) We consider the expected behavior from a model that perfectly replicates the concepts, but ignores the prompts. We do by using the training set itself as the "generated samples" ("Image only"). (2) We consider a model that aligns with the prompt, but ignores the personalized concept. We do so by dropping the concept's 's pseudo word from the evaluation prompts ("Prompt only").
Textual- Inversion Finally, we consider our own setup and further evaluate our model with an increased learning rate (2e-2, "High-LR") and a decreased learning rate (1e-4, "Low-LR").
Additional setups and experiments In appendix we consider two additional inversion setups: pivotal tuning approach (Roich et al., (021), where the model itself is optimized to improve construction, and DALLE-2 (Ramesh (Kamesh et (022)'s bipartite inversion process. We further analyze the effect of the image-set size, data diversity, and importance of prompt templates (appendix ( Finally, we evaluate our model's failure rate and coverage (appendix (B), our performance on domain- adaptation tasks (appendix D) and compare to recent single-image editing approaches (appendix H
5.3 RESULTS
Evaluation results summarized in Figure 7 a). We highlight four observations: (1) The semantic reconstruction quality of our method is comparable to sampling random images from the training set. (2) The single-word method considerably improves prompt-adherence over the multi-word baselines, at a minimal cost to reconstruction. These points outline the impressive flexibility of the textual-embedding space, showing that it can capture new concepts with only a single pseudo-word.
(3) Our baselines outline a reconstruction- reconstracioon-popp-aateeeeere closer to the true word distribution (e.g. due regularization, fewer pseudo-words, or lower learn- ing rate) can be more easily modified, but fail to capture the details of the target. Notably, our single-embedding model can be moved along this curve by simply changing the learning rate, offer- ing a user degree of control over this trade-off.
trade-off curve. Embeddings that lie
8

---

Published as a conference paper at ICLR 2023
3 3-nods
CLP image Simit sertrrrty (11
2110
Similarity Rarkinng
2.3
caplige Shart copdisss
033-
Short coptions
WAegiarizatinn
038
6.8
22t
cot
15
25
20 User Text Similarity Ranking (f)
Similarit (1)
(a)
(b)
Figure 7: Quantitative evaluation results. (a) CLIP similarity metrics. Textual inversion models (blue) outperform human captioning (red) on both image and text similarity. Our baseline model (light blue) represents an appealing point on the distortion-adherenn it by changing the learning rate. (b) User study results portray similar trade-off curve. Moreover, the CLIP-based results align with human preference. Error bars are 95% confidence intervals.
curve, and can be moved along
(4) The use of human descriptions for the concepts not only fails to capture their likeness, but also leads to diminished prompt-adherence. We hypothesize that this is tied to the selective-similarity property outlined in (nfassss et al. (2022), where sssion-and-languase
models tend focus on a subset of the semantically meaningful tokens in prompt. Long captions increase the chance of the model ignoring our desired setting, focusing only on the object description itself. Our model, meanwhile, uses only single token and thus minimizes this risk.
Finally, while our reconstruction scores are on par with those of randomly sampled real images, we reiterate that our metrics compare semantic similarity using CLIP. This metric is less sensitive to shape preservation. On this front, there remains more to be done.
5.4 HUMAN EVALUATIONS
We further evaluate the models with a user study. We created two questionnaires. In the first, users were provided with four images from concept's training set, and asked to rank the results produced by five models according to their similarity these images. In the second, users were provided with text describing an image context ("A photo the beach") and asked to rank images produced by the same models according to their similarity to the text. We used the same target concepts and prompts as the CLIP-based evaluation and collected total of 600 responses to each questionnaire. Results are shown in Figure b)). See the appendix for more details on this experiment.
The user-study results align with the CLIP-based metrics and demonstrate similar reconstruction- tradcoff. Moreover, they outline the same limitations of human-based captioning when attempting to reproduce concept, as well when modifying it.
CONCLUSIONS
We introduced the task of personalized, language-guided generation, where a pre-trained text-to- image model is leveraged create images of specific concepts novel settings. Our approach, "Textual Inversion", operates by inverting the concepts into new pseudo-words within the text- embedding space the model. These can be used new prompts, allowing for intuitive modi- fications of the concept. In a sense, our method leverages multi-modal information: text-driven interface for ease of editing, coupled with visual cues when nearing the limits natural language.
While method offers increased freedom, it may still struggle with learning precise shapes, in- stead incorporating the "semantic" essence of a concept. For artistic creations, this is typically enough. Another limitation is lengthy optimization times. Learning single concept requires roughly an hour. This could likely be shortened by training an encoder to map a set of images to their textual embedding. We aim to explore this line of work in the future.
We hope our approach paves the way for future personalized generation works. These could be core multitude of downstream applications, from providing artistic inspiration to product design.

---

Published as a conference paper at ICLR 2023
7 ETHIC STATEMENT
The model discussed this work part of the larger family of text-to-image models, and such models may be used to generate content that would be misleading or promote disinformation. Per- sonalized creation could allow a user to forge more convincing images of non-public individuals. However, our model does not preserve identity the extent where this is concern.
Learned text-to-image models have been shown to be susceptible to biases found in the training data. Examples include gender biases when portraying "doctors" and "nurses", racial biases when requesting images of scientists, and more subtle biases such as an over-representation of heterosex- ual couples and western traditions when prompting for a "wedding" (Mishkin et al. (022). Since our own work builds on such models, it may exhibit similar biases. However, we demonstrate in Appendix A.2 since our model allows creators to describe specific concepts more accurately, could serve means for reducing such biases.
Finally, the ability to learn artistic styles may be misused for copyright infringement. Rather than paying an artist for their work, user could train on their images without consent, and produce images in a similar style. While generated artwork is still easy to identify, in the future such in- fringement could be difficult detect legally pursue. However, are offset by the new opportunities that these tools could offer an artist, such as the ability to license out their unique style, the ability to quickly create early prototypes for work.
hope that such shortcomings
REPRODUCIBILITY STATEMENT
Our work makes the following effort to ensure reproducibility: (1) We will release our code, training sets, and all human captioning.
(2) We provide details on hyperparameter choices for both our training and evaluation setups appendix F (3) We provide details our human evaluation setups in appendix J.1 (4) Where image curation is involved, we provide details on the scale (section 4
show uncurated samples for assessment (appendix[I). I and offer quantitative evaluations on the effects of curation (appendix B
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296-8305, 2020.
Rameen Abdal, Peihao Zhu, John Femiani, Niloy J Mitra, and Peter Wonka. Clip2stylegan: Unsu- pervised extraction of stylegan edit directions. arXiv preprint arXiv:2112.055219, 2021.
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021.
AUTOMATICIII1 Stable diffusion webui.
stable diffusion ceausiccn-ebbiii 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.0222 02779, 2022a.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218 2022b.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. 38(4), 2019. ISSN 0730- 0301. doi: 10.11145/3303443332302,, URL hattps://de http://00.....000.11111//////// 33221123
10

---

Published as a conference paper at ICLR 2023
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv22108.02938, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen lision-language representations. In European Conference on Computer Vi- sion (ECCV), 2022.
Katherine Crowson. CLIP guided diffusion HQ 256x256.
https://colas research
google e. www.drmm/rrr.oo weeriieelee Ne wtwpg//wwwwwwww///w.eee....
2021.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.085583 2022.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04885, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794 2021.
Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the elip model. arXiv e-prints, pp. arXiv-2207, 2022.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nade nada: Clip- guided domain adaptation of image generators. arXiv preprint arXiv:2108.00944 2021.
Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint 1eXiv:2100005500 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jinjin Gu, Yujun Shen, and Bolei Zhou, Image processing using multi-code gan prior, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-68511 2020.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NewrIPS, 2020.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint 2wXiv:2210.09276: 2022.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426-2435. 2022.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-disttibution. arXiv preprint arXiv:2202. 10054, 2022.
11

---

Published as a conference paper at ICLR 2023
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with single text condition. arXiv preprint arXiv:2112.00374, 2021.
Adam Letts, Chris Scalf, Alex Spirin, Tom Mason, Chris Allen, Max Ingham, Mike Howles,
Nate Baer, and David Sin. Disco diffusion. https:///iituu..com//aaa/
disco di dusiini 2021.
Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Xing Fan, Edward Guo, and Yang
Liu. Overcoming catastrophic forgetting during domain adaptation of seq2seq language gen-
cration. In NAACL 2022. Utthttss::/ amwzon. /cieacceppbliiiciiiiins overcoming wat.stttttttttac hwwpopphhh--oogereeeeeeeee forgetting duriiniiiinidduinggggggg domain adaptation of Laag2see--anguaae
Zhi-Song Liu, Li-Wen Wang, Wan-Chi Siu, and Vicky Kalogeiton. Name your style: An arbitrary
artist-aware image style transfer. arXiv preprint arXiv:2202. 13562, 2022.
Pamela Mishkin, Lama Ahmad, Miles Brundage, Gretchen Krueger, and Girish Sastry. Dall-e preview risks and limitations. 2022.
Ryan Murdock. The big sleep, 2021.
13510380530334064
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 2021.
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. arXiv preprint ar arXiv:2203.1722 17272, 2022.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 10743-10752, 2021.
Roni Paiss, Hila Chefer, and Lior Wolf. No token left behind: Explainability- image classifi- cation and generation. arXiv preprint arXiv:2204.0400 0449000 2022.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 11410-11420 11420, 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint 11Xi:22003.12299 2021.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 103. 10.000020, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp. 8821-883 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022.
N. Rathvon. Early Reading Assessment: A Practitioner's Handbook. Guilford Publications, 2004. ISBN 9781572309845. URL httpp:://books.cottto wtt.:/..........oooooooooooooee.
ZX8Y EX8YZWEACAAJ
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for amage-to-image translation. arXiv preprint arXiv:2008. 00951, 2020.
12

---

Published as a conference paper at ICLR 2023
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint wwXiv:220511148/ 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Shivram Shrirao. Imagic stable diffusion implementation. hhttps:/ thub. com/
varShrr iia.mhhrrraoiiiiiiiii faaseriifuusrr/tteeee ma http///mmaeeeeaammeemeeeeemmeeee http///ewaapppppaaaapaeeeeeeeeee 2022.
Gabriel Skantze and Bram Willemsen. Collie: Continual learning of language grounding from
singuage-image embeddings. Journal of Artificial Intelligence Research, 74:1201-1233 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuai Jing, Fei Wu, and Bingkun Bao. Df- gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.05880 058655 2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing encoder for stylegan image manipulation. arXiv preprint arXiv:2 2021.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Conzalez-Garcia. and B. Ra-
ducanu. Transferring gans: generating images from limited data. In ECCV, 2018.
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In The IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2020.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey, 2021.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316-1324, 2018
Paul Yacoubian. Avocado bag, 2022. httpep////uttterrccc/////aaaaaaa
[54286771807177558
Hui Ye, Xiulong Yang. Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to- image synthesis using contrastive learning. arXiv preprint arXiv:2107.0244 2021.
13

---

Published as a conference paper at ICLR 2023
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 333-842, 2021.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. arXiv preprint wxxiv22099001 134, 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. arXiv preprint arXiv:2004, 00004, 2020a.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation the natural image manifold. In European conference computer vision, pp. 597-613
Springer, 2016.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yang. Dm-gan: Dynamic memory generative ad-
versarial networks for rext-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5802-5810, 2019.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Improved stylegan embedding: Where are the good latents?, 2020b.
14

---

Published as a conference paper at ICLR 2023
Appendices
A ADDITIONAL APPLICATIONS
In the following section we investigate set of additional lextual-inversion applications. These include the composition of multiple visual concepts, bias reductions, and the application of our method to downstream tasks.
A.1 CONCEPT COMPOSITIONS
Sstgle
Seloek
Sent
Seraft
"Photo of Sctonk "Photo of Seat "Photo of Serafe "Photo of Setoek in the style Sstyte" in the style Satyte" in the style of Sscyte" the style of Seat"
"Photo of Setaok "Photo of Sear the style of Serajt" in the style of Seraft"
Figure 8: Compositional generation using two learned pseudo-words. The model is able to combine the semantics of two concepts when using a prompt that combines them both. It is limited in its ability to reason over more complex relational prompts, such as placing two concepts side-by-side. Image credits: @ QinniAAtt (left), @ Leslie Manlapig (right).
In Figure 8 we demonstrate compositional synthesis, where the guiding text contains multiple learned concepts. We observe that the model can concurrently reason over multiple novel pseudo- words at the same time. However, it struggles with relations between them (e.g. fails to place two concepts side-by-side). We hypothesize that this limitation arises because our training considers only single concept scenes, where the concept is at the core of the image. Training on multi-object scenes may alleviate this shortcoming. However, we leave such investigation to future work.
A.2 BIAS REDUCTION
A common limitation of text-to-image models is that they inherit the biases found in the internet- scale data used train them. These biases then manifest in the generated samples. For example, the DALLE-2 system card (Mishkin et al.] 2022) reports that their baseline model tends to pro- duce images of people that are white-p and male-passing when provided with the prompt "A CEO". Similarly, results for "wedding". tend to assume western wedding traditions, and default to heterosexual couples.
Here, we demonstrate that we can utilize small, curated dataset in order to learn new "fairer" word for a biased concept, which can then be used place of the original to drive a more inclusive generation.
Specifically, in Figure Te we highlight the bias encoded in the word "Doctor", and show that this bias can be reduced (i.e. we increase perceived gender and ethnic diversity) by learning new embedding from a small, more diverse set.
15

---

Published as a conference paper at ICLR 2023
goonn
"A stock photo of a doctor" (Base model)
"A photo of S," (Ours)
Figure 9: Bias Reduction. Uncurated samples synthesized with pretrained biased embeddings (left) and our debiased embeddings (right). Our approach can be used to reduce bias by learning new pseudo-words for known concepts. These can be optimized using small datasets, which can be carefully curated for diversity.
Input Samples
Target Image With Mask
Output Image
"An oil painting of S.
"A black and white photo of S."
"A S."
"A S."
Figure 10: Our words can be used with downstream models that build on LDM. Here, we perform localized image editing using Blended Latent Diffusion (Avrahami al. 2022a
A.3 DOWNSTREAM APPLICATIONS
Finally, we demonstrate that our pseudo-words can be used in downstream models that build on the same initial LDM model. Specifically, we consider the recent Blended Latent Diffusion (Avrahami al., 2022a) which enables localized text-based editing images via a mask-based blending pro- cess in the latent space of an LDM. In Figure I0 we demonstrate that this localized synthesis process can also be conditioned on our learned pseudo words, without requiring any additional modifications of the original model.
16

---

Published as a conference paper at ICLR 2023
B CONCEPT COVERAGE
While our approach can handle a wide array of concepts, we find that not everything can be encoded into the embedding space with the same fidelity. Here, we investigate how likely concept is be covered by our model. To do so, we compare the CLIP-space distribution of images representing concept. to the distribution of our generated images. Formally, we conduct the following experiment:
(1) We gathered set of 65 different concepts which were uploaded by unaffiliated users to public API of our model. Let {x:] Ji=1 denote the set of images uploaded for concept j E [1,65], where nj the number of images the set.
(2) We embed all the concept-specing images into CLIP's embedding space, and compute all pair- wise distances between these images: d=.q.. D (C, (x+), (= where r E [L. nyl, 9 [r + 1, n,y,, C₁ is the CLIP image encoder and D denotes an L. distance between normalized vectors.
(3) We invert the concept into LDM using our single-word method, and sample 256 images {u/ 256 using the prompt "A photo of S.". We then compute the distance between every sampled image, and the images in the original set: d = D (Cr(z)), Cr())) where E [1, my). 9 E (56).
(4) We compare the two distance distributions, {d } and {d,"), using standard t-test. We consider the concept be "covered" if we cannot reject the hypothesis that the two samples belong to the same distribution (i.e. p >> 0.05).
(5) We repeat this experiment over all concept sets, and report the percent of sets for which the concept was successfully "covered".
Finally, we investigate the effect of "cherry picking' on these results. Specif- ically, we repeat the above experiment, but reject outliers by keeping subsets of {df.y) which are closest to the mean. In fig. II report results when using all 256 generated samples per set, and when using only the 'best' 2' 1" 6
Success
0.6
1 and of re-
sults. We additionally labeled each con- cept representing either style or an object, and we report coverage success rates for each class.
All concepts
0.4
Styles
abjects
Without cherry picking, our model can successfully capture 55.4% of tested con- cepts. However, even mild cherry pick- ing can improve these values significantly (80% and 89. 2% for factors of 1 2 and : respectively). These indicate that for most concepts, one can produce reasonable semantic reconstructions if they willing to filter the results. Another observation is that the model fares considcrably better with styles than with objects (75.8% vs. 34.3% at a factor 1 : 1). This is likely result of our model's difficulty in reproducing exact shapes. When dealing with styles, the concept training set will typically contain large object variation, and thus our metric will be less susceptible to any -naccurrccss in the generated imagery. Indeed, for objects, cherry picking appears particularly effective as it rejects the most distorted shapes.
'Cherry picking' factor
Figure 11: Evaluation of concept coverage as a function of the cherry-picking factor. A factor of 1 :: denotes that we keep one of every k samples.
17

---

Published as a conference paper at ICLR 2023
C EFFECT OF TRAINING SET PROPERTIES
C.I TRAINING SET SIZE
666-
Image only
We investigated the effect of the con- cept's training set size on the results. Specifically, we consider the headless sculpture object of Figure [ (top row).
010
30
Irage
We inverted object using our standard model but swept over dataset sizes rang- ing from a single image to 25 samples. For case of comparison, the image-only, prompt-only, and human caption-based scores for the same single object. The results are shown in Fig- ure I2
040
Long captions
Short ceptions
000
further report
Prompt only
8.2
330
0.36
0.35
0.56
Teat Ser.e
Figure 12: Quantitative evaluation of the effects of the training set size. Significant increases to dataset sizes leads larger deviation from the real-word distribution. This impacts editability and offers paltry improvement in reconstruction. Our approach shows the best results with ~ images.
When learning specific object, using additional images may lead optimized embeddings which reside farther away from real word embeddings, harming ed- itability. In this scenario, our method op- erates best when provided with images.
C.2 TRAINING IMAGE DIVERSITY
We investigate the effect of training image diversity using the CLIP-based image and text similarity metrics. We collected datasets two objects, the cat-toy and the headless sculpture of fig. [ two scenarios: (1) Images taken with the same, uniform background, but including variety of object poses, and (2) images taken with different backgrounds, but sharing the same object pose. In both cases, we compare image similarity to the original, diverse sets used throughout the of the paper. The results are shown in [able
Without background diversity, the model prioritizes the shared background, resulting reduced understanding of the concept and poor reconstructions. In contrast, lack of pose diversity results in generated images portraying the same pose, but their quality is better maintained. Ensuring variation on both ends allows the model to focus on the subject and re-create it in greater variety of poses.
Table Effects of data set diversity and prompt templates. Without background diversity, the model favors capturing background information in the tokens, leading diminished object similarity. Lack of pose diversity leads models which favor the same pose, but do not otherwise harm quality. Removing the prompt templates leads weaker embeddings which are easily overwritten by other words a prompt.
Dataset
Model
Image similarity 1
Text Similarity 1
Baseline
0.768
0.292
Fixed background
0.724
0.281
Cat
Fixed pose
0.752
0.278
No prompt templates
0.722
0.282
Baseline
0,806
0.278
Fixed background
0.786
0,263
Sculpture
Fixed pose
0.799
0.262
No prompt templates
0.604
0.287
18

---

Published as a conference paper at ICLR 2023
140
1 mugge Images
mage
Images
130
100
FDxx4)
60
80-
70-
1000 2000
5000
10000 Traiting iterations
20000
2000
2000
5000
10000 Irainin Retations
20000
AFHQ Cat
Choi et al. 2020
Face Sketchs (Ojha et al. 2021
Figure 13: Fréchet inception distance (FID, 1) for two domain -adaaiitd sets, as a function of training iterations and the number of training images. Long training periods may not help, harm the results. Using additional images can improve coverage for diverse sets.
C.3 TRAINING PROMPTS
We investigate the effect of replacing our prompt templates with a simple "S,". The results are shown in table ! Removing all templates guides the optimization towards weaker embeddings which are often ignored by the model when merged with other words.
D DOMAIN ADAPTATION COMPARISONS
further quantify our model's performance by comparing to generator domain-adaptation ap- proaches. We conduct inversion the sketch dataset of Ojjha et al. (2021) and using the few-shot AFHQ (Choi et al. 2020) split of Gal et al. (2021). For the sketch dataset, we create additional splits of 1,5, 50, and 100 images.
For each set and split, we invert the training set into a single token using our standard parameters and single V100 GPU. Following Ojha al. (2021), generate 5, 000 images using each inverted model, and evaluate the Fréchet Inception Distance (FID) (Heusel et al.) 2017) between them and the full sketch and AFHQ sets, consisting of 300 and 153 images respectively. FID is calculated
using the clean-fid library (Parmar et al. 2022). Figure 13 reports our method's FID as a function of training steps and the number of images. Table further compares our results 10 images and 5, 000 training steps to few-shot GAN domain adaptation approaches which were trained on the same sets. These results demonstrate that our method competitive with GAN-based few- shot adaptation methods, despite not training any part of the network. The FID values demonstrate considerable jitter. However, for small sets, particular, we observe that longer training typically fails to improve, and may even harm FID.
Ojha et al. (202T) additionally evaluate the diversity of their model by employing a clustering-base metric. There, generated images assigned using LPIPS (Zhang et al.) (0188) as distance metric. The diversity metric is then the average pairwise distance between images in each cluster, averaged again over all the clusters. Methods which simply memorize the training set will be assigned score of zero.
cluster containing their nearest training image,
Table reports diversity results using method and select GAN few-shot domain approaches, trained on 10 image set. Our method considerably outperforms the GAN based adaptation ap- proaches. This is likely on account of LDM's larger initial knowledgebase, our limited intervention in the model, and the greater recall typically observed in diffusion models when compared to GANs.
these models evaluate reconstruction quality and prompt-adherence using the CLIP based similarity scores. Figure 14 shows the results for both metrics, across different numbers of training images and training steps. Here, we average over both (cats, sketches) reduce noise, and further plot the trend-lines along with the fit's confidence intervals.
Finally,
When dealing with broad concepts (cats, rather than a specific cat) we observe that additional data can help improve the results. Additional data can further benefit from additional training steps.
19

---

Published as a conference paper at ICLR 2023
Table 2:
FID and diversity comparisons select few-shot GAN domain adaptation approaches.
Despite not modifying any model parameters, our approach can yield FID scores comparable to few-shot GANs, and significant increase diversity. We report our results 5, 000 iterations. Our FID may be lower at other earlier points in training (e.g. 61.11 for the cat set at 2, 000 steps).
FID +
Diversity f
Model
AFHQ Cat
Face Sketch
AFHQ Cat
Face Sketch
CDC (Ojha et al. 2021
45.13
72.74
0.52
0,45
MineGAN Wang et al.
2020)
79.31
62.27
0.21
0,40
TGAN Wang et al.
2018)
87.11
69.44
0.28
0.39
TGAN ADA Karras et al.
2020
52.70
56.76
0.34
0.41
Textual Inversion
79.72
75.61
0.62
0.53
0.62
0.8
0.265
0.60
0232
0.75
0250
0.74
imgge
imuge
Images
Images
Images
Imaees
0.25
imaqus
images
0.70
Images
imuqes
0.30
1000
3000
5000
10000
2C00D
1500 7002
5000
100n0
20203
Twining
Baiiing lenations
Figure 14: CLIP image (left) and text (right) similarity averaged across the domain admainnadaptation sets. Additional training iterations may harm training with few images, but benefit larger sets.
E ADDITIONAL INVERSION A APPROACHES
In addition to the setups outlined in the core paper, we investigated two recent approaches to in- version: Bipartite DDIM-inversion (Ramesh al. 2022: Dhariwal & Nichol 202I and pivotal tuning (Roich et al. 202T). Below we outline both methods and our experimenta results.
Bipartite inversion Dhariwal & Nichol (202I) demonstrated that the DDIM sampling (Song et al, 2020) process can be inverted through closed- iterative approach. Specifically, their approach can find a latent noise vector žr which will be denoised into a specific target image when the denois- given code co co(y). In (Ramesh et al.) 2022), they further demonstrate
ing process is conditioned that when the conditioning code is an output of CLIP, one can later modify this code using text- derived directions in CLIP's multi-modal embedding space, while keeping the initial noise, ar, fixed. This induces semantic changes in the image while maintaining the general structure of the original object.
Here, we investigate similar approach. However, rather than modifying the conditioning code co(y) directly, we change the conditioning text y. Specifically, we first find an appropriate pseudo- word for our target concept. Then, we find 2y for a given image of the concept using the text "A photo of 5." " and the closed-form solution of Dhariwal & Nichol (202T) Finally, we modify the conditioning text but keep æт frozen. The results are shown in Figure 15 (left). Here, we observe that when using LDM's typical guidance (Ho & Salimans 2021) scales (0-, the denoiser network unable maintain the original object's structure through prompt changes. When reducing the guidance scale, the outline of the original image becomes visible. However, alignment with the prompt is poor.
Such deetance-dependeen structure drift has also been demonstrated for GLIDE (Nichol et al. 2021) However, this effect is reduced in DALL-E2 (Ramesh al. 2022 (their Figure 9). Notably, state- of-the-art models (Saharia et al., 2022 Ramesh et al., 2022) typically employ guidance scales (~ 2) which are significantly lower than LDM's within the range where observe structure preserva-
20

---

Published as a conference paper at ICLR 2023
Input Samples
Bipartite Inversion
Pivotal Tuning
Reconstruction
==
"A painting of sitting next to serene pond"
"A watercolor painting of
Figure 15: Advanced inversion results using Bipartite Inversion (Ramesh et al. 2022 (left) and Pivotal Tuning (Roich et al. (021)) (right). is the guidance scale. Reconstructions were obtained using the prompt "A photo of S.". Bipartite inversion allows for more accurate reconstructions without modifying the model, but their structure is lost for complex prompts in high guidance scales. Pivotal tuning improves shapes at the cost of visual artifacts, and fail to adhere to simple prompts random noise input, while the
high guidance scales. Note that the pivotal tuning results bipartite results use a fixed noise for each column.
tion, but prompt-matching This gives us hope that bipartite inversion would allow better shape preservation in more powerful generative models.
Pivotal Tuning In the field of GAN inversion, it has been shown (Roich et al.) 2021 Bau al. 2019) that one may largely avoid the reconstruction-editbbin tion process. First, image is inverted into "pivot" code in well-behaved region of the latent space, using standard optimization. This typically results in highly editable code, but with poor identity preservation. As a second step, the generator is fine-tuned that the first step's pivot code will more accurately reproduce the inverted image. It was further demonstrated that such localized tuning can maintain the appealing properties of the latent space and retain similar latent-editing capabilities.
tradeoff using a two-stage optimiza-
Here, we investigate similar approach in order improve reconstruction. We first optimize pseudo-word using our baseline method. Then, we fine-tune the generator such that sentences of the form "A photo of S, will better reconstruct the concept-specific training set images.
Our initial investigation reveals that naïve applications of this approach lead to improved shape preservation, but also severe collapse of editing at high guidance scales. See Figure 15 (right) for examples. However, as outlined in appendix G this failure is model-dependent,, and other archi- tectures may even benefit from its use.
21

---

Published as a conference paper at ICLR 2023
A more involved application of this same principle (e.g. by combining it with similar process to the inversion outlined below, or by tuning around results produced with higher guidance scales) might overcome the issues observed in LDM. We leave such investigation to future work.
F IMPLEMENTATION DETAILS
Unless otherwise noted, retain the original hyper-paramette choices of LDM (Rombach al, 2021]. Word embeddings were initialized with the embeddings of single-word coarse descriptor of the object (e.g- "sculpture" and "cat" for the two concepts in Figure D Our experiments were conducted using 2x V100 GPUs with batch size of 4. The base learning rate was set to 0.005. Following LDM, we further scale the base learning rate by the number of GPUs and the batch size, for an effective rate of 0.04. All results were produced using 5, 000 optimization steps. We find that these parameters work well for most cases. However, for some concepts, better results can be achieved with fewer steps or with an increased learning rate.
For all CLIP-based evaluations, we use the official CLIP ViT-B/32 checkpoint.
G STABLE DIFFUSION
In addition the LDM results, we demonstrate that our method can be adapted work with Stable Diffusion, a recently released large-scale latent diffusion model (Rombach et al. (002]). However, in contrast to the baseline LDM model which trains a BeRT et al. (2018) text encoder alongside with the generative network, Stable Diffusion makes use of a pre-trained, frozen CLIP text encoder.
We implement our method over Stable Diffusion using the same training setup, losses, and parame- ters that were used for LDM. We notice that inverted Stable Diffusion embeddings tend to dominate the prompt and become more difficult to integrate into new, simple prompts. These limitations can be mitigated in one of two approaches: First, by reducing the weight of the learned embedding. This can be done through the use of longer, more complex prompts, or by using prompt re- re-weighting methods [AUTOMATICIIT] 2022). Second, through the use of the Pivotal Tuning mechanism outlined in appendix E We hypothesize that Stable Diffusion's 's decreased malleability is an artifact of either its text encoder's training its size. While LDM's text encoder was trained with the dense visual reconstruction task, here the encoder is a pre-trained CLIP, which had a simpler contrastive learning task for which focusing on a subset of image content was typically enough. Similarly, a concurrent work (Ruiz et al. 2022) reports difficulty in capturing the appearance of subject without model-tuning when using Imagen (Saharia et al.) 2002). model whose text-encoder was pre-trained on a purely linguistic task and kept frozen. On the size front, the CLIP text encoder em- ployed by Stable Diffusion contains 128M parameters, roughly a fifth LDM's BERT. Moreover, it has embedding dimension of 768 compared to LDM's 1280. These could combine to create less expressive latent space. which forces new concepts further out of the domain of real words and negatively impacts prompt adherence. We plan to further investigate this matter in the future.
In fig. 16 we provide samples generated using Stable Diffusion and the outlined methods. Note that while the pivotal tuning approach can typically provide more appealing visual outputs, it requires significantly greater computational resources, consuming roughly 30GB of VRAM with a batch size of one. The baseline inversion method, meanwhile, consumes little 12GB using a similar batch size. Moreover, the model-tuning approach leads to checkpoints that are several GB in size, compared less than 4KB for a learned embedding. Learned embeddings are thus easier share and use for large-scale collaborative work.
H IMAGE EDITING COMPARISON
To better highlight the distinction between our approach and single-image editing methods, we com- pare our approach to the concurrent Imagic (Kawar et al. 2022). We use the unofficial Stable Dif- fusion implementation of their method (Shrirao 2022). where we swept over model parameters and prompt strengths in an attempt to match our results. comparison our cat model provided in fig. I7
22

---

Published as a conference paper at ICLR 2023
Prompt weighting
Painting of two fishing boat
S. Backpack
S. banksy
8. lunchbox
Prompt weighting
PTI
Input samples
themed hotwheels car
S. Irooman
on the moon
Pablo Picasso
Figure 16: Injecting user-specific concepts into Stable Diffusion. contrast LDM, using our baseline approach with Stable Diffusion leads to less editable results. These can be overcome through prompt engineering and weighting, by using Pivotal Tuning (Roich et al. 202T.
As can be seen, the single-image editing method successfully captures details such the background and the object's 's location and pose. However, struggles with creating large changes or putting the object in new contexts while preserving key object details. Our method meanwhile is not restricted to a single scene, and instead aims to capture the semantics of a concept and apply them to new scenes.
I ADDITIONAL RESULTS
We provide additional results of personalized generation using our method. In Figure 18 we show additional text-guided synthesis results.
In Figure 19 we show large-scale galleries of uncurated results generated with the prompt "A photo of S, " In Figures 20 and [2 we provide large-scale galleries of uncurated results generated with wide assortment of prompts. These are intended provide sense of the quality of images produced and cherry-picken also contain demonstrations of typical failure cases, such difficult relational prompts (Figure 20
involved when generating the samples in the core paper. Note that these results
rows 2, 5).
23

---

Published as a conference paper at ICLR 2023
Ours (PTI)
Painting
S. Backpack
S. banksy art
S. lunchbox
S. fishing on beat
Imagic
Painting of two fishing
Cat themed backpack
Banksy
Cat lunchbox
Figure 17: Text-guided synthesis comparison to Imagic Kawar et al. 2022), Our method captures the semantics of concept and uses them to create new scenes. Imagic successfully maintains specific image details, such as the object's 's location or the background, but fails to create complex transformations while maintaining fidelity the source concept.
24

---

Published as a conference paper at ICLR 2023
"Girl with Pearl Earring by Johannes Vermeer style S.
Input samples
"The Scream in the style S."
"Sa Starry Night"
"The Great Wave in the style of S."
Input samples
"Anime painting of Kawaii S.
"A massy
"S_ skeleton in the National Science Museum"
"Negative space painting of S."
Input samples
"Painting S. the style of Monet"
"A banana S.
"Advenisement for S,
"S, cookies"
Input samples
"5, in Minecraft"
stained glass window depicting
"Sketch of steampunk S. "
"A S, suhmarine"
TEAROT
"A poster for the movic "The Teapot' starring S.
Input samples
"Manga drawing a steaming S.
watering caa"
"S. Death Star"
Injecting user-specific concepts into new scenes. Our method can change concept's style, composition, or use it to inspire new creations. Top row image credits: @Øyvind Holmstad
Figure 18:
25

---

Published as a conference paper at ICLR 2023
Input samples
"A photo of S,"
Input samples
"A photo of S, "
NEW HETT
FIIE REAN
GITO
PROTU RATOERIS
Input samples
"A photo of S. "
Input samples
"A photo of S."
Input samples
"A photo of S,"
Input samples
"A photo of S, "
Figure 19: Uncurated samples of object variations created using the prompt "A photo of S,".
26

---

Published as a conference paper at ICLR 2023
"Banksy of Seat"
S Seat
"Painting of Seat riding a dragon"
"Painting of two Seat fishing on a boat"
Selock
"A Seut themed lunchbox"
"A painting of Seat holding Sctock"
Spard
"Ukiyo-e painting of Stand"
Figure 20: Uncurated samples generated with context prompts. Quality and prompt-matching varies within the sample. However, we observe that a batch size of 16 typically sufficient to ensure several good samples.
27

---

Published as a conference paper at ICLR 2023
"Crochet Ssculpture
Seculpture
SUNG
"S "Scullture ink wash calligraphy"
"A black and white sketch of Ssculpture
Seraft
"A Scrafe backpack"
"A painting of Seraft invading from space"
S Sotyle
"A waterfall in the style of Sstyle"
Additional uncurated samples generated with context prompts. Quality and prompt- matching varies within the sample. However, we observe that a batch size of 16 is typically sufficient ensure several good samples. Image credits: @ (onniiot(bbotto)) authorized for non-commercial use only.
Figure 21:
28

---

Published as a conference paper at ICLR 2023
J TRAINING PROMPT TEMPLATES
Below we provide the list of text templates used when optimizing worudo-word:
a photo of a S..",
a rendering of a S..",
a cropped photo of the S..",
the photo of S..".
a photo of clean S..,,
a photo of a dirty S.."
a dark photo of the S..",
a photo of my S..",
photo of the cool S...,
close-up photo of S..",
a bright photo of the S..",
a cropped photo of a S..",
a photo of the S..", a good photo the S..",
photo of S..". a close-up photo of the S..",
a rendition of the S..".
a photo of the clean S..,,
a rendition S..",
a photo of a nice S..",
good photo S..".
a photo of the nice S..".
a photo of the small S..".
a photo of the weird S..",
a photo of the large S..",
a photo of a cool S..",
photo of small S_.".
J.1 DETAILS CAPTIONS CROWDSOURCING AND USER STUDY
We collected captions describing the training concepts using Amazon Mechanical Turk (AMT).
provided annotators with four images a concept and asked them to describe it in a manner that could allow artist to recreate it. We collected five short (S 12 words) and five long (< 30 words) captions per concept. In total, we collected 10 captions per concept five short and five long.
Figure 22 illustrates the data collection interface for each type of questionnaire.
When evaluate the models through a user study, we created two questionnaires. the first, users were provided with four images from concept's 's training set, and asked to rank the results produced by five models according to their similarity to these images. In the second questionnaire, users were provided with text describing image context ("An oil painting") and asked to rank the results produced by the same models according to their similarity to the text. We used the same target concepts and prompts the CLIP-hased evaluations and collected total of 600 responses each questionnaire. A single Turker was not permitted to rank more than 7 prompts per questionnaire.
Figure 23 illustrrae the data collection interface for the user study.
29

---

Published as a conference paper at ICLR 2023
Carttally stall Interrienn
lnd timer be witk tal Toaala
Bubrit
daekk ly redd rurruct ligh Fyuu grAcis
inpetter Tixusie caniic magess Thei mpii
bock ootur rige SnACCE
So devobe las grrvee sorrnctillect
Aig lipe and and
"cremite
Sertence
Subiiit
Figure 22: Screenshots the caption crowdsourcing interface with full instructions given to users.
(a, top) Interface for short captions (b, bottom) Interface for long captions
For both the the data captioning and the user study, we paid $0.2 per response. This sum ensures that, regardless of their world-wide location, workers are paid more than the US federal minimum wage (the median response time was 70 scconds, indicating a wage of roughly $10.3 per hour of work).
To maintain the quality of the queries, we only picked users with AMT "masters" qualification, demonstrating high degree of approval rate over wide range of tasks.
Furthermore, we also executed qualification test with few examples and verified that the annota- tors follow the instructions. We rejected queries from two qualified user that abused the system by submitting the same score ordering in every questionnaire (c.g. 5, 3, 1).
All tasks were flagged containing adult content. This required due to the risk that some generated images may unintentionally contain offensive contents, as the underlying LDM model was trained uncurated web data. We also included the following warning in the title "WARNING: This HIT may contain adult content. Worker discretion is advised"
30

---

Published as a conference paper at ICLR 2023
Instructions
Read the instructions carehillyy
or cuampleo, coject,
pietures by porney onject
misiooo
offect
INPORTAKIT You otoould assign of offerant han' (1.5) each agg.
may Do rot eds se the stem, ow hak mensumee me to spot that
selection.
Examples:
Rank how well do the objects in each following image resemble the object in the examples above?
wontt ...
I word, beet
I worst 5 boot
wortl B bost
T wont) ... bost
Instructions
You givat sentence that describes a picture, and low images that portay this descrition. masemoo rena images rezwiing lisur eech image sgreen deactppiin Your-il Do not select score of Your HiT may be suected such a selection.
Do select the same score for two
rejersed for sush selection
Rank how wall does each imaaee match the following description.
Description: An all painting
9
3
worst
worst. 5
worst
worsst ...
Figure 23: Screenshots of the study questionnaires with full instructions given users. (a, top) Visual similarity to the examples of the personalized object (b, bottom) Context similarity to the context description
31


Reviews:
[{"review_id": "_CYhN_EFs0", "reviewer": "Reviewer_ah2h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a small number of image samples in the text-to-image generation process, where these samples can be converted into pseudo-word to enable novel image generations, related to the given image samples.", "summary_of_the_review": "See above weaknesses. I am happy to raise my rating based on authors' responses. ", "strength_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow.\n2. Good qualitative results shown in the paper.\n\nWeaknesses:\n1. Although authors claim their method is to invert several image samples into new pseudo-words, it more likes to use some image features extracted from given image samples, and use these features along with a given text to generate new images. There are already couple of works focusing on image+text -> new images. It might be better that author can discuss the differences between them.\n2. I would like see more quantitative evaluation metrics adopted in the paper, and comparison with state-of-the-art text-to-image generation methods.\n3. How is the diversity of proposed method, does the given image samples constrain the diversity of synthetic results?\n4. If authors provide more image samples (> 3), would this further improve the performance? If these samples are unrelated and do not describe the same object, would this degrade the quality of output images?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YfSRL7Ye5D", "reviewer": "Reviewer_W72Q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for utilizing a pre-trained text-to-image diffusion model to generate novel images with a specific concept referenced with few samples.\nBy training an additional text token with the reconstruction loss for reference images, the model gets to generate various images with newly trained concept.\nSince this model does not fine-tune the generator itself, the new concept can softly incorporate with other concepts the original model can generate already.", "summary_of_the_review": "Even though it is not clear whether this method would be working consistently well, the method is simple and novel, and the results are impressive enough.\n", "strength_and_weaknesses": "Strengths\n- By the nature of the model, users only could manipulate a text prompt to get a specific style of image in mind. But it is hard to find such a prompt as can be seen in Fig 3. The proposed method solved the issue by finding a way to use a reference image set as a prompt, which is to train a new token for the concept in the images. Further, since the process does not hurt the original model’s image generation performance, the newly trained token can mingle with another concept, whether it is the one that the model could generate already or the one that was trained with another sample set likewise.\n\nWeaknesses\n- The results are mostly analyzed qualitatively. Considering the stochasticity of diffusion models, more analysis on failure cases and scores on quantitative metric that can measure the failure rate would be needed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X4bVpP66eil", "reviewer": "Reviewer_xRCj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to personalize large, pre-trained text-to-image diffusion models based on specific objects/styles presented via a small amount (3-5) of images. To achieve this, a new text-embedding is obtained by optimizing it via the reconstruction loss for the small set of training images. After some fine-tuning steps the new text-embedding can be used in normal captions to describe the original object/style in novel setting and surroundings. The experiment show that this approach is more successful than other baselines such as Guided Diffusion or VQGAN+CLIP loss.", "summary_of_the_review": "Overall I like the paper and it seems to obtain good results and be somewhat novel.\nHowever, I believe it would benefit from some more clarity about robustness and generalization capability, as well as how easy it would be to apply it to other models (not based on BERT) and how it compares (generally speaking) to approaches that finetune the model directly instead of finetuning a text embedding.", "strength_and_weaknesses": "The paper proposes a novel approach to personalize pre-trained text-to-image models. This allows users to generate specific objects or styles which were not seen during training. Compared to several baselines the approach performs much better and is able to generalize the new concepts to different surroundings and settings at test time.\n\nSeveral ablation studies also show that encoding a novel concept into a single text embedding (as opposed to several text embeddings or applying specific regularization terms) obtains the best trade-off between semantic reconstruction and text-image alignment.\n\nMy main questions are related to the robustness of the approach and its applicability to other text models.\n\nRegarding robustness: how much is the final results and generalization affected by\n* the chosen caption templates during training\n* the number of fine-tuning steps\n* the number of training images and the diversity in the training images (e.g. background, lighting, etc)\n* learning rates\n\nRegarding applicability to other text models:\n* you mention in the supplementary that this approach doesn't work as well for Stable Diffusion due to its use of CLIP instead of BERT, what does this mean for other models? Do you think this approach would work for DALLE2 which isn't conditioned on a sequence of tokens?\n* what are the characteristics that are needed to make this approach work, generally speaking?\n\nYou also briefly mention Pivotal Tuning in the supplementary which includes fine-tuning the model itself, too. I guess this is somewhat similar to DreamBooth (concurrent work, so I don't expect a comparison). It would be helpful to include more thoughts about the comparison between those two appraoches in general (finetuning a text embeddings vs finetuning a model itself), what the trade-offs between those two appraoches are, etc..", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5pyO0u533i", "reviewer": "Reviewer_a4CY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the task of textual inversion which in essence tries to capture a concept in a set of images (either style, abstract, object, or relations) as a single new \"word\" which can then we used to guide the generation of the generative models based on this new \"word\". An example would be to extract out abstract things in Pablo Picasso's style and apply it to new images. The paper qualitatively shows how superior it is compared to past approaches and does human evaluations to also verify the same.", "summary_of_the_review": "The paper's contributions are significant as it allows using existing pretrained generative models to empower a new use case and leverage the latent space of these models from the corners it wasn't accessible before. I am impressed by the results and contributions of this papers towards understanding these models better and thus suggest an accept.", "strength_and_weaknesses": "### Strengths\n- This is a very novel and important task for guided generative models and the results are impressive. \n- The approach is very simple to understand and extend.\n- Improves our knowledge about the text embedding in the generative models and how can we \"interpolate\" in between the embeddings to leverage the true latent space of text embedding by learning specific and new \"concepts\" and \"words\". The title is very appropriate in that sense.\n- The human evaluations are done and show case how well the approach works. \n- The approach also works in comparison to human written prompts which don't capture nuances in the image\n- It is nice how one can further exploit this \"word\" by specifying style of S* to guide the generation in certain directions.\n\n### Weaknesses\n- Lack quantitative analysis and guidance for future work on how to continue working on this task and have proper evaluations. I had expected the authors to create a dataset for quantitatively evaluating there generation using FID score based on the concepts and generations they have. (Though indeed it will require some creativity to build this)\n- It is unclear how this approach works with images that contain multiple objects or COCO-style scenes. Is one word enough to capture complex scenes. \n- The training setups are not super clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%