Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on the task of textual inversion which in essence tries to capture a concept in a set of images (either style, abstract, object, or relations) as a single new "word" which can then we used to guide the generation of the generative models based on this new "word". An example would be to extract out abstract things in Pablo Picasso's style and apply it to new images. The paper qualitatively shows how superior it is compared to past approaches and does human evaluations to also verify the same.
strength_and_weaknesses: ### Strengths
- This is a very novel and important task for guided generative models and the results are impressive. 
- The approach is very simple to understand and extend.
- Improves our knowledge about the text embedding in the generative models and how can we "interpolate" in between the embeddings to leverage the true latent space of text embedding by learning specific and new "concepts" and "words". The title is very appropriate in that sense.
- The human evaluations are done and show case how well the approach works. 
- The approach also works in comparison to human written prompts which don't capture nuances in the image
- It is nice how one can further exploit this "word" by specifying style of S* to guide the generation in certain directions.

### Weaknesses
- Lack quantitative analysis and guidance for future work on how to continue working on this task and have proper evaluations. I had expected the authors to create a dataset for quantitatively evaluating there generation using FID score based on the concepts and generations they have. (Though indeed it will require some creativity to build this)
- It is unclear how this approach works with images that contain multiple objects or COCO-style scenes. Is one word enough to capture complex scenes. 
- The training setups are not super clear.
clarity,_quality,_novelty_and_reproducibility: - It is unclear if the approach is reproducible as a lot of diffusion models are proprietary 
- The paper is clearly written and easy to follow.
- The approach is novel and the reader actually learns something from the paper.

summary_of_the_review: The paper's contributions are significant as it allows using existing pretrained generative models to empower a new use case and leverage the latent space of these models from the corners it wasn't accessible before. I am impressed by the results and contributions of this papers towards understanding these models better and thus suggest an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use a small number of image samples in the text-to-image generation process, where these samples can be converted into pseudo-word to enable novel image generations, related to the given image samples.
strength_and_weaknesses: Strength:
1. The paper is well written and easy to follow.
2. Good qualitative results shown in the paper.

Weaknesses:
1. Although authors claim their method is to invert several image samples into new pseudo-words, it more likes to use some image features extracted from given image samples, and use these features along with a given text to generate new images. There are already couple of works focusing on image+text -> new images. It might be better that author can discuss the differences between them.
2. I would like see more quantitative evaluation metrics adopted in the paper, and comparison with state-of-the-art text-to-image generation methods.
3. How is the diversity of proposed method, does the given image samples constrain the diversity of synthetic results?
4. If authors provide more image samples (> 3), would this further improve the performance? If these samples are unrelated and do not describe the same object, would this degrade the quality of output images?
clarity,_quality,_novelty_and_reproducibility: The paper is well presented, and shows sufficient details about the proposed method. 
summary_of_the_review: See above weaknesses. I am happy to raise my rating based on authors' responses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for utilizing a pre-trained text-to-image diffusion model to generate novel images with a specific concept referenced with few samples.
By training an additional text token with the reconstruction loss for reference images, the model gets to generate various images with newly trained concept.
Since this model does not fine-tune the generator itself, the new concept can softly incorporate with other concepts the original model can generate already.
strength_and_weaknesses: Strengths
- By the nature of the model, users only could manipulate a text prompt to get a specific style of image in mind. But it is hard to find such a prompt as can be seen in Fig 3. The proposed method solved the issue by finding a way to use a reference image set as a prompt, which is to train a new token for the concept in the images. Further, since the process does not hurt the original model’s image generation performance, the newly trained token can mingle with another concept, whether it is the one that the model could generate already or the one that was trained with another sample set likewise.

Weaknesses
- The results are mostly analyzed qualitatively. Considering the stochasticity of diffusion models, more analysis on failure cases and scores on quantitative metric that can measure the failure rate would be needed.

clarity,_quality,_novelty_and_reproducibility: This work tackles the important issue and broadens the applicability of text-to-image diffusion models with simple, novel method.
Though it is unclear whether this method would be reproducible for more various concepts, the authors clearly demonstrate their method and its efficacy. 


summary_of_the_review: Even though it is not clear whether this method would be working consistently well, the method is simple and novel, and the results are impressive enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to personalize large, pre-trained text-to-image diffusion models based on specific objects/styles presented via a small amount (3-5) of images. To achieve this, a new text-embedding is obtained by optimizing it via the reconstruction loss for the small set of training images. After some fine-tuning steps the new text-embedding can be used in normal captions to describe the original object/style in novel setting and surroundings. The experiment show that this approach is more successful than other baselines such as Guided Diffusion or VQGAN+CLIP loss.
strength_and_weaknesses: The paper proposes a novel approach to personalize pre-trained text-to-image models. This allows users to generate specific objects or styles which were not seen during training. Compared to several baselines the approach performs much better and is able to generalize the new concepts to different surroundings and settings at test time.

Several ablation studies also show that encoding a novel concept into a single text embedding (as opposed to several text embeddings or applying specific regularization terms) obtains the best trade-off between semantic reconstruction and text-image alignment.

My main questions are related to the robustness of the approach and its applicability to other text models.

Regarding robustness: how much is the final results and generalization affected by
* the chosen caption templates during training
* the number of fine-tuning steps
* the number of training images and the diversity in the training images (e.g. background, lighting, etc)
* learning rates

Regarding applicability to other text models:
* you mention in the supplementary that this approach doesn't work as well for Stable Diffusion due to its use of CLIP instead of BERT, what does this mean for other models? Do you think this approach would work for DALLE2 which isn't conditioned on a sequence of tokens?
* what are the characteristics that are needed to make this approach work, generally speaking?

You also briefly mention Pivotal Tuning in the supplementary which includes fine-tuning the model itself, too. I guess this is somewhat similar to DreamBooth (concurrent work, so I don't expect a comparison). It would be helpful to include more thoughts about the comparison between those two appraoches in general (finetuning a text embeddings vs finetuning a model itself), what the trade-offs between those two appraoches are, etc..
clarity,_quality,_novelty_and_reproducibility: The paper is well written, novel, and reproducible.
summary_of_the_review: Overall I like the paper and it seems to obtain good results and be somewhat novel.
However, I believe it would benefit from some more clarity about robustness and generalization capability, as well as how easy it would be to apply it to other models (not based on BERT) and how it compares (generally speaking) to approaches that finetune the model directly instead of finetuning a text embedding.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel approach to personalize large pre-trained text-to-image diffusion models based on specific objects/styles presented via a small number (3-5) of images.

After author rebuttal, it received scores of 6688. All the reviewers commented that the task is novel and important, the paper is clearly presented, and the results are impressive. The primary concerns shared by reviewers is the request for additional quantitative results. The authors have done a good job during rebuttal, and some new evaluation results are added.

Diffusion models are really popular these days, and the general audience will be interested in reading this paper. The idea in this paper is novel and interesting. Overall, the AC would like to recommend acceptance of the paper. 
justification_for_why_not_higher_score: While this paper is novel and interesting, comprehensive quantitative results and analysis are lacked, therefore, the AC thinks it could be a good spotlight paper, but not to the oral paper level yet.
justification_for_why_not_lower_score: Diffusion models for text-to-image (T2I) generation are popular, this submission is a timely paper that studies how to personalize a frozen large T2I model. The AC thinks that the general ICLR audience will be interested in reading this paper, and the idea behind this paper is simple and novel, and qualitative results in the paper are appealing.
Decision: Accept: notable-top-25%