DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
Authors: Brian Nlong Zhao
Yuhang Xiao
Jiashu Xu
XINYANG JIANG
Yifan Yang
Dongsheng Li
Laurent Itti
Vibhav Vineet
Yunhao Ge
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Image Generation
Personalized Generation
Primary area: generative models
TL;DR: Generation of novel and diverse images/3Ds following a set of user input images trhough prompt distribution learning

Abstract: The popularization of Text-to-Image (T2I) diffusion models enables the generation of high-quality images from text descriptions. However, generating diverse customized images with reference visual attributes remains challenging. This work focuses on personalizing T2I diffusion models at a more abstract concept or category level, adapting commonalities from a set of reference images while creating new instances with sufficient variations. We introduce a solution that allows a pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution. These prompts offer text-guided editing capabilities and additional flexibility in controlling variation and mixing between multiple distributions. We also show the adaptability of the learned prompt distribution to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our approach through quantitative analysis including automatic evaluation and human assessment.

Published as a conference paper at ICLR 2025
DREAMDISTRIBUTION: LEARNING PROMPT DISTRIBUTION FOR DIVERSE IN-DISTRIBUTION GENERATION
Brian Nlong Zhao', Yuhang Xiao Jiashu Xue* Xinyang Jiang", Yifan Yang", Dongsheng Li Laurent Itti', Vibhav Vineet Yunhao Ge
University of Southern California Harvard University Miicosooft Research Asia "iicrosoft Research Redmond
Equaa Contribution 'Equal Advising
Fow-shot
Training Images
20 generation
Sample prompt from Jumping 0
Sample prompt from D Lago
Sample prompt from D museum
Learn
generation
Sample prompt Jumping
Sample
Sample prompt
Figure DreamDistribution learns a prompt distribution D* that represents a distribution of de- scriptions corresponding to set of reference images. We can sample prompts from D or modified D by text-guided editing to generate images of diverse new instance that follows the vi- sual attributes of reference training images (top). We also apply leamed distribution flexibly to, for example, a pretrained text-to-3I) model, and generate diverse new 3D) assets following the reference images (bottom).
ABSTRACT
The popularization of Text-to-Image (T2I) diffusion models enables the genera- tion of high-quality images from text descriptions. However, generating diverse customized images with reference visual attributes remains challenging. This work focuses personalizing T2I diffusion models at more abstract concept or category level, adapting commonalities from a set of reference images while creating new instances with sufficient variations. We introduce solution that al- lows pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution. These prompts offer text-guided editing capabilities and additional flexibility in controlling variation and mixing between multiple distributions. We also show the adaptability of the learned prompt distribution to other tasks, such as text- to-3D. Finally we demonstrate effectiveness of our approach through quantitative analysis including automatic evaluation and human assessment.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Dreams have long been a source of inspiration and novel insights for many individuals (Edwards itt al., 2013; Von Grunebaum & Caillois| (023). These mysterious subconscious experiences often reflect our daily work and life (Freud) 1921). However, these reflections are not mere replicas; they often recombine elements of our reality in innovative ways, leading to fresh perspectives and ideas. We aim to emulate this fascinating mechanism in the realm of text-to-image generation.
Text-to-image (T2I) generation has recently been popularized due to the astonishing performance of state-of-the-at diffusion models such as Stable Diffusion (Rombach et al. (021) and DALL-E 2 (Ramesh et al. 2022). Variations of the T21 models have enabled several fascinating applica- tions that allow user to control the generation, such as conditioned generation based on other input modalities (Zhang et al., 2023 Li et al. 2023; Yang al. 2003), inpainting (Lugmayr et al. 2022
Xie et al. 2023), image editing (Mokady et al., 2023) Brooks et al.. 2023). One such interesting application is personalization of T2I models, where user provides
reference images of the
same instance (e.g. their pet dog), and the personalized model can generate images based on the references, with the flexibility of text-guided editing for new context. This is generally achieved by associating token with the personalized concept through fine-tuning the model parameters (Ruiz
ctt al., 2022 Kumari et al. 2023) or newly added learnable token embeddings (Gal al., 2022
Voynov et al. 2023).
In many cases, however, user may want to personalize T2I generation over more abstract visual attribute instead of specific instance-level personalization. For example, a designer may seek inspiration by generating variety of novel cartoon characters or scenery images following similar visual attributes presented in their previous works. In this case, trying over text prompts is not scalable and hard to get desired result that follows the desired visual attributes. On the other hand, using the existing personalization methods afforementioned is likely to fail when training images do not represent the same instance, but rather encompass a distribution sharing certain, yet challenging- to articulatt,, commonalities. Additionally, existing personalization methods often result in limited diversity and variation during generation (Fig. 3). Since the associated token is fixed, these methods will typically learn a token that is either overlitted to combination of visual features, or learn token that is overly generalized, which introduces more randomness into the uncontrollable diffusion process, thereby failing follow desired visual attributes in generated images.
In this work, we propose DreamDistribution, a prompt distribution learning approach on T21 diffu- sion model for various downstream tasks (Fig. Ip. Our proposed solution has three key components (Fig. 2). First, to adapt a pretrained fixed T2I model, instead of fine-tuning diffusion model parame- ters, our method builds on prompt tuning (Zhou al. 2022a b), where we use soft learnable prompt embeddings with the flexibility to concatenate with text, to associate with the training image set. This design have several advantages: (1) prevents catastrophic forgetting of the pretrained model, enabling it to learn
almost infinite variety of target prompt distributions using the same T21 dif- fusion model. (2) It is highly efficient in terms of parameters, requiring only the prompt itself as the learnable element. (3) The learned prompts remain within the semantic space of natural language, offering text- -atdd editing capabilities and generalizing to other pre-trained diffusion models, such as text-to-3ID. (4) The learned distribution increased flexibility in managing variations. Second, we introduce a distribution of prompts to model various attributes described by reference images at a broader level. The prompt distribution is modeled by a set of learnable prompt embeddings as- sociate with the training image set as a whole. The learned prompt distribution can be treated as distribution of learned "descriptions" of the reference images and should be able to model the commonalities and variations of visual attributes, e.g., foreground, style, background, texture, pose. During inference, we sample from the prompt distribution. which should have a similar semantic meaning, understood by the downstream denoising network, to produce in-distribution outputs with appropriate variations. Lastly, to effectively optimize the set of soft prompts that models the dis- tribution, we apply simple reparameterization trick (Kingma & Welling (0033) and an orthogonal loss to update the prompts at token embedding space simultancously and orthogonally.
We first demonstrate the effectiveness of our approach in customizing image generation tasks ($4) By taking a small set of images of interest as training images, we demonstrate that our approach can generate diverse in-distribution images where baseline methods fail to generate desired output. The diversity and the quality of our synthetic images are verified via automatic and human evaluation (.22) We show that the learned distribution holds the capability of text-guided editing, as well as
2

---

Published as a conference paper at ICLR 2025
further controllability such as scaling the variance and composition of distributions $4.3 Next highlight that the learned prompt distribution can be easily applied to other text- generation tasks such as pretrained text-to-3D models ( ($4.4). Lastly we show the effectiveness of our method on personalized distribution generation through classification task with synthetic training data as a proxy ($4.5). In summary, our contributions
We propose a sittiibutton-hasee prompt tuning method for diverse personalized generation by learning prompt distribution using T2I diffusion model.
Using public available pretrained T2I diffusion model, we experiment our approach on customization T2I generation tasks and show that our approach can capture visual attributes into prompt distribution and can generate diverse in-distribution images that follows text- guided editing.
Further experiments show that our method is flexible in terms of diversity mixing and easy to be adapted to other text-guided generation tasks.
We further quantitatively demonstrate the effectiveness of our approach using synthetic image dataset generation tasks as proxy and also through automatic evaluation metrics and human evaluation.
2 RELATED WORKS
2.1 TEXT-TO-IMAGE DIFFUSION MODELS
2021 Sohl-Dickstein et al. 2015) have achieved great success in various image generation tasks. State-of-the-art T2I models such as Ima- gen (Saharia et al., 2022) and DALL-E 2 (Ramesh et al. 2022 trained on large scale data demon- strate remarkable synthesis quality and controllability. Latent Diffusion Models (Rombach et al.) 202T) and its open-source implementation, Stable Diffusion (Rombach et al., 2021p. have also be- come prevailing family of generative models. In these T2I diffusion models, text is encoded into latent vectors by pretrained language encoders such as CLIP (Radford et al.) 2021), and the de- noising process is conditioned on latent vectors achieve text-to-image synthesis. However, such models trained on large scale text-image pairs are not designed to generate personalized images such images of one's pet dog, therefore only the text conditioning cannot provide fine-grained control over the generated images.
Diffusion models (Ho et al. 2020 Dhariwal Nichol
2.2 PERSONALIZED TEXT-TO-IMAGE GENERATION
Various approaches proposed to better control the text-guided diffusion models and achieve personalization. Textual Inversion (Gal et al. 2022) proposed to search for a new token in the em- bedding space representing visual concept via optimizing word embedding vector. DreamBooth (Ruiz et al. 2022) fine-tunes all parameters of the model to associate a personalized subject into an rarely used token. Custom Diffusion (Kumari et al., (023) employs that fine-tuning method but only fine-tune cross-attention layers to reduce the training time, with the ability to learn multiple concepts jointly. Subsequent works (Voynov et al., 2023 Wei al., [2023] Shi al., 2023aa Alaluf
et al. 2023 mainly follow the ideas from these works and focus on solving their drawbacks.
2.3
PROMPT LEARNING
Prompt learning is a popular method in natural language processing (NLP). The main idea is to transfer various downstream NLP tasks to masked language modeling problems via adopting proper prompt templates (Brown et al.) 2020 Lester al. 2021 Li & Liang] 2021 Radford et 2019) instead of fine-tuning the pretrained language model. Searching for the appropriate prompls is the
key of this method. Prompt engineering (Brown et al. 2020 Radford al. 2019) adopts carefully- designed discrete (hard) prompts crafting by human, while prompt tuning (Lester et al. [202]] Li & Liang 2021 automatically searches for the desired prompts in the embedding space via learning continuous (soft) prompts. The great success of NLP inspires computer vision researchers and prompt engineering is explored in pretrained vision-language models such as CLIP [Radford et al.) 2 and ALIGN (Jia et al.) 2021). CoOp (Zhou et al.) 2022b) applies the idea of prompt tuning in vision- Languaelanguage tasks, which learns a continuous prompt via minimizing the classification loss of the downstream tasks. ProDA (Lu et al. 2022) learns distribution of diverse prompts to capture various representations of a visual concept instead of a single prompt in CoOp (Zhou et al. 2022b), which achieves better generalization.
3

---

Published as a conference paper at ICLR 2025
Learnable Prompts
Generated Image
Sample by Reparameterization
Distribution
Text-to image Diflusion
Text Encoder
0
NO.1
Orthogonal Lose
Noise
Reference Imagee
Figure 2: Overview of DreamDistribution for learning prompt distribution. We keep set of K learnable soft prompts and model distribution of them at the CLIP text encoder feature space. Only prompts are learnable, CLIP encoder and the T2I diffusion model are all fixed. We use reparame- terization to sample from the prompt distribution and update the prompts through backpropagation. The training objective to make the generated images aligns with the references. An additional orthogonal loss is incorporated to promote differentiation among learnable prompts. For inference, we sample from the prompt distribution text feature space to guide the pretrained T2I generation.
Most relevant to our work are Textual Inversion Gal et al. 2022 and ProDA (Lu et al. 2022) Textual Inversion learns fixed token embedding associated with a pseudo-word. Ours learns a dis- tribution of prompts in the CLIP feature space like ProDA, allowing for learning the visual concept with diverse representations and capturing the details for reconstructions and plausible synthesis.
METHOD
Given a of images with some common visual attributes (e.g. same category, similar style), our goal is to capture the visual commonalities and variations and model by a prompt distribution in the text feature space. The commonalities among reference images may be challenging to articulate with natural language prompts. We can thus sample prompts from the distribution to guide T2I model to generate diverse unseen images while the same time following the common traits distribution. The inherent characteristics of the learned prompts are compatible with natural language instructions and other pretrained text- exxtgguied generation models.
3.1 TEXT-TO-IMAGE DIFFUSION
Text-to-image diffusion models are a class of generative models that learns image or image latent distribution by gradually denoising noise sampled from Gaussian distribution. Specifically, given a natural language text prompt, tokenizer followed by a text embedding layer maps the input text sequence of embedding vectors p. A text encoder converts the text embedding into text features {p) used for conditioning the generation process. An initial noise is sampled from (O, I), and the denoising model ea predicts the noise added noisy version image of image latent x. The denoising model eg is optimized using the objective:
C Ex,,,,,, [lee
eo (x,, c, t]11
where x is the ground-trute -ruth image or image latent obtained from learned autoencoder, x; is the noisy version of x at time-step and I).
3.2 PROMPT TUNING
Our proposed method is grounded in the notion of prompt tuning, which aims to learn a soft con- tinuous prompt on target task and is widely used in fine-tuning NLP models. (Lester et al. 202T Liu et al. 2023 & Liang 2021| Hammbadzumyann et al., 2021 Liu al., 2021) Specifically, for pretrained model that takes natural language prompt input, we can formulate prompt with continuous learnable token embeddings P [PREFIX] [SUFFIX] E RRdd where [PREFIX] and [SUFFIX] are word embeddings of natural language prefix and suffix if needed, and L rep- resents the prompt length or the total number of tokens, and represent the dimension of word embeddings. V [v] ... .vl...//aar I Mxd represents sequence of M learnable token embed- ding vectors with same dimension as word embeddings. During fine-tuning, the parameters of the pretrained generation model remain fixed, and only the learnable token embeddings V are updated
4

---

Published as a conference paper at ICLR 2025
through direct optimization employing the corresponding loss function backpropagated through gen- erator Eg and text encoder E. Formally, prompt tuning aims to find optimized embedding vectors V arg maxy P(Y PP. .)), where X and Y are input data and output label, respectively.
Prior works have shown the efficacy of adopting prompt tuning techniques on rision-langual mod- els for image classification tasks (Zhou et al. 2022b Jia et al. 2022 Zhou et al. 2022a Gal et al.
(Ga et al., 2022) adopts similar prompt tuning methods that enable personalized generation. How- ever, the limitation of this approach lies in its constraint personalize only one particular concept, such as a specific dog, as it employs fixed token embedding for concept encoding.
3.3 LEARNING PROMPT DISTRIBUTION
We aim to model more general commonalities and variations presented in the reference image set and generate diverse images new instances that visually align, therefore we propose to model a learnable distribution of prompts for the reference images. Inspired by Lu et al. (Lu et al. 2022), ) which proposed to estimate a distribution of prompt for image classification tasks, we propose to model distribution of learnable prompts over sequence of M token embeddings capture the distribution of visual attributes on T2I generation task leveraging diffusion model.
Our methods builds Stable Diffusion (Rombach et al., 2021), where pretraind CLIP (Radford iet al. 2021)) text encoder is used for obtaining text feature of the prompt. Due to the contrastive training objective of CLIP, features of texts that have similar semantic meaning have high cosine similarity and therefore close to each other in CLIP feature space (Radford et al. 2021 Lu et al.
(Lu al.. 2022) and some other previous works (Chen et al. 2022 Ma et al. (0023) have also shown that for text prompts that describe images of the same category, the CLIP text feature output from pretrained CLIP text encoder are adjacent to each other in a cluster. Therefore, is sufficient to model a Gaussian distribution of e that describes images of same category or with shared attributes without extra restriction. To do so, instead of keeping one learnable soft prompt to optimize during training, we maintain a set of K learnable prompts PK = {Px [PREFIX] V& [SUFFIX] ) corresponds to of similar reference images. Our goal is optimize the set of learnalee token embeddings [V With K learnable prompts, we can estimate the mean μt = p (E (PK)) e R.xxss and standard deviation ae = a (E (P*))= E L.dg at E text encoder space, where de is the feature dimension of text encoder space.
Applying to the training objective of T21 diffussion model, eq. T becomes:
C [Pk)) = -xx,,ct [||e Eo (x,, (x,,,,1113 011
where ē N (pe: o²) and e 10, is the sampled Gaussian noise added to the image or image latent. However, sampling è from distribution makes it not differentiable for optimization, therefore we apply the reparameterization trick similar to that used in VAE (Kingma & Welling 2013). Formally, since غ ~ N' (Me, o²), we can rewrite the optimization objective eq. as:
C <(P)) Ex.w.e. ||e es (xt, Me + wo. |
(3)
where N/ (0, I) has the same dimension as Mee and σe Since the exact computation of C (PK) is intractable, we use a Monte Carlo approach to sample w for times to approximate the expected value for optimization:
1 S Elle
C (P*)
|l en (x:, Me + W.G., 1)l
(4)
In order to avoid the scenario wherein multiple prompt features converge to a same vector, which will result in non-representative low-variance distribution, we apply a similar orthogonal loss proposed in (Lu et al. 2022) to penalize on the cosine similarity and encourage orthogonality between each
pair of prompts:
Lontoo
(E(P.).
(5)
K(K
i=1 s=it1
where (+--) is cosine similarity between a pair of vectors. The total loss is therefore:
C = L(p*) + Xootho
(6)
where X is hyperparameter.
5

---

Published as a conference paper at ICLR 2025
Reference mmges
Short Caption
No Trainnn
Zero-d
Long Caption
Dreamt- Booth
Foor Fios-ahot Tune U-iiet
Diflerion
Tran Embesiiing Only
Fow shat
Inversion
Oreamn- [Ours]
Figure 3: Comparison of results with existing methods. Given set of training images (typically 5-20, we only show 4 here), we compare generation results with other existing methods. We use Stable Diffusion version 2.1 for all methods. As can be seen on the bottom row, our method is able to generate more diverse and coherent images (also quantitatively analyzed by automatic and human evaluation
Implementation Details In all experiments, we use Stable Diffusion 2.1 (Rombach et al. 2021 and keep all the default hyperparameters. We use S 4 and À 5 10- We use K 32 prompts in all personalized generation experiments, and K = 10 prompts to reduce computation in synthetic dataset experiments. We use 1,500 steps with constant learning rate of 10-3
EXPERIMENTS
In this section, demonstrate several experiments and applications of our approach and show vi- sual results of generated images. We show the ability of our approach to capture a distribution of reference images and generate in-distribution novel images in $4.1 We present additional quantita- tive results including automatic evaluation and user studies in $4.2 We also show the flexibility and
effects of manipulating and text-guide editing learned prompt distribution in $4.3 We further high- light easy application of our learned prompt distribution to other text-based generation tasks using text-to-3D as an example $4.4. Finally in $4.5 we present experiments that show the effectiveness of our approach in generating synthetic training dataset.
4.1 DIVERSE PERSONALIZED GENERATION
We first demonstrate the ability of approach to generate images that preserve general visual features shown in training set and at the same time generate new images with high diversity. Given diverse set of few training images (typically 5-20) that not easily describable in texts and the same time share some similar visual attributes, we can generate diverse in-distribution images by simply sampling from the learned distribution the input prompt text embedding to T2I diffusion model. Our learned prompt distribution can be therefore treated as a distribution of descriptions corresponding to the of training images.
Baselines. We compare with popular instance-level personalization methods including Textual Inversion (Gal et al. 2022). DreamBooth (Ruiz et al. 2022). Custom Diffusion (Kumari et al.) 2023 We also evaluate against Short Caption that uses a short description as text prompt, and Long Caption that uses longer text caption with detailed descriptions. These comparisons empha- size our method's 's ability to take care of both similarity and diversity referencing the training images. We use the same pretrained Stable Diffusion version 2.1 with default hyperparameters provided in baseline works. We use M 8 context vectors without adding any prefix or suffix texts in either training inference process for DreamDDitributton.

---

Published as a conference paper at ICLR 2025
Textual Fivession
11.7%
Custom Diffusion
CHUC
Disamboanii
0.35
031
234.90
6%
Tostal looaiion
0.33
0.27
0.45
1.28
0.82
234.33
Casoon Dillesion
0.80
0.33
0.46
1.45
0.57
236.61
Ties
038
0.9
029
139
21315
53.3%
10 4%
DreamtBooth
(a) Our method achieves best number on most au- tomatic metrics. Mean value of the metrics across 12 scenarios are reported.
(h) Human Evaluation on image diversity (.22) aligns
with automatic evaluation (Fig. 4a). Our method shows significantly greater diversity. which may ex-
plain why it can train better image classifiers in Tab.
Figure 4: Quantitative and human evaluation.
Results. Fig. shows visualized comparison with baselines. In general, both short and long text prompting methods fail to generate results that visually follow the reference images since there is training involved and the image details are hard to describe in language. Images generated using baseline methods generally show limited variation or inconsistent visual attributes in all examples. All these methods try associate different visual concepts with fixed token, which does not pro- vide any semantic variations itself. Although the denoising process enables some randomness, the training objective of associating various concepts with fixed token will either fail to capture distribution due to non-convenn conerenve,,,,c,, leading to underfitting to generic image category information, or overfits to visual combination of the training images. By modeling multiple concepts using multiple prompts and optimizing the prompt distribution, our proposed method is able to produce substantial variations of style and view points, for example, following the reference images in the cathedral example (first column). Ours method can also model the texture and background informa- tion and generate new instance with significant variations in color and pose following the reference images of the Gundam example (second column), well as patterns, lines, style as whole and generate novel artistic creations as shown in the Basquiat's 's painting example (third column). In all, DreanDistribution is able to produce substantial variations style, viewpoints, pose, layout, etc., with appropriate visual attributes following the reference images.
4.2 GENERATION QUALITY AND DIVERSITY EVALUATION
We quantitatively assess methods in terms of diversity and quality, and further synthetic ImageNet classification performance as proxy in $4.5 We also conduct human evaluation study on the generation result compare with baseline methods.
Datasets Different from existing datasets such as DreamBooth Dataset (Ruiz al. 2022 which focus on same-instance personalization, we construct a dataset that consists of different instances in category set. Our dataset consists of 12 diverse image scenarios including photos of real objects in large and small scales, works of famous artists, and illustrations with prominent styles, sourced from illustrators from online communities. Each scenario consists of about 20 real reference images. A comprehensive list and visualization is shown in the appendix. We train our method and baselines including DreamBooth, Textual Inversion, and Custom Diffusion using the default settings provided. For our approach we use M = 4 learnable context with no text prefix suffix in both training and generating stages. For baselines, we generate image using the prompt "a photo/drawing of 5 depending on the nature of the reference set, where S* is the learned token(s). We generate 40 images per methods per scenario, resulting a total of 1, 1,920 images in the evaluation set.
We evaluate the generative images on established automatic evaluation met- rics that measure the diversity of synthetic images and the similarity between real and synthetic images. Following prior works (Heusel et al., 2017] Zhang et al. 2018 Naeem et al. 2020 Ruiz
Automatic Metrics
et al.) 2022), in Fig. 4a we evaluate image quality using CLIP-1 and DINO (Ruiz al.) 2022) that measures average pairwise cosine similarity between CLIP (Radford et al. 2021) and DINOvi (Caron et al. 202 embeddings, and FID (Heusel et al. 2017) that measures the distance between the distribution of generated images and the distribution of real images via Inception V3 (Szegedy al. 2016). Our method achieves the best quality across all three quality measurements, suggesting

---

Published as a conference paper at ICLR 2025
Reforence Images
Chineee Painting
Van Gogh
(a) Effect scaling variance of learned prompt distribution. Image diversity increases as the scaling factor increases.
(b) Composition two prompt distributions through interpolation. Mixing ratio changes linearly from left to right. Middle columns show mixtures of two styles.
Figure 5: Scaling variance and composition of prompt distributions.
that our method is capable of creating more high-quality images that fulfill the prompt requirement. CLIPPT measures the image-text alignment in CL.P space for text-editing generation. We use the 25 object prompts provided in (Ruiz et al. 2022) and generate images per prompt per scenario, resulting 1,200 text-editing images. Our method may not show best performance due to the random nature of distribution but still gives comparable result. Additionally, we report Density and Cover- age (Nacem et al. 2020) in Fig. 4al Density measures samples in regions where real samples are densely packed, while coverage calculates fraction of real samples whose neighbourhoods contain at least generated sample. Both metrics are calculated with DINOv2 (Oquab et al. 2023). Our method achieves the best coverage and diversity across the board.
Human Evaluation Admittedly, automatic evaluation does not fully capture the richness per- ceived by human observers. We further investigate if Fig. 4a correlates with human perception via conducting human evaluation based the evaluation dataset. We assign 20 independent annotators. For each of the 12 reference sets, annotators are asked to choose the most preferable set of gener- ated images based on their perceived similarity with the reference set and the diversity within the generated set. The methods are anonymized so annotators are unaware of which generated set corre- sponds which method. We collect a total of 240 samples and count the frequency of preferences. Fig. dbldtoontssstes that our generated images exhibit superior diversity compared to three baseline models, reinforeing our intuition that by learning distribution we are able to generate diverse images with coherent content and visual attributes presented in the reference image.
4.3
CONTROLLABILITY OF PROMPT DISTRIBUTION
Since learned prompt distribution is in the CLIP text feature space, it is natural manipulate the learned distribution based on the property of CLIP text feature space. We show several interesting distribution manipulation methods, including text-guided editing, scaling the variance for diversity control, interpolation between multiple distributions.
Text-guided Editing Similar to existing personalization methods (Ruiz al. 2022 Gal et al,
2022 Kumari et al. (023), our learned distribution preserves the flexibility of text-guided editing able to generate diverse in-distribution Gundam figures
As shown in Fig. and Fig. 6) that follows the style of reference images but with different pose, style, context, using user provided text-guidance at inference time. With a set of learned prompt, we concatenate them with the same text prefix and/or suffix to fit a new distribution at the CLIP text feature space to enable text-guided editing of a prompt distribution. Application includes but not limited to, generating objects of in- terests in a different background or context, transferring style using text, and controlling the pose, viewpoints, layout, of objects of interests.
Scaling Variance for Diversity Control Once a prompt distribution is learned, we can easily control the diversity of generated images by changing the variance or standard deviation of the learned distribution. We show an example of the effect of multiplying different scale factors y to the variance of learned prompt distribution in Fig. 5a When 7 0, the generated images show very similar patterns following some of the reference images. As y increases, more different layouts emerge, and when we further scale the variance for 7 = 2, the generated images become more diverse with significant randomness.
8

---

Published as a conference paper at ICLR 2025
Reference images -- D*
D burger shop
D Seattle
D Solar System
Reference D
Scallop D*
Tank
Pink Pikachu D*
Figure 6: Results on text-editability our methods. Left column shows samples of reference im- ages, right columns are generated results with corresponding prompts.
Composition of Distributions Given multiple prompt distributions in CLIP feature space, we can composite distributions by finding linearly interpolated distribution between them. This distribu- tion in the CLIP feature space should represent a text with semantic meaning that is a weighted mix- ture of the given prompt distributions, thereby showing a mixture of visual attributes in the generated images. We naively use weighted sum of the distributions to interpolate between distributions:
N
N
Σ 0ipep i=1
a:
Va,
where 2a and o are mean and standard deviations of the interpolated distribution, and a, is the weight of i-th prompt distribution with mean and standard deviation He, and de, respectively, and E a; = are weight parameters. We show an example of mixing Chinese paintings and Van Gogh paintings in FFg.. 5b From the left column right, we adjust the mixing ratio to increase the weight of Van Gogh and decrease the weight of Chinese painting.
4.4 APPLYING TO TEXT-TO-3D 3D GENERATION
Our learned distribution be flexibly applied to other text-driven tasks, long the generation pipeline uses the same pretrained text encoder as the text feature extractor. In this section, we high- light and demonstrate the flexibility of method by using prompt distribution trained on T21 diffusion for text-to-3D task. We use MVDream Shh et al. 2023b). a state-of-the-art text-to-3D model that train a NeRF (Mildenhall et al. 2021 and render 3D asset following text prompt. which in our case is prompt sampled from prompt distribution. As shown in Fig. I and Fig. 7a although MVDream incorporates some extra prior in its modified multi-view difrusion model that leads to reduced diversity, our prompt distribution can still generate 3D assets with significant vari- ation in design details. Moreover, as shown Fig. 7b the pipeline possesses text-guided editing capabilities akin to those of DreamBooth3D (Raj et al. 2023). yet it can generate instances that exhibit more diverse appearances.
4.5 APPLYING TO SYNTHETIC DATASET GENERATION
Our proposed method can be effectively used in generating synthetic image classification datasets. By giving several dozens to hundreds images that correspond a class in classification dataset, method can capture and encode distributions of the dataset images into the learnable prompt distributions, and thereby generate diverse training images similar to the training set.
We generate "synthetic copy" (Ge et al. 2022a Sariyildiz et al. 2022 Ge et al. (023) of ImageNet (Russakovsky et al. 2015) via DreamDistrin Distribution using Stable Diffusion version 2.1 with default hyperparameters. Due to the large size of ImageNet- 1 we follow previous works (Sariyildiz al.) 2022) to mainly experiment on ImageNet-100 (IN) (Tan et al. 2020), 100-class subset. For each class, generate 2,000 synthetic images and CLIP (Radford al. 2021 select top

---

Published as a conference paper at ICLR 2025
Reference Images
with stone pottem
Reference
2
Sample
Reference images
Rainbow color
(a) 3D generation results by learning a prompt distri- bution over reference images and then inference (with- out extra texts).
(h) 3D generation results by learning a prompt distri- bution over reference images and then inference with text text-guided editing.
Figure 7: Results of our method applied to 3D generation using MVDream (Si et al. 2023b)
IN
IN-V2
IN-S
IX-R
IN- Tog5
Deasty?
Cweragef
PID
Topl
Top5
Topl
Tog5
Tapl
Tog5
Topl
Top5
Tapl
Keal
13
96.7
13.1
15.1
63.9
56.1
85.2
26.7
65.
Closs Names
88.5
7011
86.2
725
26.1
45.3
55.
75.8
8.1
188
075
0.3
0.71
u.1
14937
CLIP Pronpi Radionl es al. 5021
45.6
60.2
46.1
69.6
36.2
68.1
58.8
$1.1
12.2
45.7
0.7L
0.37
0.83
0.40
165-42
InigeNe-S Sanvine 2r
55.4
77.5
55.8
72.5
29.4
40.0
59.8
$0.0
15.9
49.4
0.72
0.38
0.54
0.36
173.58
[euua] Ireen F1 A424
173
35.7
122
253
4.34
13.2
102
46.6
32
35.2
0.71
0.33
2033
0.21
214.14
Dosmbuoth 20
11.4
59.2
0.50
0,89
0.51
Cialom Dilleom Kansnn
540
59.7
6.1
157
50.1
10.1
0.47
0.64
0.42
141.5
DacenDachblan (Ckm)
$1.
188
Table I: Classification accuracy and other metrics on different real test sets by training classifier on synthetic ImageNet (IN) generated by different methods. The rows with gray background are non-training methods. When training on images generated from our method, the resulting classifier performs better on most test sets, indicating that the images synthesized by our method allowed the classifier to learn those object categories better.
1,300 images with highest cosine similarity to the embedding vector of the corresponding class name, resulting the same total number of images real ImageNet training set. We compare with non-training baselines: Real uses the real ImageNet training set, Class Names and CLIP Prompts generate images by feeding Stable Diffusion class name of each class with 80 diverse text prompts provided by CLIP, e.g. "a drawing of c", where is the class name. ImageNet-SD (Sariyildiz et al., 2022) generates images using prompts in the form of "c, he inside 6". where e represents the class name, he represents the hypernym ( (WordNet parent class name) of the class, and b is random background description from the Places365 dataset (Zhou al., (017). We also experiment with other aforementioned personalization baselines. For personalization baselines and our method, we use same random 100 images per class to train the generation pipeline. Then we train a ResNet- 50 (He et al., 2016) classifier synthetic images only for 300 epochs using 0.2 alpha for mixup augmentation (Zhang et al., 2017 and auto augment v0 via timm (Wightman, 2019).
To analyze generalizability, also evaluate the trained model validation set of ImageNet vari- ants including ImageNetV2 (IN-V2) (Recht et al.) 2019), ImageNet-Sketch (IN-S) (Wang et al, 2019) ImageNet-R (IN-R) (Hendrycks al. 202 aa). and ImageNet-A (IN-A) (Hendrycks al. 2021b). Top-1 and top-5 accuracy is reported in Tab. In all settings, the classifier is exclusively exposed to synthetic images, but images generated using our method shows the highest classification accuracy on ImageNet validation set. This is because DreamDistribution can generate a diverse set of high-quality images following training set distribution, while other prompt engineering methods cannot follow the real image distribution and tend to show limited diversity, resulting in perfor- mance degradation. We also achieve the best results on ImageNet-V2 and ImageNet-A. For the Sketch and Rendition variants, in contrast to our method, CLIP Prompts and ImageNet-SD offer specific prompts to generate images of other domains, which may account for our comparatively lower performance.
5 CONCLUSION
We introduced DreamDistributin distribution based prompt tuning method for personalizing T2I diffusion models to generate diverse in in-distribution images following small set of reference images. The key idea of our methods lies in modeling the commonalities and variations of visual attributes using a prompt distribution at text feature space. We show a variety of experiments and application that is enabled by our method.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported by the National Science Foundation (award 2318101), C-BRIC (one of six centers in JUMP, Semiconductor Research Corporation (SRC) program sponsored by DARPA), the Army Research Office (W91 INF2020053), and Amazon ML Fellowship. The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.

---

Published as a conference paper at ICLR 2025
REFERENCES
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representa- tion for rext-to-image personalization. ACM Transactions on Graphics (TOG), 42(6):1-10, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:221000155. 2022.
Gabriele Corso, Yilun Xu, Valentin de Bortoli, Regina Barzilay, and Tommi Jaakkola. Particle guidance: non-i..d. diverse sampling with diffusion models. arXiv, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis, Advances in neural information processing systems, 34:8780-877 8894, 2021.
Christopher L Edwards, Perrine M Ruby, Josie Malinowski, Paul D Bennett, and Mark T Blagrove. Dreaming and insight. Frontiers in Psychology, 4:979, 2013.
Sigmund Freud. Die traundeutung. F. Deuticke, 1921.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 22Xh::2008.01618 2022.
Yunhao Ge, Harkirat Behl, Jiashu Xu, Suriya Gunasekar, Neel Joshi, Yale Song. Xin Wang. Laurent Itti, and Vibhav Vineet. Neural-sin Learning generate training data with nerf. In European Conference on Computer Vision, pp. 477-493. Springer, 2022a.
Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Laurent Itti, and Vibhav Vineet. Dall-e for detection: Language-driven context image synthesis for object detection. arXiv preprint arXiv:2206.0955 09592, 2022b.
Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi, Laurent Itti, and Vibhav Vineet. Beyond generation: Harnessing text to image models for object detection and segmentation, 2023.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. In Fei Zong, Chengqing andXia, Wenjie Li, and Roberto Navigli (eds.), Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4921-4933, Online, August 2021. Association for Computational Linguistics. doi: 10.18653) v1/2021 1.81.11.....
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of dustr--distriuution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3349, 2021a.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262- 15271, 2021b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916 PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Diederik P Kingma and Max Welling. arXiv:1312.6114 14,, 2013.
Auto-encoding variational bayes.
arXiv preprint
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Man-Yan& Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEESCVF Conference on Computer Vision and Pultern Recognition (CVPR), 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning, arXiv preprint
2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2100 110.2111100000000 2021.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-225211 2023.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5206-5215, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Wenxuan Ma, Shuang Li, Jinming Zhang, Chi Harold Liu, Jingxuan Kang, Yulin Wang. and Gao Huang. Borrowing knowledge from pre-trained language model: A new data-efficient visual learning paradigm. In Proceedings of the IEEF/CVF international conference on computer vision, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
13

---

Published as a conference paper at ICLR 2025
Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning. pp. 7176-7185. PMLR, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini 1rXIvv:304.07793. 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ily Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp- 8748-8760 PMLR, 2021.
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXiv:2303.13500 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 0rXii:2204406125: 1(2):3, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arxiv: 22242 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22) 12222, 2015. doi: 10.1007/s 12000//11122---5-001166---
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494 2022.
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning (8) from a synthetic imagenet clone. arXiv preprint arXiv:2212.084200 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.0 30330000..11..... 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023b.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCA 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794, Springer, 2020.
14

---

Published as a conference paper at ICLR 2025
Gustave E Von Grunebaum and Roger Caillois. The dream and human societies. Univ of California Press, 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint urXiv:2303.09522 2023.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Yuxiang Wei, Yabo Zhang, Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13844, 2023.
Ross Wightman. Pytorch image models. pytorch- wwwiiiaaaeeeeeee models 2019.
https://github. wwm. hwtn///ightaa..
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22428-22453 June 2023.
Zhengyuan Yang. Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246 14255, 2023.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Pate mixup: Beyond empirical risk minimization. arXiv preprint arXhv:1710.09412, 2017.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13009):2337-2348, 2022b.
15

---

Published as a conference paper at ICLR 2025
A MORE IMPLEMENTATION DETAILS
In all experiments, we use Stable Diffusion 2.1, which is the latest version. We use the default parameters, including 7.5 guidance scale and 50 denoising steps. For all visual results, we generate images in 768 x 768 resolution, and for synthetic dataset experiments, we generate 256 x 256 images to save time and resources. We provide pseudocode for learning prompt distribution in Algorithm
Algorithm Training prompt distribution
Require: Set of reference images I {x; 1.
Require: Set of learnable prompts pK = (P:)
Require: Text encoder ε, noise predictor 60, hyperparameter
1: Random initialize all learnable embeddings V in pk
2: for image X I do
3: Sample time step
4: Sample noise с -0 I)
5: X, with noise added based on and
6: Compute Me k -=1 & (P:)
7: Compute o² K
Σ 1= (E - Mee)
8:
for E [S] do
9:
Sample w, ~ I),
10:
L. (PK) ||e (Xt, Me + w.
11:
end for
12:
C (P*)
E. (P(
13:
Lonto = K(K-I) E E (((()))
C L= C= (PK) + XLrttho
15:
Update learnable embeddings V in pk based on C
16: end for
FIG
Dare by
Coverag [1)
872
Prompts
Figure 8: In general, with more prompts, the performance increases in terms of both quality and diversity.
Donelty
Cowerape
2.04
122
8.00
115
E00
100
Orthogoral Regulonzation
Figure 9: Choice of value between x 10 and 1 x 10-2 generally achieves good balance of quantitative metrics.
16

---

Published as a conference paper at ICLR 2025
Donetty
Coverag
6.52
6623
0.7
817
6425
Prompt
Figure 10: Similar to number of prompts, increasing number of prompt tokens also shows increasing image quality and diversity.
B EVALUATION SET
We show samples of reference images from our evaluation set in Fig. 11 reference images from the same
Each row shows samples
C ADDITIONAL RESULT
C.I DIVERSE IMAGE INSTANCE GENERATION
We show additional image generation results of our method in Fig. 12 using the reference images from our evaluation set. Each row is generated using reference images of the corresponding row in Fig. We also show additional examples in Fig. 13 that demonstrate the ability of method to capture visual similarities in reference images and generate diverse images, specifically on the attributes that are shown different reference images.
C.2 TEXT-GUIDED EDITING
We show more results on the ability of our method generate diverse images with text-guided editing in Fig. 14juiing different sets of reference images.
C.3 SCALING VARIANCE FOR DIVERSITY CONTROL
We show more results on generating images from prompt distribution with scaled standard devia- tions
Fig.
C.4 COMPOSITION OF DISTRIBUTION
In Fig. 16 we show more results on composition of two different learned prompt distributions with various weights.
C.5 SAME INSTANCE PERSONALIZATION
Our method solves more generalized task compared to same instance personalization, where ref- erence images is one or more images of the same instance and the generation expected to follow user provided text prompts of different contexts. Although reconstruction of the same instance is not our main focus, we still show that our method can achieve on par results on this task compared with baselines, using DreamBooth dataset (Ruiz et al. 2022) with identical prompts and evaluation settings. The quantitative results are show Tab.
C.6 NAIVE ADAPTATION OF BASELINES
We compare our approach with a naive adaptation baseline methods, where the training set is randomly divided into 4 subsets, and generation model is trained each subset. During inference,
17

---

Published as a conference paper at ICLR 2025
(18)
(15)
(18)
(17)
(11)
(24)
(17)
(37)
(16)
(8)
(16)
(26)
Figure 11: Samples of reference images from evaluation set. Numbers on the right represent the number images each set.
18

---

Published as a conference paper at ICLR 2025
Figure 12: Samples of generated image results using reference images from the evaluation set. Each row is generated using reference images of the corresponding row in Fig. II
19

---

Published as a conference paper at ICLR 2025
Images
Ours
noisa
Reference Imagea
Oura
with noise
Figure 13: More results that shows the ability of our method to capture same visual attributes and generate diverse images specifically on the attributes that are shown different in reference images. The top example shows reference images with same shape outline but different foreground texture and pattern, and our method also generates the same shape with diverse new foreground patterns. The bottom example shows reference images of same object in different background, and our method generates the same object in diverse background as well. We additionally show comparison with naive adaptation of Textual Inversion, where add Gaussian noise with 0.5 variance to the learned tokens for generation, and the result shows that although it may increase diversity due to introduction of extra randomness, it fails to capture the common attributes that should not be varied in both examples, such as constant shape and background in first example and color and appearance in second example.
Method
CLIP-I
CLIP-T1
DINO
DreamBooth
0.80
0.31
0.67
Textual Inversion
0.78
0.26
0.57
Custom Diffusion
0.80
0.36
0.54
Ours
0.80
0.31
0.59
Table 2: Experiment results of personalized generation on DreamBooth dataset.
the final output is mixture of results from the models trained these different subsets. Based on the evaluation results shown in Tab. 3) our method outperforms this naive adaptation of three baseline methods on all metrics. Moreover, this naive adaptation takes significantly more extra disk space since it requires the storage of multiple sets of model weights.
Method
CLIP-It
CLIP-T*
DINO!
Density
Coveraget
FIDI
DreamBooth
0.26
0.33
232.13
Textual Inversion
0.80
0.25
0.44
0.78
0.66
243.93
Custom Diffusion
0.75
0.27
0.39
0.62
0.57
268.48
Ours
0.84
0.29
0.93
215.15
Table 3: Comparison of our methods with naive adaptations of baseline methods.
20

---

Published as a conference paper at ICLR 2025
Ingut images - D
D" flcating the water
D" in the GrandCannyn
D' in the forest
Ingut image
D
D In the ocean
sand
D' flying the sky
More results text-editability of our methods. Left column shows samples of reference images to learn distribution D right columns are generated results using prompts sampled from corresponding text-edited distribution.
Figure 14:
C.7 EXPLORING WITH DIFFERENT GRANULARITY OF CONCEPTS
We experiment on image sets different class granularity show that our method is capable of capturing distribution at different levels of concepts. Specifically, we composed 3 different reference image sets, forced from ImageNet images: French Bulldog, dogs, and quadruped animals, where dogs set includes images from different ImageNet dog species such Bulldog, Saluki, Chihuahua, etc., and animals set includes images from different ImageNet quadruped classes, such as tiger, cat, bison, fox, etc. We run DreamDistribution on these image sets respectively. The result visualization shown Fig. 17 For French Bulldog, our method able to generate different instances of French Bulldogs with different fur colors and appearances. For dogs reference set, our method able to generate different dog species with mixed attributes. For quadrupeds reference set, our method generate variety of quadruped animals, including non-existing animals that show mix of attributes from different animals. Quantitative analysis presented in Tab. 4 demonstrates that the most granular level, such as French Bulldog, method significantly outperforms the personalization baselines. However, at coarser- coarser-grained levels, such as Dogs or Quadrupeds, the advantage of our method diminishes. This could be attributed to the increased diversity of the reference images, which makes it challenging for our method to effectively capture a coherent or meaningful distribution, leading higher likelihood of sampling outliers from the learned prompt distribution during generation.
C.8 EXPERIMENT WITH SMALL REFERENCE SET
We conduct experiments on image sets with only four reference images to simulate scenarios where users are unable to provide larger set for better distribution capture. Our method is compared against the personalization baselines used in previous sections. Visualizations of generated re- sults are shown in Fig. 18 which shows that our method still generate results with more diversity and at the same time within the distribution reference images.
C.9 3D GENERATION DIVERSITY
Our learned prompt distribution extend to text-to-3D generation pipelines, such as MVDream Shi et al. (2023b), which utilize text-to-image models as their backbone, enabling more diverse 3D generation. As illustrated in Fig. 19 unlike the Textual Inversion baseline, which relies
21

---

Published as a conference paper at ICLR 2025
Reference Images
y=
y=1
y=
Reference Images
Y=O
= y=1
Figure 15: More results on scaling standard deviation of learned prompt distribution.
Image Set
Method
CLIP-I*
DINOT
Density
Coveraget
FIDJ
Textual Inversion
0.82
0.49
0.10
0.12
98.76
French
DrcamBooth
0.85
0.56
0.02
0.04
100.24
Bulldog
Custom Diffusion
0.84
0.54
0.09
0.14
98. 39
Ours
0.85
0.61
0.23
0.34
88.81
Textual Inversion
0.75
0.20
1.00
0.56
194.74
Dogs
DreamBooth
0.76
0.20
0.68
0.44
211.34
Custom Diffusion
0.77
0.21
1.00
0.64
179.51
Ours
0.77
0.21
0.93
0.64
173.74
Textual Inversion
0.70
0.18
1,50
0.23
214.70
Quadrupeds
DreamBooth
0.69
0.16
2.35
0.23
228.39
Custom Diffusion
0.70
0.15
2.20
0.21
217.58
Ours
0.71
0.18
1.89
0.24
210.93
Table Comparison of our method with baselines on feature-based metrics over different granular- ities of reference images.
22

---

Published as a conference paper at ICLR 2025
Dog
Cat
Sesame Street
Gundam
Figure 16: More results on composition of multiple prompt distributions using different weights.
23

---

Published as a conference paper at ICLR 2025
Textual inversion
DreamBcoth
Custom Diflusion
Ours
Real
Bulking
Dags
Quadruseds
Figure 17: Visualization of reference and generated images using different methods at different class granularity.
fixed learned token prompt the text-to-3D model, method samples from the learned prompt distribution, resulting in greater diversity in the generated 3D assets.
Following Corso et al. 2023). we perform quantitative analysis of in-batch cosine similarities among features of rendered images. Specifically, we compute the average pairwise cosine similar- ity of DINO and CLIP-I features within each image set. The results shown in [Tb. 5 demonstrate that 3D assets generated using our learned prompt distribution exhibit significantly greater diver- sity compared to those generated using Textual Inversion's 's learned token, which aligns with visual results. All samples are generated without using random seeds.
Method
CLIP-1I
DINO
Textual Inversion Ours
0.63
0.31
0.53
0.14
Table 5: Quantitative comparison of 3D generation diversity using averaged pairwise cosine simi- larity of features of rendered images. Lower similarity means the generated set is more diverse.
D ABLATION STUDY
Number of Prompts We additionally ablate the number of prompts in personalized generation. We randomly select 4 sets of reference images from our evaluation set and compute the average main Section 4.1. In
performance based on automatic quality and diversity metries introduced Fig. we show the effect of K in terms of both generation quality and diversity. We observe positive correlation between the performance (in terms of both quality and diversity) and the number of prompts. More prompts offer more flexibility for methods to model a better distribution of prompts, thus enabling the model to encapsulate content better (quality) and adapt to various nuances of the training images (diversity).
24

---

Published as a conference paper at ICLR 2025
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Reference
Figure 18: Visual results of training on only reference images.
Textual Inversion
Ours
Textual
Inversion
Ours
Figure 19: Qualitative comparison of diverse 3D generation.
Orthogonal Loss We also test different value of weight of orthogonal loss added during prompt distribution training. As shown Fig. our choice À (x-axis) can achieve good balance on quantitative metrics.
25

---

Published as a conference paper at ICLR 2025
# truining images
IN
IN-V2
IN-S
IN-R
IN-A
per class
Top1
Top5
Top1
Top5
Topl
Top5
Topl
Top5
Topl
Top5
T0
44.8
73.8
48.5
71.4
27.7
48.8
57.1
78.1
15.3
44.4
T00
56.7
79.8
62.0
81.5
30.4
50,9
54.7
78.4
188
$5.0
500
64.3
84.0
61.7
81.6
25.2
45.8
53.0
14.8
15.7
50.4
300)
63.6
82.9
60.5
80.0
67.8
11.6
45.0
Table 6: ImageNet classification accuracy on different real test sets by training a classifier on syn- thetic ImageNet (IN) generated by DreamDistribution using different number of real training images per class. Our experiments show that using 500 training images per class yields the highest final clas- sification accuracy on real validation set.
Number of Prompt Token Vectors We evaluate the impact of varying the number of token vectors M used in each learnable prompt. As shown in Fig. 10 similar to number of prompts, we observe positive correlation between performance and the number of learnable prompt tokens. However, this effect is less pronounced compared to the impact of changing the number of prompts, and increasing the number of learnable tokens results longer training times.
E
SYNTHETIC DATASET
E.1 MORE ANALYSIS ON SYNTHETIC DATASET
Number training images We experiment with different number of training images. We use randomly selected 10, 100, 500 images per class, well as all ImageNet training images to train our learnable prompts and generate same size synthetic dataset. From results shown in Tab. 6 and Fig. 20 column & 2, we found that using about 100-500 images per class (7%-38% of the real training set) would be enough to reach high classification accuracy on real validation set, while using more data would not further improve accuracy. For validation sets of ImageNet variants, less training data would obtain higher accuracy due to the domain gap between the real training set and different validation sets.
Mixing synthetic data with real training data We also experiment with mixing different sizes of synthetic image data with the real training images. We mix additional 20%, 40%, 60%. 80% and 100% synthetic data with real training data, where 100% means the size of the mixed dataset twice of the size of the ImageNet training set, and the ratio of the number of real images to the number of synthetic images is approximately 1:1. As shown in Fig. columm 3 & 4, adding more synthetic data would improve the accuracy on ImageNet validation set. On the validation sets of different domains, however, adding more synthetic data would not show significant improving trends on accuracy.
E.2 IMPLEMENTATION DETAIL
For training ImageNet dataset, use the same training hyperparameters except for reducing the number learnable prompts to 10 per class. train for epochs for training prompt distribution and 300 epochs for training ResNet-50 using generated or mixed dataset. All results are averaged over runs of training using the generated or mixed dataset. For ImageNet-R and ImageNet-A, we only evaluate the overlapping classes with ImageNet-100.
E.3 MORE VISUAL RESULT
We show some generated training images using our method and compare them with generated im-
ages using baseline methods in FFg. 21 Fig. 22 Fig. 23 Fig. 24 Fig. 25 Fig. 26 Compared to the images generated using baseline methods, non-learning methods could generate images with diver- sity, however, the appearances are generally far from real images. Learning based personalization baselines generate images with high fidelity, but with very limited diversity. Our method in contrast can generated images that looks more like real image samples and at the same time with significant diversity.
26

---

Published as a conference paper at ICLR 2025
Imagehet
Imaguiteet
InaqeeNet
ImageNit
66.90
00.00
66.50
60.2
W01
06.25
00.00
90.0
106
1000
930
30:0
25
100
25
100
Murbbr cwt.arcctrrrrrr.. Images das
twwtiingntttinnggggg mapis
Fancerappe of tynttiiccccngg Images added
Perzontage of synthhettcc Images adde
V2
maaellle
Imsgehet 92
Imageher V2
30
60-2
95.4
66.2
Took
T523
95.0
300
2000
300
3000
23
30
10C
25
50
100
InageNet-3al
Impoofiet Swotch
50.0
47.5
63.25
13
00.0
20.5
500
3000
Imagehiet
Imagefiet
Imagelle
Imagelet
64.5
80.5
10
85.0
mumber
ImageNoc54
Teat
Tass
Tip!
Taas
100
Mumber
Ramber
Figure 20: Column & 2: Top-1 and top-5 accuracy ImageNet validation set versus using different number of training images to train prompt distribution. Column & 4; Top-1 and top-5 accuracy ImageNet validation set versus percentage of synthetic images added to the real training set.
27

---

Published as a conference paper at ICLR 2025
F LIMITATIONS
Despite the ability of our method to generate diverse novel in-distribution images, it does have certain limitations. Specifically, our method may struggle capture visual features when the number of training images is limited and very diverse, Moreover, the Gaussian distribution assumption could be overly restrictive depending on the training images and the text encoder's s latent space. In the future, we hope to find a more robust approach to learning distributions from a few, highly diverse images, with more accurate assumptions and resilient distribution forms.
28

---

Published as a conference paper at ICLR 2025
00
Class Names
CLIP Prompts
ImageNet-SD
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 21: Visualization of generated images on ImageNet ambulance.
Class Names
CLIP Prompts
ImageNet-SD
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 22: Visualization of generated images on ImageNet cabbage.
29

---

Published as a conference paper at ICLR 2025
Class Names
CLIP Prompts
ImageNet-Si
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 23: Visualization of generated images on ImageNet honeycomb.
Class Names
CLIP Prompts
ImageNet-SD
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 24: Visualization of generated images ImageNet lorikeet.
30

---

Published as a conference paper at ICLR 2025
Class Names
CLIP Prompts
ImageNet-SD
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 25: Visualization of generated images on ImageNet papillon.
Class Names
CLIP Prompts
ImageNet-SD
Textual Inversion
DreamBooth
Custom Diffusion
Ours
Real
Figure 26: Visualization of generated images on ImageNet pirate ship.
31


Reviews:
[{"review_id": "DngTicljcm", "reviewer": "Reviewer_RW1y", "summary": "This paper proposes a method to learn personalized prompts that not only capture the common concept embedded in the given reference images but also reflect variations between them. Specifically, instead of learning just one soft prompt, multiple soft prompts are learned, and their mean and deviation are used to generate diverse input prompts. To ensure that each prompt learns significantly different values, an additional objective function that encourages orthogonality between the prompts was introduced during training. The authors demonstrate that this method successfully learns prompts that reflect diverse characteristics of given reference compared to previous works and suggest that it can be extended to 3D generation models.", "questions": "- In models like Custom Diffusion, if various text prompts containing the learned special token are used, isn't it possible to generate diverse images?\n- What are the results when only one learned soft prompt is used? Can we confirm what characteristics each soft prompt has learned?\n- How the results change according the value of M?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The explanation of the target problen, the method proposed to solve it, and the experimental setup are clear.\n- The effectiveness of the proposed method is convincingly validated through various experimental results.", "weaknesses": "- The baseline methods do not share the target problem of this work. Models like DreamBooth or Custom Diffusion focus on learning only the common concept by eliminating variation, so they may not be fair comparison targets.\n- In Figure 5-(a), when only the mean value is used, it seems that this model also fails to generate variable images. This suggests that simply learning one soft prompt and randomly perturbing it might suffice. However, without ablation studies, it is hard to be sure whether the proposed method itself is truly meaningful.\n- In task like this, human evaluation metrics are particularly important. However, the result in Figure 4-(b) is based on evaluations from only 10 participants, which undermines reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3ptf3Yq15", "reviewer": "Reviewer_3gbS", "summary": "This paper presents DreamDistribution, a solution for personalized text-to-image diffusion-based image generation. The key goal is to model the textual distribution of a training set, and guide the diffusion models to sample images that are adhere to the semantics of the reference images. Some learnable soft textual prompts and the reparameterization trick are introduced, enabling the generation of novel images by sampling prompts from the learned distribution. The authors also show the adaptability of the proposed DreamDistribution by applying it to various downstream tasks. Quantitative and qualitative analysis demonstrates the superiority of DreamDistribution against competing tuning based methods.", "questions": "Please refer to the Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing of the paper is clear and easy to follow.\n2. Extensive visual results are shown for intuitive comprehension of the proposed method.\n3. The intuition presented in the paper is simple yet quite straightforward, supported by the comprehensive comparative experiments.\n4. The proposed method only requires maintaining and updating a few learnable text prompts, which greatly preserves the image prior of the diffusion model.", "weaknesses": "1. The model relies heavily on the text-image alignment of diffusion, and since the diffusion model itself has not been tuned, I am curious about whether DreamDistribution can generate the new concepts if the diffusion model has not seen the reference images before.\n2. I did not observe a significant visual improvement compared to textual inversion. And maybe the performance gap will become even smaller when stronger base models are utilized.\n3. The novelty of proposed method is quite limited. Learnable tokens are very common in NLP and visual-language tasks, and the orthogonal loss is borrowed from some dictionary learning based methods, the reparameterization trick is borrowed from VAE training. These narrow the technical contribution of the proposed DreamDistrition.\n4. DreamDistribution requires large amount of reference domain images to converge. I wonder how it performs when training samples are limited to 1~10, and how can it beats textual inversion in such scenarios.\n5. The method is based on the assumption that the text distribution corresponding to a concept follows a Gaussian distribution. Is this a reasonable assumption? Also, does the number of learnable prompts required for different concepts need to be the same?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DvCaFH6vwh", "reviewer": "Reviewer_BPjf", "summary": "This paper proposes distribution learning in the prompt space to increase the diversity for personalized generation problems.\nThe effectiveness of the proposed method has been proved on text-to-image, text-to-3D and synthetic dataset generation.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing of this paper is good and easy to follow.\n2. The proposed soultion for increasing the generation diversity is straightforward and effective.\n3. With application to different scenarios (text to image, text to 3D, and synthetic dataset generation), the authors provide different perspectives to demonstrate the effectiveness of the method.", "weaknesses": "1. The methods compared in this manuscript are no longer the state-of-the-art for personalized generation.  It would be better if the authors can compare with more recent and widely used methods, eg, IP-Adapter. \n2. The main contribution of this paper happens in the prompt space. However, I didn't see too much discussion and experiments that focus on the prompt space. For example, how does the orthogonal loss affect the generation quality and diversity? How different the K learnable prompts will be if the prompt distribution learning is not used?\n3. I didn't see either qualitative or quantitative comparisons with our personalized text-to-3D methods, although the authors claim the diversity is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ta87pS7QvY", "reviewer": "Reviewer_TdJX", "summary": "The paper proposes a method for personalized image generation. Unlike prior approaches that rely on a fixed text embedding, this paper introduces a probabilistic formulation that samples from a prompt distribution, parameterized by a set of learned soft prompts. They construct a benchmark containing 12 visual concepts (e.g., instances of real objects and illustrations). They also provide a few qualitative examples on text-guided editing, composing prompt distributions, and text-to-3D. Finally, they use their method to generate a synthetic variant of ImageNet, where they demonstrate that a classifier trained on their synthetic images outperforms those produced by prompting or other personalization baselines.", "questions": "- Below are a few minor comments.\n    - L375: missing space; should be “…text-editing **generation. We**…”\n    - L413: should be “Text-**guided** Editing”\n    -  L50-53 is very tangentially related to the rest of the paper; I think the first paragraph of the introduction could benefit more from the motivation and technical context of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is novel and clearly explained; the method introduces a probabilistic formulation that helps with sample diversity. The paper is also careful about the scope of its claims; it studies “different instances in a same category set” not “same-instance personalization” (L357-358).\n- The synthetic data generation experiment in Sec 4.5 is a nice metric. Existing personalization methods perform much worse than text-only prompting for producing synthetic data, but the proposed method is able to perform better.", "weaknesses": "- The paper could benefit from providing more details about Sec 4.5.\n    - For each class, how many images was each personalization method trained on?\n    - What is the distribution of noise seeds used to generate the images in Table 1 (e.g., is each synthetic image derived from an independent and completely random sample)?\n    - It would be nice if the paper quantitatively verified the claim in L528 (other methods perform worse because they “show limited diversity”) with a diversity metric.\n    - It would be nice if the paper provided a qualitative figure similar to Figure 3 for the ImageNet classes.\n    - In Table 1, I would be curious to see a synthetic “upper bound” where one uniquely captions each real ImageNet image and generates images from this diverse pool of captions.\n    - In Table 1, I would be curious to see a personalization baseline that introduced some randomness, e.g., textual inversion with Gaussian noise added to the learned text embedding.\n- The paper could benefit from more clearly stating the types of visual concepts that are in scope for this work.\n    - It would be nice if the investigation personalizing to ImageNet classes were extended to class hierarchies. For example, the paper could compare results when considering the class granularity to be “animal” vs. “dog” vs. “Rough Collie” vs. “Lassie”. Then it would be easier to see at what granularity the probabilistic formulation is vs. is not necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)