Diverse Text-to-Image Generation via Contrastive Noise Optimization
Authors: Byungjun Kim
Soobin Um
Jong Chul Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Noise Optimization
Diverse Generation
Primary area: generative models

Abstract: Text-to-image (T2I) diffusion models have demonstrated impressive performance in generating high-fidelity images, largely enabled by text-guided inference. However, this advantage often comes with a critical drawback: limited diversity, as outputs tend to collapse into similar modes under strong text guidance. Existing approaches typically optimize intermediate latents or text conditions during inference, but these methods deliver only modest gains or remain sensitive to hyperparameter tuning. In this work, we introduce Contrastive Noise Optimization, a simple yet effective method that addresses the diversity issue from a distinct perspective. Unlike prior techniques that adapt intermediate latents, our approach shapes the initial noise to promote diverse outputs. Specifically, we develop a contrastive loss defined in the Tweedie data space and optimize a batch of noise latents. Our contrastive optimization repels instances within the batch to maximize diversity while keeping them anchored to a reference sample to preserve fidelity. We further provide theoretical insights into the mechanism of this preprocessing to substantiate its effectiveness. Extensive experiments across multiple T2I backbones demonstrate that our approach achieves a superior quality-diversity Pareto frontier while remaining robust to hyperparameter choices.

Published as a conference paper at ICLR 2026
DIVERSE TEXT-TO-IMAGE GENERATION VIA CON- TRASTIVE NOISE OPTIMIZATION
Byungjun Kim* 11
Soobin Um Jong Chul Ye
1 Graduate School of AI, KAIST, Daejeon 34141, Republic of Korea 2 Department of AI, Kookmin University, Seoul 02707, Republic of Korea (app54781) jong. ye @kaist. ac.kr
soobin um8 kookmin ac kr
(a) DDIM
(b) CADS
(c) Ours
Figure 1: Example results from our diverse image generation approach. Three distinct prompts are used: (top) "A person skiing on a very snowy slope", (middle) "A cow sits in a truck with hay barrels in it", and (bottom) "A sitting a couch next to dog". Standard DDIM (a) exhibits pronounced mode collapse, producing repetitive images and often failing to capture complex com- positional details. CADS (Sadat et al. 2024) (b) improves diversity but still yields limited variation and occasional prompt misalignment. Our method (c) delivers markedly greater diversity and fi- delity, generating wide range of images that remain strongly aligned with the input text.
ABSTRACT
Text-to-image (T2I) diffusion models have demonstrated impressive performance in generating high-fidelity images, largely enabled by text-guided inference. How- ever, this advantage often comes with a critical drawback: limited diversity, outputs tend to collapse into similar modes under strong text guidance. Exist- ing approaches typically optimize intermediate latents inference, but these methods deliver only modest gains or remain sensitive to hy- perparameter tuning. this work, we introduce Contrastive Noise Optimization, simple yet effective method that addresses the diversity issue from distinct perspective. Unlike prior techniques that adapt intermediate latents, our approach shapes the initial noise to promote diverse outputs. Specifically, we develop a contrastive loss defined the Tweedie data space and optimize a batch of noise latents. Our contrastive optimization repels instances within the batch to maximize diversity while keeping them anchored to a reference sample to preserve fidelity. We further provide theoretical insights into the mechanism of this preprocessing to substantiate its effectiveness. Extensive experiments across multiple T21 back- bones demonstrate that our approach achieves superior quality-divers Pareto frontier while remaining robust to hyperparameter choices.
conditions during
'These authors contributed equally to this work. 1 "oorrsponniig author.

---

Published as a conference paper at ICLR 2026
Mc-WOULD
Altrocttion reguinted by Ropulston
(zi - (reehh((h))
ja.
fon (Hirr)
ex Cg ( (mn)) E mp (V Par) /1))
8
Tweedie Domain
Figure 2: Conceptual overview of contrastive noise optimization. Our method enhances gen- eration diversity by optimizing the initial latent vectors, Zr, prior to the DDIM sampling process. We employ an InfoNCE loss that operates on a batch of noise vectors. This loss function pushes the optimizing sample (blue dot) away from all other negative samples in the batch to maximize separation. To preserve semantic fidelity, this repulsion counterbalanced by an attraction force that pulls the anchor towards its original, non-optimized version (the positive pair), which acts as a fixed reference point. The attraction coefficient y regulates this anchoring force, stabilizing the fidelity-diversif trade-off. This pre- pre-processin step effectively diversifies the final image outputs without fine-tuning or altering the foundational diffusion sampler.
INTRODUCTION
In recent years, diffusion models (Ho et al. 2020 Song et al. 2021 Rombach et al.) 2022) have emerged the leading paradigm for rext-to-image (22I) generation. A key driver of their success the use of text-guided inference, which steers the generation process to produce images that are not only high-fidelity but also closely aligned with a given prompt. To maximize this alignment and en- hance image quality, practitioners often employ strong guidance mechanisms, with techniques like Classifier-Free Guidance (CFG) (Ho Salimans 2021 becoming a standard practice. However, this pursuit of high fidelity comes significant cost: pronounced lack of diversity. Under strong textual guidance, the model's 's outputs often collapse into few dominant modes, failing to capture the rich variety of interpretations text prompt have. This fdelity-diversite trade-off (Dhari- wal & Nichol, 2021) remains a critical bottleneck, severely restricting the creative potential of T21 models.
To address this challenge, common line of work has focused on interventions during the iterative denoising process. These approaches typically optimize intermediate latents (Corso al. 2024 Kirchhof et al. 2025) or manipulate text embeddings (Sadat et al.) 2024 Um & Ye 2025b) to en- force separation between samples, while other strategies rely multi-agent systems or complex fine-tuning schedules (Ghosh et al. (217)) Although these methods have shown promise, they often require repeated adjustments during sampling and thus suffer from notable limitations, including substantial computational overhead (e.g., Diversity Prompt (Um & Yel 2025b); see Table (8) and lim- ited gains under accelerated few-step models, where opportunities for intervention are intrinsically scarce.
In this work, we tackle the diversity problem at its fundamental source by shifting the paradigm from inference-time interventions to initial-noise selection. We introduce Contrastive Noise Opti- mization (CNO), a simple yet powerful one-shot preprocessing framework that optimizes a batch of initial noise latents before sampling begins. Unlike prior methods that intervene repeatedly dur- ing the denoising trajectory, CNO performs a single, lightweight optimization step on initial noise and requires modifications adjustments during sampling. The key idea is to incorporate
2

---

Published as a conference paper at ICLR 2026
contrastive objective to shape Snveersit--encurraee initial noise distribution, drawing inspiration from the structure of InfoNCE-basee contrastive learning (van den Oord et al., 2019). Building on this foundation, introduce structural modifications tailored to the unique demands of diverse T2I generation, enabling diversity to be enhanced while retaining strong semantic fidelity.
At the core of our framework lies balance between two complementary forces: attraction term that anchors each optimized noise to its original counterpart, and a repulsion term that encour- ages semantic separation across samples. Crucially, impose this contrastive structure not in the raw noise latent space but in the Tweedie denoised prediction space, which provides the diffusion model's best estimate of clean data. Operating in this space allows the optimization act directly on meaningful semantic signals, making the refinement of initial noise substantially more effective.
To further control this interplay, we introduce a novel balancing parameter 7, which modulates the relative strength of attraction and repulsion. Its behavior is analytically grounded through an perspective (Proposition revealing how y governs the contributions of positive and negative pairs. We additionally provide a closed-form selection rule for that ensures robust behavior across batch sizes. Practical components such as adaptive latent pooling and a stop-gradient mechanism further enhance the efficiency and scalability of CNO for modern '21 backbones.
extended nutual-informaate
Comprchensive experiments demonstrate that our lightweight preprocessing substantially improves diversity across modern T2I frameworks (including Stable Diffusion (Rombach et al. 2022) while maintaining high image quality and text alignment. Notably, CNO delivers consistent improve- ments even under accelerated few-step samplers such as FLUX (Labs et al., (2025) and SDXL- Lightning (Lin al., 2024), where existing diversity approaches exhibit limited gains (see Sec- tion D.I.
We introduce paradigm shift from inference-time interventions to initial-noise selection, ad- dressing the diversity problem at its source while eliminating repeated per-step adjustments and remaining effective even in accelerated few-step samplers.
We develop contrastive noise optimization framework tailored for diverse T2I generation, fea- turing the balancing parameter y, extended mutual- nutual-information analysis, and efficient heuristics such as adaptive latent pooling and stop stodggradeet..
We demonstrate state-of-the-art diversity-quality trade-offs across major T2I backbones, in- cluding SD1.5, SDXL, SD3, and fast few-step samplers such as FLUX and SDXL-Lighuning
2 RELATED WORK
Improving the diversity of diffusion models has recently attracted much attention, mainly due to their increasing use critical applications such as text-to-image generation (Rombach et al.] 2022). One prominent effort is CADS (Sadat et al. 2024], which enhances sample diversity by gradually annealing noise perturbations on conditional embeddings. Although effective, their approach is sen- sitive to the noise annealing schedule and requires laborious hyperparameter searches see the diversity gain. A fundamentally different strategy is seen in Particle Guidance (PG) (Corso et al.] 2024). The idea is to repel intermediate latent samples that share the same condition, thereby en- couraging the final generated samples to exhibit distinct features. While it does not require difficult parameter searches like CADS, it often provides limited diversity gains (Kirchhol et al. 2025). This approach shares similar spirit PG (Corso et al. 2024) and incorporates diversity-improvine guidance for repelling intermediate latent instances during inference, yet in sparse manner, iee, not at every inference timestep. A key distinction from ours is that its diversity optimization (by injecting guidance) performed over inference time, which may be more expensive compared to ours that focuses the initial latent space.
related yet different task is to generate minority samples low-density instances in the data manifold (Sehwag et al.) 2022; Um & Ye 2023 2025b). Pioneer works in this area are offered by /Sehwag et al. (2022); Um & Ye (2023) which share similar idea of incorporating classifier guidance (Dhariwal & Nichol 202 to push intermediate samples toward low-density regions. The reliance on external classifiers was addressed in Um & Ye (2024) 2025b); Um et al. (2025). offering self-contained approaches for producing minority samples with diffusion models. While relevant,
3

---

Published as a conference paper at ICLR 2026
the task of generating low-density minority samples is distinct from improving diversity and does not guarantee distinct outputs. Notably, MinorityPrompt (Um & Ye 2025b) considers sot-tto-mmag generation and provides prompt optimization framework that can also be used to enhance the diver- sity of generated samples. However, it requires optimizing the diversity-improt inference, which imposes substantial computational overhead (Um & Ye 20256).
prompt during
Initial noise optimization in diffusion models has been explored in various contexts (Guo et al, 2024 Ahn et al. 2024). One instance InitNO (Guo et al. 2024), where the idea is optimize the initial noise latent to promote improved prompt alignment in texi-to-image generation. A distinction with respect to ours is that their focus is on enhancing text adherence, unlike ours. Another notable work was done by Ahn et al. (2024), who aim to characterize the influence of classifier- ffreeerr-eeeeee guid- ance (Ho & Salimans| 2021) through a properly optimized latent noise, enabled by an additional neural network that maps to the optimal noise. While interesting, their focus is inherently distinct from ours. To the best of our knowledge, our framework the first incorporate the idea of noise optimization for addressing the diversity challenge of diffusion models.
3
PRELIMINARIES
3.1 LATENT DIFFUSION MODELS
Latent Diffusion Models (LDMs) (Rombach al., 2022
improve upon traditional Denoising Dif- fusion Probabilistic Models (DDPMs) (Ho et al.) 2020) by performing the diffusion process a computationally efficient, lower-dimensional latent space. LDMs first use pre-trained autoencoder to map a high-resolution image X into compressed latent representation, Zn {xx)) The diffu- sion process is then applied directly these latent vectors.
The forward process is Markov chain that gradually adds Gaussian noise to an initial latent vector Zo over series of I' discrete timesteps. At each step t, the transition is defined as:
qz; 1 (z, v - 1,,
where {8 is a fixed variance schedule that controls the noise level at each step. A key property of this process is that the marginal distribution at any arbitrary step can be expressed closed form conditioned only on the initial latent Zg:
azz Zu) = NNzz v Vāzzo, (1 II
a. As t increases towards the signal term va, approaches zero, and the variance 1 - approaches 1. This ensures that the noised latent 2r reliably converges to an isotropic Gaussian distribution (N I), regardless of the initial latent vector Zo. Once the reverse process generates clean latent, the decoder D is used to map it back to the pixel space.
where we define a, = Br and a. II'= a,
3.2 REVERSE PROCESS AND DENOISING VIA TWEEDIE'S FORMULA
The generative process is achieved by reversing the forward process, conditioned external in- formation such as text embedding с for Text-to-Image (T2I) synthesis. This involves learning a model Po (z,-1 Zz, c) that upproximates the true posterior. In DDPM (Ho et al. 2020). this condi- tional reverse process is parameterized as a Gaussian whose mean is learned by a neural network eg c.),
1
Be VI- coccz,,,,, t,
Po 2), = N 2t-1:
of
/
The core of this process is the network Es. which is trained to predict the noise component from the noisy latent vector z, based on the condition The key insight is that this trained network can be used to directly estimate the original clean latent Zu at any timestep t. This denoised estimate 2n is implemented via Tweedie's formula (Chung et al. 2025 noise model takes the form:
Um & Ye 2025a), which for specific
1
20 2 c)
(z+ v1 dn t,c))
Voz
This equation forms the foundation of the iterative denoising process in many conditional diffusion models, allowing for the generation of latent vectors that align with the given context c.
4

---

Published as a conference paper at ICLR 2026
3.3 INFORMATION NOISE- CONTRASTIVE ESTIMATION (INFONCE)
Information Noise-Contrastive Estimation (InfoNCE) (van den Oord et al. 2019) is fundamen- tal objective for self-supervised representation learning (Chen al. 2020 Kim et al. 2021; Jang
et al,, 2023] Kim et al. 2025). aims to construct an embedding space that maximizes mutual information (Cover) 1999) between representations of positive (similar) pairs while minimizing
for negative (dissimilar) pairs. The learning process can be viewed as a classification task in which,
for a given anchor sample, the model must correctly identify its positive counterpart from set of
negative samples.
Specifically, for an anchor embedding vector Z;, its positive pair z!", and set of B 1 negative samples { the InfoNCE loss is formulated as
B
exp(f(z;, )/T)) EE= exp(f 2/)/T) ).
LInENNCE == B - log
Σ 1=1
(5)
where -(:--) denotes similarity measure (e.g.. cosine similarity) between two representation tors, and T is a temperature parameter controlling the sharpness of the distribution. Intuitively, the loss encourages the anchor to be close to its positive pair while pushing it away from all negative samples, thereby tightening intra-class similarity and enlarging inter-class separation in the embed- ding space.
4 PROPOSED METHOD
4.1 OPTIMIZING INITIAL NOISE WITH CONTRASTIVE LOSS BETWEEN TWEEDIES
core challenge in text-to-image diffusion models is that independently sampled initial noises žr often lead to generations that collapse into similar modes, even under varied stochasticity. Rather than intervening during the denoising trajectory as prior diversity methods, we enhance diversity at its fundamental source by optimizing the initial noises themselves before sampling begins. Our approach, which we call Contrastive Noise Optimization (CNO), refines a batch of initial noises using contrastive objective applied in the Tweedie denoised prediction space, enabling the noises to be semantically well-separated while remaining faithful to their original distribution. The full procedure is provided in Algorithm I in Section B.1
The algorithm proceeds as follows. First, we sample a batch of initial latent codes Zy = ( from standard Gaussian distribution (0, Hereafter, we denote the denoised estimate from Equation Eq. (4) as Zojc- Using this, we compute the initial target latents, ) by applying the denoising estimator defined in Equation Eq. 44) to this initial noise at timestep T. Each resulting % therefore the model's one-step prediction of the clean latent Zy from the noise z/7. These pre-computed latents then serve as fixed anchors, each defining a unique identity for its respective sample throughout the optimization.
Before computing the loss, we employ a practical optimization to enhance efficiency. The high di- mensionality of the latents (B, C,S, 8) makes the pairwise similarity calculation computationally intensive. We found experimentally that applying an adaptive average pooling operation to down- sample the latents to sufficiently smaller spatial resolution (B, C, w, w), where compromise performance (Section[5.I). This step substantially reduces memory usage and acceler- ates the similarity matrix computation, making the optimization process more tractable. Downsam- pled latents { J1=1+ )/ contrastive loss LeNo:
S, did not
1B are then normalized, and the noise {z} BB is updated using
exp(S(2) O 20/05 )/T
1 Σ
LeNO
log
(6)
B
EA_ (s) 2/T))
This loss function is designed to achieve two objectives simultaneously.
Attraction (Numerator). It encourages the current latent Z//r to remain similar to its corresponding initial target latent af This ensures that each sample maintains coherence with its initial concept and does not drift away during optimization.
5

---

Published as a conference paper at ICLR 2026
Repulsion (Denominator). pushes the current latent z//r to be dissimilar from all other current latents ( in the batch. This directly promotes diversity by forcing the latent representa- tions disperse within the batch.
By iteratively updating 27 with the gradient of this loss x, LcNo), we guide the initial noise vec- tors positions in the latent space that are predisposed to generating a diverse of images. Once the optimization is complete, this well-distributed batch of noise [2 is fed into a standard, pre-trained DDIM denoiser to produce the final images. Consequently, our method effectively en- hances output diversity through simple pre- pre-rrcessing stage that modulates the starting point of the generation, all without requiring any modifications the pre-trained diffusion model itself.
Stop-gradient for computational efficiency. Optimizing the loss in Eq. (6) requires backpropa- gation through diffusion models, which can incur substantial computational overhead. To mitigate this, we apply a stopgrad operator (Chen & Hel (22T)) on the model path used in computing the Tweedie's estimate. As also demonstrated in Ahn et al. (2024), this simple strategy yields significant savings in training cost with only marginal impact performance (see Table 8p.
4.2 GAMMA EFFECT: REGULATED ATTRACTION FOR STABLE IMAGE DIVERSIFICATION
In our proposed algorithm, the InfoNCE loss for a single sample within a batch of size B consists of attraction term (to itself) and B repulsion terms (from all other samples in the batch). When the batch size B is large, the cumulative repulsion force can become excessively strong. This risks pushing the optimized noise of the intended distribution, potentially leading the generation of less plausible or cut-of-distributiin images.
To mitigate this issue and achieve a more stable optimization, we introduce a coefficient, 7 to dy- namically regulate the attraction force. This is done by dividing the similarity term in the numerator of the loss function by 7. The modified InfoNCE loss is follows:
2 2
1 B
CNo
Σ
log
Σラ= (22 2//T)/T)
Empirically, we found that in our framework behaves similarly to Gaussian regularizer (Guo et al. 2024), which penalizes large deviations from the Gaussian prior. detailed analysis is pro- vided in Section C.3
Desirable Value for 7 The desirable value for y is derived by creating a balance between the regulated attraction force and the cumulative repulsion forces. We achieve this by equating the maximum value of the attraction term (numerator) with the sum of the maximum values of the B 1 repulsion terms. Assuming the maximum similarity score this balance can be expressed as:
cxp(L/(77)) (B 1)exp(1/7).
Solving for gives the following relationship:
7 =
For instance, in our common experimental setting where = 0.1 and 5, the calculated 7 is approximately 0.88, which is very close the fixed value of = 1.0 we have consistently used. For a fixed r = 0.1, the optimal y changes moderately with batch size B:
B 13 + 7 0.8 B= > = 0.7 B 775 + > 0.6
This shows that the batch size B grows larger, 7 is not highly sensitive. Therefore, using single, appropriately chosen fixed value for 7 can also yield stable results without significant performance degradation.
4.3 THEORETICAL INTUITIONS
We provide mathematical insights into our contrastive framework by establishing its connection to mutual information. We begin with the classical view of InfoNCE as variational lower bound

---

Published as a conference paper at ICLR 2026
mutual information, shown by van den Oord al. (2019). Specifically, the InfoNCE loss in Eq. (5) satisfies
LINONNEE 2 log B T(Z; pus))
(10)
where Y( is the mutual information between random variables X and Y. This inequality im- plies that minimizing LinfNCE indirectly maximizes (Z; Zpos). encouraging the learned embedding space to cluster positive pairs. However, this classical relationship does not clarify how negative pairs shape the embedding space an aspect that is critical in our framework, where negative sam- ples drive diversity.
To capture this effect, we augment the traditional bound to incorporate mutual information with respect to negative pairs. The following proposition formalizes this result.
Proposition The InfoNCE loss in Eq. (5) satisfies
LInfONCE 2 -I(Z; Zus) + I(Z; Zneg) + log 1)
(11)
where B denotes the batch size, and ((;Y) is the mutual information between random variables X and Y:
I(X;Y)= ExX,Y log p(X,Y) EXxXY log p(X 1Y) p(X)p(Y)
p(X)
The proof is provided in Section A.1 This result shows that the InfoNCE loss is inherently linked negative samples as well as positive ones: minimizing the loss decreases the mutual information with negatives while increasing that with positives.
Extension with Gamma. We further analyze our modified loss in Eq. which introduces coefficient to control the relative strength of positive pairs.
Proposition 2. For the loss function defined in Eq. 7 the following inequality holds:
CCO 2 - I(2; 7pos) + T(2; Zoog) log( B -1).
(12)
7
The proof is given in Section A.2 This proposition indicates that y scales the positive mutual infor- mation term, serving as a control knob to modulate the influence of positive pairs in our contrastive objective. We provide empirical results demonstrate the impact of y in the appendix; see Sec- tion C.2
5 EXPERIMENTS
Implementation Details. Our experiments are conducted on three distinct pre-trained text-to-image diffusion frameworks: Stable Diffusion v1.5 (SD1.5), SDXL, and SD3. We compare our method with state-of-the-arr zero-shot diversity samplers, including Condition- Annealed Diffusion Sampler (CADS) (Sadat et al. 2024) and Particle Guidance (PG) (Corso et al.) 2024), as well as the prompt- optimization-ban based diversity method of [Um & Ye (2025b). referred to as DiversityPrompt. All evaluations use text prompts randomly sampled from the MS-COCO (Lin et al. 2014) validation set. For each prompt, generate 3-5 images, yielding total of roughly 6-10 samples.
Evaluation Metrics. The goal of our research to enhance the Pareto frontier between image quality and diversity of generated images while maintaining high degree of relevance the text prompt. To quantitatively assess this, we used the following key metrics: CLIPScore, PickScore, Inwge-Reward for evaluating image quality, and Vendi Score, Mean Pairwise Similarity (MSS) for diversity. Details for those metrics appear in Section B.2
5.1 RESULTS
Comparison with Existing Zero-Shot Diversity Samplers. The results in Table demonstrate the effectiveness of our approach. Our method achieves high performance on the key diversity met- rics, Vendi Score and MSS, consistently outperforming all baselines across the different foundation models. While performance on Density and Coverage is highly competitive, our approach's 's strong results on Vendi Score and MSS prove its robust, model-agnostic ability to mitigate mode collapse.

---

Published as a conference paper at ICLR 2026
Method
CLIP
MSS
DDIM
0.7018
0.6706
0.6033
0.7382
31,5863
21,5081
0.2222
0.1657
4,6949
PG
0.6940
0.7024
0.5975
0.7446
31,3222
21,2086
0.1712
0.1426
4,7630
SDI.5
CADS
0.6866
0.7240
0.5686
0.7292
31,4663
21.2933
0.1137
0.1330
4.7805
Diversity
0,5839
31,5457
21.3510
0.1332
0.1393
4,7599
0.7308
31,4525
21.3779
0.1284
4.7855
DDIM
0.6858
0.6538
0.5713
0.7368
31,8788
22.4761
0.7302
0.2169
2.8377
PG
0.5820
0.7088
0.3855
0.5606
31,5679
22.1631
0.6950
0,2050
2.8545
SDXL
0.5262
22.2078
0.1765
2.8864
0.6720
0.5553
0.7273
0.1623
FM-ODE
0,7184
0.5828
0.6472
0.6770
31,7783
22,5763
1,301
0.3028
4,2205
PG
0.7782
0.3900
0.8110
0.7370
32,0463
22,3500
1,0577
0.3066
4,2097
SD3
0.6110
0.6682
22.4987
0.2960
4,2487
0.7100
0.6573
31,7713
22.5647
0.2909
Table I: Quantitative results of zero-shot diverse samplers. Our proposed method is bench- marked against standard samplers (DDIM and Flow-Matching ODE) and state-of-the- diversity- enhancing techniques: PG (Corso et al. 2024) and CADS (Sadat et al. 2024). DiversityPrompt refers to the prompt-optimization- evaluation demonstrates that our approach consistently achieves superior performance in diversity metries, including MSS(J) and Vendi Score(1), across Stable Diffusion 1.5, XL, and 3. Notably, enhances diversity while effectively preserving image quality and prompt fidelity, successfully nav- igating the fidelity-diversin trade-off by optimizing the initial latent space.
diversity approach developed in Um & Ye (2025b). The
DDIM
CADS
PG
Ours
4.850
4,850
4.825
4.825
Vondi Score
Vendli Store
Vendi Score
4.800
4.800
4.800
0.775
4.775
4.778
4.750
4.750
4.700
4.725
4.725
4.725
31.25
21.4
0.1
ClipScore
PickScore
ImageReward f
Figure 4: Pareto curves of diverse sampling methods between Vendi Score and text-to-image align- ment metrics. For our methods, we use N Nops 5, 1.0, w 8, 0.1 in common.
Crucially, these substantial gains diversity do not com- promise generation quality. Our method maintains strong prompt fidelity, evidenced by competitive CLIP scores, and sustains competitive superior Pick-Score pared to CADS across all Stable Diffusion models. This indicates our outputs are only more varied but also aesthetically preferable. This quantitative strength is mir- rored in our qualitative results (see Figure 5), where our model shows particular strength on complex com- positional prompts that cause competitor methods to fail. Where gains in diversity often come at the cost of quality, our method achieves both, delivering outputs that are not only more varied but also consistently high in fidelity.
4.50
4.70
Vendi Score
4.78
117
4.77
16
64
21
225
21,250
21,255 21.350 PickSrore
21.255
21,000
Figure 3: Ablation the window size w. The Pareto frontier PickScore vs. Vendi Score.
Figure fillustrates the quality-diverse trade-off by plot- ting the Pareto frontiers for our method and key baselines. The plots reveal a clear and compelling advantage for our approach, which establishes a domi- nant frontier across metrics trained large-scale human preferences. This is most evident in the PickScore and amage-Reward charts, where our method is strictly superior to competitors like CADS, indicating outputs
aesthetically pleasing for any given level of diversity. While
8

---

Published as a conference paper at ICLR 2026
@ DDIM
(b) Particle Guidance
(c) CADS
(d) Ours
Figure 5: Qualitative comparison with pre-existing zero-shot diverse generative methods For the prompt "A white rabbit on the moon. "(left) and "A green unicorn in snowy forest" "(right), we compare our method (d) with baseline approaches. Our method successfully generates high-fidelity images that are strongly aligned with the text prompts. In contrast, the other methods exhibit various failures.
CADS may achieve marginally higher peak CLIPScore, this metric known to favor rigid seman- tic alignment rather than creative or aesthetically superior interpretations. In contrast, our method's dominance across both PickScore and Image-Rewatt demonstrates more intelligent trade-off. prioritizes what human user would find visually appealing and contextually appropriate over mechanical, word-for-word adherence to prompt. This quantitative strength is mirrored in our qualitative results (see Appendix D.4), where our model uniquely succeeds complex composi- tional prompts that cause competitors to fail.
Effect of Window Size We conduct an ablation study to investigate the effect of the downsam- pling window size, w, applied to the Tweedic latent shape " [ Line 1Op. This step is crucial for capturing the global structure of the initial noise prediction while reducing computational cost. We experiment with E (4, 8, 16) and compare these against the baseline of 64, which effectively uses the full-resolution latent shape.
The results are illustrated in the PickScore-Vente Score Pareto frontier in Figure 3 As shown, an ag- gressive downsampling with wo leads to noticeable performance degradation, failing to match the frontier established by larger window sizes. In contrast, moderate downsampling with and w 16 achieves highly competitive performance compared to the = 64 baseline. This suggests that moderate downsampling successfully preserves the essential structural information for diversification while benefiting from increased computational efficiency. Excessive downsampling (w = 4), however, appears to discard critical details necessary for the optimization process. Based on these findings, we select 16 for our main experiments, as it provides the best trade-off' between performance and efficiency.
CONCLUSION
We introduced Contrastive Noise Optimization, a simple yet effective pre-processing method to address mode collapse in text-to-image (T2I) diffusion models. By applying a contrastive loss rectly to the initial noise vectors for given text prompt, approach ensures diverse starting points for generation, eliminating the need for the complex sampling guidance or laborious hyperparameter tuning required by prior work. Our method sets state-of-the-a the quality-diverse Pareto frontier, outperforming strong baselines on key diversity metrics without compromising prompt fi- delity image quality.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by the National Research Foundation Korea (NRF) under Grant RS- 2024-00336454, and the AI Computing Infrastructure Enhancement (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25- 120217). This work also supported by the Institute of Information & Communications Tech- nology Planning & Evaluation (ITP) grant funded by the Korean government (MSIT) (No. RS- 2019-11190075, Artificial Intelligence Graduate School Program(KAIST); No. RS-2025-02304967, AI Star Fellowship (KAIST); No. RS-2025-02219311 AI Star Fellowship (Kookmin University)).
REFERENCES
Donghoon Ahn, Jiwon Kang, Sanghyun Lee, Jaewon Min, Minjae Kim, Wooseok Jang, Hyoungwon Cho, Sayak Paul, SeonHwa Kim, Eunju Cha, et al. A noise is worth diffusion guidance. arXiv preprint arXiv:2412.038855, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. CFG++: manifold-constrrinned classifier free guidance for diffusion models. In The Thirteenth Interna- tional Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. Open- www. festt/foru?iii dooorummd==F7. 77.bbTTT
Review.net, 2025. URL httttss://openneeettttt
Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi S. Jaakkola. Particle guidance: non-i.idd diverse sampling with diffusion models. In The Twelfih International Confer- ence on Learning Representations, 2024. URL htttp:://openreetee:ee wwt.t/rfrrfummddd
KqbCvIFBY7
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pp- hcc/paper/2c2111 //hash/ 4 1aa223leec922e0ddd///22222223623 -ttp:///de2eeaafffcaaaaattteeeee html
8780-8794, 2021. URL https: httppppps/ppccccaagss.o...o newr.........n.................
Abst
Sandra Zhang Ding, Jiafeng Mao, and Kiyoharu Aizawa. Training-free free sketch-guided diffusion with latent optimization. CoRR, abs/2409.0033 2024. doi: 1044550AARXI.22000000011. URL
https://doi wtt.grgg............o 20..550/axxv..244. 00311
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. Trans. Mach. Learn. Res., 2023, 2023, httthttts///peeeeeee
http:wwweeeoooeeeeeoooeeeeeeeeoe foomm??
Id=g970Hb@ykk
Arnab Ghosh, Viveka Kulharia, Vinay P. Namboodiri, Philip H. S. Torr, and Puneet Kumar Dokania. Multi-agent diverse generative adversarial networks. CoRR, abs/1704.02906 2017. h
62906
Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. CoRR, abs/2310.1151x 2023. doi: 10.48550/ARXTVV2310.. 11513.
orgg (0555//aaxxv.2553 111.1.1.1111.111
Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang. and Di Huang. Initno: Boost- ing text-to-image diffusion models via initial noise optimization. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA. USA, June 16-22, 2024, pp. 9380-9389. IEEE, 2024. doi: 10.1109/CVRR3333322220000 10. 1111111000//19330 200.2.2224400 00896
URL https:
10

---

Published as a conference paper at ICLR 2026
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In StarieFFrancing Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7514-7528. Association for Computational Lin- guistics, 2021, doi: 10. 118.63///2222.EEELPMAMAMAAA WEN.LLLANNNPMI....... URL 18653/v vl 2021. emnlp main. 595
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URUhtttps://ppeneezai
met/forum: i 1d=
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpp::/pprocse///e/ htwp:pooeeeednnnnn.rrrrr.......e @c5bcffcc584af006l
wwtpssccpppp.eeeeee cc/ppaper/222//aastt
ab 01 79ca44-44--- Abstrbbbbbarr........ htm
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu, T2i-comphench: A com- prehensive benchmark for open-world compositional text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference
Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:: pip.rrs.npppp.. nips. www.iips.ccppppeee cttpcc/pppppeeceeeeeccceeeeee.ee capaper_fillee fiwes/ wttps//apprr//23/// Berchmarks.
Jiho Jang, Seonhoon Kim, Kiyoon Yoo, Chaerin Kong, Jangho Kim, and Nojun Kwak. Self-distilled self-supervised representation learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2829-2839, 2023.
Dachee Kim, Youngjun Yoo. Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9619-9628, 2021.
Euyoung Kim, Soochahn Lee, and Kyoung Mu Lee. Disentangled contrastive learning from syn- thetic matching pairs for targeted chest x-ray generation. IEEE Access, 2025.
Michael Kirchhof, James Thornton, Louis Béthune, Pierre Ablin, Eugene Ndiaye, and marco cuturi. Shielded diffusion: Generating novel and diverse images using sparse repellency. In Forty-second International Conference on Machine Learning, 2025. URL
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htLhttp::/ http:///s.pppp............cppees 1baccdbb0000 Aot.::aaaaaaaaaaatattaaat--o-oo
Abst Cenoraact-Coofffeecee httml
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. htttltt/t////rxxttt www.iii.orrraaaao abs htw.g//aaaaaa...a 220...15.66611.. 15742
o
Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, and Yadong Mu. Diffsplat: Repurposing image diffusion models for scalable 3d gaussian splat generation. In International Conference on Learning Representations (ICLR), 2025.
11

---

Published as a conference paper at ICLR 2026
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.33929 2024.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. CoRR, abs/1405.0311 2014. 1505.0312
Zheyuan Liu, Munan Ning, Qihui Zhang, Shuo Yang, Zhongrui Wang, Yiwei Yang, Xianzhe Xu, Yibing Song, Weihua Chen, Fan Wang, and Li Yuan. Cot-lized diffusion: Let's reinforce t2i gen- eration step-by-step. In The Thirty-nin minth Annual Conference on Neural Information Processing Systems, 2025. URL /otttps://openreveett
net fort/forum?e wwwp:mmmd-d999990000000
Yushu Pan and Elias Bareinboim. Counterfactual image editing with disentangled causal latent space. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 14512- 14522. IEEE, 2022, doi: (t//////VR2228.22222220000
battts::/doi.crgg/// URL http:: /ttp:
1109/CVPR52688 202.9922226688222222222222 01413
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. Forum?ld=F"Nys5ccTVy
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Com- puter Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, URL https://doi.
pp. 10674-10685 IEEE, 2022. doi: 221100//CPPS5268822222000 201..
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. We- ber. CADS: Unleashing the diversity of diffusion models through condition-amnealed sam- pling. In The Twelfth International Conference on Learning Representations, 2024. URL
https: /LEE::://openeeiew net htgpa///weeennnne//ooooorre/o/da ==r
Vikash Sehwag. Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11492-11501, 2022.
Jonathon Shlens. Notes on kullback-leibler divergee and likelihood. CORR, /bs/1444.2000, 2014.
URL http:tt//aaxxi..ro/// httpg/gggb///////// 2000
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttss://pperreeeeeses
net. wwrum? d=
EXTIG12RRHS
Soobin Um and Jong Chul Ye. Don't play favorites: Minority guidance for diffusion models. arXiv preprint arXiv:2301.12334 2023.
Soobin Um and Jong Chul Ye. Self-guided generation of minority samples using diffusion models. In European Conference on Computer Vision, pp. 414-330. Springer, 2024.
Soobin Um and Jong Chul Ye. Minority-focused text-to-image generation via prompt optimiza- tion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA. June 11-15, 2025, pp. 2025a. doi: 10.110//CVPR5223422225.. 2099099999 com/ Gereeration_via htt.:ion_iiia..oooooooooooiii wtw.:/attttncaanaa__ee
Computer Vision Foundation / IEEE,
URL Tee-ooooootcddttttt-eet-ee Image
thecvf.
12

---

Published as a conference paper at ICLR 2026
Soobin Um and Jong Chul Ye. Sinority-focusee text-to-image generation via prompt optimization. In Proceedings of the Computer Vision and Patter Recognition Conference, pp. 2025b.
Soobin Um, Beomsu Kim, and Jong Chul Ye. Boost-and-skip: simple guidance-free diffusion for minority generation. arXiv preprini arXiv:2502.06516, 2025.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019. httpp:ttts///ariii..oo wtt/ abtp//b///1////
037488
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent
optimization improves classifier guidance. In IEEE/CVF International Conference Com- puter Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. IIC.0///C.CV..... doi: 10.1099/CCCV510002222.00000
IEEE, 2023.
[ttpssts::///o1 http:///oo..rr/1..1111 1ttp://1.1111111//11 10000//110..0010101111001001.1..
2023.00669
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Prucessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htt:htt::// wwwp::pppppppppppppppppp n wiwp. http://aapppaaaappccccpppppepp
html
13

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF PROPOSITION
1
Proposition
1
The InfoNCE loss in Eq. (5) satisfies
LInfONCE 2 -I(Z; pos) + I(Z; Zneg) + log 1),
(13)
where B denotes the batch size, and I(X;Y) is the mutual information between random variables X and Y:
((; PXXX log ((XY) p(X)p(Y)
EXXxY log p(X (Y)
p(X)
Proof. In general, we can formulate the infoNCE loss Eq. (4) by setting g(: exp(f(:-))/)
from Eq.
g(z. Zpos
IINONCE = -Bp Byos Ford
log
(14)
g(z. Zpos) + > I (Z, Zneg
Let us simply notate Zposs Zuag as Zp. Zn respectively. This can be split into two terms:
}}
LinfONCE -Ep(azp) [log g(, Zp)] + Bp(x,,,, x..) log g(z, Zp) +
Σ g(z 2 (1)
(15)
Given that g(::) non-negative due to its exponential form, Eq. the positive pair similarity from second term:
15 has lower bound by omitting
LISOONCE 2 - -pp(x.z,) [log (2, kp)) + (p(z,.))
log
g 2. (0)
(16)
According to van den Oord et al. 2019), function g(z, z') estimates the probability density ratio p(x [x)) related to mutual information maximization. This formulation of induces the follow- p(2)
ing Eq.
p(z zp)
B-1 1 p(z (+)
LinfoNCE 2 -E -B((,,,) log p(z)
+ E Bp(n,x..)
log 1=1
(17)
p(z)
II(Z Zp) + Ep(a,
log
p(z 2(10)
Ep(x) loog(z)).
Using the property of logarithm and Jensen's Inequality, then
p(z
LInfONCE 2 -I(ZZp)) Ep(z,e.))
log
-lo((B- -1) -))-Ea -Ep(e) [log p(z)]
B
B B-1
2 -1(2;Zp))
Ep(z.e..))
[logp(z 2((0)] + +lg(B 1) Ep(z) [log p(z)].
B
(18)
Note that negative sample S,,S are sampled same distribution. According to Law of large numbers, 2Z!²) Ep(x.x.) [logp(z] 2n)].
can approximate B 2
B-1
E Elt,.... log
i=l
Therefore, the last inequality Eq. (19 holds.
LInCANCE 2 (Z; 7p) + Ep(z.2.,) [log knn) + log B 1) Exiz)) [log((z))
(19)
=1( k,) + I(Z: Zn. + log 1).
(20)
14

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF PROPOSITION 2
Proposition
2
For the loss function defined in Eq. 77. the the following inequality holds:
Cco - [(z; pos) T(Z; Zneg) log B -1)-
(21)
7
Proof. Compared to Equation Eq.
14 similarity function g(z='') is replaced with on (2,=²)
=
expl (z,z') = 29(2, Therefore, Equation Eq. (16) can be rewritten as:
B
Cooo 2 - Ep(a.,,) log(g(2,2p)) Ep(a..))
log
g
z, 2n
(1)
(22)
i=1
B
-- Bplan,)) [log (p) + pp.
log
g
(1)
(23)
Following similar derivations with Appendix A.1 we can simply show that Loxo T(Z; 7n) + log( B 1).
-p)(Z
L
B
IMPLEMENTATION DETAILS
B.1 PSEUDOCODE
Detailed algorithm for our sampling method is provided in Algorithm
T
B.2 EVALUATION METRICS
Image Quality and Prompt Alignment. To measure the quality and textual relevance of the generated images, we employ a suite of widely-recognized automated metrics.
CLIPScore. This metric evaluates the semantic consistency between generated image and its corresponding text prompt by calculating the cosine similarity of their embeddings from pre-trained CLIP model (Hessel et al., 2021).
PickScore. We use PickScore (Kirstain et al. 2023), a reward model trained on large-scale human preferences, to assess the overall aesthetic quality and prompt alignment of the im- ages.
Emage-RRearde As complementary metric, Image-Reward Reward (Xu et al. 2023 is another preman-preference-baaee based reward model that provides scores reflecting the general quality of the generated content.
Diversity, To evaluate the intra antra-prompt diversity the generated images, we utilize two distinct metries that capture different aspects of variation.
Vendi Score. The Vendi Score (Friedman & Dieng 2023 measures the diversity of a set of samples by analyzing the eigenvalue distribution of their similarity matrix. It provides holistic assessment of both the variety and balance of the generated images.
Mean Pairwise Similarity (MSS). This metric directly quantifies the average similarity be- tween all unique pairs of images generated for a single prompt. We first extract image features using the self-supervised descriptor for image copy detection (SSCD) model (Pizzi et al., 2022). Then, we compute the pairwise cosine similarity matrix of these features and calcu- late the mean of its off-diagonal elements. lower MSS value indicates higher diversity, as images in the set are, on average, less similar to one another.
B.3 HYPERPARAMETER SETTINGS
For our main experiments, we use a set of 2K prompts, with each prompt generating a batch of B images. The number of inference steps set to 50 for Stable Diffusion 1.5 and XI., and 28
15

---

Published as a conference paper at ICLR 2026
Table 2: Model-specific hyperparameters for our proposed method.
Hyperparameter
Stable Diffusion 1.5
Stable Diffusion XI.
Stable Diffusion 3
CFG Scale
6.0
6.0
7.0
Optimization Steps (Nopt)
3
3
3
Gamma (?)
1.0
1.0
1.0
Window Size (w)
16
16
32
Learning Rate (7))
0.01
0.01
0.001
for Stable Diffusion The batch size (B) was set for SD1.5 and SD3, and for SDXL. The specific hyperparameters for our proposed method are detailed in Table [ As shown in the table, most settings are shared across different T2I backbones, highlighting the robustness our approach to hyperparameter choices.
C FURTHER ANALYSES AND DISCUSSIONS
C.1 STEPWISE MECHANISM OF CONSTRASTIVE NOISE OPTIMIZATION
To clarify the exact role the attraction coefficient y and the gradient dynamics within our frame- work, provide stepwise breakdown using minimal batch example.
Loss Mechanism: Attraction and Repulsion. Consider a minimal batch of size = Let L1 denote the loss for the first initial noise vector z/-- Given a fixed reference anchor 2 2_.ref and the similarity function sim(-.-), the loss decomposes into two distinct forces:
21.r exp(sim(2 2017 (/(T)) (
L1 log ΣE= exp(sim(&) 2/)))/ /T)
sim sim(2))rr 21oof log (exp(sim($) (oppsimm(b/p/:/2br///)) exp(sim 1rrr//
7T
(A) Attmaction
(B) Repulsion
(A) Attraction.
This term encourages alignment between the current estimate 2/jr and its fixed "o/"" preserving semantic fidelity to the original concept.
reference 2l,eef
(B) Repulsion. It pushes 2/r away from other samples in the batch (e.g., () enforcing diversity by maximizing semantic distance.
Role of Gamma (?). The coefficient y serves regulator for the fidelity-diversity trade-off by exclusively scaling the attraction term. As derived in Proposition the effective objective is lower- bounded by:
Coxo 2 [(2; 2pmc) + T(Z; [(Z:7meg) + log B - 1).
7
This reveals that y modulates the strength of the positive mutual information. Specifically:
Decreasing < 1: Amplifies the attraction force (1/7 > 1). This is crucial for larger batch sizes (e.g., B = 3), where cumulative repulsion from B negative pairs can overshadow the single attraction term. A lower restores balance, preventing the sample from drifting too far from the anchor.
Increasing 7 1: Dampens the attraction, allowing the repulsion term dominate. This promotes greater diversity but risks reducing fidelity.
Practically, we find that setting 7 ss In - 1) + 1) provides a robust baseline for balancing these forces across varying batch sizes.
16

---

Published as a conference paper at ICLR 2026
Gradient Flow and the Stop-Gradient Strategy. To optimize 2y efficiently, analyze the gradi- ent flow. For the loss L1 defined above:
Active Gradient Flow: Gradients propagate through o/1r in both numerator and denomi- nator, driving it toward the anchor and away from negatives.
No Flow to Anchor: The reference & oo1
sef is fixed; thus, gradients flow through this term,
ensuring it remains stable guidepost.
Crucially, to backpropagate from 20TT to z7. we utilize a stopgrad operation on the diffusion model output Eg. By the chain rule:
T
22 C =
VoOoT
òzt
Deg (Zs(ZT) (ZT) āT
T Vioi
I
/OT
dzt
Calculating the full Jacobian aeg/ /år is computationally prohibitive. By applying st opogaad, we set this term to zero, simplifying the update to:
Ve, 2
VāT V2077
This approximation effectively updates the noise directly in the direction of the semantic gradi- ent. This strategy mirrors the effective Jacobian approximation used Score Distillation Sampling (SDS) (Poole al. 2023) and NoiseRefine (Ahn al. 2024), ensuring stable and efficient guidance without the cost of backpropagating through the U-Net.
C.2 GAMMA EFFECT: STABILIZING OPTIMIZATION PROCESS
To validate the stability of our proposed method, con- duct an ablation study the hyperparameter to ana- lyze its impact on output variability. We set y values of (1.0, 0.9, 0.8, 0.7}. To ensure that our findings are not contingent on a specific learning rate, we vary the learn- ing rate within the range of [0.01, 0.02). For each set- ting of 7, we generate 5 images per prompt, collecting a total of 5K images using SD1.5 model. We then compute evaluation metrics and calculate their sample variance to quantify the statistical variability of the outputs.
4.8
1.8
Vendi Score
4.1
4.80
7= 1.0
4.79
7
477
7
0.7
The results of this experiment are summarized in Table where we calculate the sample variance of those metries in E [0.01, 0.02). We observe a clear saturation effect: as y decreased from 1.0, the variance of the evalua- tion metrics stabilizes. Specifically, the most significant change variance occurs when 7 reduced from 1.0 0.9. Further decreasing to 0.8 and 0.7 yields diminishing changes in variance, indicating that the metries enter a stable regime. For instance, s exhibits a steady downward trend as y decreases, while the other metrics maintain relatively consistent level of variance for < 0.9.
4.77
21
21.25
21:30
21.35
PickScare
Figure 6: Impact of The Pareto fron- tier of PickScore vs. Vendi Score.
To further investigate the boundaries of the fidelity- diversity trade-off, we extended our analysis to ex- treme values 7. We evaluated y = 0.01 and 100.0 using the SD1.5 backbone and compared their performance against the standard DDIM sampler and our nominal setting (- 1.0).
Table 3: Gamma effect. Subscripts VS, PS, IR mean Vendi Score, CLIPScore, PickScore, Image-Reward, respectively.
Sumple variance (x10-1)
The quantitative results are summarized in Table We observe two distinct behaviors these extremes:
ays
sps
$/
1.0
0.076
5.49
1.54
0.60
0.9
0.068
2.47
1.00
0.61
Strong Attraction (y = 0.01). When y is extremely
0.8
0.061
3.89
0.78
0.34
0.7
0.050
0.41
0.11
small, the attraction term becomes overwhelmingly
17

---

Published as a conference paper at ICLR 2026
Method
CLIPScore 1
PickScore
ImReward ?
MSS +
Vendi Score 1
DDIM
31.3940
21.3937
0.0890
0.1485
4.7413
Ours ( 0.01)
31.3934
21.3945
0.0888
0.1485
4.7412
Ours ( 1.0)
31.3424
21.2907
0.0360
0.1276
4.7945
Ours (? 100.0)
31.3764
21.2696
0.0185
0.1279
4.7933
Table 4: Ablation on extreme gamma values. compare performance under extreme settings (n 0.01 and y 100.0) against DDIM and our nominal parameter (y 1.0). Extremely low y reverts performance to the DDIM bascline, while extremely high degrades fidelity without providing the optimal diversity gains achieved at y 1.0.
dominant (1/y > 1), rigidly anchoring the noise to the initial Tweedie estimate. This effectively suppresses the diversity-inducing repulsion, resulting in metrics nearly identical the DDIM base- line.
Weak Attraction (y 100.0). Conversely, a very large y negates the anchoring force. While this allows for marginal diversity gains, it causes significant drop in fidelity (PickScore and ImageRe- ward) compared to 7 = 1.0, indicating that the optimization drifts away from semantic alignment without sufficient regularization.
These findings confirm that our nominal value (y Rs 1.0) strikes an effective balance, leveraging suf- ficient attraction maintain quality while allowing enough repulsive freedom to enhance diversity.
C.3 LEVERAGING KI. DIVERGENCE FOR NOISE REGULARIZZTTIN
To further analyze the stability of our method, investigate the effect of an explicit regularization term. This can be achieved by penalizing the deviation of the optimized noise batch {z-] B from the standard Gaussian prior, A (o0, I), using Kullback-Leibler (KL) divergence term (Shlens 2014).
For single noise tensor Zr. we treat all of its constituent elements as a single population of data points estimate an underlying distribution. First, compute the sample mean (î²) and sample variance ( across all D = C x H x W elements within the tensor:
H W C
W hr|c, w]. 0² -²= 82-D-I-1 3I
P DE
(zrfc h, w| p2)
Here, Zr|c, h, wi represents the pixel value allocated in c-th channel and (h, w) -position of the latent tensor Zy. These statistics define an estimated univariate Gaussian distribution, P ((, d²), that characterizes the single noise tensor. We then measure the divergence of this distribution from the standard normal prior, Q N(O, 1). The KL divergence for these univariate Gaussian distributions is:
f12
D (N (3) 1)) = log g
By minimizing this KL penalty, we enforce constraint that en- courages the internal statistics of the optimized noise tensor to remain close those of standard normal distribution. Integrated algorithm is shown in Algorithm
Verci Soore
As shown in Figure 7 incorporating this KL penalty shifts the quality-diversity Pareto frontier to the lower-right, indicating a trade-off towards higher textual fidelity at the cost of lower diver- sity. Interestingly, we observe an analogous phenomenon in our analysis of the attraction coefficient 7, As detailed in Section C.2 lowering the value of similarly shifts the frontier to the lower- right and stabilizes performance; see Figure 6 for details. This parallel suggests that the coefficient in contrastive loss im- plicitly functions as a regularizer, controlling the diversity- trade-off in manner similar to explicit KI. divergence penalty.
No. KL KL. mag
Figure 7: Ablation study on ap- plying Kullback-Leibler diver- gence. Weight for KL diver- gence set as À 1000.
18

---

Published as a conference paper at ICLR 2026
Table 5: Ablation of window size w across prompt types. We compare the effect of full spatial resolution (w = 64) versus aggressive downsampling (w CompBench (complex).
1) on GenEval (simple) and T2I-
Dataset
Setting
CLIPScore f
PickScore
ImReward f
MSS
Vendi Score
64
30.7080
21.3857
0.0585
0.1017
4.8376
GenEval
30.8394
21.3320
0.0083
0.1016
4.8374
W 64
30.8639
20.0979
-0.1697
0.1218
4.7931
T21-CompBench
30.9414
20.0856
-0.2335
0.1208
4.7879
Table 6: Effect of adjusting the initial noise variance. We evaluate the influence of scaling the initial Gaussian prior variance by a factor T (Zr ~ W0 Naively increasing variance fails to improve diversity and degrades quality higher values, whereas framework achieves the best diversity-quality trade-off.
Method
CLIPScore f
PickScore
ImReward
MSS
Vendi Score
1.0 (DDIM)
31,5041
21.4358
0.1324
0.1620
4. 4.7029
1.01
31,4950
21.4381
0.1597
0.1608
4.7060
1.025
31,4668
21.4313
0.1814
0.1599
4.7090
1.05
31,4469
21.3237
0.2058
0.1621
4.7058
1.1
31.4757
20.8230
0.0817
0.1811
4.6539
1.15
30.0744
19.6513
-0.6910
0.2114 14
4.5561
Ours
31.3424
21,2907
0.0360
0.1276
4.7945
C.4 INFLUENCE OF WINDOW SIZE PROMPT TYPES
To address the question of whether aggressive downsampling disproportionately affects specific prompt categories, we expanded our ablation study on the window size w. While Figure 4 demon- strates the global impact of w, here specifically examine the performance difference between preserving full spatial resolution (w 64) and aggressive downsampling (w = 1) across two dis- tinct prompt domains:
GenEval (Ghosh et al. 2023): representing general, simple captions.
T21- T2I-CompBench (Huang al. 2023): representing complex, compositional prompts that require spatial reasoning.
The comparative results are summarized in Table 5 We observe that for both prompt categories, using results in degraded performance compared to 64. This confirms that spatial struc- ture in the initial noise prediction is valuable for optimization. Notably, the degradation in fidelity (PickScore and ImageReward) and diversity (Vendi Score) is observed in both, but the preserva- tion of spatial dimensions (w = 64) is particularly critical for maintaining the quality of complex prompts in T2l-CompBench. Aggressive downsampling (w 1) effectively collapses spatial infor- mation into single vector, which hinders the model's 's ability to optimize for compositional elements that rely on spatial layout. Thus, moderate large window size is essential ensure robustness across varying prompt complexities.
C.5 IMPACT OF INITIAL NOISE VARIANCE ON DIVERSITY
The initial latent 2y is standardly sampled from Gaussian distribution (O, To investigate whether simply increasing the prior variance promotes diversity, we introduced a scaling factor such that 27 v2).
As shown in Table 6 slight increases variance (r E [1.01, 1.05)) yield negligible diversity gains, suggesting that small perturbations fail escape dominant modes. Conversely, larger variances
19

---

Published as a conference paper at ICLR 2026
Prompt: "photo of a cat and dog running, mountain background."
(a) DDIM
(b) CADS
(c) TweedieMix
(d) Ours
Figure 8: Qualitative comparison with concept fusion methods. We compare DDIM, CADS, Tweed- ieMix, and ours using the prompt: "photo of cat and dog rumning, mountain background." While TweedieMix (c) attempts to fuse concepts, it suffers from low text-image alignment, often failing to generate of the subjects (e.g., omitting the cat) due the limitations of personalization-baase tuning. In contrast, Ours (d) successfully captures all semantic elements while providing diverse variations.
Method
CLIPScore
PickScore 1
ImReward
MSS
Vendi Score 1
DDIM
35.9353
23.1232
1.4163
0,2319
2.8302
CADS
35.2270
22.7042
1.1303
0.1860
2.8816
TweedieMix 29.7348
20.1795
-0.4640
0.2112
2.8531
Ours
35.3717
22.7858
1.2632
0.1645
2.9040
Table 7: Quantitative comparison with TweedieMix. We evaluate performance on compositional prompts using personalized concepts. Our method achieves a better balance of alignment (CLIP, PickScore) and diversity (MSS, Vendi) without requiring the pre-training or segmentation steps mandated by TweedieMix.
(T 2 1.1) cause sharp decline in fidelity and paradoxically reduce diversity, indicating a failure converge meaningful image manifolds. This observation aligns with recent findings that vari- ance inflation deteriorates ODE-based sampling (Um et al. (0025). In contrast, our method achieves superior diversity (Vendi 4. 7.79) without such quality collapse, demonstrating that contrastive op- timization is far more effective than naive noise scaling.
C.6 INVESTIGATION OF ALTERNATIVE DIVERSITY APPROACHES
While our primary comparisons focus on zero-shot diversity samplers, the landscape of diversity- enhancing techniques also includes methods based on multi-concept fusion, 3D-aware generation, and counterfactual interventions. In this section, we extend our analysis to these broader families of approaches, incorporating both (i) direct empirical comparison where feasible, and (ii) conceptual discussion for methods whose goals or training regimes differ fundamentally from ours.
To provide more comprehensive assessment, we therefore evaluate our method against Tweed- ieMix, recent method that fuses personalized concepts via Tweedie's formula. We conducted ex- periments using compositional prompts derived from the official personalized concepts provided by TweedieMix (e.g., "dog", "cat", "mountain"). As shown in Table 7 and Figure 8 CNO demonstrates a superior trade-off between diversity and text-image alignment.
We observe that TweedieMix often exhibits degraded fidelity, such omitting requested subjects (e.g., the cat in Figure ()). This limitation largely stems from the catastrophic forgetting inher- ent in personalization-basee approaches; optimizing for specific concepts can degrade the model's ability to generate general concepts or complex compositions outside the pre-learned set. Further- more, TweedieMix relies on concept embeddings learned through additional training stage. This naturally biases generation toward those memorized representations, restricting the model's ability explore the diverse variations that zero-shot method like CNO can access. Computationally.
20

---

Published as a conference paper at ICLR 2026
TweedieMix incurs significant overhead as it requires segmenting latents using external segmen- tation model (e.g., Text-SAM) prior to fusion. In contrast, our framework requires no extra training external models, applying directly to arbitrary prompts in lightweight, she-shot manner.
Beyond multi-concept fusion, several recent approaches advance generative controllability from orthogonal directions, namely 3D consistency, layout reasoning, and counterfactual structure. Diff- Splat (Lin et al. 2025) introduces a differentiable 3D splatting pipeline designed to improve multi- View coherence and geometric accuracy. CoT-lized Diffusion (Liu al., 2025) incorporates multi- modal LLM-based chain-of-thought reasoning into the denoising trajectory to refine spatial arrange- ments and relational structure. Pan & Bareinboim (225)) propose a causally grounded latent space enabling counterfactual manipulation with invariant factors preserved across interventions. While these approaches significantly enhance compositional fidelity, geometric structure, or causal inter- pretability, they do not target the diversity-qualig trade-off in T2I generation. Their methods typ- ically modify the generative process itself, whereas our approach directly reshapes the initial noise distribution to mitigate mode collapse without altering the sampling trajectory or model architecture, making CNO complementary to these orthogonal research directions.
C.7 COMPUTATIONAL ANALYSIS
Table 8: Computational cost and performance comparison on Stable Diffusion v1.5.
Method
Time (sec / batch) +
PickScore f
VendiScore
DDIM
11.131
21.2398
4,8024
Particle Guidance
11.164
21.2164
4.8016
CADS
11.167
21.2254
4.7964
DiversityPrompl
18.703
21.3067
4.7599
Ours (w/o stopgrad)
12.853
21.3125
4.8010
Ours (with stopgrad)
11.866
21.3044
4.8039
To evaluate the practical efficiency and computational overhead of proposed method, we con- ducted a comparative analysis against several baseline and state-of-the-art techniques. All experi- ments in this section were performed using the Stable Diffusion v1.5 model.
Our evaluation focuses on the trade-off between computational cost and performance. We generated total of 5K samples for each method to measure the average time per batch, along with key per- formance indicators for quality (PickScore) and diversity (VendiScore). The results, summarized in Table 8 provide clear overview of each method's performance profile.
As presented in Table our approach demonstrates a highly compelling profile. With an optimization step of Nopt our method incurs a modest computational overhead of approximately 5% relative to the standard DDIM sampler.
Despite this, our approach is notably faster and achieves superior metric scores compared Mi- norityPrompt. It also remains significantly more efficient than computationally intensive methods like Particle Guidance. Crucially, this slight increase latency highly acceptable trade-off. Our method achieves unique point on the Pareto frontier of efficiency, quality, and diversity. The combination of high PickScore and VendiScore delivered by our approach represents a sfate-of-the- art balance unmatched by any other method at any computational cost. This result underscores the practical value of our method, offering solution that is both powerful and efficient for real-world applications.
Impact of Batch Size (B). To provide practical guidelines for real-world usage, we further analyzed how the batch size B influences both generation quality and computational overhead. Our empirical findings suggest that the optimal strategy is to set equal to the number of images intended for generation per prompt (NIPP).
Table 9 illustrates the performance trade-offs. Increasing B from to allows the repulsion term to act larger set of samples, pushing them into increasingly diverse directions. Consequently, B = achieves a superior fidelity-diversity balance compared to B = 3, yielding comparable diversity (Vendi Score) with improved quality metrics (CLIPSccore, PickScore, and ImwgeReward).
21

---

Published as a conference paper at ICLR 2026
Table Impact of batch size on performance. 5 yields better balance, improving quality metrics while maintaining high diversity compared to B = 3.
Method
CLIPScore
PickScore t
ImReward t
MSS
Vendi Score
DDIM
31.3940
21.3937
0.0890
0,1485
4.7413
Ours (B 3)
31.3298
21.2640
0.0238
0.1267
4.7949
Ours (B
31.3424
21,2907
0.0360
0.1276
4.7945
Table 10: Influence of batch size B on computations. We report Peak Memory (MiB) and Time (sec/batch). Our method remains efficient even at larger batch sizes compared DiversityPrompt.
B
Method
Memory (MiB)
Time (sec/batch) |
Memory (MiB)
Time (sec/batch)
DDIM
7694
8.091
10978
12.778
Ours
7672
8.809
10828
13.427
DiversityPrompt
13070
15.063
19006
21.631
We generally do not observe stagnation in diversity improvement to the NIPP limit; thus, utilizing the largest feasible B is recommended.
Regarding computational cost, while increasing B naturally raises memory usage and inference time, the overhead for our method remains relatively marginal. Crucially, even at 5, our ap- proach is significantly more efficient than iterative baselines such as DiversityPrompt, which incurs nearly double the memory and time cost (see Table 10). This efficiency stems from our one-shot op- timization of the initial noise, avoiding the heavy cost of iterative interventions during the sampling process.
C.8 LIMITATIONS AND FUTURE WORK
Despite its effectiveness in mitigating mode collapse, method like other liversity-focusee ap- proaches - exhibits limitations when handling prompts that require strong compositional grounding. In scenarios involving specific spatial relations interactions, the optimization for diversity can oc- casionally compromise text-image alignment. For instance, illustrated in Figure 9 while standard sampling successfully captures specific relational details (e.g., a dog holding frisbee), diversity methods including CADS, PG, and our method (at = 1.0) may struggle to fully preserve this structural coherence.
Prompt: "A dog holding a yellow frisbee in its mouth." "
(a) DDIM
(b) CADS
(c) PG
(d) Ours (7 = 1.0)
(e) Ours ( < 1.0)
Figure 9: Failure case analysis. We illustrate a scenario requiring strong compositional grounding (specifically, the relation of holding). While the standard DDIM sampler (a) correctly captures the relational structure, diversity-enhancii
methods (b-d) including Ours with the nominal 7 = 1.0 may struggle to maintain this fine- fine-grained alignment. However, unlike other baselines, framework provides a remedy: by lowering the attraction coefficient (y < 1.0), we can recover the correct semantic alignment (e) by trading off small degree of diversity.
22

---

Published as a conference paper at ICLR 2026
Crucially, however, our framework offers novel advantage to address this challenge: the attrac- tion coefficient 7. Unlike existing baselines where the didelity-diversity trade-off is often rigid, our method enables dynamic control via 7 As demonstrated in Figure (c). by lowering Y below 1.0, we can intensify the anchoring force toward the initial Tweedie estimate. This effectively recovers the correct compositional alignment (e.g., the holding relation) at the cost of modest reduction in diversity. This capability confirms that y functions as an effective fidelity controller, allowing users flexibly navigate failure cases that otherwise difficult resolve in competing frameworks.
For future work, while our method focuses on the initial noise 27, we believe that applying a sim- ilar optimization strategy intermediate latents Z, (where < T) could be promising avenue for further enhancing diversity for a single prompt. Some studies have explored optimizing these intermediate latents generate images with high fidelity to complex textual conditions (Wallace er al. 2023 Ding et al. 2024). The effectiveness of such an approach may depend on the model and the degree to which its Tweedie prediction already structured to reflect the semantic content of the input prompt at intermediate timesteps. This direction may warrant deeper investigation approach.
D ADDITIONAL EXPERIMENTAL RESULTS
D.I PERFORMANCE ON ACCELERATED MODELS
Model
Method
CLIPScore
PickScore f ImageReward +
Vendi Score
DDIM
31.5536
22.6598
0.7231
0.2865
2.7470
SDXII-Lightming
CADS
31,4425
22.5767
0.6876
0.2674
2.7749
(4-step)
Ours
31.4474
22.5659
0.6740
0.2289
2.8258
FM-ODE 32,1012
22.7411
1.0499
0.3012
2.7220
FLUX-I-Schnell
(4-step)
CADS
31.7153
22.5431
0.8622
0.2287
2.8250
Ours
32.0664
22.6137
1.0070
0.2231
2.8316
Table 11: Quantitative comparison on few-step accelerated models. We evaluate performance on SDXL-Lightning and FLUX-1-Schnee under 4-step inference setting. Our method is compared against DDIM, CADS, and Flow-Mas Now-Matching ODE (FM-ODE), the default sampler used in FLUX. The results indicate that approach maintains robust diversity improvements (higher Vendi Score, lower MSS) even in aggressive few-step regimes where iterative interventions like CADS are less effective.
Prompt: "An astronaut on the moon" "
(a) SDI.5
(b) SDXL.
(c) SD3
(d) SDXL-Lightning (e) FLUX-I-Schmell
Figure 10: Visualization of initial Tweedie estimates (Zoyz) ) across different backbones. We compare the one-step denoised predictions from the initial Gaussian noise ž7.
Few-step accelerated models, while computationally efficient, are prone to severe mode collapse due to limited stochasticity in their shortened trajectories. To validate the robustness of our approach in this regime, we evaluated CNO on two representative accelerated models: SDXL-Lightning (4-step distilled diffusion) and FLUX-1-Schnell (4-step rectified flow).
23

---

Published as a conference paper at ICLR 2026
As summarized in Table 11 CNO delivers consistent improvements in diversity metrics (Vendi Score and MSS) across both architectures. Notably, we observe that iterative diversity samplers like CADS yield limited gains in this 4-step setting. This performance gap stems from fundamental operational difference: iterative methods rely on accumulating guidance over many timesteps, a mechanism that becomes ineffective when the sampling trajectory is drastically condensed.
In contrast, CNO performs a one-shot optimization of the initial noise 27 prior to sampling, making independent of the inference step count. Crucially, we found that accelerated models tend to pro- duce significantly clearer and more deterministic Tweedie predictions (forr) at the initial timestep compared to standard many-step models (see Figure[TO) This characteristic renders the initial noise optimization particularly effective, as the contrastive gradients derived from these sharp Tweedie estimates are highly semantically meaningful. These results confirm that our framework model- agnostic and robust even under aggressive sampling acceleration.
D.2 USER STUDY RESULTS
Alignment & Quality (%)
Diversity & Creativity (%)
Baseline
CNO (Ours)
Baseline
CNO (Ours)
CADS
21.29
78.71
16.45
83.55
PG
41.29
58.71
21.94
78.06
Table 12: Human preference evaluation. We compare CNO against CADS and Particle Guidance (PG). The values represent the percentage of user preference. CNO demonstrates strong advantage in both image quality and generation diversity.
To complement our quantitative evaluation, we conducted a human preference study comparing CNO against two fiverset baselines: CADS (Sadat et al. 2024] and Particle Guidance (PG) (Corso et al.) 2024). A total of 32 participants evaluated 30 randomly ordered image pairs, each generated using identical prompts and seeds. For each pair, participants selected the preferred sample according to two criteria: (i) Alignment & Quality (text adherence and visual fidelity) and (ii) Diversity & Creativity (semantic distinctiveness and variation).
As shown in Table 12 CNO is consistently preferred over both baselines. Notably, against CADS, our method achieved preference rates of 78.71 % for quality and 83.55% % for diversity. A similar trend is observed against PG, confirming that the quantitative gains of CNO translate into perceptibly superior generation quality and diversity.
D.3 EXPERIMENTS ON DISTINCT PROMPT DOMAINS
Table 13: Quantitative evaluation on the GenEval benchmark.
Method
CL.IScore f
PickScore t
ImReward
MSS +
Vendi Score t
DDIM
32.0443
21.7006
-0.1422
0.1389
4.7550
CADS
31.6739
21.4260
-0.3117
0.1020
4,8433
Ours
31.6738
21.4923
-0.2841
0.1006
4.8508
To verify that our method's effectiveness not limited the daily scenes typical of MS-COCO, we extended our evaluation to the GenEval benchmark (Ghosh et al.) (202).. GenEval is designed to test compositional reasoning and includes diverse set prompts distinct from standard captioning datasets. We evaluated Stable Diffusion v1.5 using this benchmark, comparing CNO against DDIM and CADS.
The quantitative results presented in Table 13 Consistent with our main findings, CNO achieves the highest diversity scores (Vendi Score and MSS), outperforming both the standard sampler and the baseline diversity method (CADS). Crucially, while maintaining superior diversity, CNO also
24

---

Published as a conference paper at ICLR 2026
retains higher image quality compared to CADS, evidenced by higher PickScore and ImageRe- ward values. This demonstrates that the robustness our contrastive noise optimization extends beyond specific datasets and effectively generalizes to diverse textual domains.
D.4 ADDITIONAL GENERATED SAMPLES
Figure and Figure 12 represent that CNO shows high image quality and textual fidelity compared DDIM, CADS, and Particle Guidance which the hyperparameters of our method are equivalent to
Table 2
E
USE OF LARGE LANGUAGE MODELS
We used Large Language Models (LLMs) to aid in the verbal refinement and polishing of our paper. This usage was only limited to improving readability and fixing some grammar errors. The core research, including the formulation of our method, experimental design, and analysis of results, was conducted solely by the authors.
25

---

Published as a conference paper at ICLR 2026
Algorithm Diverse T2I Generation via Contrastive Noise Optimization
Inputs: Text embedding c, batch size B, optimization steps Nopt, learning rate 7, temperature 7, attraction coefficient 7, window size w {x batch of diverse images.
Outputs:
1: Models:
Diffusion model cØ. DDIM sampler DDDMM
2: Initialize batch of trainable noise vectors ZT {z) 10,
3: Let T Zr
4: for = to Nap do
5: for 1 to B do
6:
OOJT + var (z)
v1 Trstooggrakee((t
7:
if n iitel + Zolr
then
8:
20/T
9:
end if
oor 0/7
DownSample w)), 2 w)
Lrtel ofTT O/T
Normalize(2/ (())), (zrmalize
12:
end for
Σe= (x @ //rr//r) )
exp(s(E) 20 2// ///(()))
13:
CNo = 1 B
B EE=
log
14:
Zr t- ZT - Zz,CiNo
15:
end for
16:
1o11 LB
t DDDIM (z,,c
(x))==
Decode (x0)==1)
return [
(xi1
Algorithm 2 Contrastive Noise Optimization with KL. Regularization
Inputs:
Text embedding c, batch size B, optimization steps Nogt, learning rate 7) temperature т, attraction coefficient y, window size w, KL divergence weight {x
Outputs:
batch of diverse images.
Models: Diffusion model Eg. DDIM sampler DDDIM
2: Let D be the number of pixels in single noise tensor (channel C height H x width W)
3: Initialize a batch of trainable noise vectors Zr = { (0,
4: Let Z Zz
5: for = 1 to Nog do
6: for 1 to B do
7:
%/1 Var (2)
V1 arstopgrad(e (c)))
8:
if st.eef t L
1 then
9:
201
10:
end if
11:
2 upr
irref
DownSample ri w), DownSamplo(2hrd1 w)
12:
2 O/T
(2 Normalize(2)h1) (2f..c)
13:
end for
exp(s(2mm: 2111///))))))))
14:
B DE log
EjL exp(f( / )/////))
P
D Eh=1 I
H
zr/c, h w] for 1....,
t D-1 ΣH w=1 c=
16:
07
-==1
Σ (Zr [c, h w/ 2 for i = 1, 1,..,,B B
LKL = 1 EB
log #;
a. +
2
1
2
18:
Lioal t- L2 +XCII.
19: Zp t Zy -
end for
21: [ &- DDDIM Zr,c)
22:
[x)) + Decode( (z) 1
23:
return [
26

---

Published as a conference paper at ICLR 2026
"A dog sits on boat floating
"A batter goes hit the ball just thrown him from the pitcher"
"A cut sitting leuther couch wedd to two remotes."
"A man smiles while holding a large turkey."
building clock with roman numerals on it."
"A person that flying a kite the
"Two dogs lying on blanket sleeping on couch."
ripe banana sitting in black bowl!" "
(a) DDIM
(b) CADS
(c) Ours
Qualitative comparison of images generated from the same of text prompts by (a) DDIM, (b) CADS and (c) our proposed method. Images with the same position in individual grids share the same prompt and seed.
Figure 11:
27

---

Published as a conference paper at ICLR 2026
"An astronout the moon" "
"A cal the grass"
DDIM
DDIM
CADS
CADS
PG
PG
CNO
CNO
"Motorcycle on display in parking lot for many to
"A close-up of wearing hut and tie."
see'
DDIM
DDIM
CADS
CADS
PG
PG
CNO
CNO
person a black jacket skiing through deep snow."
"A portrait of a tiger"
DDIM
DDIM
CADS
CADS
PG
PG
CNO
CNO
Figure 12: Additional results of synthesized images across different diffusion sampling methods.
Each row represents specific diversity method: DDIM. CADS, Particle Guidance (PG), and our proposed CNO. CNO consistently demonstrates high diversity compared to baseline methods.
28


Reviews:
[{"review_id": "gHzXY18JMA", "reviewer": "Reviewer_XTmn", "summary": "This paper addresses the persistent issue of limited diversity in text-to-image (T2I) diffusion models under strong text guidance. Instead of prior approaches that tinker with intermediate latents or guide text embeddings during or across inference steps, the authors introduce Contrastive Noise Optimization (CNO): a lightweight, pre-processing method that optimizes the initial noise latents before sampling. By defining a contrastive InfoNCE loss in the Tweedie denoised latent space—incorporating both attractive (anchoring) and repulsive (diversifying) forces—CNO aims to produce batches of noise that yield diverse yet faithful image outputs. The method is anchored in theoretical reformulation of InfoNCE to balance fidelity and diversity, is computationally efficient, and demonstrates robust gains across several quantitative and qualitative metrics on Stable Diffusion variants and benchmarks.", "questions": "The loss function as written in Section 4.1/Algorithm 1 and its implementation in Algorithm 2 (for KL regularization) are slightly inconsistent in the summation indices and argument ordering, which could lead to confusion for others seeking to re-implement the approach.\n\nThe diversity-boosting methods compared against cover PG, CADS, and DiversityPrompt; however, some recent methods involving multi-concept fusion, 3D-aware T2I generation, and counterfactual interventions (cf. TweedieMix (Kwon & Ye 2025), DiffSplat (Lin et al. 2025), CoT-lized Diffusion (Liu et al. 2025), and Pan & Bareinboim (2025)) are not considered or even discussed. These methods, while not identical, are highly relevant to the diversity and compositional generalization challenge, and their exclusion diminishes the scope of empirical claims.\n\nThe main conceptual innovation of optimizing the initial noise with a contrastive loss heavily draws from existing InfoNCE/contrastive techniques, which are increasingly used in generative and diffusion-based frameworks. The extension to the Tweedie latent space is a clever adaptation, though it may be seen as an incremental, albeit meaningful, refinement of existing paradigms (see Guo et al. 2024, Ahn et al. 2024). The contribution would have benefited from a more careful positioning, clarifying the distinction between novelty and adaptation.\n\nThis paper provides a practical contribution to advancing diversity in text-to-image generative models.  The empirical evaluation is thorough, with solid quantitative and qualitative evidence supporting the claims of improved Pareto frontiers.\n\nHowever, the work is somewhat incremental in terms of conceptual novelty, with the main technical move (contrastive optimization of initial noise) being an adaptation of ideas familiar within the generative/representation learning community. There are also missing references to closely related state-of-the-art works on multi-concept and compositional generation, diversity quantification, and alternative diversity control strategies, some of which could serve as strong baselines. Additionally, the exposition could benefit from clarified notation and more critical discussion of potential failure corners.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is technically sound, mathematically rigorous in its core claims, and offers strong theoretical motivation for adopting a contrastive loss in the denoised Tweedie latent space. The derivations for the mutual information bounds (Propositions 1 & 2) and the use of a regularization parameter ($\\gamma$) are helpful and clear (see Sections 4.2–4.3 and Appendix A).\n\nExtensive experiments (see Table 1 and Figure 3) across SD1.5, SDXL, and SD3 show the method consistently outperforms recent strong baselines like CADS, PG, and DiversityPrompt, especially on diversity metrics (Vendi Score, MSS) while retaining high-quality image generation (PickScore, ImageReward).", "weaknesses": "The loss function as written in Section 4.1/Algorithm 1 and its implementation in Algorithm 2 (for KL regularization) are slightly inconsistent in the summation indices and argument ordering, which could lead to confusion for others seeking to re-implement the approach.\n\nThe diversity-boosting methods compared against cover PG, CADS, and DiversityPrompt; however, some recent methods involving multi-concept fusion, 3D-aware T2I generation, and counterfactual interventions (cf. TweedieMix (Kwon & Ye 2025), DiffSplat (Lin et al. 2025), CoT-lized Diffusion (Liu et al. 2025), and Pan & Bareinboim (2025)) are not considered or even discussed. These methods, while not identical, are highly relevant to the diversity and compositional generalization challenge, and their exclusion diminishes the scope of empirical claims.\n\nThe main conceptual innovation of optimizing the initial noise with a contrastive loss heavily draws from existing InfoNCE/contrastive techniques, which are increasingly used in generative and diffusion-based frameworks. The extension to the Tweedie latent space is a clever adaptation, though it may be seen as an incremental, albeit meaningful, refinement of existing paradigms (see Guo et al. 2024, Ahn et al. 2024). The contribution would have benefited from a more careful positioning, clarifying the distinction between novelty and adaptation.\n\nThis paper provides a practical contribution to advancing diversity in text-to-image generative models.  The empirical evaluation is thorough, with solid quantitative and qualitative evidence supporting the claims of improved Pareto frontiers.\n\nHowever, the work is somewhat incremental in terms of conceptual novelty, with the main technical move (contrastive optimization of initial noise) being an adaptation of ideas familiar within the generative/representation learning community. There are also missing references to closely related state-of-the-art works on multi-concept and compositional generation, diversity quantification, and alternative diversity control strategies, some of which could serve as strong baselines. Additionally, the exposition could benefit from clarified notation and more critical discussion of potential failure corners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVyU4VXzBC", "reviewer": "Reviewer_qqYa", "summary": "The paper presents Contrastive Noise Optimization (CNO), a training-free diversity-boosting wrapper for text-to-image diffusion models. By marrying Tweedie-space contrastive repulsion with anchor-guided fidelity control, CNO produces markedly varied outputs in a single, lightweight pre-processing stage. Comprehensive evaluations on SD 1.5, SDXL and SD3 reveal consistent gains in Vendi score with negligible CLIPScore degradation, surpassing the latest zero-shot samplers in MSS and human-preference metrics and establishing a new Pareto frontier for quality-diversity trade-offs.", "questions": "The proposed method shows strong qualitative and quantitative performance across multiple models (SD1.5, SDXL, SD3). However, as highlighted in Diffusion2GAN and Loopfree, few-step models often exhibit more severe diversity degradation. It would be valuable to further analyze whether the proposed method remains effective under such few-step models.\n\nDiffuison2GAN: Distilling Diffusion Models into  Conditional GANs \\\nLoopfree: One-Way Ticket : Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tDiversity is an important and interesting topic. Many models tend to overlook diversity issues during pretraining. \n2.\tWith complex prompts like “A cow sits in a truck with hay barrels in it,” other methods either fail or repeat elements, but our method produces diverse and semantically accurate images.", "weaknesses": "1. The proposed method improves generation diversity by optimizing the initial latent, which is sampled from a Gaussian distribution. As the initial latent is determined by this distribution, adjusting the variance of the Gaussian may also affect diversity. It would be interesting to discuss whether such a change could further enhance diversity.\n2. It might be worth exploring whether the proposed method can further improve the diversity of the few-step distilled models.\n3. A user study could help verify, through human evaluation, whether the method achieves higher generative diversity than the baseline.\n4. It would be helpful if the authors could report the additional computational resources required as B increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QfjEku79D9", "reviewer": "Reviewer_A3GX", "summary": "This paper addresses the pervasive problem of limited diversity in text-to-image (T2I) diffusion models, particularly under strong text guidance. The authors propose Contrastive Noise Optimization (CNO), a pre-processing framework that directly optimizes the batch of initial noise latents to encourage diversity, applying a contrastive InfoNCE loss in the denoised Tweedie space. The approach balances a repulsion term (encouraging diversity) with an attraction to the original sample (preserving semantic fidelity), and introduces a coefficient $\\gamma$ to stabilize this trade-off. Theoretical analysis and comprehensive experiments across multiple diffusion backbones demonstrate improvements in both image diversity and quality.", "questions": "1. Could the authors provide additional quantitative and qualitative results on how extreme values of $\\gamma$ (especially $<<1$ and $>>1$) impact the fidelity/diversity trade-off? Are there prompts or settings where the method consistently fails or collapses into poor outputs?\n2. What are the empirical impacts and best practices for choosing the batch size $B$ in real-world usage? Is there a threshold beyond which diversity improvements stagnate or degrade?\n3. Is the optimization of the initial noise vectors consistently stable across different diffusion model architectures, or are there architectures/settings for which CNO performs suboptimally?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the diversity challenge at its root by optimizing initial noise, rather than intervening mid-sampling or in text embedding space. This leads to a simple, efficient method that is model-agnostic and does not require modifications to the diffusion backbone.\n2. The authors extend the mutual information perspective of InfoNCE to simultaneously account for positive and negative pairs (Section 4.3), providing formal justification for how diversity and fidelity are balanced. They further detail how the $\\gamma$ parameter scales this effect.\n3. Substantial empirical evidence is provided, including Table 1 and Figure 3, benchmarking against DDIM, Particle Guidance (PG), CADS, and DiversityPrompt across three major diffusion models (SD 1.5, XL, 3). The proposed method consistently outperforms baselines in key diversity metrics (Vendi Score, MSS) and maintains or improves quality/fidelity (PickScore, CLIPScore).\n4. The exposition is clear, diagrams such as Figure 2 provide an intuitive grasp of the architecture/mechanism, and the methodology is described in practical, replicable pseudocode (Algorithm 1, Appendix B.1).\n5. The method shows low sensitivity to hyperparameter choices (Section 5, Table 2), making it appealing for real-world adoption.", "weaknesses": "1. While the theoretical analysis is a strength, there is a (potentially confusing) notational inconsistency when switching between $\\mathcal{L}{\\text{CNO}}$, $\\mathcal{L}{\\text{InfoNCE}}$, and $\\mathcal{L}_{\\text{CNO}}^\\gamma$ across pages 5-6. The exact role of $\\gamma$ in the contrastive loss numerator/denominator, and how gradients are stopped or propagated through each component, could be clarified further. For readers less familiar with contrastive objectives applied to image generation, a stepwise, worked example would reduce ambiguity.\n2. Section 5.1 references an ablation in Figure 4 testing $w\\in{4, 8, 16, 64}$, yet the impact on specific image types, prompt complexity, or semantic regions is not deeply probed. There is no discussion of whether aggressive downsampling disproportionately harms long-tail or compositional prompts.\n3. While automated metrics (Vendi, CLIPScore, etc.) are robust and reflect recent trends, there is a notable absence of human evaluation for subjective diversity, creativity, or aesthetic ranking beyond PickScore. As text-to-image generation is ultimately a user-facing task, such a study would strongly solidify the real-world value of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J9pOqdryCo", "reviewer": "Reviewer_ZwsY", "summary": "The paper addresses T2I diffusion models’ mode collapse (limited diversity under strong text guidance) via Contrastive Noise Optimization (CNO). It optimizes initial noise latents (before DDIM sampling) using a contrastive loss in Tweedie denoised space: repelling batch samples to boost diversity, while anchoring each to its unoptimized version to preserve fidelity. A γ coefficient balances repulsion/attraction for stability. It achieves a superior quality-diversity Pareto frontier, enabling creative applications (e.g., diverse concept generation) without complex tuning, advancing practical T2I usability.", "questions": "refer to the above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It innovatively addresses T2I diversity by optimizing initial noise (not intermediate latents) via Contrastive Noise Optimization (CNO)—a contrastive loss in Tweedie denoised space (not latent space) with repulsion (batch sample separation) and attraction (anchoring to unoptimized noise). This differs from CADS (text embedding perturbation) and PG (intermediate latent repulsion), solving diversity at the source while avoiding hyperparameter fragility.\n\n2. It sets a superior quality-diversity Pareto frontier, with 5% overhead vs. DDIM but outperforming costly baselines (DiversityPrompt). Enabling diverse, high-fidelity generation (e.g., creative concept design) without complex tuning, it advances practical T2I usability and guides future noise-optimization research.\n\n3. Rigorous validation on SD1.5/SDXL/SD3 shows top Vendi Score (diversity) and PickScore (quality). From Fig.3, the CNO is outperforming DDIM/CADS/PG consistently. Such observations are also true after reading the supplementary.", "weaknesses": "1. Experiments only use MS-COCO validation prompts (daily scenes) but lack other-domain prompts. Maybe the authors could consider to include other kind of prompt datasets to broaden this setup.\n\n2. The CNO is only tested on the SD series models, I wonder would it be applicable to other structure based T2I models? Flux, DeepFloyd, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)