ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization
Authors: Luca Eyring
Shyamgopal Karthik
Karsten Roth
Alexey Dosovitskiy
Zeynep Akata
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation
Diffusion Models
Test-Time Training
Reward Models
Learning From Human Feedback
Primary area: diffusion_based_models
TL;DR: Optimizing initial noise of one-step diffusion models leads to significantly improved results

Abstract: Text-to-Image (T2I) models have made significant advancements in recent years, but they still struggle to accurately capture intricate details specified in complex compositional prompts. While fine-tuning T2I models with reward objectives has shown promise, it suffers from "reward hacking" and may not generalize well to unseen prompt distributions. In this work, we propose Reward-based Noise Optimization (ReNO), a novel approach that enhances T2I models at inference by optimizing the initial noise based on the signal from one or multiple human preference reward models. Remarkably, solving this optimization problem with gradient ascent for 50 iterations yields impressive results on four different one-step models across two competitive benchmarks, T2I-CompBench and GenEval. Within a computational budget of 20-50 seconds, ReNO-enhanced one-step models consistently surpass the performance of all current open-source Text-to-Image models. Extensive user studies demonstrate that our model is preferred nearly twice as often compared to the popular SDXL model and is on par with the proprietary Stable Diffusion 3 with 8B parameters. Moreover, given the same computational resources, a ReNO-optimized one-step model outperforms widely-used open-source models such as SDXL and PixArt-alpha, highlighting the efficiency and effectiveness of ReNO in enhancing T2I model performance at inference time.

ReNO: Enhancing One-step Text-to-lmage Models through Reward-based Noise Optimization
Luca Eyring" .2,,, 1.23, Shyamgopal Karthik 122,,4 Karsten Roth
Alexey Dusovitskiy
Zeynep Akata 1,2,3 1223
Technical University of Munich
Munich Center of Machine Learning UUniversity of Tübingen & Tübingen AI Center "Inceptive
Helmholtz Munich
luca. eyringStum.de de shyamgopal. karthikkuni karphiggupl.kkkthikkuuuiii -wwbiingteebbnnncccmm
Abstract
Text-to-lmage (T2I) models have made significant advancements in recent years, but they still struggle to accurately capture intricate details specified in complex compositional prompts. While fine-tuning T2I models with reward objectives has shown promise, it suffers from "reward hacking" and may generalize well to unseen prompt distributions. In this work, we propose Reward-based Noise Optimization (ReNO), novel approach that enhances T2I models inference by optimizing the initial noise based on the signal from one or multiple human preference reward models. Remarkably, solving this optimization problem with gradient ascent for 50 iterations yields impressive results on four different one- step models across two competitive benchmarks. T2I-CompBench and GenEval. Within a computational budget of 20-50 seconds, ReNO one-step models consistently surpass the performance of all current open-source Text-to-Image models. Extensive user studies demonstrate that our model is preferred nearly twice as often compared to the popular SDXL model and on par with the proprietary Stable Diffusion 3 with 8B parameters. Moreover, given the same computational resources, a ReNO-optimized one-step model outperforms widely- used open-source models such as SDXL and PixArt-a, highlighting the efficiency and effectiveness of ReNO in enhancing T2I model performance at inference time. Code is available at https: / //g github. cow E/////xppaanbllleeeeeea
Introduction
Advancements in Text-to-mmage (T2I) models have been achieved in recent years, largely due the availability of massive image-text datasets [26, 82, 83] and the development of denoising diffusion models [19, 36, 76, 84]. Despite these improvements, T2I models often struggle to accurately capture the intricate details specified in complex compositional prompts [3, 37]. Common challenges include incorrect text rendering, difficulties with attribute binding, generation unlikely object combinations, and color leakage. While recent models have begun to address these issues by employing enhanced language encoders, larger diffusion models, and better data curation [6. 12, 13, 22]. these approaches typically involve training larger models from scratch, making them inapplicable existing models.
As more efficient alternative, fine-tuning T21 models has gained significant attention. This approach can be tailored either toward specific preferences [32, 78, 105] or general human preferences. Inspired by the success of Reinforcement Learning from Human Feedback (RLHF) [16, 29] in Large Language
equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

PisArb DMD
ReNO
SD-Turba
RaNO
SDXI -Turbo
ReNO
HyperSn
ReND
rennts zloy toeather is aen grew neader
aleinatict
mester feasan distiact, for
aartlicity and clarity, salcable
toserits the moe"
Inpaore sylee wolering wartar ast caarry blatea
saic Netartatic dityscape all petrtieg: My sver asoy exutsarra
Figure 1: Qualitative results of four different one-step text-to-Imase models with and without ReNO over different prompts. The same initial random noise is used for the one-step generation and the initialization of ReNO. ReNO significantly improves upon the initially generated image with respect to both prompt faithfulness well aesthetic quality for all four models. Best viewed zoomed in.
Models (LLMs), several works [11, 18, 23, 74, 109] propose aligning T2I models by fine-tuning them on hreferpreferred prompt-image sets using RLHF-inspired techniques. Additionally, human preference reward models, such as PickScore [46], HPSv2 [97]. and ImageReward [100], have gained popularity. These models are trained output a score reflecting human preference for an image given a specific prompt, typically by measuring human preferences for various images generated from the same prompt. The scores predicted by these models have been utilized as evaluation metries for the quality of generated images. Furthermore, Clark et al. [17]. Li et al. [51]. Prabhudesai et al. [72] directly fine-tune T2I models on these differentiable reward models to maximize the predicted reward of generated images. This approach is efficient due to the directly differentiable objective.
Fine-tuning T2I models with reward objectives has major drawback of "reward hacking", which occurs when a reward model gives a high score to an undesirable image. Reward hacking points to deficiencies in existing reward models, highlighting gaps between the desired behavior and the behavior implicitly captured by the reward model, which is especially prone to appear when explicitly fine-tuning for reward [17, 20, 51]. Additionally, these models are often fine-tuned on small scale (c.g., <50 prompts in some cases [8, 23]) and thus may not generalize well to unseen prompt distributions with complex compositional structures. In this work, our aim is enhance T2I models at inference for each unique generation, similar to the paradigm of test-time training for classification models [27, 88). Fine-tuming diffusion models for every single prompt would both be computationally expensive (Dreambooth [78] takes 5 minutes on 1 A100), and susceptible to treaardhhackig
We sidestep the challenge of fine-tuning the model's parameters and instead explore optimizing the initial random noise during inference without adapting any of the model's parameters. To obtain more optimal noise and higher-quality generated image, we introduce Reward-based Noise Optimization (ReNO), where the initial noise is updated based on the signal from a reward model evaluated on the generated image. The main challenges in this approach are twofold. First, backpropagating the gradient through the denoising steps can lead to exploding/vanishiin gradients, rendering the optimization process unstable. Our insight that by employing distilled one-step T2I model [12, gradients since backpropagation is
75, 81, 102], we circumvent the issue of exploding/vanishinn performed through single step. Second, naively optimizing the initial latent for an arbitrary objective can lead to collapse due to reward hacking. To mitigate this, we propose the use of combination of reward objectives to not overfit to any single reward. Moreover, given a well-calibrated one-step T2I model with frozen parameters, the generated images should not exhibit reward hacking if the initial noise remains in the proximity of the initial noise distribution. Therefore, we propose an optimization scheme with limited steps, regularization of the noise to stay in-distribution, and gradient clipping.

---

In essence, ReNO involves optimizing the initial latent noise given an one-step T2I model (e.g., SD/SDXL-Turbo) and a reward model (e.g., ImageReward) for limited number of iterations (10-50 steps). On the popular evaluation benchmarks T21-Compbench and GenEval, our noise optimization strategy (ReNO) significantly improves performance, increasing scores by over 20% in some cases. This enhancement allows SD2.1-Turbo models to approach the performance of closed-source proprictary models such as DALL-E [6] and SD3 [22]. We demonstrate that ReNO substantially improves the performance of four different one-step T2I models (e.g. Figure 1), both in terms of quantitative evaluation and extensive user studies, while only requiring 20-50 seconds to generate image. Moreover, given the same computational budget, ReNO surpasses the performance of competing multi-step models, offering an attractive trade-off between performance and inference speed. ReNO not only motivates the development of more robust reward models but also provides compelling benchmark for their evaluation. Finally, our results highlight the importance of the noise distribution in T2I models and encourage further research into understanding and adapting it.
2 Reward-based Noise Optimization (ReNO)
Despite the remarkable progress in Eext-to-Image (T2I) generation, current state-of-the the-att art models still struggle to consistently produce visually satisfactory images that fully adhere to the input prompt. Recent studies have highlighted the significant impact of the initial noise vector on the quality of the generated image [19, 85]. In fact, selecting and re-ranking images generated from set of initial noises based on reward models has been shown to substantially improve performance [41, 46). This observation naturally leads to the question of whether it is possible identify an optimal noise vector that maximizes a given goodness measure for the generated image. In this section, we first provide an overview of one-step diffusion models, which serve as the foundation for our work. We then introduce our simple yet principled approach that enables practical noise optimization to enhance the performance of one-step T21 models based on uuman-preference reward models, addressing the challenge of generating high-quality images that align with the input prompt.
2.1 Background: One-Step Diffusion Models
T2I models aim to generate images xy conditioned on a given textual prompt p. generative model Go parameterized by 9 takes as input a noise vector e ~ (V(0I) I) and prompt p. and outputs an image Go(e, p) = Xo. The objective to learn the parameters 0, such that the generated image xo aligns with the semantics of the prompt p. This is typically achieved by training the model on large dataset of paired text and images. Recent models are based on a time-dependent formulation between standard Gaussian distribution ~ NO, I), and data Xo (x)).. These models define a probability path between the initial noise distribution and the target data distribution, such that
x 012X0 + otE,
where oe is a decreasing and 0r is an increasing function of f. Score-based diffusion [40, 45, 86] and flow matching [1, 54, 57] models share the observation that the process X, can be sampled dynamically using a stochastic or ordinary differential equation (SDE or ODE). Consider the forward SDE that transforms data into noise as increases dx, = u(x,, t) dt + g(t) dw,, where uz(2,, 6) denotes the drift, w, is Wiener process and g() represents the diffusion schedule. Then, the marginal probability distribution pt (x) of x, in (1) coincides with the distribution of the probability flow ODE [45, 86], well as the reverse-time SDE [2]
dx;= = [u(x,, t) - s(x)) t)] dt + gt(() dw:.
(2)
where s(x, t) = V log pe (x) is the score function. By solving either the ODE or SDE backward in time from XT E (0, I), we can generate samples from po(x). This relies on good estimate of the parameterized score So so(x,, l). The choice of functions aa and ae are defined implicitly based on the forward SDE [40, 45, 85, 87]. Furthermore, the process x, is considered with I sufficiently large such that xT approximates the initial noise distribution A (0, I). Then, has been shown that the score can be approximated efficiently based on, e.g. the denoising loss [36]
interval [0, T]
(0) e- (0.1). (l (x,, () + +
During inference, these models simulate an ODE/SDE through discretization for a number of steps. This can be computationally expensive the trained model must be evaluated sequentially.
3

---

p: "A blue scooter is parked near curb in front of a green vintage car"
Reward Models R3 ge.....
R.
One-Step T21 Model Gg(e,p)
ImageReward
0.35
Initial O E
Latent Noise
R.
HPSv2
0.28
Iteratively update Noise to maximize Reward Scores
g+ = Ves Σ, ARR
(
RV ImageReward
One-Step T21 Model Gg(E,
1.95
ReNO Updated 50
Latent Noise
R
HPSv2
0.36
Figure Overview of our proposed ReNO framework. Given reward models based on human preferences, we optimize the initial latent noise to maximize the reward scores (consisting HPSv2 [97]. PickScore [46], ImageReward [100], and CLIP [73]) for the images generated by the one-step T2I model. Over 50 iterations, the quality of the images and the prompt faithfulness are improved.
Distillation. As means to reduce inference time, distillation techniques have recently gained traction with the intent to learn student model that approximates the solution of the simulated differ- ential equation with a trained teacher model given fewer inference steps, c.g. score distillation [71] penalizes the estimated score the real data distribution. Furthermore, several methods have been proposed to distill models into one-step generators, which learn to approximate the full ODE or SDE in one step. Our work builds upon the following one-step T21 models which refer as Gg- Adversarial Diffusion Distillation (ADD) [81] combines score distillation with an adversarial loss and is employed to train SD-Turbo based SD 2.1 [76] teacher and SDXL-Turbo [81] based on SDXL [69]. Diffusion Matching Distillation (DMD) [102] additionally leverages a distributional loss based
approximated KL divergence and is applied for PixArt-a DMD [12, 13J. Lastly, Trajectory Segmented Consistency Distillation (TSCD) [75] introduces a progressive segment-wise consistency distillation [44, 87] loss to train HyperSDXL [75] with reward fine-tuning. All these models are trained in latent space such that during inference, an image is generated by first generating sample latent space and then decoding it Gg(E. D(Go p)) with a pre-trained decoder D.
2.2 Initial Noise Optimization
Given a Text-to-Image Image generative model Go(E,p) that generates images based on a noise and a prompt p. defined the following optimization problem following previous work [5, 43, 80, 91] with the objective of optimizing the noise e based on a criterion function C Hxx - evaluated on the generated image
e" arg max (G)
Then, given a differentiable C, (4) can be solved through iterative optimization via standard gradient ascent techniques. However, backpropagating through C ((Ga(E, is non-trivial as current Text-to- Image models are based the simulation of ODEs SDEs (Section 2.1). Several methods have been proposed to enable backpropagation through time-dependent generative models [14, 17, 60, 91], based on e.g., the adjoint method [70]. Our method, in contrast, leverages the crucial observation that selecting a one-step model as Go enables efficient backpropagation through (4). Although this realization may initially appear trivial, proves to be fundamental step facilitating practical noise optimization in Text-to-Image Image models. Current methods require between 10 [91] and 40 [5] minutes to optimize noise and thus, to generate single image. Our approach achieves image generation, including noise optimization, in 20-50 seconds, making it suitable for practical applications.

---

Noise regularization. One important consideration, is that it is desirable for e to stay within the proximity of the initial noise distribution .V I) as otherwise Gg might provide unwanted generations. This can be realized by including regularization term inside of C. Samuel et al. [79] propose instead of directly optimizing the likelihood of (r( to instead consider the likelihood of the norm of the noise r = ||e||, which is distributed according to X distribution p(r). Thus, following Ben-Hamu et al. [5], Samuel et al. [80] we maximize the log-likelihood of the norm of a In our framework, this corresponds to employing a regularized criterion function given by C(xo, E) = ((xo) + ((), which can be plugged into (4),
noise sample K (e) = (d |)) -
In Figure 3, we provide an illustrative example where we chose the criterion to maximize a selected color channel e of the generated image while minimizing the other two ε,, C2
Optimize Noise Wrt. Criturion C
30
(oso)=E,,)
A.i.c X
S. x
-xxx 1,0.22 (5)
where x.) x..de denotes the channel of the pixel at (1, Note that due to the calibration of the trained model and the noise staying in- distribution, the noise does not collapse to the optimal e*, which would result in the genera- tion of a fully blue or red image. Also, the op- timization first adapts the color of the car and then starts changing the background. Here, 10 optimization steps provide satisfactory re- sults illustrating the efficacy of the proposed one-step noise optimization framework.
Ge
"A sportscar"
Figure 3: Initial noise optimization for one-step Go HyperSDXL with two color channel criterions (5),
2.3 Human Preference Reward Models and Our Reward Criterion
Inspired by the success of Reinforcement Learning From Human Feedback [16, 29] in aligning LLMs with human preferences, similar methods have been explored for T21 generation. The underlying idea is to train a model Ry that takes in an input along with the generated output (in this case a prompt and the corresponding image) and provides a score for the "goodness" of the generated output. Notable open- human preference reward models for T2I include ImageReward [100] based on BLIP [49] and human preferences collected for the DiffusionDB dataset, PickScore [46], and HPSv2 [97] both based on a CLIP [73] ViT-H/14 backbone. These reward models provide a quantitative measure of the image's quality and relevance to the prompt through a prediction by differentiable neural network. Thus, they have not only been employed for the evaluation of T21
models but also fine-tune them [17, 18, 100] as means of achieving higher reward scores. Lastly, CLIPScore [35] has also been leveraged to measure the prompt alignment of generated image.
To generally enhance the performance of text-to-Imager models without any fine-tuning, we propose to leverage a Reward- based criterion function C for Noise Optimization (ReNO). Specifically, we propose to use weighted combination of a number of pre-trained reward models Rg.... R as the criterion function
Č(xo, p) = =,, R.R (xo. p),
(6)
where >j denotes the weighting for reward model Rd Employing a combination of reward models can help prevent "reward-hh "reward-hacking" and allow capturing various aspects of image quality and prompt adherence, as different reward models are trained on different prompt and preference sets. This not only effectively combines the strengths of multiple reward models, but also helps mitigate their weaknesses. ReNO then boils down to iteratively solving (4) with gradient ascent
e+1 = + e [[ (K(E') + Σ ( -)),))
(7)
where 17 is the learning rate. Similar to the color example in Figure 3, it is actually not desirable to find the optimal e* as we want to prevent adversarial samples that exploit the reward models. We find that already a few optimization steps (<50) of ReNO lead to significant improvements in both prompt following and visual aesthetics, striking a good balance between reward optimization and the prevention of reward hacking. Due to the efficacy of the proposed framework, generating one image, including noise optimization, takes between 20-50 seconds, depending on the model and image size, enabling its practical use. We provide sketch of ReNO in Figure 2 and full details in Algorithm 1.
5

---

3 Related Work
Initial Noise Optimization. The initial noise optimization framework was first introduced in DOODL [91] for improved guidance in Texl-to-Image models. Subsequently, it was leveraged by Karunratanakul et al. [43] for 3D universal motion priors, for rare-concept generation [61, 79, 80] and enhancing image quality [31, 89] in text-to-image models, music generation [62, 63], and by D-Flow [5] for solving inverse problems in various settings. While these methods mainly focus on controlling the generated sample for specific applications, our proposed method is designed to gener- ally improve text-to-Image models without the need for additional techniques to mitigate exploding or vanishing gradients on the optimization process. Most related our work is DOODL [91], which also proposes to improve the textual alignment of text-to-image models using a CLIP-score- criterion function, which similarly employ in our method. These existing methods, however, take 10 (DOODL) to 40 (D-Flow) minutes to generate a single image due to their application on time-dependent generative models with large number of denoising steps. To mitigate this, Samuel et al. [80] propose a hootstrap-based method to increase the efficiency of generating a batch of images. However, this method is limited to settings where the goal is to generate samples including a concept jointly represented by a set of input images.
Reward Optimization for Lext-to-Image Models. Reward models [46, 47, 97, 98, 100] were first introduced to mimic human preferences given an input prompt and generated images. There have been several attempts at incorporating these signals to enhance texl-to-image generation. One notable direction is the idea of using reinforcement learning based algorithms to fine-tune text-to-image models to better align with these rewards either with an explicit reward model [8. 11, 18, 23, 30, 109] or by bypassing it entirely with Direct Preference Optimization [50, 74, 92, 101]. However, this can be expensive, requiring thousands of queries to generalize, and therefore a lot of work has explored directly fine-tuning diffusion models [17, 51, 72] using differentiable rewards [39, 46, 47, 98, 100]. Additionally, there has also been works exploring the concept of using reward models to perform classifier-guidanee [4, 34] as well using rewards to distill diffusion models into fewer steps [48, 75]. Differently from these works, we focus on adapting a diffusion model during inference by purely optimizing the initial latent noise using differentiable objective.
4 Experiments
Experimental Setup. We evaluate the effectiveness of our proposed method, ReNO, using four open- source one-step image generation models: SD-Turbo, Turbo, PixArt-a DMD, FLUX-schnell and HyPerSDXL. HyperSDXL generates images of size 1024 x 1024 while the others generate 512 x 512. To assess the performance across diverse scenarios. we consider three challenging tasks. First, we evaluate on T21-CompBench [37]. which comprises 6000 compositional prompts spanning six categories, using a VQA, object detection, and image-text matching scores. Second, we employ GenEval [28], consisting of 552 object-focused prompts, measuring the quality of the generated images using a pre-trained object detector. Finally, we utilize Parti-Prompts [103], a collection of more than 1600 complex prompts, and assess the generated images using both reward-basse based metrics and extensive human evaluation. Throughout all experiments, we optimize Equation (7) for 50 steps using gradient ascent with Nesterov momentum and gradient norm clipping for stability. Lastly, we select the image with the highest reward score from the optimization trajectory for evaluation.
4.1 Effect of Reward Models
Table 1: SD-Turbo evaluated on the attribute binding categories of T21-CompBench and the LAION aesthetic score predictor [83] for different reward models.
We analyze the effect of various reward models in Table 1. We see that opti- mizing ImwaeReward or CLIPScore alone improves the rext-image faithfulness (i.e., attribute binding from T21-Compbench). However, this comes at the cost of de- creased aesthetic score. PickScore and HPSv2 improve the image quality, how- ever the gains in faithfulness are modest. Combining all the rewards leads to having strong improvements in faithfulness, while
Attribute Binding
Reward
Aesthetic
Color
Shape 1
Texture
SD-Turbo
0.5513
0.4448
0,5690
5.647
CLIPScore
0.6625
0.5501
0.6621
5,475
HPSv2
0.6443
0.5451
0.6859
5.752
ImageReward
0.7720
0.6104
0.7334
5.611
PickScore
0.6341
0.5069
0.6242
5.711
All
0.7830
0.6244
0.7466
5.704

---

Table Quantitative Results on 121-CompBench. ReNO combined with (1) PixArt-a
DMD [12, 13, 102], (2) SD-Turbo [81], (3) -oXL-Turbo [81], (4) HyperSD [75] demonstrates
superior compositional generation ability both attribute binding, object relationships, and complex
compositions. The best value is bolded, and the second-best value is underlined. Multi-step results
taken from [13, 22].
Attribute Binding
Object Relationship
Model
Complex
Color
Shapet
Texture
Spatialt
Non-Spatial
SD v1.4
0.38
0.36
0.42
0.12
0.31
0.31
SD v2.1
0.51
0.42
0.49
0.13
0.31
0.34
SDXL
0.64
0.54
0.56
0.20
0.31
0,41
PixArt-a:
0.69
0.56
0.70
0.21
0.32
0.41
DALL-E2
0.57
0.55
0.64
0.13
0.30
0.37
DALL-E3
0.81
0.68
0.81
(I) PixArt-a DMD
0.38
0.34
0.47
0.19
0,30
0.36
(1) ReNO (Ours)
0.64
0.57
0.72
0.25
0.31
0,46
(2) SD-Turbo
0.55
0.44
0.57
0.17
0.31
0,41
(2) ReNO (Ours)
0.78
0.62
0.75
0.22
0.32
0.48
(3) SDXL-Turbo
0.61
0.44
0.60
0.24
0.31
0.43
(3) ReNO (Ours)
0.78
0.60
0.74
0.26
0.31
0.47
(4) HyperSDXL
0.65
0.50
0.65
0.25
0.31
0.46
(4) ReNO (Ours)
0.79
0.63
0.77
0.26
0.31
0.48
Table Quantitative Results on GenEval. ReNO combined with (1) PixArt-a DMD [12, 102],
(2) SD-Turbo [81]. (3) SDXL-Turbo [81], (4) HyperSDXL [75] improves results across all categories.
The best value bolded, and the second- best value is underlined. Multi step results taken from [22].
Model
Mean
Singlet
Twot
Counting
Colorst
Position?
Color Attribution
SD v2.1
0.50
0.98
0.51
0.44
0.85
0.07
0.17
SDXL.
0.55
0.98
0.74
0.39
0.85
0.15
0.23
IF-XL
0.61
0.97
0.74
0.66
0.81
0.13
0.35
Aix
0.48
0.98
0.50
0.44
0.80
0.08
0.07
DALL-E 2
0.52
0.94
0.49
0.77
0.10
0.19
DALL-E
0.67
0.96
0.83
0.43
0.45
SD3 (8R)
0.68
0.98
0.84
0.66
0.74
0.40
0.43
(1) PixArt-a DMD
0.45
0.95
0.38
0.46
0.76
0.05
0.09
(I) ReNO (Ours)
0.59
0.98
0.72
0.58
0.85
0.15
0.27
(2) SD-Turbo
0.49
0.99
0.51
0.38
0.85
0.07
0.14
(2) ReNO (Ours)
0.62
1.00
0.82
0.60
0.88
0.12
0.33
(3) SDXL-TTurb
0.54
1.00
0.66
0.45
0.84
0.09
0.20
(3) ReNO (Ours)
0.65
1.00
0.84
0.68
0.90
0.13
0.35
(4) IXL
0.56
1.00
0.76
0.43
0.87
0.10
0.21
(4) ReNO (Ours)
0.65
1.00
0.90
0.56
0.91
0.17
0.33
Fl.UX-schnel
0.64
0.98
0.80
0.64
0.78
0.18
0.43
FLUX-schnell ReNO (Ours)
0.72
0.99
0.90
0.79
0.87
0.21
0.56
FLUX-dev
0.68
0.99
0.85
0.74
0.79
0.21
0.48
also increasing the image quality. Thus, we employ ReNO with all four reward models. We report further details in Appendix C, including the performance of all combinations of reward models.
4.2 Quantitative Results
Table 2 presents the quantitative results ReNO on T21- 721-Compbench. Most notably, we observe that for both Pixart-a DMD and SD-Turbo, there are improvements of over 20% in the Color, Shape, and Texture Categories. For instance, on Color SD-Turbo improves from 55% to 78%, which only slightly below DALL-E.3 3. Similar improvements can also be seen for SDXL-Turbo and HyperSDXi. models where performance increases by over 10 percentage points these categories. Even outside this, there are significant boosts in the Spatial, Non-Spatial, and Complex categories, highlighting

---

both the efficacy of the noise optimization framework, as well as the utility of human preference models for improving T2I generation at inference. Similar trends can also be noticed for GenEval in Table 3, where applying our noise optimization framework helps improve the performance of various one-step diffusion models. For instance, SD-Turbo improves its mean score from 0.49 to 0.62. Notably, our strongest model, HyperSDXXL + ReNO, comes very close to the proprietary DALL--E 3 and SD3, i.c. beating DALL-E 3 on 4/6 categories in GenEval. In the case of FLUX-schnell, ReNO improves the performance (0.72) to even surpass that of the base FLUX-dev model (0.68). Most notably, this is the strongest open-source results reported on the GenEval benchmark. both of these benchmarks, the categories. It is also important to note that both Compbench and GenEval use a variety of methods unrelated to human preference rewards, such as VQA models and object detectors, to detect different objects in the generated images. We report further quantitative results including comparisons other test-time-bie based methods in Appendix B. Additionally, these quantitative results supported by the qualitative results reported in Figure I and Appendix A. Lastly, we report full details for the conducted FLUX-schnell experiments in Appendix E.8.
noise optimization framework improves results for all the models in all
4.3 User Study Results
Results RobiO Various Nethods
ReNO (hyper5DXt)
54, .%%%%
45.4%
vs 5D3
To further validate ReNO we perform user study on the commonly used Parti-
ReNO (hyper5DXt)
60.0%
39.4%
vs MyperStXi, 1I Step!
Prompts [103] with Amazon Mechanical Turk
Reto (SD-Turbe))
62.95
37.15
vs SDXL 150 Stess)
(AMT). Parti-Prompts generally includes longer complex prompts that test artistic generation ca- pabilities as opposed to 121-Compbench and
ReAo (SD-Turbb))
67.0%
33.0%
vs 50 (50 Stops!
ReAo (SD-Turbe)
68.9%
31.1%
vs SDX, Turbo (1 Stapl
GenEval, which purely focus on faithfulness. We conducted user studies with ReNO applied SD-Turbo for 512> x 512 and HyPerSDXL for
ReAo (SD-Tubbe)
77.35
22.7%
vs Burbo (1 Stop!
Prefemence Hake (56)
1024 x 1024 generation. We compare SD>-uubo + ReNO against SD-Turbo, SDXL-Turbo, SD2.1 (50 Steps), and BDxe-Base (50 Steps). The results in Figure 4 confirm our findings in the quantitative evaluation. SD-Turbo + ReNO has an above 60% win rate against all benchmarked models reaching up to 77% against the SD-Turbo base. To contextualize these results, SD3 [22] conducts a similar user study on Parti-Prompts and reports a 70% win rate against SDXL (50 steps). Our strongest base model, HyperSDXL, already beats SDXL (50 steps) [75] without ReNO. Thus, compare it with and without ReNO as well as against the proprietary SD3 (8B) [22]. Again, HyperSDXL + ReNO achieves an above 60% win rate, and notably, it also narrowly beats SD3 with 54%. This confirms our finding in ReNO, which substantially improves overall generative quality, pushing results at least close to the ones of even current state-of-the-ard proprietary models. Lastly, we note that user studies on AMT can potentially be noisy and, therefore, view the results holistically along with quantitative evaluation. We provide detailed breakdown of the preference for image quality and faithfulness, as well as full details of the user study in Appendix D.
Figure 4: User Study Results for ReNO
Performance Increase with Iteration Steps
4.4
Computational Cost of ReNO
Score [Color, Shape, Texture)
0.70
The primary concern of our proposed method is the increased inference cost since existing methods (e.g. DOODL, D-Flow) are impracti- cal for regular T2I generation usage. However, we circumvent this issue through our restriction to one-step models and 50 optimization steps, which makes ReNO run in 20-50 seconds. To analyze the performance of ReNO with respect to the number of optimization steps we evaluate its performance over set of reference points. We report results on the attribute binding part of 121-ComBBench for SD-Turbo + ReNO in Figure 5 and visually corroborate these results with Figure 6. Note that even when restricted to the same compute budget as SDXL (50 steps, ~7sec), SD-Turbo + ReNO significantly outperforms while in this comparison PixArt-a: (20 steps, ~7sec) lies shortly below the
0.85
0.80
ReNO (SD-Turbo)
SDXL
0.55
PiAAri-
Computstiona Coet (SDXL)
15
S0
70
Iteration Steps
Figure 5: Attribute binding results T2I-
CompBench with varying number of iterations.
of ReNO.
8

---

pt "A yellow reindeer and a blue elephant"
ReNO
10
20
30
.4
50
E
G) Xo
Reward Model Feedback R2(
PixArt-a DMD
SD Turbo
SDXL Turbo
HyperSD
Figure 6: The initial images generated with four different one-step models Gg given the prompt p "A yellow reindeer and a blue elephant" and randomly initialized noise e". Each column shows the result of optimizing the noise latent ex for steps with respect to our reward-bass criterion.
4.5 Effect of ReNO on the Diversity of Generated Images
To investigate the effect of noise optimiza- tion output diversity, we evaluate im- ages generated across 50 different random seeds for 1 10 prompts from Parti- Parti-Pront Prompts. Specifically, we generate a batch of images and use LPIPS [106] and DINO [9, 64] scores to compute the average similarity the generated batch, where lower simi- larity score corresponds to higher diversity. As shown in Table 4, one-step models (SD- Turbo, SDXL-Turbo) exhibit lower diver- sity compared their multi-step counter- parts (SD2.1, 1, SDXL), likely due to adver- sarial training. However, applying ReNO not only maintains but actually increases diversity. For both SD-Turbo and SDXL-Turbo. ReNO achieves diversity levels approaching their respective multi-step base models highlighting an unexpected benefit of noise optimization increasing diversity. Figure 10 illustrates these improvements qualitatively. We hypothesize that the reason for this increased diver- sity is that ReNO adds structure to the noise, thus optimizes away from the zero mean of the noise distribution and creating more diverse noises compared to sampling from the standard Gaussian.
Table
We measure the average LPIPS and DINO similarity scores over images generated for 50 different seeds for 100 prompts from Parti-Promtps.
LPIPS
DINO 1
SDrborbo
0.382 00.043
0.770
10.101
SD-Turbo + ReNO
0.246 +0.046
0.712 +0.132
SD2.1 (50-step)
0.243 10.049
0.623
10.150
SDXL-Turbo
0.391 00.044
0.835
+0.073
SDXL-Turbo + ReNO
0.291
10.041
0.763
+0.116
SDXL (50-step)
0.351 00.02
0.700 =0.128
4.6 Comparison to Multi-Step Noise Optimization
We benchmark ReNO against DOODL [91]. which performs noise optimization using the 50-step SD2.1 model. Due to DOODL's computational demands, we evaluate on the first 50 prompts from T21-CompBench. Despite using the same CLIPScore objective, ReNO achieves four times larger improvements in the optimized criterion while requiring 75% less GPU memory and running 100x faster, highlighting the effectiveness of our one-step approach. Moreover, ReNO's multi-reward objective leads to substantially larger gains in attribute binding accuracy (21.0-28.9%; (%.% compared to solely using CLIPScore, reiterating the efficacy of ReNO's optimization objective.

---

Table 5: Performance comparison of ReNO and DOODL over the first 50 prompts of each of the Attribute Binding categories in D2l-CompBench. We report scores from default 12I-Compbench evaluation using BI.I--QQA as well as the optimized CLIPScore before and after optimization.
Attribute Binding (Change)
Model
CLIPScore
secfiter (total)
VRAM
Color f
Shapet
Texturet
SD2.1
33.4 38.5 (+5.1)
52.4
63.4
0.261 0.289 (+0.03)
SD2.1 DOODL (CLIP)
$1.6 (-0.8)
64.6 (+1.2)
24s (20min)
40GB
-D-Turbo
48.5
61.8
0.362
SD-Turbo ReNO (only CLIP)
70.1 (19.7)
66.9 (+18.4)
79.6 (+18.2)
0,483 (+0.12)
0.2 (10s)
10GB
SD-Turbo ReNO (all)
82.1 (+21.7)
77.4 (+22.99
82.8 (+21.0)
0,437 (+0.08)
0.4 (20s)
15GB
4.7 Limitations
An interesting observation in our experiments is that despite using different image generation models of varying architectures and sizes, they broadly converge to similar performance on both T21- Compbench and GenEval. In addition to the limitations of the generative models, we hypothesize that this could be due to the limitations of the reward models themselves, ves, given their limited compositional reasoning abilities [104]. Stronger reward models [53, 107] and preference data [15, 33, 42, 99, 108] would be crucial in enhancing results further.
Secondly, not only the runtime but also the amount of needed GPU VRAM is signif- icantly higher when using ReNO. We re- duce it by leveraging fp16 quantization and the pytorch [67] memory reduction tech- nique introduced in Bhatia and Dangel [7]. which for ReNO lowers the VRAM by an- -15%. Then, all of the models can be optimized on a single A100 GPU in 20-50 seconds, and c.g., SD-Turb requires only 15GB VRAM for the entire optimization process. Note, however, that the amount of VRAM also scales with the size of the generated image. Thus, HyperS- DXL needs 39GB of VRAM. We provide summary of the computational cost of ReNO in Table 6, which lays out ReNO's 's main limitation. Finally, current T2I models struggle with generating humans, rendering text, and also modeling complex compositional relations. While our work attempts to alleviate these issues and provides a flexible framework for further improvements, future work is required to resolve these issues.
Table 6: Computational cost comparison of ReNO opti- mizing four reward models on an A100 GPU.
Method
sec/iter (total)
VRAM
#params
Img size
SD-Turbo
0.4s (20s)
15GB
860M
512 x 512
SDXL-Turbo
0.6s (30s)
21GB
2.6B
512 512
PixArt-a DMD
0.5s (25s)
25GB
600M
512 x 512
FLUX-schmell
0.6s (30s)
50GB
12B
512 x
HyperSDXL
1.08 (50s)
39GB
2.6B
1024 x 1024
5 Conclusion
We introduce ReNO, a test-time optimization strategy for enhancing text-to-image generation without any fine-tuning. Not only do we achieve the strongest results among all open-source models T21-Comphench and GenEval, but images from ReNO on a single-step SDDTurbo have over a 60% win rate against a 50-step SDXL model and competitive with the 8B parameter SD3 model on user studies. We also demonstrate that ReNO outperforms SDXL even when restricted to the same computational budget, highlighting the benefits of ReNO for practical use cases. The performance gains from ReNO underscore the importance of developing even better and more robust reward models and, moreover, establish valuable benchmark for assessing their effectiveness. Furthermore, the substantial impact of optimizing the initial noise distribution motivates further research into understanding, manipulating, and controlling this crucial aspect of generative models.
Acknowledgements
This work was supported by BMBF FKZ: 011S18039,, by the ERC (853489 DEXIM), by EXC number 2064/1 project number 390727645. Shyamgopal Karthik and Karsten Roth thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. Luca Eyring and Karsten Roth would also like to thank the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support.
10

---

References
[1] Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In ICLR, 2023.
[2] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 1982.
[3] Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny, Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. arXiv preprint arXiv:2304, 053900 2023.
[4] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In ICLR, 2024,
[5] Heli Ben-Hamin Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. In ICML, 2024.
[6] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. OpenAl Technical Report, 2023,
[7] Samarth Bhatia and Felix Dangel. Lowering pytorch's 's memory consumption for selective differentiation. 2024.
[8] Kevin Black, Michael Janner, Yilun Du, lyy Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[10] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. SIGGRAPH, 2023.
[11] Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffusion models with text-encoder reinforcement learning. arXiv preprint arXiv:231..15657 2023.
[12] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao. Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403 04692, 2024.
[13] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[14] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018
[15] Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Mj-bench: Is your multimodal reward model really good judge for text-to-image generation?, 2024. URL https: /arxxv..vvv. org/ abs/ /2077 04842.
[16] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NIPS, 2017.
[17] Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards. In ICLR, 2024.
[18] Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, and Tingbo Hou. Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models. In CVPR. 2024.
[19] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
[20] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023.

---

[21]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021.
[22]
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint SaKiv::20030332066 2024.
[23] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lec, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. NeurlPS, 2023.
[24] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fres -free structured diffusion guidance for compositional text-to-image synthesis. In ICLR, 2023.
[25]
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. In NeurlPS, 2023.
[26] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. NeurIPS, 2023.
[27] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. In NeurIPS, 2022.
[28] Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused frame- work for evaluating text-to-image alignment. In NeurIPS, 2023.
[29] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. NIPS, 2013.
[30] Jianshu Guo, Wenhao Chai, Jie Deng. Hsiang- Wei Huang, Tian Ye, Yichen Xu, Jiawei Zhang, Jenq-Neng Hwang, and Gaoang Wang. Versat2i: Improving text-to-image models with versatile reward. arXiv preprint arXiv:2403. 18493, 2024.
[31] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In CVPR, 2024.
[32] Yuwei Guo. Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
[33] Xu Han, Linghao Jin, Xiaofeng Liu, and Paul Pu Liang, Progressive compositionality in text-to-image generative models. arXiv preprint arXiv:2410.167. 2024.
[34] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongiun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In ICLR. 2024.
[35] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022.
[36] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[37] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In NeurIPS. 2023.
[38] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL htthtt///doo orw.1orggg..o 5281/ zenodo zenodo.6143773.
[39] Arman Isajanyan, Artur Shatveryan, David Kocharyan, Zhangyang Wang, and Humphrey Shi. Social reward: Evaluating and enhaneing generative ai through million-user feedback from an online creative community. In ICLR, 2024.
12

---

[40] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In NeurIPS, 2022.
[41]
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. arXiv preprint arXiv:2305. 13308, 2023.
[42] Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for text-to-image generation. arXiv preprint arXiv:2410. Stti::240..8803,, 2024.
[43] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwa- janakorn, and Siyu Tang. Optimizing diffusion noise can serve universal motion priors. In CVPR, 2024.
[44] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. In ICLR, 2024.
[45] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurlPS, 2021.
[46] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS, 2023.
[47] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 23X.:2302..2//99 2023.
[48] Jiachen Li, Weixi Feng, Wenhu Chen, and William Yang Wang. Reward guided latent consistency distillation, 2024.
[49] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 2022.
[50] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.0 04465, 2024.
[51] Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Textcraftor: Your text encoder can be image quality controller. In CVPR. 2024.
[52] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. TMLR. 2024.
[53] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv 2024.
preprint arXiv:2404.011
[54] Yaron Lipman, Ricky T. Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.
[55] Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-image alignment with deep-fusion large language models, 2024. URL https:// arxiv. orriv.orr/abbee 2aabb/2400 10695.
[56] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. ECCV, 2022.
[57] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
[58] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
13

---

[59] Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eije Eijnden. and Saining Xie. Sit: Exploring flow and diffusion-baate generative models with scalable interpolant transformers, 2024.
[60] Pierre Marion, Anna Korba. Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet. Implicit diffusion: Efficient optimization through stochastic sampling. arXiv, 2024.
[61] Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari. Fixed-poin inversion for text-to-image diffusion models. arXiv preprint arXiv:2312.12540, 2023.
[62] Zachary Novack, Julian McAuley, Taylor Barg-Kirkpatrick. and Nicholas J. Bryan. DITTO-2: Distilled diffusion inference-time t-optimization for music generation. In International Society of Music Information Retrieval (ISMIR), 2024.
[63] Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrien and Nicholas J Bryan. Ditto: Diffusion inference-time t-optimization for music generation. In ICML, 2024.
[64] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07198, 2023.
[65] Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin' ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In CVPR, 2023.
[66] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In NeurIPS Datasets and Benchmarks Track, 2021.
[67] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high sesshperffrmance deep learning library. NeurlPS, 2019.
[68]
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023.
[69]
Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[70] L.S. Pontryagin. Mathematical Theory of Optimal Processes. Classics of Soviet Math- ematies. 1987. ISBN 9782881240775 URL https://books. .oogle de/books?id- kwZqOF4cBVAC.
[71] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion, 2022.
[72] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 2023.
[73] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[74] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. NeurlPS, 2023.
[75] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, 2024.
[76] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[77] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[78] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
14

---

[79] Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm corm-guided latent space exploration for text-to-image generation. In NeurIPS, 2023.
[80] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. In AAAI, 2024.
[81] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.1704. 2023.
[82] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1,,02/ 14, 2021.
[83] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
[84] Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
[85] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[86] Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[87] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
[88] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
[89] Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung- Hui Chang. Tuning-free aligmment of diffusion models with direct noise optimization. arXiv preprint 1xXi:22405.188// 2024.
[90] Patrick von Platen. Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https: //t thub. con/huggingface/ diffusers, 2022.
[91] Bram Wallace, Akash Gokul, Stefano Emnon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In ICCV, 2023.
[92] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In CVPR, 2024.
[93] Ruichen Wang. Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Com- positional text-to-image synthesis with attention map control of diffusion models. In AAAI, 2024.
[94] Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. arXiv preprint orXiv:2401.15660 2024.
[95] Zijie J Wang. Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horgg Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprini arXiv:2210. 14896, 2022.
[96] Zirui Wang. Zhizhou Sha. Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Grounding diffusion with token-level supervision, 2023.
[97] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng
Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.00 09040, 2023.
[98] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. In ICCV, 2023.
15

---

[99] Xun Wu, Shaohan Huang, and Furu Wei. Multimodal large language model is a human-aligned annotator for text-to-image generation. arXiv preprint arXiv:240..15100 2024.
[100] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurIPS, 2023.
[101] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In CVPR, 2024.
[102] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv 2023.
preprint a.
[103] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022.
[104] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalturi, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of- -ords,, and what do about it? In ICLR, 2022.
[105] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[106] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[107] Sixian Zhang. Bohan Wang, Jungiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimensional human preference for text-to-image generation. In CVPR, 2024.
[108] Xinchen Zhang, Ling Yang, Guohao Li, Yaqi Cai, Jiake Xie, Yong Tang, Yujiu Yang, Mengdi Wang, and Bin Cui. Itercomp: Iterative composition-awand feedback learning from model gallery for text-to-image generation. arXiv preprint arXhv:2410.07171, 2024.
[109] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models, arXiv preprint arXiv:2401. 12244, 2024.
16

---

Appendix
The Appendix is organized follows:
Section A provide further qualitative results.
Section B presents further quantitative analysis.
Section C provides an analysis of the different employed reward models.
Section D describes the full details for our user study setup.
Section outlines the implementation specifics, including the algorithm.
A Further Qualitative Results
Here, report further qualitative results. We separate them into 512 x 512 and 1024 X 1024 generated images. First, in Figure 7, we show examples of ReNO applied to SD-Turho, SDXIL-Turbo, and PixArt-a DMD. Then, in Figures 8 and methods. Broadly, we see that ReNO not only fixes the artifacts occurring in one-step models but also improves compositional understanding (color attribution, spatial reasoning) as well as the quality of generated faces and pushes current one-step models to be broadly on par with proprietary models in these settings. Lastly, we report qualitative results for the effect of ReNo on the diversity of generated images in Figure 10.
report HyperSIXXI. ReNO against competing
PixArt-e DMD
ReNO
SD-Turbo
ReNO
-uuhoTurbo
ReNO
SD2 (50 steps)
"A gream dag and
gray cos"
"A arange chade to the right airplass"
Joheson desicted millesopher ucadesic
Suift depicted minister peinting
"A teaster ridiig bike
Figure 7: Comparison of images generated with and without ReNO at 512 x 512 resolution across various one-step models and SD 1. The noise used to generate the initial image is the same one that is used to initialize ReNO.
17

---

HDDerSD
ReNO
SDXL (50 steps)
SD3
DALL-E 3
"A green dog and red
"A pink slaphant and grey caw"
"A orange chair to the right airplane"
depicted
philosopher king acadenic painting Grag Rutkouski"
"Taylar Swift depicted prine ninister in an acadenic painting by Kandinsky"
riding bike'
Figure 8: Images generated with and without ReNO using HyperSDXL at 1024 x 1024 resolution compared competing T2I models SDXL, SD3, and DALL-E 3. ReNO helps fix artifacts and generates images of comparable quality to even closed-source models. The noise used to generate the initial image the same one that is used initialize ReNO.
18

---

HyperSD
HyperSD ReNO
SDXL steps)
DALLE
curious, orange fox and fuufy, white rabbit, playing to- gether green meadow filled with yellow dandilions.
An epic, futuristic cityscape
painting: solitary figure, and a color- ful sky over snowy mountains.
painting:
escapes glont shark's leap. right: eminous mountains, bloe
A majestic, ruiilient sea ship navigates the ley wilderness the style
An expansive future city, flonting Japenese-style spaceship, bebind a med- itating Mahershala All.
Figure 9: Comparison of generated images from different models (HyperSD, HyperSD ReNO, SDXL (50 steps), SD3, DALLE-3) for various prompts. Each row corresponds to a specific prompt, and each column represents different model.
SD-Turbo
"A van parked on grass"
+ ReNO
SD-Turbo
'A photo of Light bulb outer space traveling Ling the gel salaxy with seiling bost inside Light bulb."
+ ReNO
Figure 10: Kon-cherry-picked results for SD-Turbo with and without ReNO for two different prompts
over the first 5 seeds. ReNO increases the diversity of generated images w.r.t. content and layout.
19

---

B Further Quantitative Results
B.1 Comparison to Direct Preference Optimization
Direct Preference Optimization has recently been applied in the context of Diffusion models [8, 50, 101]. Here, we compare against an SDXL model that has been preference-tunee on dataset of over 800k preferences in Table 7. We see that while DPO improves both attribute binding and the aesthetic score of the generated images, it underperforms the SIXL-TTurbo with ReNO. This highlights the potential of dest-time/onlnn optimization compared to traditional fine-tuning, since it can generalize much better to unseen prompt distributions.
Table 7: Comparison of ReNO and Direct Preference Optimization (DPO) with SDXL-based model. SDXL Base result taken from [13].
Method
Attribute Binding
Color t
Shape
Texture
Aesthetic
SDXL Base [69]
0.6369
0.5408
0.5637
5.604
DPO-SDXL [92]
0.6793
0.5316
0.6513
5.687
SDXL-Turbo ReNO
0.7800
0.5955
0.7396
6.024
B.2 Compositional Text-to-lmage Methods.
We show the results for several methods that have been tailored compositional Text-to-Image generation in Table 8. These methods either explicitly finetune the model for improved compositional generation, or modify the inference process, or repeat the sampling over multiple iterations. We see that ReNO consistently outperforms specific methods tailored for this task.
We also note that some methods LLMs and other tools (image-editing, customization etc.) to plan out or correct generations [25, 52, 94]. However, these methods significantly impact the generation process through, e.g., iterative generation and planning. In contrast, ReNO only changes the initial noise and doesn't t alter the generative model at all. Thus, our method could also be incorporated into these tools further improve performance.
Table 8: Quantitative Results on 12I-CompBench. Full comparison against different Compositional Text-to-Image methods. The best value is bolded, and the second-best value is underlined. Results for compositional methods taken from [66, 94].
Attribute Binding
Object Relationship
Model
Complext
Color f
Shapet
Texture
Spatialt
Non-Spatial
SD2.1
0.5065
0.4221
0.4922
0.1342
0.3096
0.3386
Composable Diffusion [56]
0.4063
0.3299
0.3645
0.0800
0.2980
0.2898
Ma--Mass-Connt [93]
0.4119
0.4649
0.4505
0.1249
0,3046
0.3779
Diffusinn [24]
0.4990
0.4218
0.4900
0.1386
0,3111
0.3355
+ TokenCompose [96]
0.5055
0.4852
0.5881
0,1815
0,3173
0,2937
+ Attn-Ei Attn-Exet [10]
0.6400
0.4517
0.5963
0,1455
0,3109
0.3401
GORS [37]
0.6603
0.4785
0.6287
0.1815
0.3193
0.3328
3u-Turoo + ImageSelect [41]
0.7222
0.5552
0.6919
0.2216
0.3154
0.4618
(1) PixArt-a DMD
0.3824
0,3414
0.4691
0.1906
0,3060
0.3643
(1) ReNO (Ours)
0.6454
0.5658
0.7186
0.2508
0.3138
0,4554
(2) SD-Turbo
0.5513
0.4448
0.5690
0.1739
0,3101
0.4052
(2)+ ReNO (Ours)
0.7830
0.6244
0.7466
0,2235
0,3161
0.4829
SDXL-Turbo
0.6149
0,4366
0.6001
0.2401
0.3118
0.4250
(3)) + ReNO (Ours)
0.7800
0.5955
0.7396
0.2551
0.3147
0.4690
(4) HyPerSDXL
0.6535
0.4956
0.6496
0.2509
0.3108
0.4582
ReNO (Ours)
0.7904
0.6324
0.7671
0.2616
0.3145
0.4766
20

---

Comparison to Multi-step Noise Optimization Methods. In Section 4.6, we report quantitative comparison between the multi-step noise optimization method DOODL and ReNO. In Table additionally report more details on the difference in efficiency between DOODL and ReNO. Note that for the same objective and model family ReNO is 120x faster compared DOODL..
Table 9: Computational cost comparison of ReNO compared DOODI..
Model
sec/iter (total)
T2l-CompBench
VRAM
#params
SD2.1 + DOODL (CLIP)
24s (20min)
83.33 A100 days
40GB
860M
SD-Turbo + ReNO (only CLIP)
0.2s (10s)
0.63 A100 days
10GB
860M
SD-Turbo ReNO
0.4s (20s)
1.25 A100 days
15GB
860M
C Analysis of Reward Models
We show all results for all combinations of the reward models in Table 10. Broadly, adding all reward models ensures that meaningful improvement is achieved both on attribute binding and on the aesthetic score. In addition this, we perform a leave-one-out analysis on Parti-Prompts in Table 11, where one reward is excluded from ReNO and subsequently analyzed.
Even when particular reward is not optimized for, in the metrics, and in most cases, at least 80% the images improve even on the left-out reward. This phenomenon across variety of models (e.g. CLIP, BLIP) trained differing datasets certainly indicates that there are significant improvements made by ReNO across most of the images. While the reward increase and the percental improvement, and ImageReward seem to be less correlated to the other rewards, which could be explained based on the similar backbone employed by HPSv2 and PickScore. Interestingly, PixArt-a DMD achieves the highest reward scores after optimization, which does not follow the quantitative results for "el-CompBench and GenEval reported in Section 4.2.
that there is consistent improvement
differ based the one-step model, CLIPScore
Table 10:
Full results for different reward model combinations considered in ReNO over the attribute binding categories of T'T2--Compli 211-CommBeech and the LAION aesthetic score predictor [83]. We highlight the best and second-bes, results per number of reward models.
Attribute Binding
Reward Models
Aesthetic
Color 1
Shape t
Texture
Base (SD-Turbo)
0.5513
0.4448
0.5690
5.647
CLIPScore
0.6625
0.5501
0.6621
5.475
+ HPSv2
0.6443
0,5451
0.6859
5.752
+ ImageReward
0.7720
0.6104
0.7334
5.611
+ PickScore
0.6341
0,5069
0.6242
5.711
+ CLIPScore + HPSv2
0.6691
0.5664
0.6979
5.714
CLIPScore Reward
0.7749
0.6218
0.7415
5.579
+ HPPS2 Image
0.7710
0.6228
0.7518
5.692
PickScore CLIP
0.6606
0.5500
0.6735
5.615
+ PickScore + HPSv2
0.6593
0.5571
0.6766
5.776
+ PickScore ImageReward
0.7798
0.6298
0.7354
5.662
+ CLIPScore HPSP2 + ImageReward
0.7735
0.6238
0.7524
5.677
+ PickScore CLIPScore + HPSv2
0.6886
0.5599
0.7012
5.733
+ PickScore + CLIPScore + ImageReward
0.7797
0.6218
0.7513
5.620
PickScore HPSv2 + Rewae
0.7778
0.6298
0.7457
5.713
All
0.7830
0.6244
0.7466
5.704
Reward Weighting. The four reward models that we employ output scores in different ranges. Specifically, HPSv2 mostly ranges between 0.2-0.4, while PickScore is in the range of 20-30 for
21

---

most of the mages. ImageReward is in the range of -2 to +2 for the majority of images, and CLIPScore between and For all our experiments, we use weights of 1.0 for ImageReward. 5.0 for HPSv2, 0.05 for PickScore, and 1.0 for CLIPScore. When each score range is scaled to [0, 1], then these weights correspond CLIPScore. These weights ensure that the losses from each reward model are roughly similar, with higher emphasis on ImageReward.
for ImageReward, 1.0 for HPSv2, 0.5 for PiekScore, and 1.0 for
Table II: Leave-one-out reward evaluation on Parti-Prompts. The listed reward first column left out in ReNO and subsequently analyzed with respect its change well as the percentage of generations where ReNO improves this reward.
Initial Reward
ReNO Reward
Change 1
Improve t
PixArt-a DMD
CLIPScore [0, 1]
0.332
0.386
+0.054
95.1
PickScore [20, 30]
22.235
23.788
+1.553
97.6
HPSv2 [0.2, 0.4]
0.281
0.324
+0.043
97.7
ImageReward [-2,
0.896
1.367
+0.471
88.7
SD-Turbo
CLIPScore [0, 1]
0.353
0.386
+0.033
80.4
PickScore [20, 30]
22.028
23.232
+1.204
91.1
HPSv2 [0.2, 0.4]
0.266
0.310
+0.044
94.7
ImageReward [-2,
0.552
1.243
+0.691
90.3
SDXI.-Turo
CLIPScore [0, 1]
0.360
0.379
+0.019
74.2
PickScore [20, 30]
22.505
23.185
+0.680
82.3
HPSv2 [0.2, 0.4]
0.280
0,310
+0.030
93.4
ImageReward [-2,
0.920
1.270
+0.350
81.0
D User Study
We perform our user study on Amazon Mechanical Turk, and pay participants based on prior guidelines [65], which also ensures the compensation is above the minimum wage. We pairwise preferences due to its simplicity, allowing users to mark ties between images that are equally good/bad. Each pairwise comparison is treated as individual entity and handed an individual user to minimize user biases. In particular, each pairwise comparison between the two models has involved at least 339 unique users (and 673 maximal), with the average being 495. To reduce the number of user comparisons, we perform the user study on a subset of Parti- Parti-Prompps totaling slightly above 1000 prompts (excluding challenges ('Basic', 'Imaagnattoo', 'Perspective', 'Linguistic Structures") ] and categories ['Abstract', 'Indoor Scenes', 'Produce & Plants") 'I).
We ask users to answer the following three questions:
On personal preference: "Which image would you personally prefer getting given the input text (based on your personal tradeoff between faithfulness and aesthetics)" ?
On aestheticness:
"Which image do you find more aesthetically pleasing?
On faithfulness: "Which image more faithful the input text?"
We also provide additional information on terminology:
Faithfulness:
The generated image should reflect all key concepts, their relations and their attributes given in the text prompt.
Aestheticness: Refers to the style, coloring and interpretation in the depiction of concepts (i.e. "looks better").
Personal preference: Some generations can be more faithful, but less aesthetic, or the other way around. Choose which you prefer :).
22

---

User Study keeutss for Resb vs Viitus Methors
User Shudy uils for Seho ous Merticds
NLNO (yyabb
Poho Trype SDXL
17.4%
4715
SDX,
Step)
takO 1p murbel
12.4%
37.24
5DAL (50 stepd
ReVi (S0 (orbt)
61. 24
20.5%
vs 5IXA, 150 Skepsl
ano (SD-Tumss)
64
35.00
so0 JTuto Stedl
ReND (0-3lrtv))
2/6
10
Nurbo Step)
Irage Quality Preference
Preference Sace (96)
(a) User Preferences for Image Quality
(h) User Preferences for Prompt Faithfulness
Figure 11: User study results on aestheticness and faithfulness based on Parti-Prompts
For the competing methods, guidance (CFG), and CFG = 7.5 for SD2.1 and CFG 5.0 for SDXL. For the proprietary SD3, we generate images through the API provided at ctttp://platte: compare SDXL with SD-Turbo + ReNO, we generate images in 1024 x 1024 for SDXL as this is its native resolution and then afterward downsize them to 512 512.
default parameters: one-step generation without classifier-free
stability .ai.. Note that to
In Figure 11, we report the results for the specific questions on faithfulness and aestheticness. Interestingly, for all models, the preference for aestheticness is even larger than that of faithfulness. While the quantitative results on T2I-CompBench and GenEval reported in Section 4.2 mainly benchmark the prompt following improvements of ReNO, this result confirms ReNO's benefits in improving the general quality of generated images.
E
Implementation Details
Our code is built with Pytorch [67] and is mainly based on the diffusers library [90]. It is available at https://giiie
ExxplannableM...
F.1 Algorithm
We outline the overall algorithm for ReNO in Algorithm Note that, choose gradient ascent with Nesterov momentum as we found this for our computational budget to yield the best results. Although line-search-based methods such as L-BFGS them gradient ascent provides efficient and effective optimization of the criterion function. However, L-BFGS gradient ascent without momentum might also viable optimization methods for ReNO.
viable options [5], find that even without
Algorithm ReNO
Input: p (prompt), Go (One-Step T2I Model), R....... (Reward Functions), 1o. (Reward Weights), m (# Optimization Steps), 7) (Learning Rate), Areg (Regularization Strength) Initialize 1- 0.0, E NO I), R inf. for t = 0 to m do
Generate image x² Compute reward-based criterion Rd = Σx (x
Go(E'.p) .p)
grad, = Vee D (E') + Rd
grad, = GradNormCClppgrradd 0.1)
0.9 1;-1 Vz
grad,
e 1+1
if Rt Re then
x = x/. R = R!
end if
end for return x;
23

---

E.2 ReNO hyperparameters
As detailed in Algorithm the main hyperparameters in ReNO are the learning rate μe, the regu- larization strength Aug and the choice for reward models, which we explore in Appendix C. We use Areg = 0.01 for all our experiments. For the learning rate, we use μ = 5 for all our 512 x 512 models and 10 for HyperSIXXI. that generates 1024 1024 as we found this to give good balance between exploration, improvements, and fast convergence. Note that in combination with gradient norm clipping. this also prevents major changes in the noise that would completely change the generated image. This effect can be observed in Figures and 6, and Appendix A, as the image after ReNO optimization still shares significant details with the initially generated image.
E.3 Models
SD-Turbo, SDXL-Turbo [81], and HyperSDXL [75] are built with a UNet [77] architecture similar to the one proposed in Rombach et al. [76]. On the other hand, PixArt-a DMD [12, 13] leverages a Diffusion Transformer [21, 59, 68] based architecture. We use the checkpoints of SD-Turbo, SDXL-Turbo, HyperSDXL., and PixArt-a DMD supplied through huggingface For HyperSDXI. we use the one-step UNet checkpoint (as opposed to the LoRA version).
E.4 Rewards
In this work, employ the four following reward models for ReNO.
E.4.1
Human Preference Score v2 (HPSv2)
HPSv2 [97] is an improved version of the HPS [98] model, which uses an OpenCLIP ViT-H/14 model and is trained on prompts collected from DiffusionDB [95] and other sources. Note that here we employ the further improved HPSv2. checkpoint.
E.4.2 PickScore
PickScore also uses the same ViT-H/14 model, however is trained on the Pick-a-Pic dataset which consists of 500k+ preferences that are collected through crowd-sourced prompts and comparisons.
E,4.3 ImageReward
ImageReward [100] trains a MLP over the features extracted from BLIP model [49]. This is trained on dataset of images collected from the DiffusionDB [95] prompts.
E.4.4 CLIPScore
Lastly, we use CLIPScore [35, 73], which was not designed specifically as a human preference reward model. However, it measures the text-image alignment with score between O and 1. Thus, it offers way of evaluating the prompt faithfulness of the generated image that can be optimized. We use the model provided by OpenCLIP [38] with ViT-H/14 backbone.
E.5 Metries
Apart from the user study (details in Appendix D) and the reward models themselves in Table 11, we benchmark ReNO with three different evaluation schemes as detailed in the following.
E.5.1
121-CompBench
T21-CompBench is comprehensive benchmark proposed by Park et al. [66] for evaluating the compositional capabilities of text-to-image generation models. The benchmark consists of three categories and six cut-categories of compositional text prompts: (1) Attribute binding, which includes color, shape, and texture sub-categories, where the model should bind the attributes with the correct objects to generate the complex scene; (2) Object relationships, which includes spatial and non- spatial relationship sub-categories, where the prompts contain least two objects with specified relationships; and (3) Complex compositions, where the prompts contain more than two objects
24

---

or more than two sub-cau categories. The attribute binding subtasks are evaluated using BLIP-VQA (i.e., generating questions based on the prompt and applying VQA on the generated image), spatial relationships are evaluated using an object detector, non-spatial relationships are evaluated through CLIPScore (CLIP ViT-B/32), and complex compositions are evaluated using all three models.
E.5.2 GenEval
GenEval is object-focused framework introduced by Ghosh et al. [28] for evaluating the alignment between text prompts and generated images from Text-to-Iman Image (T2I) models. Unlike holistic metrics such as FID or CLIPScore, GenEval leverages existing object detection methods to perform fine- grained, instance-level analysis of compositional capabilities. The framework assesses various aspects of image generation, including object co-occurrence, position, count, and color. By linking the object detection pipeline with other discriminative vision models, GenEval can further verify properties like object color. All the metrics on the GenEval benchmarks are evaluated using a MaskFormer object detection model with a Swin Transformer [58] backbone. Lastly, GenEval is evaluated over four seeds and reports the mean for each metric, which we follow.
E.5.3 LAION Aesthetic Score Predictor
Furthermore, we employ the improved LAION Aesthetic Predictor as an evaluation metric. It consists of an MLP trained top of CLIP [73] backbone. Importantly, this predictor does not take the prompt as joint input with the image. Thus, the aestheticness of an image is always evaluated independently of what prompt was used to generate it. This predictor can also be used as objective to improve the aesthetic quality of generated images, which we briefly investigated. We found that while numerically, the generated images achieve higher score, their actual visual quality does not seem to always be higher. We hypothesize that this is because the predictor is independent of the given prompt and thus might be more prone to reward-hacking.
E.6 Diversity Analysis
We generated images with 50 different seeds for prompts from each of the 11 challenges of PartiPrompts, totaling 110 prompts. Then, for each prompt, we evaluate the diversity over the 50 seeds by computing the mean pairwise LPIPS [106] and DINO [9, 64] scores. The higher these two scores are, the less diverse the generated images across seeds. We report the mean and standard deviation across all prompts.
E.7 Comparison DOODL
For our comparison to multi-step noise optimization DOODL, we use the first 50 prompts from each of the attribute binding categories of T2I-CompBench. We benchmark DOODL using the official codebase (https:// ///ggttuub cwwccoo//aaaaa/o homm/salesfrrs www.or/eee////oooeee.ee/ Ttto///lob/maiiii /doddl py), adapted to SD2.1 with 50 steps. We chose to focus on the SD2. model family because when running DOODL on SDXL, it exceeds 40GB of VRAM making it unfeasible for single GPU runs and thus inference.
E.8 FLUX-schnell results
We find that noises employed for FLUX-schnell with one step translate very well FLUX-schnell wtih four steps. Thus, due to efficiency we apply ReNO to FLUX-schnell with one step and afterward feed in the optimal noise to the four step FLUX-schnell model to obtain our final generation. Due to VRAM constraints, we generate samples in 512 x 512 including CPU-offloading such that FLUX- schnell ReNO runs within 40GB of VRAM. The FLUX-de results reported in Table 3 are taken from Liu et al. [55]. We report FLUX + ReNO results on the attribute binding categories of T2l-ChmpBenchh in Table 12 and qualitative comparison in Figure 12.
F
Broader Impact
Text-to-Image models have wide variety of uses in different settings. While they can be used for harmful purposes, practcal deployments of these models (including ours) must be made with safety checker/filter to prevent the generation of NSFW content. In our work, rely existing pretrained
25

---

Table 12: Comparison of 512 x 512 FLUX-schnell with and without ReNO on the attribute binding categories of T21-CompBench.
Attribute Binding
Color
Shape f
Texture f
FLUX-schnell
0.69
0.53 0.60
0.67 0.75
FLUX-schmeel + ReNO
0.80
"A Uspanese-style painting:
traditional wooden
orange zight black airplane"
green dog red cat"
"Two boxses and bridge, pagoda, three dogs"
lone samural Narrior, and cherzy blosson petals tranquil ziver"
FLUX-schnell (4 step)
ReNO
FLUX-dev
Figure 12: Non-cherry-picked results (seed for FLUX-schhell with and without ReNO compared to FLUX-dev.
models, and therefore would inhereit its biases. However, we believe that our reward optimization framework is flexible to also include safety and faimess and potential objectives which would be an option mitigate the harms of existing image generation models.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the method and experiments section justify all the claims made in the abstract and introduction. Additional clarifications are also provided in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: There is detailed subsection analyzing the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

There are new theoretical results that the paper provides. Existing theory has been concisely explained with all the assumptions.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper (along with the appendix) provides all the details needed to repro- duce the main experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the models and datasets we work with in the paper are all open-source. Our code will also be released upon acceptance of the work or when we make the work public, whicever is sooner.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. co/
mablic/guides /t/g///id/CoSSuui
Policy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctppblpcccccccccccccccccccccccc geiici/ggge/////ieeoeieiiicissee /ooe//ooieeeeseeeeeeeeeeee wwwssssssiiiooooeeoooooosssissii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, all experimental details, hyperparameters are clarified in the section experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The main results on T21- T21-Compbende averaged over 3 runs, GenEval over runs as specified by these benchmarks. The user study is performed over 1600 prompts and hundreds of users making it robust possible given the available resources. This follows standard practice in this topic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: There is detailed discussion this in Section 4, and 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/tpprr/////////////// cc/ http://errpppcccpplttcc//e////ee csGuidelines!
Answer:
[Yes]
Justification: Yes, the work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the broader impacts of the work are discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: There is discussion of the safeguards of the image generation models in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the existing models and datasets appropriately cited.
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.tcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets released in the paper apart from the code which will be publicly released and appropriately documented and licensed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The appendix contains all the details about the study, including details given to participants and compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to NeurIPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No potential risks were possible, the authors verified this beforehand. The only user study conducted in the paper did not require IRB approvals, since the participants provided informed consent.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "5t7r0V1TeG", "reviewer": "Reviewer_UuFQ", "summary": "The paper introduces Reward-based Noise Optimization (ReNO), a novel method to improve Text-to-Image (T2I) models by optimizing the initial noise during inference using human preference signals. This approach addresses the limitations of current fine-tuning methods, which often lead to \"reward hacking\" and poor generalization. By utilizing one-step diffusion models, ReNO enhances image quality and adherence to complex prompts without retraining, achieving significant performance improvements on benchmarks like T2I-CompBench and GenEval. Extensive user studies demonstrate that ReNO models are preferred nearly twice as often as popular models like SDXL, showcasing their efficiency and effectiveness in enhancing T2I model performance and user satisfaction.", "questions": "1. Reward model robustness:\n- How sensitive is ReNO to the choice of reward models? Have you observed any cases where optimizing for certain reward models leads to unexpected or undesirable results? This could help understand the robustness and potential limitations of the approach.\n\n2. Computational efficiency:\n- Could you provide more details on how ReNO's performance scales with the number of optimization steps and computational budget? Is there a clear point of diminishing returns?\n\n3. Diversity of outputs:\n- Does optimizing for reward models potentially reduce the diversity of generated images? Have you conducted any analysis on how ReNO affects the distribution of outputs compared to the base models?\n\n4. Integration with other techniques:\n- How might ReNO complement or interact with other techniques for improving text-to-image models, such as fine-tuning or prompt engineering? (Out of curiosity)", "limitations": "The authors acknowledge some limitations of their approach, particularly in the \"Limitations\" section. They mention:\n\n- Convergence of different models to similar performance levels, potentially due to limitations in reward models.\n- The increased VRAM requirements of their method.\n- Persistent challenges in generating humans, rendering text, and modeling complex compositional relations.\n- They briefly mention the possibility of hallucination in their method, which is a relevant concern for AI-generated content.\n\n\nI think they addressed the limitations adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Optimizing the initial noise input during inference to improve image quality and prompt fidelity, which is an innovative angle compared to typical model fine-tuning approaches.\n2. Conducts extensive experiments on multiple challenging benchmarks (T2I-CompBench, GenEval, Parti-Prompts) to evaluate the method.\n3. Compares against a wide range of baselines and state-of-the-art models, including proprietary ones like DALL-E 3 and Stable Diffusion. 4. Analyzes the impact of different reward models and optimization iterations.\n5. Clearly explains the motivation and approach of ReNO.\n6. Demonstrates competitive performance with proprietary models like SD3, despite using smaller open-source models as a base.\n7. Provides a practical method to enhance text-to-image models at inference time with reasonable computational cost (20-50 seconds per image).", "weaknesses": "1. Limited analysis of potential negative impacts or failure modes:\n\nThe paper does not thoroughly discuss potential downsides or risks of their approach. For example:\n\n- Could optimizing for reward models lead to unexpected or undesirable outputs in some cases?\n- Are there risks of amplifying biases present in the reward models?\n- Could this approach be misused to generate more convincing deepfakes or misleading images?\n\n2. Limited comparison to related optimization approaches:\n\nThe paper compares to some baseline models, but doesn't thoroughly compare to other test-time optimization methods for text-to-image models. Comparisons to approaches like:\n\n- DOODL (Kerras et al., 2022)\n- D-Flow (Ben-Hamu et al., 2023)\n\nWould help contextualize the novelty and advantages of ReNO.\n\n3. Insufficient analysis of impact on image diversity:\n- The paper doesn't thoroughly examine whether optimizing for rewards reduces the diversity of generated images. Some analysis of how ReNO affects the distribution of outputs would be valuable. (Specially theoretical)\n\nAside from the mentioned points, everything else was satisfactory, and I enjoyed the paper!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments. We are especially glad that they enjoyed the paper. Below, we address the concerns raised in the review. \n\n>  ***Limited comparison to related optimization approaches***\n\nWe address the comparison to DOODL in the global response. For D-Flow, the proposed method takes even longer (30-40 minutes for a 350M parameter model).\n\n> ***Insufficient analysis of impact on image diversity / Diversity of outputs***\n\nWe address this in the global response. Would you like us to analyze any other metrics to measure the distribution of generated images with ReNO compared to without ReNO?\n\n> ***Limited analysis of potential negative impacts or failure modes***\n\nReNO can successfully enhance the general capabilities of existing T2I models, including better prompt following and visual quality. As with all T2I models, this can be used for positive and negative applications. Additionally, since the objective is to optimize for the human preference reward models, the model is biased towards aspects that these models especially focus on. Reward models could also be prone to biases because of, e.g., their training data and, thus, this could be a potential risk in amplifying biases in existing models and a source of undesirable outputs. However, ReNO also opens up the possibility of including a bias-mitigating reward model.\n\n> ***Reward model robustness***\n\nWe would like to highlight Tables 1 & 7, where we benchmark the choice of different reward models. We did observe undesirable outputs **without** the noise regularization, as sometimes images with severe artifacts are generated, because of the noise going out of distribution, that still achieve a high reward score. Additionally, the reward models focus on some aspects more (e.g., colors, counting) than others (e.g., spatial understanding in the prompt). Specifically, we found that for a prompt including \"under/above\" or \"left/right\" the reward model scores can be very similar independent of which one is chosen for the same image.\n\n> ***Computational Efficiency***\n\nWe would like to highlight Figure 5, where we show the improvement in attribute binding on T2I-Compbench with increasing iterations. We see rapid improvements for the first 10-15 iterations (4-6 seconds for SD-Turbo on one A100), where it already surpasses popular open-source multi-step models like multi-step SDXL and PixArt-$\\alpha$. We support this with some qualitative examples in Figure 6. After 50 iterations (20 seconds), we see diminishing returns and see no major gains beyond 75 iterations (30 seconds).\n  \n> ***Integration with other Techniques:***\n\nOur experiments (Tables 2 & 3) indicate that we get better results with stronger base models. Therefore, fine-tuning base models to improve their performance or orthogonal techniques like prompt engineering should enhance the results further and be straightforward to incorporate together with ReNO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgFOqY8AOk", "reviewer": "Reviewer_8y7L", "summary": "The paper presents a novel approach called Reward-based Noise Optimization (ReNO) to enhance the performance of one-step Text-to-Image (T2I) models. ReNO optimizes the initial noise of T2I models using a human preference reward model, addressing the limitations of current T2I models in capturing complex details in compositional prompts. The method shows promising results across four different one-step models on T2I-CompBench and GenEval benchmarks, outperforming open-source T2I models and achieving comparable performance to a proprietary model. ReNO is computationally efficient and improves the quality of generated images, as demonstrated through user studies.", "questions": "Overall, ReNO presents a simple yet effective solution to enhance text-to-image diffusion models without the need for additional training. Generally, I have a positive outlook. Please refer to the Weaknesses section for a detailed list of questions and suggestions.", "limitations": "The authors provide Limitations and Broader Impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. It is reasonable to introduce a distilled one-step T2I model to address the notorious issue of exploding/vanishing gradients that exist in T2I diffusion models.\n2. ReNO improves the accuracy of T2I models in capturing intricate details in complex prompts. It consistently surpasses the performance of popular open-source T2I models.\n3. ReNO is applicable to existing models, avoiding the need for retraining from scratch. The approach is based on human preference, enhancing the alignment with desired outputs.", "weaknesses": "1. ReNO is essentially a runtime optimization approach that leverages advanced reward models to achieve state-of-the-art text-to-image generation capabilities. Indeed, the authors opt to optimize initial noise as a learnable parameter. Could I choose to make the parameters of a UNet learnable to achieve a similar objective?\n2. Have the authors attempted to apply ReNO to text-to-image diffusion models, such as SD, using techniques like gradient checkpoint and LoRA?\n3. Has the proposed ReNO been tested on video generation diffusion models for text-to-video tasks?\n4. The authors are encouraged to include the following references:\n   - Guided image synthesis via initial image editing in diffusion model, ACM MM 2023\n   - InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments. We are especially glad that they appreciated the simplicity and effectiveness of ReNO. Below, we address the concerns raised in the review. \n\n> ***ReNO is essentially a runtime optimization approach that leverages advanced reward models to achieve state-of-the-art text-to-image generation capabilities. Indeed, the authors opt to optimize initial noise as a learnable parameter. Could I choose to make the parameters of a UNet learnable to achieve a similar objective? Have the authors attempted to apply ReNO to text-to-image diffusion models, such as SD, using techniques like gradient checkpoint and LoRA?***\n\nYes, we also briefly tried optimizing the parameters of the U-Net with LoRA. This leads to the model generating images with visual artifacts which is caused due to \"reward-hacking\". In contrast, only optimizing the noise keeps the model untouched and, thus, does not lead to it generating adversarial images as long as the noise stays in distribution. Moreover, fine-tuning with reward models poses a number of other challenges. [lines 40-51] For example, the SDXL U-Net has 2.6B parameters, which makes it computationally infeasible to fine-tune at inference for every prompt. Even with LoRA and gradient checkpointing (as done by AlignProp[57]), this would still have 10M+  parameters to train (vs ~16k parameters for the noise optimization), which would take several minutes, if not hours (AlignProp fine-tunes SD1.5 on 4 GPUs for 24 hours).\n\n> ***Has the proposed ReNO been tested on video generation diffusion models for text-to-video tasks?***\n\nOur noise optimization framework is directly applicable to video generation. However, at the moment, while there are human preference reward models for video generation, there are no one-step video generation models publicly available, which are otherwise prohibitively expensive to train in a resource-constrained setting. Once open-source one-step video models are available, ReNO would be ideally suited to further enhance the performance of video generation.\n\n> ***The authors are encouraged to include the following references***\n\nWe thank the reviewer for pointing out these related works, especially the related concurrent InitNO, and we will add these to the paper. While InitNO also looks to optimize the initial noise, this is done by computing a loss function using attention maps as opposed to optimization with human preference reward objectives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DL2vWFooRu", "reviewer": "Reviewer_mwA2", "summary": "The paper introduces Reward-based Noise Optimization (ReNO), a novel approach to enhance Text-to-Image (T2I) models at inference by optimizing the initial noise based on human preference reward models. ReNO significantly improves model performance within a computational budget of 20-50 seconds, outperforming all current open-source T2I models and being preferred almost twice as often as the popular SDXL model in user studies. Additionally, ReNO-optimized models demonstrate superior efficiency, surpassing widely-used models like SDXL and PixArt-alpha with the same computational resources.", "questions": "if the author can significantly reduce the number of optimization iterations, it would be a good method, and the author can refer to some adversarial attack methods. I hope to see the author improve on optimizing time", "limitations": "The paper points out that the optimization time takes 20-50 seconds, The optimization time is too long and not easy to use", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A new approach that optimizes the initial noise in T2I models at inference time using gradient ascent, which enhances model performance significantly.\n\n2. outperforming all current open-source T2I models and being preferred almost twice as often as the popular SDXL model in user studies. Additionally, ReNO-optimized models demonstrate superior efficiency, surpassing widely-used models like SDXL and PixArt-alpha with the same computational resources.", "weaknesses": "Noise optimization is a bit like an \"adversarial attack\" that achieves its goal by adding noise to the original input, but the only drawback of the method is the time cost, as using gradient descent to obtain noise requires a lot of iterations. Therefore, optimizing time is a crucial factor that requires attention. The paper points out that the optimization time takes 20-50 seconds, and it says that one-step t2i is used. What if multi-step reasoning is directly used? Is the effect better than optimizing noise?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and are especially glad they emphasize the significant enhancement achieved by ReNO. Below, we address the concerns raised in the review. \n\n> ***Therefore, optimizing time is a crucial factor that requires attention. The paper points out that the optimization time takes 20-50 seconds, and it says that one-step t2i is used. What if multi-step reasoning is directly used? Is the effect better than optimizing noise?***\n\nWe would like to point out that ReNO actually provides an efficient formulation to enhance T2I models even compared to existing multi-step T2I models. For instance, in Figure 5, we show that ReNO outperforms existing open-source models such as multi-step SDXL and PixArt-$\\alpha$ with the same compute budget. Even with 10-15 iterations (4-10 seconds depending on the one-step model), we see significant improvements in prompt following and visual quality, as shown in Figure 5. Additionally, ReNO is much faster than other noise optimization methods, as mentioned in the global rebuttal.\n\n> ***Noise optimization is a bit like an \"adversarial attack\" that achieves its goal by adding noise to the original input***\n\nWhile we agree that there are connections to the literature on adversarial attacks, we would like to emphasize that the changes in the generated images are specifically not adversarial. We agree that leveraging different optimization techniques, e.g. from the adversarial attack literature, to reduce the number of iterations/time required for convergence could be an interesting future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NsE3VQAD5I", "reviewer": "Reviewer_EAgt", "summary": "In this work, the authors propose finetuning the noise that a one-step diffusion model predicts a clean image from, with respect to an ensemble of preference constraints.  Because only a one-step diffusion model is used, optimizing the noise is fast in terms of number of steps (and therefore wall clock time).  The authors demonstrate that the performance of this optimized one-step text-to-image model is comparable to diffusion models that utilize multiple levels of denoising on preference benchmarks.", "questions": "Are there ways to optimize multiple noises through ReNO to generate a batch of preference-respecting images without essentially optimizing each noise separately?\n\nWhat are the ways ReNO can enable or address the backpropagation through multiple denoising levels, and provide interesting implications for general multi-step diffusion models?  Are there ways this can be demonstrated in the rebuttal?", "limitations": "This reviewer foresees no substantial potential negative societal impact from this work.  However, in the Limitations section, the authors state that they hypothesize that the reward models may be limiting, and that stronger reward models and preference data may be crucial in enhancing results further.  This reviewer feels like this hypothesis could be directly tested within the scope of this work; building off of the initial results of Table 1, further study could be performed on utilizing subsets of the complete set of reward functions to evaluate the benefits of each.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The strength of this paper is that the approach is intuitive, and the success of the approach is reasonable.\n\n**Originality:** The paper does not appear to be very novel.  Optimizing the initial source noise has been explored before, and the authors are reusing existing preference classifiers/constraints.  The only ``novel\" component seems to be applying it to a one-step diffusion model, which may not actually be a strength (see weaknesses), which seems more as a simplified edge case, as well as utilizing multiple constraints in a weighted fashion which has limited originality.\n\n**Quality:** The paper quality is not particularly high.  The organization of the paper is rather messy.  The evaluations are also insufficient to demonstrate the benefits of their proposed approach.\n\n**Clarity:**  The clarity could be improved.  For example, Equation 7 was referenced before it was even written.  Furthermore, the background sections were structured strangely.  The section of \"Background: One-Step Diffusion Models\" spends most of its passage discussing regular Diffusion Models instead; one-step diffusion models were only mentioned in two sentences (lines 116-117). \n Furthermore, Section 2 - titled \"ReNO\", reads almost as a background/related works section (most clearly demonstrated in lines 118-125), but ReNO is only introduced two pages later (approx line 175).  Then, there is a separate Related Works section.  The paper could benefit from severe reorganization to improve its clarity.\n\n**Significance:** In this reviewer's perspective, this paper has limited significance.  The results seem expected, and do not provide deep insight (they essentially verify that optimization of the the initial noise helps performance), and the approach is only limited to one-step diffusion models and not the diffusion modeling in general.  Furthermore, the reliance on an ensemble means that this approach does not work on a singular specific preference of interest, but only on the average of multiple.", "weaknesses": "First and foremost, the comparisons showcased within this paper are flawed.  In Table 2 and Table 3, the authors compare their approach against the performance of default text-to-image models.  It should be almost expected that the performance of ReNO, by virtue of task-specific optimization, **should** have improved performance over default text-to-image models.  This is therefore not an interesting comparison; in fact, it even raises suspicions on the approach in the cases that it does not outperform default models (e.g. Attribute Binding in Dall-E 3).  The authors should instead compare their optimized approach against other optimized approach; for example, DOODL, or DOODL modified to be one-step, the approach from Samuel et al., etc.  This provides a clearer picture of ReNO as an optimization scheme compared to other optimization schemes in tackling preference-respecting generation.  The comparison to DPO is a nice small start, but we need more such comparisons - across benchmarks, and across more task/metrics.  It is really strange that the authors do not compare to the works they state are the most related to their approach.\n\nThe other weakness of this approach is that it is limited to one-step diffusion models.  As the authors state, \"backpropagating the gradient through multiple denoising steps can lead to exploding/vanishing gradients, rendering the optimization process unstable.\"  They instead limit themselves to using only \"a distilled one-step T2I model\".  The authors have not solved a fundamental problem in a general way, of how to optimize the noise for general text-to-image models; instead, they only demonstrate it for a single-step T2I model.  The impact of this work is therefore extremely limited; such insights can clearly not generalize to arbitrary diffusion models; and the authors **have not solved** a key issue of the approach of noise optimization but just ignored it entirely in favor of a more limited problem setting where the results they achieved is rather expected.\n\nThe method also seems to rely heavily on an ensemble.  However, this makes it rather ungeneralizable; under this approach it is not possible to optimize the text-to-image model with respect to one specific, particular preference.  Instead, the reliance on multiple reward functions simultaneously implicitly means that the resulting policy will balance between each of the functions used in the ensemble.  If there were a novel preference that it were important to optimize a text-to-image model to respect, this approach would not be able to be applied successfully.\n\nThis reviewer found that this work lacks severely in insight - indeed, many of the purported results are completely to be expected.  The authors repeatedly tout the fast training benefits of ReNO, but really this boils down to it being forced to work with a one-step diffusion model, which makes the fast optimization a rather expected result.  The computational cost was not improved in any way by the authors by their approach, it comes for free due to them restricting their **choice** of diffusion model to a one-step one.  Therefore, it is completely unsurprising that using a one-step diffusion model would result in faster optimization than multi-step ones; there is no new insight to be gained here.  Furthermore, the user study was conducted for SD-Turbo + ReNO against default models like SD-Turbo without any optimization; of course, there should be an improvement!  All these results have verified is that optimizing the noise helps, but this is to be expected - it is completely unsurprising.\n\nFurthermore, a severe weakness of ReNO is that once optimized, there is no diversity in the output - particularly because the approach is one-step.  For multi-step diffusion, even with the initial noise being optimized, the resulting output has variability because the other denoising steps have stochasticity (of resampling the Gaussian noise).  However, one-step diffusion has no diversity for a fixed (optimized) noise.  Even though it may be cheaper in wall-clock time to optimize one one-step image, if a batch of $n$ images were to be generated ReNO would need $n$ times the proposed time cost for one image.  On the other hand, Samuel et. al only optimize the initial generation point for multi-step diffusion, enabling a result that still has diversity while still respecting preference.  The speed is by default 1-4 minutes for Samuel et . al which is comparable or even preferable to ReNO, particularly because it can generate a batch all at once whereas ReNO needs to be reoptimized for each image which may take longer.  Also, Samuel et. al reports optimization improvements that reduce it from minutes to seconds, which makes it strictly better than RENO which takes seconds to just generate one image that has no diversity. Samuel et. al state in Section 6.1 that it takes 1-5 minutes to adapt to the new concept and 1-2 seconds to generate new semantically correct images.\n\nThe authors do not supply new criteria, nor do they innovate the approach of optimizing the noise.  Instead, they simply apply multiple existing criteria simultaneously.  Furthermore, the approach is limited to one-step diffusion models, which inhibits its generality.  Ultimately, the scope of this work feels more like a workshop paper rather than a conference-level paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to clarify the contributions of ReNO to ensure the problems ReNO is aiming to solve are understood. We tackle the question of whether we can **generally** enhance T2I models *without any fine-tuning* at test time. We propose to tackle this through the Noise Optimization framework by considering *human-preference reward models* as the optimization criterion. This is not a task-specific framework, nor are we proposing a new general method for noise optimization. Instead, we are specifically aiming to generally enhance a trained model at test time through our approach. This has not yet been considered or discussed in previous work, and how this generally performs was unclear. Additionally, it poses challenges that we propose to tackle through the use of one-step models (for computational efficiency), the use of multiple reward models (to prevent reward hacking), and by using noise regularization (for the noise to stay in distribution).\n\nWe thoroughly evaluate ReNO across four one-step T2I models (i.e SD-Turbo, SDXL-Turbo, Pixart-$\\alpha$-DMD, and HyperSDXL), across 3 benchmarks (T2I-Compbench, GenEval, Parti-Prompts), human preference evaluations (nearly 10k comparisons), and now also the diversity of the outputs. Our results show generally enhanced performance **without** optimizing for any specific task. For example, SD-Turbo is elevated to prompt-following levels close to DALLE-3 and is significantly preferred over 50-step SDXL, while HyperSDXL + ReNO is preferred over SD3 (8B). This substantial performance increase is a major contribution of ReNO, and we argue it is a **very significant** finding. It elucidates the importance of initially sampled noise and demonstrates its effective manipulation within a reasonable timeframe. We believe it is fascinating and unexpected that such a lightweight approach can significantly enhance the performance of image generation. This also differentiates ReNO from previous noise optimization work by showcasing the framework's power in a new, more general setting. Lastly, it motivates further research into understanding and controlling initial noise and provides a novel way of benchmarking future T2I reward models.\n\n> ***Comparison to SeedSelect (Samuel et al.)***\n\nSeedSelect tackles rare-concept generation, a fundamentally different problem from the one ReNO is solving. I.e. the goal of SeedSelect is, given 3-5 reference images, to generate images of rare concepts represented in these images. They propose to tackle this based on the noise optimization framework and reduce computational time, as mentioned in our related work section: \"To mitigate this, Samuel et al. [64] propose a bootstrap-based method to increase the efficiency of generating a batch of images. However, this method is limited to settings where the goal is to generate samples including a concept jointly represented by a set of input images\". On the other hand, we consider a more general setting, where each image is generated separately, given noise and text as input. Therefore, adapting SeedSelect to the general setting considered in our paper is not straightforward.\n\n> ***The authors demonstrate that the performance of this optimized one-step text-to-image model is comparable to diffusion models that utilize multiple levels of denoising on preference benchmarks.***\n\nWe disagree with this statement. We clearly show across multiple benchmarks and user studies that ReNO-optimized one-step models significantly outperform their corresponding multi-step models in all benchmarks and even their next-generation multi-step ones (See Tables 2, 3 & 5 and Figures 4 & 5).\n\n> ***Optimizing the initial source noise has been explored before, and the authors are reusing existing preference classifiers/constraint.***\n\nAs mentioned above, while we are repurposing existing reward models, as far as we are aware, the optimization criterion of human-preference reward models for noise optimization has not been considered or discussed before.\n\n> ***The results seem expected, and do not provide deep insight (they essentially verify that optimization of the the initial noise helps performance)***\n\nWe would argue that the magnitude of the performance increase makes it a **very significant** insight that is not expected at all. One-step models are known to have significantly worse visual quality and prompt following than the base multi-step models, therefore it is not expected that purely optimizing the initial noise would make them outperform even next generation multi-step models.  Further, increasing the diversity of the generated images compared to one-step models is also unexpected and a valuable finding as less diversity is also a common weakness of one-step models. \n\n> ***First and foremost, the comparisons showcased within this paper are flawed. In Table 2 and Table 3, the authors compare their approach against the performance of default text-to-image models. It should be almost expected that the performance of ReNO, by virtue of task-specific optimization, **should** have improved performance over default text-to-image models.***\n\nAs mentioned, we are not doing any task-specific optimization. We are proposing one optimization that **generally** improves a T2I model, and we thoroughly evaluate each ReNO-enhanced model over different benchmarks as well as human evaluation.\n\n> ***This is therefore not an interesting comparison; in fact, it even raises suspicions on the approach in the cases that it does not outperform default models (e.g. Attribute Binding in Dall-E 3).***\n\nWe respectfully disagree. Pushing SD-Turbo to a performance close to DALL-E 3 just by changing the initial noise is not expected and is actually a very interesting comparison as also acknowledged by Reviewer UuFQ.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their time and their detailed and insightful comments. We appreciate their recognition of the ReNO's significant enhancement in general performance (*UuFQ*,*8y7L*,*mwA2*), and its novelty (*UuFQ*, *mwA2*), clarity (*UuFQ*, *8y7L*), and practicality (*UuFQ*, *8y7L*). We want to highlight the new experiments we conducted and some of the joint questions here while additionally we address each raised issue with individual replies for each reviewer. We repeat the major contributions of ReNO here:

- Practical Noise Optimization for T2I Models: ReNO demonstrates that one-step models can make noise optimization a practical tool for generally enhancing Text-to-Image generation. It significantly improves efficiency in both time (20-50 seconds vs 20 minutes per image for previous noise optimization methods) and memory usage. This approach makes optimizing for human preference reward models during inference practically possible, substantially enhancing both prompt adherence and visual quality.
- Significantly Enhanced Performance: ReNO-enhanced one-step models achieve results that are on par with much larger, better-trained closed-source models while outperforming popular open-source models by large margins on standard benchmarks. Notably, given the same computational budget, one-step models with ReNO outperform widely used multi-step models, offering superior results with enhanced efficiency. This demonstrates the effectiveness of aligning T2I outputs with human preferences during inference, even with the constraints of a one-step model, and elucidates the power of noise optimization in a novel more general setting.

---

**Diversity evaluation.** We've conducted a diversity analysis in response to the reviewers' questions about ReNO's impact on diversity. We generated images with 50 different seeds for 10 prompts from each of the 11 challenges of PartiPrompts, totaling 110 prompts. Then, for each prompt, we evaluate the diversity over 50 seeds by computing the mean pairwise LPIPS and DINO score. The higher these two scores are, the less diverse the generated images across seeds. We report the mean and standard deviation across all prompts in the following table, as well as in the additional pdf.

|                   | LPIPS                | DINO                 |
| ----------------- | -------------------- | -------------------- |
| SD-Turbo          | 0.382 ± 0.043        | 0.770 ± 0.101        |
| SD-Turbo + ReNO   | *0.246* ± 0.046 | *0.712* ± 0.132 |
| SD2.1 (50-step)   | **0.243** ± 0.049    | **0.623** ± 0.150    |
| SDXL-Turbo        | 0.391 ± 0.044        | 0.835 ± 0.073        |
| SDXL-Turbo + ReNO | **0.291** ± 0.041    | *0.763* ± 0.116 |
| SDXL (50-step)    | *0.351* ± 0.042 | **0.700** ± 0.128    |

Remarkably, ReNO actually significantly increases the diversity of one-step models. As we believe this is a significant finding, we plan to include this in the main text of the paper. We also provide some non-cherry-picked results for the first 5 seeds in the pdf. 

We hypothesize that the reason for this increased diversity is that ReNO optimizes the noise away from the zero mean of the noise distribution, thus creating more diverse noises compared to sampling from the standard Gaussian. Even though we regularize the noise to stay in distribution, we do not enforce this. To validate this hypothesis, we compute the standard deviation across all noises before and after ReNO-optimization. As expected, the standard deviation for the initial noise across this sample size (110 prompts * 50 seeds) is *1.0000*. In contrast, the standard deviation of ReNO-optimized noise is *1.0039*, which confirms this hypothesis.

---

**Comparison to DOODL.** Due to the fact that DOODL takes more than 80 days on one A100 to evaluate on T2I-Compbench, we were unable to make comparisons to it on the standard  benchmarks. The following table illustrates this point:

|                 | sec/iter (total) | T2I-CompBench duration | VRAM | \#params |
| --------------- | ---------------- | ---------------------- | ---- | -------- |
| SD2.1 + DOODL (CLIP)   | 24s (20min)      | 83.33 A100 days        | 40GB | 860M     |
| SD-Turbo + ReNO (only CLIP) | 0.2s (10s)       | 0.63 A100 days         | 10GB | 860M  |
| SD-Turbo + ReNO | 0.4s (20s)       | 1.25 A100 days         | 15GB | 860M  |

Furthermore, when employing a larger model such as 50-step SDXL or multiple reward models, DOODL's VRAM requirement exceeds 40GB, making it infeasible even on A100 GPUs. Additionally, since DOODL optimizes multi-step models, it encounters problems with exploding gradients, which may compromise its noise optimization efficiency compared to ReNO, despite its extended runtime. To substantiate this, we evaluated DOODL on the first 50 prompts from T2I-CompBench's three attribute binding tasks. Our analysis includes both VQA evaluation results and changes in the optimized CLIPScore, effectively measuring the efficacy of the 50-step optimization process. We compare SD2.1 + DOODL (using CLIPScore) against SD-Turbo + ReNO (using CLIPScore) and ReNO with all considered reward models.

|Model|Color ↑|Shape ↑|Texture ↑|CLIPScore ↑|
|---|---|---|---|---|
|SD2.1|33.4|52.4|63.4|0.261|
|SD2.1 + DOODL (CLIP)|38.5 (+5.1)|51.6 (-0.8)|64.6 (+1.2)|0.289 (+0.03)|
|SD-Turbo|60.4|48.5|61.8|0.362|
|SD-Turbo + ReNO (only CLIP)|70.1 (*+9.7*)|66.9 (*+18.4*)|79.6 (*+18.2*)|0.483 (**+0.12**)|
|SD-Turbo + ReNO (all)|82.1 (**+21.7**)|77.4 (**+28.9**)|82.8 (**+21.0**)|0.437 (*+0.08*)|

We observe that ReNO achieves substantially higher gains compared to DOODL, both w.r.t. the CLIP loss that we optimize for and also the independent VQA evaluation. Note that the results from DOODL are in line with those reported in their paper, where they report increases in CLIPScore by 0.026 and 0.031. To provide a more comprehensive comparison between DOODL and ReNO, we plan to include this analysis and its results in the paper's Appendix.

Meta-review: **Summary:**

The paper introduces Reward-based Noise Optimization (ReNO), a method to improve Text-to-Image (T2I) models by optimizing initial noise during inference using human preferences. ReNO enhances image quality and prompt fidelity without retraining, showing significant improvements on benchmarks and user preference over models like SDXL.

**Strengths:**

- Novel approach of optimizing noise during inference for better image quality.
- Extensive experiments and comparisons with state-of-the-art models.
- Practical method with clear motivation and reasonable computational cost.

**Weaknesses:**

- Limited discussion on risks, such as bias amplification or unintended outputs. Insufficient analysis of ReNO’s impact on image diversity.
- Incomplete comparisons with other test-time optimization methods.

**Reason to Accept:**

The paper offers an innovative and effective method for improving T2I models, supported by strong experimental results, making it a valuable contribution despite some areas needing further analysis.
Decision: Accept (poster)