ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization

Authors: luca  eyring
shyamgopal  karthik
karsten  roth
alexey  dosovitskiy
zeynep  akata
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 09:06:43

Abstract: Text-to-Image (T2I) models have made significant advancements in recent years, but they still struggle to accurately capture intricate details specified in complex compositional prompts. While fine-tuning T2I models with reward objectives has shown promise, it suffers from "reward hacking" and may not generalize well to unseen prompt distributions. In this work, we propose Reward-based Noise Optimization (ReNO), a novel approach that enhances T2I models at inference by optimizing the initial noise based on the signal from one or multiple human preference reward models. Remarkably, solving this optimization problem with gradient ascent for 50 iterations yields impressive results on four different one-step models across two competitive benchmarks, T2I-CompBench and GenEval. Within a computational budget of 20-50 seconds, ReNO-enhanced one-step models consistently surpass the performance of all current open-source Text-to-Image models. Extensive user studies demonstrate that our model is preferred nearly twice as often compared to the popular SDXL model and is on par with the proprietary Stable Diffusion 3 with 8B parameters. Moreover, given the same computational resources, a ReNO-optimized one-step model outperforms widely-used open-source models such as SDXL and PixArt-alpha, highlighting the efficiency and effectiveness of ReNO in enhancing T2I model performance at inference time.

Main paper: 
 introduction: 
 Advancements in Text-to-Image (T2I) models have been achieved in recent years, largely due to the availability of massive image-text datasets [26, 82, 83] and the development of denoising diffusion models [19, 36, 76, 84] . Despite these improvements, T2I models often struggle to accurately capture the intricate details specified in complex compositional prompts [3, 37] . Common challenges include incorrect text rendering, difficulties with attribute binding, generation of unlikely object combinations, and color leakage. While recent models have begun to address these issues by employing enhanced language encoders, larger diffusion models, and better data curation [6, 12, 13, 22] , these approaches typically involve training larger models from scratch, making them inapplicable to existing models.As a more efficient alternative, fine-tuning T2I models has gained significant attention. This approach can be tailored either toward specific preferences [32, 78, 105] or general human preferences. Inspired by the success of Reinforcement Learning from Human Feedback (RLHF) [16, 29] in Large Language Models (LLMs), several works [11, 18, 23, 74, 109] propose aligning T2I models by fine-tuning them on human-preferred prompt-image sets using RLHF-inspired techniques. Additionally, human preference reward models, such as PickScore [46] , HPSv2 [97] , and ImageReward [100] , have gained popularity. These models are trained to output a score reflecting human preference for an image given a specific prompt, typically by measuring human preferences for various images generated from the same prompt. The scores predicted by these models have been utilized as evaluation metrics for the quality of generated images. Furthermore, Clark et al. [17] , Li et al. [51] , Prabhudesai et al. [72] directly fine-tune T2I models on these differentiable reward models to maximize the predicted reward of generated images. This approach is efficient due to the directly differentiable objective.Fine-tuning T2I models with reward objectives has a major drawback of "reward hacking", which occurs when a reward model gives a high score to an undesirable image. Reward hacking points to deficiencies in existing reward models, highlighting gaps between the desired behavior and the behavior implicitly captured by the reward model, which is especially prone to appear when explicitly fine-tuning for a reward [17, 20, 51] . Additionally, these models are often fine-tuned on a small scale (e.g., <50 prompts in some cases [8, 23] ) and thus may not generalize well to unseen prompt distributions with complex compositional structures. In this work, our aim is to enhance T2I models at inference for each unique generation, similar to the paradigm of test-time training for classification models [27, 88] . Fine-tuning diffusion models for every single prompt would both be computationally expensive (Dreambooth [78] takes 5 minutes on 1 A100), and susceptible to "reward-hacking".We sidestep the challenge of fine-tuning the model's parameters and instead explore optimizing the initial random noise during inference without adapting any of the model's parameters. To obtain more optimal noise and a higher-quality generated image, we introduce Reward-based Noise Optimization (ReNO), where the initial noise is updated based on the signal from a reward model evaluated on the generated image. The main challenges in this approach are twofold. First, backpropagating the gradient through the denoising steps can lead to exploding/vanishing gradients, rendering the optimization process unstable. Our insight is that by employing a distilled one-step T2I model [12, 75, 81, 102] , we circumvent the issue of exploding/vanishing gradients since backpropagation is performed through a single step. Second, naively optimizing the initial latent for an arbitrary objective can lead to collapse due to reward hacking. To mitigate this, we propose the use of a combination of reward objectives to not overfit to any single reward. Moreover, given a well-calibrated one-step T2I model with frozen parameters, the generated images should not exhibit reward hacking if the initial noise remains in the proximity of the initial noise distribution. Therefore, we propose an optimization scheme with limited steps, regularization of the noise to stay in-distribution, and gradient clipping.In essence, ReNO involves optimizing the initial latent noise given an one-step T2I model (e.g., SD/SDXL-Turbo) and a reward model (e.g., ImageReward) for a limited number of iterations (10-50 steps). On the popular evaluation benchmarks T2I-Compbench and GenEval, our noise optimization strategy (ReNO) significantly improves performance, increasing scores by over 20% in some cases. This enhancement allows SD2.1-Turbo models to approach the performance of closed-source proprietary models such as DALL-E 3 [6] and SD3 [22] . We demonstrate that ReNO substantially improves the performance of four different one-step T2I models (e.g. Figure 1 ), both in terms of quantitative evaluation and extensive user studies, while only requiring 20-50 seconds to generate an image. Moreover, given the same computational budget, ReNO surpasses the performance of competing multi-step models, offering an attractive trade-off between performance and inference speed. ReNO not only motivates the development of more robust reward models but also provides a compelling benchmark for their evaluation. Finally, our results highlight the importance of the noise distribution in T2I models and encourage further research into understanding and adapting it.2 Reward-based Noise Optimization (ReNO)Despite the remarkable progress in Text-to-Image (T2I) generation, current state-of-the-art models still struggle to consistently produce visually satisfactory images that fully adhere to the input prompt.Recent studies have highlighted the significant impact of the initial noise vector ε on the quality of the generated image [19, 85] . In fact, selecting and re-ranking images generated from a set of initial noises based on reward models has been shown to substantially improve performance [41, 46] . This observation naturally leads to the question of whether it is possible to identify an optimal noise vector that maximizes a given goodness measure for the generated image. In this section, we first provide an overview of one-step diffusion models, which serve as the foundation for our work. We then introduce our simple yet principled approach that enables practical noise optimization to enhance the performance of one-step T2I models based on human-preference reward models, addressing the challenge of generating high-quality images that align with the input prompt. 
background: one-step diffusion models: 
 T2I models aim to generate images x 0 conditioned on a given textual prompt p. A generative model G θ parameterized by θ takes as input a noise vector ε ∼ N (0, I) and a prompt p, and outputs an image G θ (ε, p) = x 0 . The objective is to learn the parameters θ, such that the generated image x 0 aligns with the semantics of the prompt p. This is typically achieved by training the model on a large dataset of paired text and images. Recent models are based on a time-dependent formulation between a standard Gaussian distribution ε ∼ N (0, I), and data x 0 ∼ p 0 (x). These models define a probability path between the initial noise distribution and the target data distribution, such thatEQUATIONwhere α t is a decreasing and σ t is an increasing function of t. Score-based diffusion [40, 45, 86] and flow matching [1, 54, 57] models share the observation that the process x t can be sampled dynamically using a stochastic or ordinary differential equation (SDE or ODE). Consider the forward SDE that transforms data into noise as t increases dx t = u(x t , t) dt + g(t) dw t , where u t (x t , t) denotes the drift, w t is a Wiener process and g(t) represents the diffusion schedule. Then, the marginal probability distribution p t (x) of x t in (1) coincides with the distribution of the probability flow ODE [45, 86] , as well as the reverse-time SDE [2] EQUATIONwhere s(x, t) = ∇ log p t (x) is the score function. By solving either the ODE or SDE backward in time from x T = ε ∼ N (0, I), we can generate samples from p 0 (x). This relies on a good estimate of the parameterized score s θ (x t , t). The choice of functions α t and σ t are defined implicitly based on the forward SDE [40, 45, 85, 87] . Furthermore, the process x t is considered on an interval [0, T ] with T sufficiently large such that x T approximates the initial noise distribution N (0, I). Then, it has been shown that the score can be approximated efficiently based on, e.g. the denoising loss [36] LEQUATIONDuring inference, these models simulate an ODE/SDE through discretization for a number of steps. This can be computationally expensive as the trained model must be evaluated sequentially. Given reward models based on human preferences, we optimize the initial latent noise to maximize the reward scores (consisting HPSv2 [97] , PickScore [46] , ImageReward [100] , and CLIP [73] ) for the images generated by the one-step T2I model. Over 50 iterations, the quality of the images and the prompt faithfulness are improved.Distillation. As a means to reduce inference time, distillation techniques have recently gained traction with the intent to learn a student model that approximates the solution of the simulated differential equation with a trained teacher model given fewer inference steps, e.g. score distillation [71] penalizes the estimated score to the real data distribution. Furthermore, several methods have been proposed to distill models into one-step generators, which learn to approximate the full ODE or SDE in one step. Our work builds upon the following one-step T2I models which we refer to as Gθ . Adversarial Diffusion Distillation (ADD) [81] combines score distillation with an adversarial loss and is employed to train SD-Turbo based on SD 2.1 [76] as a teacher and SDXL-Turbo [81] based on SDXL [69] . Diffusion Matching Distillation (DMD) [102] additionally leverages a distributional loss based on an approximated KL divergence and is applied for PixArt-α DMD [12, 13] . Lastly, Trajectory Segmented Consistency Distillation (TSCD) [75] introduces a progressive segment-wise consistency distillation [44, 87] loss to train HyperSDXL [75] with reward fine-tuning. All these models are trained in latent space such that during inference, an image is generated by first generating a sample in latent space and then decoding it G θ (ε, p) = D( Gθ (ε, p)) with a pre-trained decoder D.Given a Text-to-Image generative model G θ (ε, p) that generates images based on a noise ε and a prompt p, we defined the following optimization problem following previous work [5, 43, 80, 91] with the objective of optimizing the noise ε based on a criterion function C : R H×W ×c → R evaluated on the generated imageEQUATIONThen, given a differentiable C, (4) can be solved through iterative optimization via standard gradient ascent techniques. However, backpropagating through C(G θ (ε, p)) is non-trivial as current Text-to-Image models are based on the simulation of ODEs or SDEs (Section 2.1). Several methods have been proposed to enable backpropagation through time-dependent generative models [14, 17, 60, 91] , based on e.g., the adjoint method [70] . Our method, in contrast, leverages the crucial observation that selecting a one-step model as G θ enables efficient backpropagation through (4) . Although this realization may initially appear trivial, it proves to be a fundamental step in facilitating practical noise optimization in Text-to-Image models. Current methods require between 10 [91] and 40 [5] minutes to optimize noise and thus, to generate a single image. Our approach achieves image generation, including noise optimization, in 20-50 seconds, making it suitable for practical applications.Noise regularization. One important consideration, is that it is desirable for ε to stay within the proximity of the initial noise distribution N (0, I) as otherwise G θ might provide unwanted generations. This can be realized by including a regularization term inside of C. Samuel et al. [79] propose instead of directly optimizing the likelihood of p T (ε), to instead consider the likelihood of the norm of the noise r = ||ε||, which is distributed according to a χ d distribution p(r). Thus, following Ben-Hamu et al. [5] , Samuel et al. [80] we maximize the log-likelihood of the norm of a noise sample K(ε) = (d -1)log(||ε||) -||ε|| 2 /2. In our framework, this corresponds to employing a regularized criterion function given by C(x 0 , ε) = C(x 0 ) + K(ε), which can be plugged into (4).In Figure 3 , we provide an illustrative example where we chose the criterion to maximize x i,j,c 0EQUATIONwhere x i,j,c 0 denotes the channel c of the pixel at (i, j). Note that due to the calibration of the trained model and the noise staying indistribution, the noise does not collapse to the optimal ε ⋆ , which would result in the generation of a fully blue or red image. Also, the optimization first adapts the color of the car and then starts changing the background. Here, 10 optimization steps provide satisfactory results illustrating the efficacy of the proposed one-step noise optimization framework.Inspired by the success of Reinforcement Learning From Human Feedback [16, 29] in aligning LLMs with human preferences, similar methods have been explored for T2I generation. The underlying idea is to train a model R ψ that takes in an input along with the generated output (in this case a prompt and the corresponding image) and provides a score for the "goodness" of the generated output. Notable open-source human preference reward models for T2I include ImageReward [100] based on BLIP [49] and human preferences collected for the DiffusionDB dataset, PickScore [46] , and HPSv2 [97] both based on a CLIP [73] ViT-H/14 backbone. These reward models provide a quantitative measure of the image's quality and relevance to the prompt through a prediction by a differentiable neural network. Thus, they have not only been employed for the evaluation of T2I models but also to fine-tune them [17, 18, 100] as a means of achieving higher reward scores. Lastly, CLIPScore [35] has also been leveraged to measure the prompt alignment of a generated image.To generally enhance the performance of Text-to-Image models without any fine-tuning, we propose to leverage a Reward-based criterion function C for Noise Optimization (ReNO). Specifically, we propose to use a weighted combination of a number n of pre-trained reward models R 0 ψ , . . . R n ψ as the criterion functionEQUATIONwhere λ i denotes the weighting for reward model R i ψ . Employing a combination of reward models can help prevent "reward-hacking" and allow capturing various aspects of image quality and prompt adherence, as different reward models are trained on different prompt and preference sets. This not only effectively combines the strengths of multiple reward models, but also helps mitigate their weaknesses. ReNO then boils down to iteratively solving (4) with gradient ascentEQUATION)where η is the learning rate. Similar to the color example in Figure 3 , it is actually not desirable to find the optimal ε ⋆ as we want to prevent adversarial samples that exploit the reward models. We find that already a few optimization steps (<50) of ReNO lead to significant improvements in both prompt following and visual aesthetics, striking a good balance between reward optimization and the prevention of reward hacking. Due to the efficacy of the proposed framework, generating one image, including noise optimization, takes between 20-50 seconds, depending on the model and image size, enabling its practical use. We provide a sketch of ReNO in Figure 2 and full details in Algorithm 1.As detailed in Algorithm 1 the main hyperparameters in ReNO are the learning rate µ, the regularization strength λ reg and the choice for reward models, which we explore in Appendix C. We use λ reg = 0.01 for all our experiments. For the learning rate, we use µ = 5 for all our 512 × 512 models and µ = 10 for HyperSDXL that generates 1024 × 1024 as we found this to give a good balance between exploration, improvements, and fast convergence. Note that in combination with gradient norm clipping, this also prevents major changes in the noise that would completely change the generated image. This effect can be observed in Figures 1 and 6 , and Appendix A, as the image after ReNO optimization still shares significant details with the initially generated image. 
related work: 
 Initial Noise Optimization. The initial noise optimization framework was first introduced in DOODL [91] for improved guidance in Text-to-Image models. Subsequently, it was leveraged by Karunratanakul et al. [43] for 3D universal motion priors, for rare-concept generation [61, 79, 80] and enhancing image quality [31, 89] in text-to-image models, music generation [62, 63] , and by D-Flow [5] for solving inverse problems in various settings. While these methods mainly focus on controlling the generated sample for specific applications, our proposed method is designed to generally improve Text-to-Image models without the need for additional techniques to mitigate exploding or vanishing gradients on the optimization process. Most related to our work is DOODL [91] , which also proposes to improve the textual alignment of text-to-image models using a CLIP-score-based criterion function, which we similarly employ in our method. These existing methods, however, take 10 (DOODL) to 40 (D-Flow) minutes to generate a single image due to their application on time-dependent generative models with a large number of denoising steps. To mitigate this, Samuel et al. [80] propose a bootstrap-based method to increase the efficiency of generating a batch of images. However, this method is limited to settings where the goal is to generate samples including a concept jointly represented by a set of input images.Reward Optimization for Text-to-Image Models. Reward models [46, 47, 97, 98, 100] were first introduced to mimic human preferences given an input prompt and generated images. There have been several attempts at incorporating these signals to enhance text-to-image generation. One notable direction is the idea of using reinforcement learning based algorithms to fine-tune text-to-image models to better align with these rewards either with an explicit reward model [8, 11, 18, 23, 30, 109] or by bypassing it entirely with Direct Preference Optimization [50, 74, 92, 101] . However, this can be expensive, requiring thousands of queries to generalize, and therefore a lot of work has explored directly fine-tuning diffusion models [17, 51, 72] using differentiable rewards [39, 46, 47, 98, 100] .Additionally, there has also been works exploring the concept of using reward models to perform classifier-guidance [4, 34] as well as using rewards to distill diffusion models into fewer steps [48, 75] .Differently from these works, we focus on adapting a diffusion model during inference by purely optimizing the initial latent noise using a differentiable objective. 
experiments: 
 Experimental Setup. We evaluate the effectiveness of our proposed method, ReNO, using four opensource one-step image generation models: SD-Turbo, SDXL-Turbo, PixArt-α DMD, FLUX-schnell and HyperSDXL. HyperSDXL generates images of size 1024 × 1024 while the others generate 512 × 512. To assess the performance across diverse scenarios, we consider three challenging tasks. First, we evaluate on T2I-CompBench [37] , which comprises 6000 compositional prompts spanning six categories, using a VQA, object detection, and image-text matching scores. Second, we employ GenEval [28] , consisting of 552 object-focused prompts, measuring the quality of the generated images using a pre-trained object detector. Finally, we utilize Parti-Prompts [103] , a collection of more than 1600 complex prompts, and assess the generated images using both reward-based metrics and extensive human evaluation. Throughout all experiments, we optimize Equation ( 7) for 50 steps using gradient ascent with Nesterov momentum and gradient norm clipping for stability. Lastly, we select the image with the highest reward score from the optimization trajectory for evaluation.Table 1 : SD-Turbo evaluated on the attribute binding categories of T2I-CompBench and the LAION aesthetic score predictor [83] for different reward models. We analyze the effect of various reward models in Table 1 . We see that optimizing ImageReward or CLIPScore alone improves the text-image faithfulness (i.e., attribute binding from T2I-Compbench). However, this comes at the cost of decreased aesthetic score. PickScore and HPSv2 improve the image quality, however the gains in faithfulness are modest. Combining all the rewards leads to having strong improvements in faithfulness, while [12, 13, 102] , (2) SD-Turbo [81] , (3) SDXL-Turbo [81] , (4) HyperSD [75] demonstrates superior compositional generation ability in both attribute binding, object relationships, and complex compositions. The best value is bolded, and the second-best value is underlined. Multi-step results taken from [13, 22] . (2) SD-Turbo [81] , (3) SDXL-Turbo [81] , (4) HyperSDXL [75] improves results across all categories. The best value is bolded, and the second-best value is underlined. Multi-step results taken from [22] . also increasing the image quality. Thus, we employ ReNO with all four reward models. We report further details in Appendix C, including the performance of all combinations of reward models.Table 2 presents the quantitative results of ReNO on T2I-Compbench. Most notably, we observe that for both Pixart-α DMD and SD-Turbo, there are improvements of over 20% in the Color, Shape, and Texture Categories. For instance, on Color SD-Turbo improves from 55% to 78%, which is only slightly below DALL-E 3. Similar improvements can also be seen for SDXL-Turbo and HyperSDXL models where performance increases by over 10 percentage points in these categories. Even outside this, there are significant boosts in the Spatial, Non-Spatial, and Complex categories, highlighting both the efficacy of the noise optimization framework, as well as the utility of human preference models for improving T2I generation at inference. Similar trends can also be noticed for GenEval in Table 3 , where applying our noise optimization framework helps improve the performance of various one-step diffusion models. For instance, SD-Turbo improves its mean score from 0. To further validate ReNO we perform a user study on the commonly used Parti-Prompts [103] with Amazon Mechanical Turk (AMT). Parti-Prompts generally includes longer complex prompts that test artistic generation capabilities as opposed to T2I-Compbench and GenEval, which purely focus on faithfulness. We conducted user studies with ReNO applied to SD-Turbo for 512 × 512 and HyperSDXL for 1024 × 1024 generation. We compare SD-Turbo + ReNO against SD-Turbo, SDXL-Turbo, SD2.1 (50 Steps), and SDXL-Base (50 Steps). The results in Figure 4 confirm our findings in the quantitative evaluation. SD-Turbo + ReNO has an above 60% win rate against all benchmarked models reaching up to 77% against the SD-Turbo base. To contextualize these results, SD3 [22] conducts a similar user study on Parti-Prompts and reports a 70% win rate against SDXL (50 steps). Our strongest base model, HyperSDXL, already beats SDXL (50 steps) [75] without ReNO. Thus, we compare it with and without ReNO as well as against the proprietary SD3 (8B) [22] . Again, HyperSDXL + ReNO achieves an above 60% win rate, and notably, it also narrowly beats SD3 with 54%. This confirms our finding in ReNO, which substantially improves overall generative quality, pushing results at least close to the ones of even current state-of-the-art proprietary models. Lastly, we note that user studies on AMT can potentially be noisy and, therefore, view the results holistically along with quantitative evaluation. We provide a detailed breakdown of the preference for image quality and faithfulness, as well as full details of the user study in Appendix D. The primary concern of our proposed method is the increased inference cost since existing methods (e.g. DOODL, D-Flow) are impractical for regular T2I generation usage. However, we circumvent this issue through our restriction to one-step models and 50 optimization steps, which makes ReNO run in 20-50 seconds. To analyze the performance of ReNO with respect to the number of optimization steps we evaluate its performance over a set of reference points. We report results on the attribute binding part of T2I-CompBench for SD-Turbo + ReNO in Figure 5 and visually corroborate these results with Figure 6 . Note that even when restricted to the same compute budget as SDXL (50 steps, ~7sec), SD-Turbo + ReNO significantly outperforms it while in this comparison PixArt-α (20 steps, ~7sec) lies shortly below the Pareto-frontier of ReNO. To investigate the effect of noise optimization on output diversity, we evaluate images generated across 50 different random seeds for 110 prompts from Parti-Prompts. Specifically, we generate a batch of images and use LPIPS [106] and DINO [9, 64] scores to compute the average similarity of the generated batch, where a lower similarity score corresponds to higher diversity. As shown in Table 4 , one-step models (SD-Turbo, SDXL-Turbo) exhibit lower diversity compared to their multi-step counterparts (SD2.1, SDXL), likely due to adversarial training. However, applying ReNO not only maintains but actually increases diversity. For both SD-Turbo and SDXL-Turbo, ReNO achieves diversity levels approaching their respective multi-step base models highlighting an unexpected benefit of noise optimization increasing diversity. Figure 10 illustrates these improvements qualitatively. We hypothesize that the reason for this increased diversity is that ReNO adds structure to the noise, thus optimizes it away from the zero mean of the noise distribution and creating more diverse noises compared to sampling from the standard Gaussian.We benchmark ReNO against DOODL [91] , which performs noise optimization using the 50-step SD2.1 model. Due to DOODL's computational demands, we evaluate on the first 50 prompts from T2I-CompBench. Despite using the same CLIPScore objective, ReNO achieves four times larger improvements in the optimized criterion while requiring 75% less GPU memory and running 100x faster, highlighting the effectiveness of our one-step approach. Moreover, ReNO's multi-reward objective leads to substantially larger gains in attribute binding accuracy (21.0-28.9%) compared to solely using CLIPScore, reiterating the efficacy of ReNO's optimization objective. An interesting observation in our experiments is that despite using different image generation models of varying architectures and sizes, they broadly converge to similar performance on both T2I-Compbench and GenEval. In addition to the limitations of the generative models, we hypothesize that this could be due to the limitations of the reward models themselves, given their limited compositional reasoning abilities [104] . Stronger reward models [53, 107] and preference data [15, 33, 42, 99, 108] would be crucial in enhancing results further. Secondly, not only the runtime but also the amount of needed GPU VRAM is significantly higher when using ReNO. We reduce it by leveraging fp16 quantization and the pytorch [67] memory reduction technique introduced in Bhatia and Dangel [7] , which for ReNO lowers the VRAM by another ~15%. Then, all of the models can be optimized on a single A100 GPU in 20-50 seconds, and e.g., SD-Turbo requires only 15GB VRAM for the entire optimization process. Note, however, that the amount of VRAM also scales with the size of the generated image. Thus, HyperS-DXL needs 39GB of VRAM. We provide a summary of the computational cost of ReNO in Table 6 , which lays out ReNO's main limitation. Finally, current T2I models struggle with generating humans, rendering text, and also modeling complex compositional relations. While our work attempts to alleviate these issues and provides a flexible framework for further improvements, future work is required to resolve these issues. 
conclusion: 
 We introduce ReNO, a test-time optimization strategy for enhancing text-to-image generation without any fine-tuning. Not only do we achieve the strongest results among all open-source models on T2I-Compbench and GenEval, but images from ReNO on a single-step SD-Turbo have over a 60% win rate against a 50-step SDXL model and is competitive with the 8B parameter SD3 model on user studies. We also demonstrate that ReNO outperforms SDXL even when restricted to the same computational budget, highlighting the benefits of ReNO for practical use cases. The performance gains from ReNO underscore the importance of developing even better and more robust reward models and, moreover, establish a valuable benchmark for assessing their effectiveness. Furthermore, the substantial impact of optimizing the initial noise distribution motivates further research into understanding, manipulating, and controlling this crucial aspect of generative models.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: There is a discussion of the safeguards of the image generation models in the appendix.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All the existing models and datasets are appropriately cited.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: The appendix contains all the details about the user study, including details given to participants and compensation.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No potential risks were possible, the authors verified this beforehand. The only user study conducted in the paper did not require IRB approvals, since the participants provided informed consent.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: "A van parked on grass""A photo of a light bulb in outer space traveling the galaxy with a sailing boat inside the light bulb." + ReNODirect Preference Optimization has recently been applied in the context of Diffusion models [8, 50, 101] . Here, we compare against an SDXL model that has been preference-tuned on a dataset of over 800k preferences in Table 7 . We see that while DPO improves both attribute binding and the aesthetic score of the generated images, it underperforms the SDXL-Turbo with ReNO. This highlights the potential of test-time/online optimization compared to traditional fine-tuning, since it can generalize much better to unseen prompt distributions. We show the results for several methods that have been tailored to compositional Text-to-Image generation in Table 8 . These methods either explicitly finetune the model for improved compositional generation, or modify the inference process, or repeat the sampling over multiple iterations. We see that ReNO consistently outperforms specific methods tailored for this task.We also note that some methods use LLMs and other tools (image-editing, customization etc.) to plan out or correct generations [25, 52, 94] . However, these methods significantly impact the generation process through, e.g., iterative generation and planning. In contrast, ReNO only changes the initial noise and doesn't alter the generative model at all. Thus, our method could also be incorporated into these tools to further improve performance. Comparison to Multi-step Noise Optimization Methods. In Section 4.6, we report quantitative comparison between the multi-step noise optimization method DOODL and ReNO. In Table 9 , we additionally report more details on the difference in efficiency between DOODL and ReNO. Note that for the same objective and model family ReNO is 120x faster compared to DOODL. We show all results for all combinations of the reward models in Table 10 . Broadly, adding all reward models ensures that a meaningful improvement is achieved both on attribute binding and on the aesthetic score. In addition to this, we perform a leave-one-out analysis on Parti-Prompts in Table 11 , where one reward is excluded from ReNO and subsequently analyzed.Even when a particular reward is not optimized for, we see that there is a consistent improvement in the metrics, and in most cases, at least 80% of the images improve even on the left-out reward. This phenomenon across a variety of models (e.g. CLIP, BLIP) trained on differing datasets certainly indicates that there are significant improvements made by ReNO across most of the images. While the reward increase and the percental improvement, can differ based on the one-step model, CLIPScore and ImageReward seem to be less correlated to the other rewards, which could be explained based on the similar backbone employed by HPSv2 and PickScore. Interestingly, PixArt-α DMD achieves the highest reward scores after optimization, which does not follow the quantitative results for T2I-CompBench and GenEval as reported in Section 4.2. Reward Weighting. The four reward models that we employ output scores in different ranges. Specifically, HPSv2 mostly ranges between 0.2-0.4, while PickScore is in the range of 20-30 for most of the images. ImageReward is in the range of -2 to +2 for the majority of images, and CLIPScore is between 0 and 1. For all our experiments, we use weights of 1.0 for ImageReward, 5.0 for HPSv2, 0.05 for PickScore, and 1.0 for CLIPScore. When each score range is scaled to [0, 1], then these weights correspond to 4.0 for ImageReward, 1.0 for HPSv2, 0.5 for PickScore, and 1.0 for CLIPScore. These weights ensure that the losses from each reward model are roughly similar, with a higher emphasis on ImageReward. We perform our user study on Amazon Mechanical Turk, and pay participants based on prior guidelines [65] , which also ensures the compensation is above the minimum wage. We use pairwise preferences due to its simplicity, allowing users to mark ties between images that are equally good/bad. Each pairwise comparison is treated as an individual entity and handed to an individual user to minimize user biases. In particular, each pairwise comparison between the two models has involved at least 339 unique users (and 673 maximal), with the average being 495. To reduce the number of user comparisons, we perform the user study on a subset of Parti-Prompts totaling slightly above 1000 prompts (excluding challenges ['Basic', 'Imagination', 'Perspective', 'Linguistic Structures'] and categories ['Abstract', 'Indoor Scenes', 'Produce & Plants']).We ask users to answer the following three questions:• On personal preference: "Which image would you personally prefer getting given the input text (based on your personal tradeoff between faithfulness and aesthetics)?" • On aestheticness: "Which image do you find more aesthetically pleasing? • On faithfulness: "Which image is more faithful to the input text?"We also provide additional information on terminology:• Faithfulness: The generated image should reflect all key concepts, their relations and their attributes given in the text prompt. • Aestheticness: Refers to the style, coloring and interpretation in the depiction of concepts (i.e. "looks better"). • Personal preference: Some generations can be more faithful, but less aesthetic, or the other way around. Choose which you prefer :). For the competing methods, we use default parameters: one-step generation without classifier-free guidance (CFG), and CFG = 7.5 for SD2.1 and CFG = 5.0 for SDXL. For the proprietary SD3, we generate images through the API provided at https://platform.stability.ai/. Note that to compare SDXL with SD-Turbo + ReNO, we generate images in 1024 × 1024 for SDXL as this is its native resolution and then afterward downsize them to 512 × 512.In Figure 11 , we report the results for the specific questions on faithfulness and aestheticness.Interestingly, for all models, the preference for aestheticness is even larger than that of faithfulness. While the quantitative results on T2I-CompBench and GenEval reported in Section 4.2 mainly benchmark the prompt following improvements of ReNO, this result confirms ReNO's benefits in improving the general quality of generated images.Our code is built with Pytorch [67] and is mainly based on the diffusers library [90] . It is available at https://github.com/ExplainableML/ReNO.We outline the overall algorithm for ReNO in Algorithm 1. Note that, we choose gradient ascent with Nesterov momentum as we found this for our computational budget to yield the best results. Although line-search-based methods such as L-BFGS are viable options [5] , we find that even without them gradient ascent provides efficient and effective optimization of the criterion function. However, L-BFGS or gradient ascent without momentum might also be viable optimization methods for ReNO.Input: p (prompt), G θ (One-Step T2I Model), R 0,1...n ψ (Reward Functions), λ 0,1...n (Reward Weights), m (# Optimization Steps), η (Learning Rate), λ reg (Regularization Strength) Initialize v -1 = 0.0, ε 0 = N (0, I), R ⋆ = -inf. for t = 0 to m do Generate image x t 0 = G θ (ε t , p) Compute reward-based criterion R t = n i λ i R i ψ (x t 0 , p) grad t = ∇ ε t [λ reg K(ε t ) + R t ] grad t = GradNormClip(grad t , 0.1) v t = 0.9 • v t-1 + η • grad t ε t+1 = ε t + v t if R t > R ⋆ then x ⋆ 0 = x t 0 , R ⋆ = R t end if end for return x ⋆ 0 E.SD-Turbo, SDXL-Turbo [81] , and HyperSDXL [75] are built with a UNet [77] architecture similar to the one proposed in Rombach et al. [76] . On the other hand, PixArt-α DMD [12, 13] leverages a Diffusion Transformer [21, 59, 68] based architecture. We use the checkpoints of SD-Turbo, SDXL-Turbo, HyperSDXL, and PixArt-α DMD supplied through huggingface. For HyperSDXL, we use the one-step UNet checkpoint (as opposed to the LoRA version).In this work, we employ the four following reward models for ReNO.HPSv2 [97] is an improved version of the HPS [98] model, which uses an OpenCLIP ViT-H/14 model and is trained on prompts collected from DiffusionDB [95] and other sources. Note that here we employ the further improved HPSv2.1 checkpoint.PickScore also uses the same ViT-H/14 model, however is trained on the Pick-a-Pic dataset which consists of 500k+ preferences that are collected through crowd-sourced prompts and comparisons.ImageReward [100] trains a MLP over the features extracted from a BLIP model [49] . This is trained on a dataset of images collected from the DiffusionDB [95] prompts.Lastly, we use CLIPScore [35, 73] , which was not designed specifically as a human preference reward model. However, it measures the text-image alignment with a score between 0 and 1. Thus, it offers a way of evaluating the prompt faithfulness of the generated image that can be optimized. We use the model provided by OpenCLIP [38] with a ViT-H/14 backbone.Apart from the user study (details in Appendix D) and the reward models themselves in Table 11 , we benchmark ReNO with three different evaluation schemes as detailed in the following.T2I-CompBench is a comprehensive benchmark proposed by Park et al. [66] for evaluating the compositional capabilities of text-to-image generation models. The benchmark consists of three categories and six sub-categories of compositional text prompts: (1) Attribute binding, which includes color, shape, and texture sub-categories, where the model should bind the attributes with the correct objects to generate the complex scene; (2) Object relationships, which includes spatial and nonspatial relationship sub-categories, where the prompts contain at least two objects with specified relationships; and (3) Complex compositions, where the prompts contain more than two objects or more than two sub-categories. The attribute binding subtasks are evaluated using BLIP-VQA (i.e., generating questions based on the prompt and applying VQA on the generated image), spatial relationships are evaluated using an object detector, non-spatial relationships are evaluated through CLIPScore (CLIP ViT-B/32), and complex compositions are evaluated using all three models.GenEval is an object-focused framework introduced by Ghosh et al. [28] for evaluating the alignment between text prompts and generated images from Text-to-Image (T2I) models. Unlike holistic metrics such as FID or CLIPScore, GenEval leverages existing object detection methods to perform a finegrained, instance-level analysis of compositional capabilities. The framework assesses various aspects of image generation, including object co-occurrence, position, count, and color. By linking the object detection pipeline with other discriminative vision models, GenEval can further verify properties like object color. All the metrics on the GenEval benchmarks are evaluated using a MaskFormer object detection model with a Swin Transformer [58] backbone. Lastly, GenEval is evaluated over four seeds and reports the mean for each metric, which we follow.Furthermore, we employ the improved LAION Aesthetic Predictor as an evaluation metric. It consists of an MLP trained on top of a CLIP [73] backbone. Importantly, this predictor does not take the prompt as a joint input with the image. Thus, the aestheticness of an image is always evaluated independently of what prompt was used to generate it. This predictor can also be used as an objective to improve the aesthetic quality of generated images, which we briefly investigated. We found that while numerically, the generated images achieve a higher score, their actual visual quality does not seem to always be higher. We hypothesize that this is because the predictor is independent of the given prompt and thus might be more prone to reward-hacking.We generated images with 50 different seeds for 10 prompts from each of the 11 challenges of PartiPrompts, totaling 110 prompts. Then, for each prompt, we evaluate the diversity over the 50 seeds by computing the mean pairwise LPIPS [106] and DINO [9, 64] scores. The higher these two scores are, the less diverse the generated images across seeds. We report the mean and standard deviation across all prompts.For our comparison to multi-step noise optimization DOODL, we use the first 50 prompts from each of the attribute binding categories of T2I-CompBench. We benchmark DOODL using the official codebase (https://github.com/salesforce/DOODL/blob/main/doodl.py), adapted to SD2.1 with 50 steps. We chose to focus on the SD2.1 model family because when running DOODL on SDXL, it exceeds 40GB of VRAM making it unfeasible for single GPU runs and thus inference.We find that noises employed for FLUX-schnell with one step translate very well to FLUX-schnell wtih four steps. Thus, due to efficiency we apply ReNO to FLUX-schnell with one step and afterward feed in the optimal noise to the four step FLUX-schnell model to obtain our final generation. Due to VRAM constraints, we generate samples in 512 × 512 including CPU-offloading such that FLUXschnell + ReNO runs within 40GB of VRAM. The FLUX-dev results reported in Table 3 are taken from Liu et al. [55] . We report FLUX-schnell + ReNO results on the attribute binding categories of T2I-CompBench in Table 12 and a qualitative comparison in Figure 12 .Text-to-Image models have a wide variety of uses in different settings. While they can be used for harmful purposes, practcal deployments of these models (including ours) must be made with a safety checker/filter to prevent the generation of NSFW content. In our work, we rely on existing pretrained models, and therefore would inhereit its biases. However, we believe that our reward optimization framework is flexible to also include safety and fairness and potential objectives which would be an option to mitigate the harms of existing image generation models.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All the models and datasets we work with in the paper are all open-source. Our code will also be released upon acceptance of the work or when we make the work public, whicever is sooner. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Justification: There are no new assets released in the paper apart from the code which will be publicly released and appropriately documented and licensed.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 


Decision: **Summary:**

The paper introduces Reward-based Noise Optimization (ReNO), a method to improve Text-to-Image (T2I) models by optimizing initial noise during inference using human preferences. ReNO enhances image quality and prompt fidelity without retraining, showing significant improvements on benchmarks and user preference over models like SDXL.

**Strengths:**

- Novel approach of optimizing noise during inference for better image quality.
- Extensive experiments and comparisons with state-of-the-art models.
- Practical method with clear motivation and reasonable computational cost.

**Weaknesses:**

- Limited discussion on risks, such as bias amplification or unintended outputs. Insufficient analysis of ReNO’s impact on image diversity.
- Incomplete comparisons with other test-time optimization methods.

**Reason to Accept:**

The paper offers an innovative and effective method for improving T2I models, supported by strong experimental results, making it a valuable contribution despite some areas needing further analysis.

Reviews: review: paper_summary: paper_summary: The paper introduces Reward-based Noise Optimization (ReNO), a novel method to improve Text-to-Image (T2I) models by optimizing the initial noise during inference using human preference signals. This approach addresses the limitations of current fine-tuning methods, which often lead to "reward hacking" and poor generalization. By utilizing one-step diffusion models, ReNO enhances image quality and adherence to complex prompts without retraining, achieving significant performance improvements on benchmarks like T2I-CompBench and GenEval. Extensive user studies demonstrate that ReNO models are preferred nearly twice as often as popular models like SDXL, showcasing their efficiency and effectiveness in enhancing T2I model performance and user satisfaction.
strength_weakness: strength_weakness: 1. Limited analysis of potential negative impacts or failure modes:

The paper does not thoroughly discuss potential downsides or risks of their approach. For example:

- Could optimizing for reward models lead to unexpected or undesirable outputs in some cases?
- Are there risks of amplifying biases present in the reward models?
- Could this approach be misused to generate more convincing deepfakes or misleading images?

2. Limited comparison to related optimization approaches:

The paper compares to some baseline models, but doesn't thoroughly compare to other test-time optimization methods for text-to-image models. Comparisons to approaches like:

- DOODL (Kerras et al., 2022)
- D-Flow (Ben-Hamu et al., 2023)

Would help contextualize the novelty and advantages of ReNO.

3. Insufficient analysis of impact on image diversity:
- The paper doesn't thoroughly examine whether optimizing for rewards reduces the diversity of generated images. Some analysis of how ReNO affects the distribution of outputs would be valuable. (Specially theoretical)

Aside from the mentioned points, everything else was satisfactory, and I enjoyed the paper!
questions: questions: 1. Reward model robustness:
- How sensitive is ReNO to the choice of reward models? Have you observed any cases where optimizing for certain reward models leads to unexpected or undesirable results? This could help understand the robustness and potential limitations of the approach.

2. Computational efficiency:
- Could you provide more details on how ReNO's performance scales with the number of optimization steps and computational budget? Is there a clear point of diminishing returns?

3. Diversity of outputs:
- Does optimizing for reward models potentially reduce the diversity of generated images? Have you conducted any analysis on how ReNO affects the distribution of outputs compared to the base models?

4. Integration with other techniques:
- How might ReNO complement or interact with other techniques for improving text-to-image models, such as fine-tuning or prompt engineering? (Out of curiosity)
limitations: limitations: The authors acknowledge some limitations of their approach, particularly in the "Limitations" section. They mention:

- Convergence of different models to similar performance levels, potentially due to limitations in reward models.
- The increased VRAM requirements of their method.
- Persistent challenges in generating humans, rendering text, and modeling complex compositional relations.
- They briefly mention the possibility of hallucination in their method, which is a relevant concern for AI-generated content.


I think they addressed the limitations adequately.
paper_summary: paper_summary: The paper presents a novel approach called Reward-based Noise Optimization (ReNO) to enhance the performance of one-step Text-to-Image (T2I) models. ReNO optimizes the initial noise of T2I models using a human preference reward model, addressing the limitations of current T2I models in capturing complex details in compositional prompts. The method shows promising results across four different one-step models on T2I-CompBench and GenEval benchmarks, outperforming open-source T2I models and achieving comparable performance to a proprietary model. ReNO is computationally efficient and improves the quality of generated images, as demonstrated through user studies.
strength_weakness: strength_weakness: 1. ReNO is essentially a runtime optimization approach that leverages advanced reward models to achieve state-of-the-art text-to-image generation capabilities. Indeed, the authors opt to optimize initial noise as a learnable parameter. Could I choose to make the parameters of a UNet learnable to achieve a similar objective?
2. Have the authors attempted to apply ReNO to text-to-image diffusion models, such as SD, using techniques like gradient checkpoint and LoRA?
3. Has the proposed ReNO been tested on video generation diffusion models for text-to-video tasks?
4. The authors are encouraged to include the following references:
   - Guided image synthesis via initial image editing in diffusion model, ACM MM 2023
   - InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, CVPR 2024
questions: questions: Overall, ReNO presents a simple yet effective solution to enhance text-to-image diffusion models without the need for additional training. Generally, I have a positive outlook. Please refer to the Weaknesses section for a detailed list of questions and suggestions.
limitations: limitations: The authors provide Limitations and Broader Impact.
paper_summary: paper_summary: The paper introduces Reward-based Noise Optimization (ReNO), a novel approach to enhance Text-to-Image (T2I) models at inference by optimizing the initial noise based on human preference reward models. ReNO significantly improves model performance within a computational budget of 20-50 seconds, outperforming all current open-source T2I models and being preferred almost twice as often as the popular SDXL model in user studies. Additionally, ReNO-optimized models demonstrate superior efficiency, surpassing widely-used models like SDXL and PixArt-alpha with the same computational resources.
strength_weakness: strength_weakness: Noise optimization is a bit like an "adversarial attack" that achieves its goal by adding noise to the original input, but the only drawback of the method is the time cost, as using gradient descent to obtain noise requires a lot of iterations. Therefore, optimizing time is a crucial factor that requires attention. The paper points out that the optimization time takes 20-50 seconds, and it says that one-step t2i is used. What if multi-step reasoning is directly used? Is the effect better than optimizing noise?
questions: questions: if the author can significantly reduce the number of optimization iterations, it would be a good method, and the author can refer to some adversarial attack methods. I hope to see the author improve on optimizing time
limitations: limitations: The paper points out that the optimization time takes 20-50 seconds, The optimization time is too long and not easy to use
paper_summary: paper_summary: In this work, the authors propose finetuning the noise that a one-step diffusion model predicts a clean image from, with respect to an ensemble of preference constraints.  Because only a one-step diffusion model is used, optimizing the noise is fast in terms of number of steps (and therefore wall clock time).  The authors demonstrate that the performance of this optimized one-step text-to-image model is comparable to diffusion models that utilize multiple levels of denoising on preference benchmarks.
strength_weakness: strength_weakness: First and foremost, the comparisons showcased within this paper are flawed.  In Table 2 and Table 3, the authors compare their approach against the performance of default text-to-image models.  It should be almost expected that the performance of ReNO, by virtue of task-specific optimization, **should** have improved performance over default text-to-image models.  This is therefore not an interesting comparison; in fact, it even raises suspicions on the approach in the cases that it does not outperform default models (e.g. Attribute Binding in Dall-E 3).  The authors should instead compare their optimized approach against other optimized approach; for example, DOODL, or DOODL modified to be one-step, the approach from Samuel et al., etc.  This provides a clearer picture of ReNO as an optimization scheme compared to other optimization schemes in tackling preference-respecting generation.  The comparison to DPO is a nice small start, but we need more such comparisons - across benchmarks, and across more task/metrics.  It is really strange that the authors do not compare to the works they state are the most related to their approach.

The other weakness of this approach is that it is limited to one-step diffusion models.  As the authors state, "backpropagating the gradient through multiple denoising steps can lead to exploding/vanishing gradients, rendering the optimization process unstable."  They instead limit themselves to using only "a distilled one-step T2I model".  The authors have not solved a fundamental problem in a general way, of how to optimize the noise for general text-to-image models; instead, they only demonstrate it for a single-step T2I model.  The impact of this work is therefore extremely limited; such insights can clearly not generalize to arbitrary diffusion models; and the authors **have not solved** a key issue of the approach of noise optimization but just ignored it entirely in favor of a more limited problem setting where the results they achieved is rather expected.

The method also seems to rely heavily on an ensemble.  However, this makes it rather ungeneralizable; under this approach it is not possible to optimize the text-to-image model with respect to one specific, particular preference.  Instead, the reliance on multiple reward functions simultaneously implicitly means that the resulting policy will balance between each of the functions used in the ensemble.  If there were a novel preference that it were important to optimize a text-to-image model to respect, this approach would not be able to be applied successfully.

This reviewer found that this work lacks severely in insight - indeed, many of the purported results are completely to be expected.  The authors repeatedly tout the fast training benefits of ReNO, but really this boils down to it being forced to work with a one-step diffusion model, which makes the fast optimization a rather expected result.  The computational cost was not improved in any way by the authors by their approach, it comes for free due to them restricting their **choice** of diffusion model to a one-step one.  Therefore, it is completely unsurprising that using a one-step diffusion model would result in faster optimization than multi-step ones; there is no new insight to be gained here.  Furthermore, the user study was conducted for SD-Turbo + ReNO against default models like SD-Turbo without any optimization; of course, there should be an improvement!  All these results have verified is that optimizing the noise helps, but this is to be expected - it is completely unsurprising.

Furthermore, a severe weakness of ReNO is that once optimized, there is no diversity in the output - particularly because the approach is one-step.  For multi-step diffusion, even with the initial noise being optimized, the resulting output has variability because the other denoising steps have stochasticity (of resampling the Gaussian noise).  However, one-step diffusion has no diversity for a fixed (optimized) noise.  Even though it may be cheaper in wall-clock time to optimize one one-step image, if a batch of $n$ images were to be generated ReNO would need $n$ times the proposed time cost for one image.  On the other hand, Samuel et. al only optimize the initial generation point for multi-step diffusion, enabling a result that still has diversity while still respecting preference.  The speed is by default 1-4 minutes for Samuel et . al which is comparable or even preferable to ReNO, particularly because it can generate a batch all at once whereas ReNO needs to be reoptimized for each image which may take longer.  Also, Samuel et. al reports optimization improvements that reduce it from minutes to seconds, which makes it strictly better than RENO which takes seconds to just generate one image that has no diversity. Samuel et. al state in Section 6.1 that it takes 1-5 minutes to adapt to the new concept and 1-2 seconds to generate new semantically correct images.

The authors do not supply new criteria, nor do they innovate the approach of optimizing the noise.  Instead, they simply apply multiple existing criteria simultaneously.  Furthermore, the approach is limited to one-step diffusion models, which inhibits its generality.  Ultimately, the scope of this work feels more like a workshop paper rather than a conference-level paper.
questions: questions: Are there ways to optimize multiple noises through ReNO to generate a batch of preference-respecting images without essentially optimizing each noise separately?

What are the ways ReNO can enable or address the backpropagation through multiple denoising levels, and provide interesting implications for general multi-step diffusion models?  Are there ways this can be demonstrated in the rebuttal?
limitations: limitations: This reviewer foresees no substantial potential negative societal impact from this work.  However, in the Limitations section, the authors state that they hypothesize that the reward models may be limiting, and that stronger reward models and preference data may be crucial in enhancing results further.  This reviewer feels like this hypothesis could be directly tested within the scope of this work; building off of the initial results of Table 1, further study could be performed on utilizing subsets of the complete set of reward functions to evaluate the benefits of each.

Comments: title: Reviewer Response [3]
Reviewer Response [2]
Reviewer Response [1]
End of Discussion Period
Response to rebuttal
Rebuttal by Authors [2/2]
comment: Thank you for your insightful feedback. Your comments have helped us recognize areas where we can better articulate our existing work. We plan to refine the presentation in the Abstract, Introduction, and Section 2 to more clearly communicate our motivation and contributions. Specifically, we will emphasize:

1. Our primary goal is to maximize the performance of T2I models within significant resource constraints.
2. Test-time optimization emerges as a promising direction for this goal. However, our experiments with DOODL reveal limitations in achieving the desired balance of quality and efficiency.
3. Consequently, we focus on one-step models as a tractable starting point, introducing a novel human preference reward model based approach that leverages complementary strengths to boost overall image generation performance.
    
For example, we plan to update the following sentence in the Abstract to better reflect this narrative:

"In this work, we propose Reward-based Noise Optimization (ReNO), a novel approach that enhances T2I models at inference by optimizing the initial noise based on the signal from one or multiple human preference reward models."

->

"In this work, we provide a new perspective on generally improving T2I generation through Reward-based Noise Optimization (ReNO), a novel approach that enhances one-step T2I models at inference. ReNO optimizes the initial noise based on signals from multiple human preference reward models, offering a unique solution to improve generation quality within strict computational constraints."

> ***Currently there also seems to be a big focus on preference optimization (through an ensemble), which distracts from the overall goal of simply "improving generation quality of existing T2I models" which everyone can appreciate.***

Additionally, we plan to lower the emphasis on the ensemble of reward models and discuss it as a tool to enhance image generation quality in our updated manuscript.

> ***The focus is not on source noise optimization because its implications do not extend (currently) to general diffusion models - the source noise optimization is simply a mechanism for improving test-time generation quality. The one-step diffusion model is not used to sidestep multi-step diffusion models (which raises suspicions of the authors avoiding the worthwhile interesting questions, and raises concerns that their insights are not generally useful or applicable across all T2I models), but a choice made for tractability.***

We agree with your assessment and will more clearly clarify in our revised manuscript that our focus is on improving test-time generation quality, with source noise optimization as a mechanism and one-step models chosen for tractability, rather than to sidestep multi-step models or avoid broader questions in noise optimization for diffusion models.

We believe this revised framing will more effectively communicate the significance and broader impact of our work in the context of practical T2I model deployment and optimization.
Surprisingly, the storytelling and motivation outlined in the general comment is more appealing and clear than what was ultimately presented in the paper.  This reviewer is actually on board with the ultimate motivation of "obtain[ing] the best possible/extremely high-quality text-to-image generation using open-source models and tools".  Leading it towards cheaper models and alternatives while considering memory constraints and navigating closed-source models to motivate "test-time optimization methods [as] a compelling alternative to enhance the generation quality of existing multi-step T2I models".  And then going from test-time optimization of large multi-step models to a single one.  Focusing on numerical insights about expensiveness of existing models (e.g. their RAM) would be compelling - a numbers-focused argument would be welcomed.

This story reads much better than the one that was initially provided; where the central focus seems to be about T2I noise optimization broadly - but the only thing demonstrated was for one-step T2I models.  And that mismatch is a big source of motivating confusion because the findings for one-step T2I models do not translate necessarily to general source-noise optimization insights for T2I models.  Currently there also seems to be a big focus on preference optimization (through an ensemble), which distracts from the overall goal of simply "improving generation quality of existing T2I models" which everyone can appreciate.

If the storytelling were structured like the roadmap provided above from the onset, this reviewer would appreciate the work and its scope much better.  This reviewer actually strongly encourages the authors to write their paper (esp their Introduction, Abstract, etc.) with this motivating story in mind - to avoid any potential confusion and disappointment in what the authors really seek to tackle.  The focus is *not* on source noise optimization because its implications do not extend (currently) to general diffusion models - the source noise optimization is simply a mechanism for improving test-time generation quality.  The one-step diffusion model is *not* used to sidestep multi-step diffusion models (which raises suspicions of the authors avoiding the worthwhile interesting questions, and raises concerns that their insights are not generally useful or applicable across all T2I models), but a choice made for tractability.  This presentation would allow the reviewer to actually appreciate the results rather than be focused on the one-step T2I choice from the get-go and be disappointed at the lack of useful general insights.

This reviewer appreciates the proposed story here and can see the pieces fit in - but as the main paper is written currently, the reviewer still feels uncomfortable directly recommending acceptance.  A slight increase of the score will be made out of a common understanding - but still, the reviewer would highly recommend rewriting the motivating portions of the paper (results can obviously remain the same) to fit this proposed storyline.
> ***The comment remains the same, that something that has explicit optimization with respect to preference intuitively should outperform something that has not been optimized whatsoever. This reviewer still believes the user study is unfair, and that it is completely intuitive and expected that an SD-Turbo + ReNO should outperform non-optimized SD-Turbo with respect to preference, because SD-Turbo+ReNO was explicitly optimized to resepct it! In fact, it is troubling that even with optimization it does not outperform DALL-E 3; intuitively these optimized techniques should always outperform non-optimized ones.***

We agree that SD-Turbo + ReNO should outperform SD-Turbo as long as the optimization was done correctly given robust T2I reward models. However, the margin of increase is not clear apriori, see e.g. the performance increase of DOODL in the author rebuttal and Table 3 in the additional PDF. Additionally, outperforming 50-step models that are 2-5x bigger with 10x the compute used for training is not at all to be expected purely from noise optimization.


> ***It is strange that the authors are adamant on staying with a one-step diffusion model, refusing to even try some distilled technique that can generate samples in 2 steps or 6 steps (e.g. some LCM models).***

Even with a 2-step model, the VRAM requirement will significantly increase. Thus, e.g., 2-step HyperSDXL will not fit into 40GB anymore, making it impractical as a general image generation model. We thank the reviewer for the suggestion and agree that this is an interesting future research direction.
We thank the reviewer for engaging with our rebuttal. We would like to first clarify our motivation for approaching this work: Our goal was to obtain the best possible/extremely high-quality text-to-image generation using open-source models and tools. To this end, models such as the 50-step SD2.1/SDXL models would be the first choice. However, the shortcomings of these models are well-documented and are further corroborated in our paper. Alternately, one could train bigger models on larger datasets with higher-quality data (as is the trend with SD2.1 -> SDXL -> SD3). Unfortunately, these require large-scale GPU resources unavailable to most research groups (e.g. DALL-E 2 itself uses 40000+ A100 GPU days) and additionally, current SOTA models are paid services and not open source (DALLE-3, SD3 (8B)). Therefore, test-time optimization methods are a compelling alternative to enhance the generation quality of existing multi-step T2I models. These methods (e.g. DOODL) work with multi-step models and enhance the quality of the generated images. However, not only are they computationally expensive, they also provide limited improvements in the metrics that they optimize for(on average CLIPScore increases by 0.03 with DOODL on 50 step SD2.1). Our hypothesis was that the lack of effective optimization was due to exploding gradients and other challenges of optimizing multi-step diffusion models. Therefore, we made the unconventional decision of optimizing the initial noise of a one-step model. Apriori, it was unclear if these models could even match the corresponding multi-step model even after noise optimization, let alone surpass them. To our surprise, not only was it 60x faster to optimize, but the gains in CLIPScore were 4x (0.12) that were obtained from optimizing multi-step models (e.g., DOODL). Enhancing this further, we incorporated other models that could provide complementary signals to improve both visual quality (e.g., HPSv2) and prompt following (e.g., ImageReward). As a result, we obtained image generation results that were the highest reported results among any open-source method on T2I-Compbench and GenEval. Further, even for the same time that a 50-step SDXL takes for generation, we show better prompt following by performing noise optimization with SD-Turbo (Fig 5). We believe that providing a recipe to enhance the quality of text-to-image generation (noise optimization of the best-distilled models) in a cost-effective manner is our key contribution.


> ***In Tables 2 it is pretty apparent that DALL-E 3 outperforms ReNO, while being a multi-step denoising diffusion model. There is some confusion on why the authors disagree with the initial statement, and suggests that “ReNO-optimized one-step models significantly outperform their corresponding multi-step models in all benchmarks”. [...] In fact, it is troubling that even with optimization it does not outperform DALL-E 3;***

Our finding is for one-step models and their **corresponding** multi-step models. SD-Turbo is based on SD2.1 with an 800M U-Net and a 336M params CLIP ViT-H text encoder, which makes it the multi-step model to compare to. SD-Turbo and DALLE-3 are models trained with very different resources and architectures. While specific details for DALLE-3 are not available, e.g. SD3 leverages a DiT of 8B params with a T5-XXL (4B params) text encoder. We show that in all experiments conducted, SD-Turbo outperforms 50-step SD2.1 and even 50-step SDXL, which is the next-generation multi-step model. Based on our findings, a one-step model based on SD3/DALLE-3, like SD3-Turbo, enhanced with ReNO should outperform multi-step SD3/DALLE-3. Unfortunately, these models are proprietary, and thus, we could not benchmark with SD3-Turbo. To summarize, DALLE-3 and SD3 are "two generations" after SD-Turbo and thus, do not constitute a fair comparison between one-step and multi-step models. Similarly, a method improving LLaMa2-7B isn't expected to outperform GPT-4 to be a meaningful research contribution.

> ***Regarding multiple reward models: it still appears that ReNO depends on an ensemble of reward models. And that performance does not increase significantly without the ensemble. Relying on an ensemble suggests that this method may not perform well for a specific, novel criteria (in situations where other existing criteria in the ensemble are not as important)***

As mentioned in our previous answer, ReNO can be flexibly used based on given preferences. While we benchmark all current T2I reward models we are aware of, adapting this to new models is straightforward as long as the novel criteria are expressed in a differentiable function. See, for example, the color example in Figure 3, or given a new, more robust, and stronger T2I reward model, ReNO can be employed with just this reward. Table 1 does show that already a single reward model achieves significant improvements.
A major point of contention is that the authors have “side-stepped” a crucial challenge of noise optimization for general T2I models, rather than address it head-on, by limiting their scope to one-step T2I models.  But then, the implications of their findings are severely limited to just one-step cases.  Nothing seems to be able to be generalized to the multi-step case, and no insights can be made about noise optimization for general-purpose T2I models.  But since the capabilities of one-step T2I models are inherently limited, and even with this optimization scheme it cannot outperform default multi-step diffusion models such as DALL-E 3, this reviewer fails to see the discoveries of this work as impactful or useful - especially since the findings in this work cannot generalize to arbitrary T2I model settings.  “Side-stepping” is not something to be celebrated, if it severely limits the implications and impact of their work.  The insight into the behavior of general diffusion modeling is limited, if existent at all.

Regarding multiple reward models: it still appears that ReNO depends on an ensemble of reward models.  And that performance does not increase significantly without the ensemble.  Relying on an ensemble suggests that this method may not perform well for a specific, novel criteria (in situations where other existing criteria in the ensemble are not as important)
The comment remains the same, that something that has explicit optimization with respect to preference intuitively should outperform something that has not been optimized whatsoever.  This reviewer still believes the user study is unfair, and that it is completely intuitive and expected that an SD-Turbo + ReNO should outperform non-optimized SD-Turbo with respect to preference, because SD-Turbo+ReNO was explicitly optimized to resepct it!  In fact, it is troubling that even with optimization it does not outperform DALL-E 3; intuitively these optimized techniques should always outperform non-optimized ones.

Generating a batch: once again, to generate a batch of $n$ images ReNO needs to optimize $n$ independent noises; because of the one-step nature of the underlying diffusion model.  Compare this with a multi-step diffusion model, even where the number of levels is 2 - diversity is automatically provided through the resampling of the source noise which adds stochasticity.  Utilizing only a one-step diffusion model prohibits generalization of a batch in a tractable manner and is deeply unsatisfying.

Optimizing multiple noise levels: by describing the implications of noise optimization for multi-step generation as out of scope, the authors have inherently painted the scope of their work to a very small, unsurprising, and rather insightful scope.  Again, their findings cannot generalize to the common, more powerful multi-step diffusion model case.  Nor is it surprising that an optimized approach beats unoptimized approaches in terms of preference. Furthermore, focusing only on one-step diffusion models introduces a litany of new issues, such as bad default generation behavior and also inability to generate a batch from the same optimized source noise.

It is strange that the authors are adamant on staying with a one-step diffusion model, refusing to even try some distilled technique that can generate samples in 2 steps or 6 steps (e.g. some LCM models).  This would naturally address complaints about generating batches, would provide initial insights into noise optimization for the general case over multiple timesteps, and the few steps would also hopefully avoid gradient explosion/vanishing.

The comments regarding writing remain, e.g. one-step diffusion models are only mentioned in lines 116-117, but the entire section of 2.1 is titled One-Step Diffusion models, despite mostly covering general Diffusion Modeling.

As the majority of the fundamental key concerns this reviewer has with the work remain, this reviewer cannot recommend the paper for acceptance.
This reviewer appreciates the detailed rebuttal, and provides thoughts in response:

This reviewer understands that exploring the optimization of the noise vector is the main problem ReNO is aiming to solve.  A limitation is that, the analysis seems only to hold for one-step models.  It is not obvious that optimizing the noise is generally useful for diffusion modeling; especially since over multiple timesteps, the effect of each particular noise sample is minimized - potentially including the initial one.  The impact of this work is therefore severely limited.

To that note, it is not particularly impressive if by default, one-step T2I models are capped in terms of modeling capability.  If even with the best optimization, T2I models cannot outperform multistep diffusion models without optimization (e.g. DALL-E 3), then one-step T2I models are essentially a dead-end, barring amazing new developments in the space.  This reviewer therefore disagrees that this is a particularly significant finding.  The purported importance of initially sampled noise appears only to hold for this limited, toy example and is not supported for diffusion models in general.  The authors do not provide real insights into how to perform noise optimization over multiple timesteps, nor can they provide real insights into noise optimization for the general case of (potentially multi-step) diffusion models as the usefulness of the source noise is then in question.

In Tables 2 it is pretty apparent that DALL-E 3 outperforms ReNO, while being a multi-step denoising diffusion model.  There is some confusion on why the authors disagree with the initial statement, and suggests that “ReNO-optimized one-step models significantly outperform their corresponding multi-step models in all benchmarks”.  Furthermore, Table 5 doesn’t seem relevant to the discussion.

This reviewer would like to clarify that when the authors state that they propose an optimization that “generally improves a T2I model”, they are really referring to an optimization that improves a one-step T2I model only.  It is not obvious that such benefits extend to the multi-step case.
Dear Reviewers,

As the discussion period draws to a close, we would like to express our sincere gratitude for your initial review of our paper. We appreciate the recognition of our work's contributions in terms of performance enhancement, novelty, clarity, and practicality.

We hope that in our rebuttal, we have successfully addressed each point raised and provided valuable additional analysis based on your feedback. This includes a diversity study of ReNO, where we found that ReNO significantly increases diversity, and a more comprehensive comparison with DOODL (ReNO is 60x faster with 4x better metric optimization). Additionally, we want to further highlight that ReNO-enhanced one-step models outperform multi-step models with the same computational budget by summarizing Figure 5 in the following table, illustrating the efficacy of our proposed method:

| Model                      | T2I-CompBench Average | Time    |
| -------------------------- | --------------------- | ------- |
| SDXL                       | 0.5804                | ~7 sec  |
| SD-Turbo                   | 0.5217                | 0.4 sec |
| SD-Turbo + ReNO (10 steps) | 0.6315                | 4 sec   |
| SD-Turbo + ReNO (15 steps) | 0.6497                | 6 sec   |
| SD-Turbo + ReNO (20 steps) | 0.6653                | 8 sec   |
| SD-Turbo + ReNO (25 steps) | 0.6804                | 10 sec  |
| SD-Turbo + ReNO (30 steps) | 0.6966                | 12 sec  |
| SD-Turbo + ReNO (40 steps) | 0.7082                | 16 sec  |
| SD-Turbo + ReNO (50 steps) | 0.7180                | 20 sec  |

We believe these clarifications and additions further demonstrate the strengths of our approach.

We would greatly appreciate your consideration of our rebuttal as you formulate your final evaluations. We again want to thank the reviewers for their time and expertise throughout the review process. The reviewers' suggestions have, without a doubt, improved the breadth and quality of our submission.

Kind regards,

Anonymous Authors
Thank you for addressing my concern. I think the paper is now is a good shape (especially after the new experiments). I'd like to maintain my score.
The reviewer mentions that our method is not universal and time-consuming, and points to the other reviews. It would be much appreciated if the reviewer could provide more details, since in our rebuttal to each of the other reviews, we have thoroughly clarified these points.

> ***Universality of ReNO:***

In this work, we are tackling the most general form of Text-to-Image generation. ReNO-enhanced one-step models consistently surpass the performance of all current open-source Text-to-Image models across a variety of **general** T2I benchmarks and a comprehensive user study.

> ***Time-consuming:***

We address this point in the rebuttal above. ReNO enhanced SD-Turbo outperforms existing open-source models such as multi-step SDXL and PixArt-$\alpha$ **with the same compute budget**. This shows that ReNO is not time-consuming but actually time-efficient.
Thank you for the responses. 

Referring to the opinions of other reviewers, I think the proposed method is not universal and time-consuming, so I keep my score.
Dear Reviewers,

Thanks for your efforts in reviewing our paper. Specifically, we want to thank you for asking important questions, which led us to study the change in diversity through ReNO and provide a more thorough comparison to DOODL. Please let us know if our response addresses all your concerns or if you would like us to provide any other clarifications.
> ***the authors **have not solved** a key issue of the approach of noise optimization but just ignored it entirely in favor of a more limited problem setting where the results they achieved is rather expected.***

While we agree that solving the issue of computational efficiency for multi-step generation with noise optimization is a very interesting research question, we never claim to solve this problem generally with ReNO. We sidestep this challenge by considering one-step diffusion models. Note that this is still a major insight because applying noise optimization in the form of DOODL/D-Flow is not practical for general T2I generation, as mentioned in the global rebuttal.

> ***Furthermore, the reliance on an ensemble [...] ungeneralizable [...]***

We report the performance of all the combinations of reward models in Table 1 & 7. As can be seen, also without an ensemble with only HPSv2 (better image quality) or ImageReward (better prompt following), ReNO achieves significant performance improvements. We are unsure if we understand why this would make ReNO not generalizable. In the general setting we are considering, the goal is no one specific preference but a general improvement of the used model.

> ***If there were a novel preference that it were important to optimize a text-to-image model to respect, this approach would not be able to be applied successfully.***

On the contrary, our approach is designed to be flexible and adaptable to various objectives. There's no inherent limitation preventing the application of ReNO to novel preferences or optimization goals. In fact, we demonstrate in Figure 3 how "personalized" objectives can be effectively incorporated. With just 10 optimization iterations, we show significant increases in specific color attributes (redness/blueness) of generated images.

> ***Furthermore, the user study was conducted for SD-Turbo + ReNO against default models like SD-Turbo without any optimization; of course, there should be an improvement!***

In the user study, we specifically compare SD-Turbo + ReNO against SDXL-Turbo, SD2.1 (50-step), and SDXL (50-step). All of these models are usually preferred over SD-Turbo. SD-Turbo + ReNO is significantly preferred over all of them, even the next-generation SDXL with 50 steps! Additionally, HyperSDXL + ReNO is preferred over SD3 (8B). 

> ***Are there ways to optimize multiple noises through ReNO to generate a batch of preference-respecting images without essentially optimizing each noise separately?***

Theoretically, this could also be incorporated into ReNO as a further Criterion function. However, the problem we are tackling is general T2I generation, which is by default only per single prompt and noise. Thus, this is out of the scope of our work but an interesting future direction.

> ***What are the ways ReNO can enable or address the backpropagation through multiple denoising levels, and provide interesting implications for general multi-step diffusion models? Are there ways this can be demonstrated in the rebuttal?***

While we agree that solving the challenges of noise optimization for multi-step generation is a very interesting research question, we never claim to solve this problem generally with ReNO, and it is also out of the scope of this work. We show how the noise optimization network can be leveraged to effectively arrive at a **very significantly** better model. We leave how to best adapt our findings to multi-step diffusion models to future work.

> ***Equation 7 was referenced before it was even written***

Thanks for this pointer, it was supposed to refer to Equation 4 and we will update it in the paper accordingly.

> ***one-step diffusion models were only mentioned in two sentences (lines 116-117).***

The paragraph [lines 112-125] completely serves to introduce the different one-step diffusion models we employ in this work and how they work in general.

> ***reads almost as a background/related works section (most clearly demonstrated in lines 118-125)***

In this part, we introduce the four different one-step models we use to benchmark ReNO. We consider this as background for ReNO, as the section is also titled.

> ***Then, there is a separate Related Works section. The paper could benefit from severe reorganization to improve its clarity.***

Thank you for this comment; we will consider how to improve the clarity. With the separate related work section after the introduction of ReNO we aim to contextualize ReNO within the scope of all related work. Which part of the way we introduce ReNO exactly was unclear? We would be happy to incorporate any specific suggestions to enhance the clarity.