UniFL: Improve Latent Diffusion Model via Unified Feedback Learning

Authors: jiacheng  zhang
jie  wu
yuxi  ren
xin  xia
huafeng  kuang
pan  xie
jiashi  li
xuefeng  xiao
weilin  huang
shilei  wen
lean  fu
guanbin  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:26:38

Abstract: Latent diffusion models (LDM) have revolutionized text-to-image generation, leading to the proliferation of various advanced models and diverse downstream applications. However, despite these significant advancements, current diffusion models still suffer from several limitations, including inferior visual quality, inadequate aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present **UniFL**, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL.
Notably, UniFL consists of three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which accelerates inference.
In-depth experiments and extensive user studies validate the superior performance of our method in enhancing generation quality and inference acceleration. For instance, UniFL surpasses ImageReward by 17\% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57\% and 20\% general preference with 4-step inference.

Main paper: 
 introduction: 
 The emergence of diffusion models has led to remarkable advances in the field of text-to-image (T2I) generation, marked by notable milestones like DALLE-3 [1] , Imagen [2] , Midjourney [3] , etc, elevating the generation quality of images to an unprecedented level. Particularly, the introduction of open-source image generation models, exemplified by latent diffusion model (LDM) [4] , has inaugurated a transformative era of text-to-image generation, triggering numerous downstream applications such as T2I personalization [5, 6, 7, 8] , controllable generation [9, 10, 11 ] and text-tovideo (T2V) generation [12, 13, 14] . Nevertheless, despite these advancements achieved thus far, current latent diffusion-based image generation models still exhibit certain limitations. i) Inferior visual quality: The generated images still suffer from poor visual quality and lack authenticity. Examples include characters with incomplete limbs or distorted body parts, as well as limited fidelity in terms of style representation. ii) Inadequate aesthetic appeal: The generated image tends to lack aesthetic appeal and often fails to align with human preferences, especially in the abstract aesthetic concepts aspects such as color, lighting, atmosphere, etc. iii) Slow inference speed: The iterative denoising process employed by diffusion models led to inefficiencies during inference that significantly impede generation speed, thereby limiting the practicality of these models in various application scenarios. Recently, numerous works have endeavored to address the aforementioned challenges. For instance, RAPHAEL [15] resorts to the techniques of Mixture of Experts) [16, 17, 18] boost the generation performance via stacking the space MoE and time MoE block. Works [19, 20, 21, 22, 23] represented by ImageReward [23] propose incorporating human preference feedback to guide diffusion models toward aligning with human preferences. SDXL Turbo [24] , PGD [25] , and LCM [26, 27] , on the other hand, targets on achieve inference acceleration through techniques like distillation and consistency models [28] . However, these methods primarily concentrate on tackling individual problems through specialized designs, which poses a significant challenge to the elegant integration of these techniques. For example, MoE significantly complicates the pipeline, making the acceleration method infeasible to apply, and the consistency models [28] alter the denoising process of the diffusion model, making it arduous to directly apply the ReFL preference tuning framework proposed by ImageReward [23] . Therefore, a natural question arises: Can we devise a more effective approach that comprehensively enhances diffusion models in terms of image quality, aesthetic appearance, and generation speed?To tackle this issue, we present UniFL, a solution that offers a comprehensive improvement to latent diffusion models through unified feedback learning formulation. UniFL aims to boost the visual generation quality, enhance aesthetic attractiveness, and accelerate the inference process. To achieve these objectives, UniFL features three novel designs upon the unified formulation of feedback learning. Firstly, we introduce a pioneering perceptual feedback learning (PeFL) framework that effectively harnesses the extensive knowledge embedded within diverse existing perceptual models to provide more precise and targeted feedback on the potential visual defects of the generated results. Secondly, we employ decoupled aesthetic feedback learning to boost the visual appeal, which breaks down the coarse aesthetic concept into distinct aspects such as color, atmosphere, and texture, simplifying the challenge of abstract aesthetic optimization. Furthermore, an active prompt selection strategy is also introduced to choose the more informative and diverse prompt to facilitate more efficient aesthetics preference learning. Lastly, UniFL develops adversarial feedback learning to achieve inference acceleration by incorporating the adversarial objective in feedback tuning. We instantiate UniFL with a two-stage training pipeline and validate its effectiveness with SD1.5 and SDXL, yielding impressive improvements in generation quality and acceleration. Our contributions are summarized as follows:• New Insight: Our proposed method, UniFL, introduces a unified framework of feedback learning to optimize the visual quality, aesthetics, and inference speed of diffusion models.To the best of our knowledge, UniFL offers the first attempt to address both generation quality and speed simultaneously, offering a fresh perspective in the field.• Novelty and Pioneering: In our work, we shed light on the untapped potential of leveraging existing perceptual models in feedback learning for diffusion models. We highlight the significance of decoupled reward models and elucidate the underlying acceleration mechanism through adversarial training.• High Effectiveness: Through extensive experiments, we demonstrate the substantial improvements achieved by UniFL across various types of diffusion models, including SD1.5 and SDXL, in terms of generation quality and inference acceleration. 
related works: 
 Text-to-Image Diffusion Models. Text-to-image generation has gained unprecedented attention over other traditional tasks [29, 30, 31, 32, 33] . Recently, diffusion models have gained substantial attention and emerged as the de facto mainstream method for text-to-image generation, surpassing traditional image generative models like GAN [34] and VAE [35] . Numerous related works have been proposed, including GLIDE [36] , DALL-E2 [1] , Imagen [2] , CogView [37] etc.. Among these, Latent Diffusion Models (LDM) [4] extend the diffusion process to the latent space and significantly improve the training and inference efficiency of the diffusion models, opening the door to diverse applications such as controllable generation [9, 10] , image editing [11, 38, 39] , and image personalization [5, 7, 6] and so on. Even though, current text-to-image diffusion models still have limitations in inferior visual generation quality, deviations from human aesthetic preferences, and inefficient inference. The target of this work is to offer a comprehensive solution to address these issues.Improvements on Text-to-Image Diffusion Models. Given the aforementioned limitations, researchers have proposed various methods to tackle these issues. Notably, [40, 15, 41] focuses on improving generation quality through more advanced training strategies. Insipred by the success of reinforcement learning with human feedback (RLHF) [42, 43] in the field of LLM, [20, 21, 44, 23, 45] explore the incorporation of human feedback to improve image aesthetic quality. On the other hand, [25, 24, 28, 27, 26] concentrate on acceleration techniques, such as distillation and consistency models [28] to achieve inference acceleration. While these methods have demonstrated their effectiveness in addressing specific challenges, their independent nature makes it challenging to combine them for comprehensive improvements. In contrast, our study unifies the objective of enhancing visual quality, aligning with human aesthetic preferences, and acceleration through the feedback learning framework. 
preliminaries: 
 Latent Diffusion Model. Text-to-image latent diffusion models leverage diffusion modeling to generate high-quality images based on textual prompts, which generate images from Gaussian noise through a gradual denoising process. During pre-training, a sampled image x is first processed by a pre-trained VAE encoder to derive its latent representation z. Subsequently, random noise is injected into the latent representation through a forward diffusion process, following a predefined schedule {β t } T . This process can be formulated asz t = √ α t z + √ 1 -α t ϵ, where ϵ ∈ N (0, 1) is the random noise with identical dimension to z, α t = t s=1 α s and α t = 1 -β t . To achieve the denoising process, a UNet ϵ θ is trained to predict the added noise in the forward diffusion process, conditioned on the noised latent and the text prompt c. Formally, the optimization objective of the UNet is:EQUATIONReward Feedback Learning. Reward feedback learning (ReFL) [23] is a preference fine-tuning framework that aims to improve the diffusion model via human preference feedback. It consists of two phases: (1) Reward Model Training and (2) Preference Fine-tuning. In the Reward Model Training phase, human preference data is collected to train a human preference reward model, which serves as a proxy to provide human preferences. More specifically, considering two candidate generations, denoted as x w (preferred generation) and x l (unpreferred one), the loss function for training the human preference reward model r θ can be formulated as follows:EQUATION) where D denotes the collected feedback data, σ(•) represents the sigmoid function, and c corresponds to the text prompt. The reward model rθ is optimized to produce a reward score that aligns with human preferences. In the Preference Fine-tuning phase, ReFL begins with an input prompt c, initializing a random latent variable x T . The latent variable is then progressively denoised until reaching a randomly selected timestep t. Then, the denoised image x ′ 0 is directly predicted from x t . The reward model obtained from the previous phase is applied to this denoised image, generating the expected preference score r θ (c, x ′ 0 ). ReFL maximizes such preference scores to make the diffusion model generate images that align more closely with human preferences:L refl (θ) = E c∼p(c) E x ′ 0 ∼p(x ′ 0 |c) [-r(x ′ 0 , c)](3) Our method follows a similar learning framework to ReFL but devises several novel components to enable comprehensive improvements. 4 UniFL: Unified Feedback LearningOur proposed method, UniFL, aims to improve the latent diffusion models in various aspects, including visual generation quality, human aesthetic quality, and inference efficiency. our method takes a unified feedback learning perspective, offering a comprehensive and streamlined solution. An overview of UniFL is illustrated in Fig. 2 . In the following subsections, we delve into the details of three key components: perceptual feedback learning to enhance visual generation quality (section 4.1); decoupled feedback learning to improve aesthetic appeal (section 4.2); and adversarial feedback learning to facilitate inference acceleration (section 4.3). 
perceptual feedback learning: 
 Current diffusion models exhibit limitations in achieving high-fidelity visual generation, for example, object structure distortion. These limitations stem from the reliance on reconstruction loss(MSE loss) solely in the latent space, which lacks structural supervision on the high-level visual quality.To address this issue, we propose perceptual feedback learning (PeFL). Our key insight is that various visual perception models already embed rich visual priors, which can be exploited to provide feedback for visual generation and fine-tune the diffusion model. The complete PeFL process is summarized in Algorithm 1. In contrast to ReFL, which starts from a randomly initialized latent representation and only considers the text prompt as a condition, PeFL incorporates image content as an additional visual condition for perceptual guidance. Specifically, given a text-image pair, (c, x), we first select a forward step T a and inject noise into the ground truth image to obtain a conditional latent x 0 → x Ta . Subsequently, we randomly select a denoising time step t and denoising from x Ta , yielding x Ta → x Ta-1 ... → x t . Next, we directly predict x t → x ′ 0 . By incorporating the visual condition input, the denoised image is expected to restore the same high-level visual characteristics, such as object structure, and style, which existing perception models can capture. For instance, in the case of object structure, the instance segmentation model can serve as a valuable resource as it provides essential descriptions of object structure through instance masks. Consequently, the feedback on the generation of such visual characteristics on x ′ 0 can be obtained by comparing it with the ground truth segmentation mask via:EQUATIONwhere m I is the instance segmentation model, GT(x 0 ) is the ground truth instance segmentation mask and L instance is the instance segmentation loss. Note that our PeFL differs from ReFL as indicated by the red font in Algorithm 1. With the visual condition input and perception model, the diffusion model is allowed to get a detailed and focused feedback signal on a specific aspect, instead of the general quality feedback offered by ReFL. Moreover, the flexibility of PeFL allows us to leverage various existing visual perceptual models, more examples can be found in the Appendix A.Decoupled Aesthetic Fine-tuning. Existing text-to-image diffusion models exhibit shortcomings in images that satisfy human aesthetic preferences. While PeFL prioritizes objective visual quality, aesthetic quality is inherently subjective and abstract, requiring human aesthetic feedback to steer the generation process. Despite ImageReward's attempt to incorporate human aesthetic preferences through a reward model, its performance is hindered by oversimplified modeling that fails to capture the multidimensional nature of human aesthetic preferences. Generally, humans consider the aesthetic attractiveness of an image from various aspects, such as color, lighting, etc, and conflating these aspects without distinguishing during preference tuning would encounter optimization conflicts as evidenced in [46] . To address this issue, we follow [23] to achieve aesthetic preference tuning but suggest decoupling the various aesthetic aspects when constructing preference reward models. Specifically, we decomposed the general aesthetic concept into representative dimensions and collected the corresponding annotated data, respectively. These dimensions include color, layout, lighting, and detail. Subsequently, we train a separate aesthetic preference reward model for each annotated data according to Eq.2. Finally, we leveraged these reward models for aesthetic preference tuning: wi+1 ← wi // Update LDMw i using PeFL loss 15: end for Active Prompt Selection. We observed that when using randomly selected prompts for aesthetic preference fine-tuning, the diffusion model tends to rapidly overfit the reward model due to the limited semantic richness, leading to diminished effectiveness of the reward model. To address this issue, we further propose an active prompt selection strategy, which selects the most informative and diverse prompt from a prompt database. This selection process involves two key components: a semantic-based prompt filter and nearest neighbor prompt compression. By leveraging these techniques, the overfitting can be greatly mitigated, achieving more efficient aesthetic reward fine-tuning. More details of this strategy are presented in the Appendix.B.2.EQUATIONThe inherent iterative denoising process of diffusion models significantly hinders their inference speed. To address this limitation, we introduce adversarial feedback learning to reduce the denoising steps during inference. Specifically, to achieve inference acceleration, we exploit a general reward model r a (•) to improve the generation quality of fewer denoising steps. However, as studied in [23] , the samples under low inference steps tend to be too noisy to obtain the correct rewarding scores. To tackle this problem, rather than freeze the reward model during fine-tuning, we incorporate an extra adversarial optimization objective by treating r a (•) as a discriminator and update it together with the diffusion model. Concretely, we follow a similar way with PeFL to take an image as input and execute the diffusion and denoising consecutively. Afterward, in addition to maximizing the reward score of the denoised image, we also update the reward model in an adversarial manner. The optimization objective is formulated as:EQUATIONwhere θ and ϕ are the parameters of the diffusion model and discriminator. With the adversarial objective, the reward model is always aligned with the distribution of the denoised images with various denoised steps, enabling the reward model to function well across all the timesteps. Note that our method is distinct from the existing adversarial diffusion methods like SDXL-Turbo [24] . These methods take the adversarial distillation manner to accelerate the inference, which tends to require another LDM as the teacher model to realize distillation, incurring considerable memory costs. By contrast, we follow the reward feedback learning formulation, which integrates adversarial training with the reward tuning and achieves the adversarial reward feedback tuning via the lightweight reward model.We employ a two-stage training pipeline to implement UniFL. The first stage focuses on improving generation quality, leveraging perceptual feedback learning and decoupled feedback learning to boost visual fidelity and aesthetic appeal. In the second stage, we apply adversarial feedback learning to accelerate the diffusion inference speed. To prevent potential degradation, we also include decoupled feedback learning to maintain aesthetics. The training objectives of each stage are summarized as follows:EQUATION5 Experiments 
implementation details and metrics: 
 Dataset. We utilized the COCO2017 [47] train split dataset with instance annotations and captions for structure optimization with PeFL. Additionally, we collected the human preference dataset for the decoupled aesthetic feedback learning from diverse aspects (such as color, layout, detail, and lighting). 100,000 prompts are selected for aesthetic optimization from DiffusionDB [48] via active prompt selection. During the adversarial feedback learning, we use data from the aesthetic subset of LAION [49] with image aesthetic scores above 5.Training Setting. We utilize the SOLO [50] as the instance segmentation model. We utilize the DDIM [51] scheduler with a total of 20 inference steps. Baseline Models. We choose two representative text-to-image diffusion models with distinct generation capacities to comprehensively evaluate the effectiveness of UniFL, including (i) SD1.5 [4] ;(ii) SDXL [40] . Based on these models, we pick up several state-of-the-art methods(i.e. Im-ageReward [23] , Dreamshaper [52] , and DPO [22] for generation quality enhancement, LCM [27] , SDXL-Turbo [24] , and SDXL-Lightning [53] for inference acceleration) to compare the effectiveness of quality improvement and acceleration. All results of these methods are reimplemented with the official code provided by the authors.Evaluation Metrics. We generate the 5K image with the prompt from the COCO2017 validation split to report the Fréchet Inception Distance (FID) [54] as the overall visual quality metric. We also report the CLIP score with ViT-B-32 [55] and the aesthetic score with LAION aesthetic predictor to evaluate the text-to-image alignment and aesthetic quality of the generated images, respectively. Given the subjective nature of quality evaluations, we further conducted comprehensive user studies to obtain a more accurate evaluation.Quantitative Comparison. Tab.1 summarize the quantitative comparisons with competitive approaches on SD1.5 and SDXL. Generally, UniFL exhibits consistent performance improvement on both architectures and surpasses the existing methods of focus on improving generation quality or acceleration. Specifically, for the generation quality, UniFL surpasses both DreamShaper (DS) and ImageReward (IR) across all metrics, where the former relies on high-quality training images while the latter exploits the human preference for fine-tuning. It is also the case when compared with the recently proposed preference tuning method DPO. In terms of acceleration, UniFL also exhibits notable performance advantages, surpassing the LCM with the same 4-step inference on both SD1.5 and SDXL. Surprisingly, we found that UniFL sometimes obtained even better aesthetic quality with fewer inference steps. For example, when applied to SD1.5, the aesthetic score is first boosted from 5.26 to 5.54 without acceleration, and then further improved to 5.88 after being optimized by adversarial feedback learning. This demonstrates the superiority of our method in acceleration.Step We also compared the two latest acceleration methods on SDXL, including the SDXL Turbo and SDXL Lightning. Although retaining the high text-to-image alignment, we found that the image generated by SDXL Turbo tends to lack fidelity, leading to an inferior FID score. SDXL Lightning achieves the most balanced performance in all of these aspects and reaches impressive aesthetic quality in 4-step inference. However, UniFL still obtains slightly better performance on these metrics.User Study. We conducted a comprehensive user study using SDXL to evaluate the effectiveness of our method in enhancing generation quality and acceleration. As illustrated in Fig. 3 , our method significantly improves the original SDXL in terms of generation quality with a 68% preference rate and outperforms DreamShaper and DPO by 36% and 25% preference rate, respectively. Thanks to PeFL and decoupled aesthetic feedback learning, our method exhibits improvement even when compared to the competitive ImageReward, and is preferred by 17% additional people. In terms of acceleration, our method surpasses the widely used LCM by a substantial margin of 57% with 4-step inference. Even when compared to the latest acceleration methods like SDXL-Turbo and SDXL-Lightning, UniFL still demonstrates superiority and obtains more preference. This highlights the effectiveness of adversarial feedback learning in achieving acceleration.Qualitative Comparison. As shown in Fig. 4 , UniFL achieves superior generation results compared with other methods. For example, when compared to ImageReward, UniFL generates images that exhibit a more coherent object structure (e.g., the horse), and a more captivating aesthetic quality (e.g., the cocktail). Notably, even with fewer inference steps, UniFL consistently showcases higher generation quality, outperforming other methods. It is worth noting that SDXL-Turbo, due to its modification of the diffusion hypothesis, tends to produce images with a distinct style.A young girl playing with a frisbee in a grassy field surrounded by trees.A man riding on the back of an elephant in a city street.A person riding a skateboard on a ramp. A boy jumping off a spaceship.A … teddy bear sits on a child's bed.A close-up of a lady with sunglasses.A skydiver jumps from a plane, Hayao Miyazaki style.Three children on a couch, full shot.A ballerina executing a pirouette ... To validate the effectiveness of our design, we systematically remove one component at a time and conduct a user study. The results are summarized in Fig. 6 (a). In the subsequent sections, we will further analyze each component. More results are presented in the Appendix.As depicted in Fig. 5 (a), PeFL leverages the instance segmentation model to capture the overall structure of the generated object effectively. By identifying structural defects, such as the distorted limbs of the little girl, the broken elephant, and the missing skateboard, PeFL provides more precise feedback signals for diffusion models. Such fine-grained flaws can not be recognized well with ReFL due to its global and coarse preference feedback, instead, the exploited professional visual perception provides more detailed and targeted feedback. As presented in Fig. 5 (b), the PeFL significantly boosts the object structure generation (e.g. the woman's glasses, ballet dancer's legs). It is also demonstrated by the notable performance drop (71.9% vs 28.1%) when disabling the PeFL.Multiple aspects optimization with PeFL. PeFL exploits various perceptual models to improve some particular visual aspects of the diffusion model and can easily be extended to multi-aspect optimization. As illustrated in Fig. 8 , the simultaneous incorporation of two distinct optimization objectives (style and structure optimization) does not compromise the effectiveness of each other. Take the prompt a baby Swan, graffiti as an example, integrating the style optimization via PeFL upon the base model successfully aligns the image with the target style. Further integrating the structure optimization objective preserves the intended style while enhancing the overall structural details (e.g. the feet of the Swan).Necessity of decoupling design. We conducted an experiment that finetuned the SD1.5 using the same prompt set but a global aesthetic reward model trained with all dimensions' collected aesthetic preference data. As depicted in Fig. 6 (b), the generated images are more harmonious and have an artistic atmosphere with the decoupled aesthetic reward tuning and are preferred by more 17% individuals than the non-decoupled counterpart. This can be attributed to the ease of abstract aesthetic learning with the decoupling design. Moreover, it also can be found that aesthetic feedback learning with actively selected prompts leads to a higher preference rate (54.6% vs 45.4%) compared with the random prompts. Further analysis of the prompt selection can be found in the Appendix.B.2.Selection of hinge coefficient α d . We select the hinge coefficient for each aesthetic reward model based on their reward distributions on the validation set. As illustrated in Fig. 7 (left). there are clear margins in the reward scores between preferred and unpreferred samples. Moreover, such margin varies across these dimensions, emphasizing the necessity of the decoupled design. Empirically, we set α d to the average reward scores of the preferred samples to encourage the diffusion model to prioritize generating samples with higher reward scores. Fig. 7 (right) demonstrates that setting a small hinge for the "color" reward resulted in only minor improvement, while substantial coefficients led to image oversaturation. Optimal results were achieved by selecting a coefficient close to the average reward score of the preferred samples. A similar trend was observed for layout and lighting aesthetics from our experiments, except for the "detail" dimension. Interestingly, a slightly lower coefficient sufficed for satisfactory detail optimization, as a higher coefficient introduced more background noise. This could be attributed to the significant reward score difference between preferred and unpreferred samples, where a high coefficient could excessively guide the model toward the target reward dimension.Analysis on adversarial feedback learning. We analyzed the mechanism for the acceleration behind our adversarial feedback learning and found that (i) Adversarial training enables the reward model to provide guidance continuously. As shown in Fig. 9 (a), the diffusion model offer suffers rapid overfitting when frozen reward models, known as reward hacks. By employing adversarial feedback learning, the trainable reward model (acting as the discriminator) can swiftly adapt to the distribution shift of the diffusion model output, significantly mitigating the over-optimization phenomenon, and allowing the reward to provide effective guidance for a longer duration. (ii) Adversarial training expands the time step of feedback learning optimization. The adversarial objective poses a strong constraint to force high-noise timesteps to generate clearer images, which allows the samples across all denoising steps to be rewarded properly. As presented in Fig. 9 (b), when disabling the adversarial objective while retaining the full optimization timesteps during rewarding, the reward model fails to provide effective guidance for samples under fewer denoising steps due to the high-level noise, which Ablation on Acceleration Steps. We examine the acceleration capacity of UniFL under various inference steps, ranging from 1 to 8 as illustrated in Fig. 10 . Generally, UniFL performs exceptionally well with 2 to 8 inference steps with superior text-to-image alignment and higher aesthetic quality.The LCM method is prone to generate blurred images when using fewer inference steps and requires more steps (e.g., 8 steps) to produce satisfied images. However, both UniFL and LCM struggle to generate high-fidelity images with just 1-step inference, exhibiting a noticeable gap compared to SDXL-Turbo (e.g., the Labradoodle), which is intentionally designed and optimized for an extremely low-step inference regime. Therefore, there is still room for further exploration to enhance the acceleration capabilities of UniFL towards 1-step inference.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: All checkpoints and code to reproduce our results will be publicly available on our project page upon the acceptance of our paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We propose UniFL, a framework that enhances visual quality, aesthetic appeal, and inference efficiency for latent diffusion models from the unified feedback learning perspective. By incorporating perceptual, decoupled, and adversarial feedback learning, UniFL can be applied to various latent diffusion models, such as SD1.5 and SDXL, and exceeds existing methods in terms of both generation quality enhancement and inference acceleration.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Implementation details can be found in Sec.5.1. All checkpoints and code to reproduce our results would be publicly available on our project page as long as our paper is accepted.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The evaluation metrics adopted in this paper do not measure accuracy but visual quality, aesthetic appeal, and text-to-image alignment.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Implementation details can be found in Sec.5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This paper definitely does not violate any Code Of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Appendix.F.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [No] Justification: This paper involved the user study for image quality evaluation in Sec.5.2. The full text of the instructions given to participants consists of only one sentence: "Given the two images below, which one do you prefer?" Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The proposed perceptual feedback learning (PeFL) is highly flexible, allowing it to utilize different existing visual perception models to offer targeted visual quality feedback on particular aspects. To showcase the scalability of PeFL, we present two additional case studies where PeFL is employed to optimize style and layout generation.i) Style: To effectively capture image style and provide feedback on style generation, we utilize the VGG-16 [56] model to encode image features and extract visual style using the well-established gram matrix in style transfer. Furthermore, we have curated a substantial dataset of approximately 150,000 high-quality artist-style text images. We leverage this dataset to conduct PeFL for style optimization.The objective of the optimization can be formulated as follows:EQUATION)where V is the VGG network, and Gram is the calculation of the gram matrix. We validate the effect of PeFL in style optimization based on SD1.5 and SDXL. Note that due to the newly introduced artist-style dataset, we compare our method with the DMs fine-tuned with the same style dataset via pre-train loss to ensure a fair comparison. As depicted in Fig. 11 , the PeFL significantly boosts style generation (e.g. 'frescos', 'impasto' style), enabling the model to generate the image with a more aligned style compared with applied pre-train loss (MSE loss). We further conduct the quantitive experiment to evaluate the effectiveness of PeFL on style optimization. Specifically, we collect 90 prompts about style generation and generate 8 images for each prompt. Then, we manually statistic the rate of correctly responded generation to calculate the style response rate. As shown in Tab.2, it is clear that the style PeFL greatly boosts the style generation on both architectures thanks to the superior style feedback provided by the VGG extracted feature, especially for SD1.5 with about 15% improvement. In contrast, leveraging naive diffusion pre-train loss for fine-tuning with the same collected style dataset suffers limited improvement due to stylistic abstraction missing in latent space.A girl, frescos a family, impasto ii) Layout: Generally, the semantic segmentation map characterizes the overall layout of the image as shown in Fig. 12 (a) . Therefore, semantic segmentation models can serve as a better layout feedback provider. Specifically, we utilize the visual semantic segmentation model to execute semantic segmentation on the denoised image x ′ 0 to capture the current generated layout and supervise it with the ground truth segmentation mask and calculate semantic segmentation loss as the feedback on the layout generation:EQUATIONwhere m s represents the semantic segmentation model, GT(x 0 ) is the ground truth semantic segmentation annotation and the L semantic is the semantic segmentation loss depending on the specific semantic segmentation model. We conducted an experiment on PeFL layout optimization based on SD1.5. Specifically, we utilize the COCO Stuff [57] with semantic segmentation annotation as the semantic layout dataset and DeepLab-V3 [58] as the semantic segmentation model. The results are presented in Fig. 12 (b ). It demonstrates that the PeFL significantly improves the layout of the generated image, for instance, the bear on the bed in a diagonal layout. Note that here we focus on the objective layout generation that is explicitly mentioned in the prompts, for example, 'stands', 'overlooking', and 'sit at'. As a comparison, the layout reward model used in aesthetic feedback learning primarily emphasizes the subjective composition from the aesthetic angle, which may not be described clearly by the textual prompt. We further conduct the user study to evaluate the effectiveness of PeFL with the semantic segmentation model quantitatively. As shown in Fig. 13 (b), we are surprised to find that the image details also observed a significant boost in addition to the improvement in the layout generation. This probably stems from the dense per-pixel feedback from the semantic segmentation objective.SD1.5GT Mask Generated ImageA woman standing on a dock with a pink umbrella in her hand.Two people riding on the back of two horses in a green field.A man and a woman standing in front of a red and white flag, holding large scissors as if they are cutting the flag.. Indeed, PeFL is an incredibly versatile framework that can exploit a wide range of visual perceptual models, such as OCR models [59, 60] and edge detection models [61, 62] , to boost the performance of LDMs. Furthermore, we are actively delving into utilizing the visual foundation model, such as SAM [63] , which holds promising potential in addressing various visual limitations observed in current diffusion models.PeFL utilizes various visual perceptual models to provide visual feedback in specific dimensions to improve the visual generation quality on particular aspects. Different visual perceptual models of a certain dimension may have different impacts on the performance of PeFL. Taking the structure optimization of PeFL as an example, we investigated the impact of the accuracy of instance segmentation models on PeFL performance. Naturally, the higher the precision of the instance segmentation, the better the performance of structure optimization. To this end, we choose the Mask2Former [64] ,another representative instance segmentation model with state-of-the-art performance to achieve structure optimization with PeFL. The results are shown in Fig. 16 (a) and Fig. 13 (a) . It is intriguing to note that the utilization of a higher precision instance segmentation model does not yield significantly improved results in terms of performance. We speculate it lies in the different architectures of the instance segmentation of these two models. In SOLO [50] , the instance segmentation is formulated as a pixel-wise classification, where each pixel will be responsible for a particular instance or the background. Such dense supervision fashion enables the feedback signal to better cover the whole image during generation. In contrast, Mask2Former [64] takes the query-based instance segmentation paradigm, where only a sparse query is used to aggregate the instance-related feature and execute segmentation. This sparse nature of the query-based method makes the feedback insufficient and leads to inferior fine-tuning results. We leave further exploration of how to choose the most appropriate visual perceptual model for feedback tuning to future work. We utilize the SOLO instance segmentation model trained with close-set dataset COCO for PeFL structure optimization. One may be concerned that this will lead to limited concepts that PeFL can optimize (i.e. only the COCO concept). However, on the one hand, although we apply the SOLO instance segmentation model trained on the COCO dataset (80 categories), we observe that PeFL exhibits exceptional generalization capability and the generation performance of many concepts not shown in the COCO dataset is also boosted significantly as shown in Fig. 14 . We believe LDM can be guided to learn general and reasonable structure generation via PeFL optimization. On the other hand, the proposed perceptual feedback learning is a very flexible framework, and it is very straightforward to replace the close-set model SOLO with other open-set instance segmentation models such as Ground-SAM to achieve further improvement for the concepts in the wild(e.g. concepts in LAION dataset). We break down the general and coarse aesthetic concept into more specific dimensions including color, layout, detail, and lighting to ease the challenge of aesthetic fine-tuning. We then collect the human preference dataset along each dimension. Specifically, we employ the SDXL [40] as the base model and utilize the prompts from the MidJourney [3, 65] as input, generating two images for each prompt. Subsequently, we enlist the expertise of 4 to 5 professional annotators to assess and determine the superior image among the generated pair. Given the inherently subjective nature of the judgment process, we have adopted a voting approach to ascertain the final preference results for each prompt. Finally, we curate 30,000, 32,000, 30,000, and 30,000 data pairs for the color, layout, detail, and lighting dimensions, respectively. Examples of the collected aesthetic feedback data of different dimensions are visually presented in Fig. 15 .Prompt Selection Process. We introduce an active prompt selection strategy designed to choose the most informative and diverse prompts from a vast prompt database. The comprehensive implementation of this selection strategy is outlined in the Algorithm.2. Our strategy's primary objective is to select prompts that offer maximum information and diversity. To accomplish this, we have devised two key components: the Semantic-based Prompt Filter and the Nearest Neighbor Prompt Compression. The semantic-based prompt filter is designed to assess the semantic relationship embedded within the prompts and eliminate prompts that lack substantial information. To accomplish this, we utilize an existing scene graph parser2 as a tool to parse the grammatical components, such as the subjective and objective elements.The scene graph parser also generates various relationships associated with the subjective and objective, including attributes and actions. We then calculate the number of relationships for each subjective and objective and select the maximum number of relationships as the measurement of the information amount encoded in the prompt. A higher number of relationships indicates that the prompt contains more information. We filter out prompts that have fewer than τ 1 = 1 relationships, which discard the meaningless prompt like 'ff 0 0 0 0' to reduce the noise of the prompt set. Upon completing the filtration process, our next objective is to select a predetermined number of prompts that exhibit maximum diversity. To achieve this, we adopt an iterative process to achieve this objective. In each iteration, we randomly select a seed prompt and subsequently suppress its nearest neighbor 3 prompts that have a similarity greater than τ 2 = 0.8 as illustrated in Fig. 16 end if 24: end for 25: SP ← RandomSelect(P, S, N ) // randomly select N diverse prompts according the retained index 26: return SP next iteration commences with the remaining prompts, and we repeat this process until the similarity between the nearest neighbors of all prompts falls below the threshold τ 2 . Finally, we randomly select the prompts, adhering to the fixed number required for preference fine-tuning.Analysis of the active prompts. As illustrated in Fig. 6 (a) in the main paper, our strategically chosen prompts yield superior performance in aesthetic feedback learning. To further comprehend the advantage of this design, we present the training loss curve in Fig. 16 (c ), comparing the use of actively selected prompts versus random prompts. It clearly shows that the diffusion model rapidly overfits the guidance provided by the reward model when using the randomly selected prompts, ultimately resulting in the loss of effectiveness of the reward model quickly. One contributing factor to this phenomenon is the distribution of prompts for optimization. If the prompts are too closely distributed, the reward model is forced to frequently provide reward signals on similar data points, leading to the diffusion model rapidly overfitting and collapsing within a limited number of optimization steps. We statistic the average nearest embedding similarity (ANS) of the 100K prompts randomly selected from DiffusionDB [48] by calculating the cosine similarity between each prompt with its most similar prompt within the embedding space and taking the average over all the prompts. The ANS of the randomly selected prompts is approximately 0.89, which delivery highly redundant. As a comparison, the prompts selected by our strategy exhibit considerable diversity with ANS of 0.73, enabling a more balanced and broad reward calculation, which eases the over-fit significantly. Therefore, with the actively selected prompts, the diffusion model obtains a more comprehensive feedback signal and can be optimized toward human preference more efficiently.To further verify the generalization of UniFL, we performed downstream tasks including LoRA, and ControlNet Specifically, we selected several popular styles of LoRAs [7] , and several types of ControlNet [9] and inserted them into our models respectively to perform corresponding tasks. As shown in Fig. 17 , our model demonstrates excellent capabilities in style adaptation and controllable generation.We present more visual comparison between different methods in Fig. 18 . It demonstrates the superiority of UniFL in both the generation quality and the acceleration. In terms of generation quality, UniFL exhibits more details (e.g. the hands of the chimpanzee), more complete structure (e.g. Superhero, modeling in Venice, Italy, with a confident smile.A delicate porcelain teacup sits on a saucer, its surface adorned with intricate blue patterns. UniFL demonstrates promising results in generating high-quality images. However, there are several avenues for further improvement:Large Visual Perception Models: We are actively investigating the utilization of advanced large visual perception models to provide enhanced supervision.Extreme Acceleration: While the current 1-step model's performance may be relatively subpar, the notable success we have achieved in 4-step inference suggests that UniFL holds significant potential for exploration in one-step inference.Streamlining into a Single-stage Optimization: Exploring the possibility of simplifying our current two-stage optimization process into a more streamlined single-stage approach is a promising direction for further investigation.The proposed framework, UniFL, has the potential to have significant broader impacts in the field of image generation and related downstream applications. The improved visual quality of image generation achieved through UniFL can enhance various applications that rely on generated images, including computer graphics, virtual reality, and content creation. This can lead to the creation of more realistic and visually appealing virtual environments, improved visual effects in movies and video games, and better-quality generated content for digital media. However, it is also important to consider the potential ethical implications and societal impacts of advancements in image generation techniques. With the ability to generate highly realistic images, there is a risk of misuse or abuse, such as the creation of deepfake content for malicious purposes. Researchers, developers, and policymakers must be vigilant and consider the ethical implications of these advancements, promoting responsible use and raising awareness about the potential risks associated with synthetic media.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/vacancy/SceneGraphParserhttps://github.com/facebookresearch/faiss 


Decision: The paper initially received mixed reviews. The reviewers appreciated the idea of introducing a unified feedback learning framework capable of integrating various priors from perceptual models to enhance different diffusion models, as well as the extensive experiments conducted. However, they expressed concerns about the limited novelty compared to previous work like ReFL, the unconvincing performance improvements and qualitative results, and the lack of explanations for some key design choices, implementation details, ablation studies, and user studies.


The rebuttal partially addressed the reviewers’ concerns by providing additional explanations, experimental results, and discussions. After the rebuttal, Reviewers dHXQ, nwEj, and af1S rated the paper as borderline accept or weak accept, while Reviewer qtQN maintained a borderline reject score. During the subsequent discussion, Reviewers dHXQ, nwEj, and af1S acknowledged that the rebuttal had addressed their concerns. However, Reviewer qtQN did not participate in the discussion and did not update their final recommendation.


The AC has read the paper, reviews, and rebuttal, and discussed with the reviewers at length. The AC found the arguments of Reviewers dHXQ, nwEj, and af1S to be the most convincing, noting that the concerns raised by Reviewer qtQN were either addressed in the original submission or adequately explained in the rebuttal. As a result, the AC weighed the paper's strengths more heavily than its current weaknesses.


The authors are encouraged to improve the camera-ready version by following the reviewers' recommendations, particularly by including the new experimental results provided in the rebuttal, offering a more in-depth discussion of the differences from prior work, and adding further explanation and analysis. Such revisions would greatly enhance the work and significantly increase its impact. This decision was discussed with and approved by the AC and SAC.

Reviews: review: paper_summary: paper_summary: This paper proposes a framework to enhance visual quality, aesthetic appeal, and inference efficiency using various methods, including perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. Good experimental results are observed.
strength_weakness: strength_weakness: The author claims that previous works primarily focus on individual problems through specialized designs and proposes a unified framework. However, three different methods are designed to solve different problems. Why is it called unified?

In Line 145, it states that image content is incorporated as an additional condition for guidance. However, as shown in Figure 1, no extra inputs are added to the diffusion model.

In the experiment, only an instance segmentation model is used for perceptual feedback learning. Have the authors tried other types of models? It would be interesting to see some analysis on this aspect.

For decoupled feedback learning in Equation 5, why is a hinge loss used instead of a winner-loser loss as in Equation 4? What do the data annotations look like? How to choose the hyper-parameters of the hinge loss?

What are the details of the semantic-based prompt filter and nearest neighbor prompt compression described in Line 189 for active prompt selection?

In Line 202, it states that samples with low inference steps tend to be too noisy to obtain correct rewarding scores in previous methods. How does the proposed adversarial feedback learning improve this and how does it accelerate inference?
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: The work introduces Unified Feedback Learning (UniFL), a unified framework to enhance diffusion models through feedback learning. It addresses three main challenges in diffusion models: visual quality, aesthetic appeal, and inference efficiency. UniFL comprises perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. 
- Perceptual Feedback Learning utilizes existing perceptual models, such as an instance segmentation model, to fine-tune diffusion models on a specific aspect. 
- Decoupled Feedback Learning decomposes the general aesthetic concept into color, layout, lighting, and detail. It fine-tunes the model by reward feedback learning in all these sub-categories. 
- Adversarial Feedback Learning exploits a general reward model as a discriminator to improve the generation quality of fewer denoising steps.
Experiments demonstrate that UniFL improves the performance of diffusion models like SD1.5 and SDXL in terms of generation quality, aesthetic preference, and inference speed.
strength_weakness: strength_weakness: - The overall method is some modification from ReFL.
- The perceptual feedback learning part is limited by the selected models, which seem to only fine-tune concepts used to train these perceptual models. 
- The performance is not significantly superior to other methods according to Table 1.
questions: questions: - If users want to generate images containing concepts not shown in COCO dataset, but shown in LAION, it seems perceptual feedback learning part will not work. 
- If users want to fine-tune the diffusion model in several different perceptual aspects simultaneously, will the priors from different perceptual models interrupt each other? 
- Weaknesses above
limitations: limitations: Yes, the authors addressed the limitations.
paper_summary: paper_summary: Considering that current diffusion models still suffer from several limitations, this paper aims to propose a unified framework, UniFL to address the main existing challenges by applying feedback learning. To respectively solve the issues of visual distortion, poor aesthetic appeal, and inefficient inference, this paper demonstrates different sub-modules, namely Perceptual Feedback Learning, Decoupled Feedback Learning, and Adversarial Feedback Learning. By fully leveraging the ability of Perceptual Feedback Learning to fine-tune the diffusion during the two training stages, the proposed model aims to achieve both remarkable generation quality and speed simultaneously. Sufficient experiments, including extensive ablation studies for different proposed sub-modules, have proved the validation and effectiveness of the model’s design.
strength_weakness: strength_weakness: The proposed qualitative visualization comparison mainly focuses on justifying the overall generation style and the structure correctness. However, a main issue that may occur during the inference speed-up is detail loss. It will be more appreciated if the author can provide some further visual comparison results to demonstrate the model’s ability to keep the visualization results consistent with that adjective in the text prompt context.

Also, according to Figure 7, an unwelcomed issue can be found that the images generated by UniFL for text prompt A cream-colored labradoodle wearing glasses and a black beret teaching calculus at a blackboard mistakenly wears a black tie, which has never been mentioned in the text prompt. It shows an unexpected trend of overfitting and inconsistency. Further explanation about the cause of such results should be addressed.
questions: questions: Please see the weakness part.
limitations: limitations: Though there are still some blemishes part for some of the proposed results, generally the experiment part is well-organized, and filled with abundant ablation studies for the proposed modules.
paper_summary: paper_summary: This paper introduces UniFL, a novel approach for improving diffusion models through unified feedback learning. The objective of UniFL is to enhance visual generation quality, preference aesthetics, and inference acceleration. To achieve this, the paper proposes three key components: perceptual feedback learning, decoupled aesthetic feedback learning, and adversarial feedback learning. UniFL is designed to be a two-stage training pipeline that can be applied to various models and yields impressive improvements in both generation quality and acceleration. The paper provides experimental results demonstrating the effectiveness of UniFL in terms of generation quality and acceleration. Additionally, the paper discusses the potential broader impacts and ethical implications of advancements in image generation techniques.
strength_weakness: strength_weakness: 1. There are some minor LaTeX formatting issues: for example, there should be a consistent space before parentheses in abbreviations, the quotes in the Appendix should be implemented using `’, and “our” should be “Our” in line 130. 
2. The values of hyper-parameters such as \alpha_d are not explicitly stated in the paper.
questions: questions: 1. Have the authors explored the impact of different aesthetics on generated images?
2. Are the user studies enough to prove the effectiveness of each module in this field?
3. Why not compare with SDXL-IR and SDXL-DPO in Ablation on Acceleration Steps?
limitations: limitations: Yes.

Comments: title: Response to rebuttal
Kindly reminder to respond to author responses
comment: Thank you for providing the response. After reading it, I have decided to keep the original score.
Dear Reviewers,

As the discussion phase is ending in one day, it would be helpful if you could respond to the authors' rebuttal if you haven’t already. This would allow the authors to address any remaining concerns.

Thanks,

AC
Thank you for your rebuttal. After reviewing your rebuttal, it addressed most of my concerns. I tend to increase my rating to Borderline accept.
Thank you for your response. I have checked the rebuttal and tend to keep my score.
Dear Reviewers,

Thank you very much again for performing this extremely valuable service to the NeurIPS authors and organizers.

As the authors have provided detailed responses, it would be great if you could check them and see if your concerns have been addressed. Your prompt feedback would provide an opportunity for the authors to offer additional clarifications if needed.

Cheers,

AC