UniFL: Improve Latent Diffusion Model via Unified Feedback Learning
Authors: Jiacheng Zhang
Jie Wu
Yuxi Ren
Xin Xia
Huafeng Kuang
Pan Xie
Jiashi Li
Xuefeng Xiao
Weilin Huang
Shilei Wen
Lean Fu
Guanbin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Feedback Learning
Acceleration
Primary area: generative_models
TL;DR: A unified feedback learning framework to enhance the performance of text-to-image diffusion model

Abstract: Latent diffusion models (LDM) have revolutionized text-to-image generation, leading to the proliferation of various advanced models and diverse downstream applications. However, despite these significant advancements, current diffusion models still suffer from several limitations, including inferior visual quality, inadequate aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present **UniFL**, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL.
Notably, UniFL consists of three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which accelerates inference.
In-depth experiments and extensive user studies validate the superior performance of our method in enhancing generation quality and inference acceleration. For instance, UniFL surpasses ImageReward by 17\% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57\% and 20\% general preference with 4-step inference.

UniFL: Improve Latent Diffusion Model via Unified Feedback Learning
Jiacheng Zhang 12.1
Jie Wu
Yuxi Ren
Xin Xia² Huafeng Kuang²
Pan Xie
Jiashi Li
Xuefeng Xiao²
Weilin Huang
Shilei Wen?
Lean Fu² Guanbin Li
Sun Yat-sen University
Bytedance Inc
PPegg Cheng Laboratory
Project Page:
https://uni wal.anii...iguuul.i .it.tuu.iio//
teeeee Frrpo Feetbaok Learaing
Lecouppe FeodBbek varring
ViSuEA Quality
Aestheric
UniFL
puannd
Advernartal Feedback Learntest
Acceleration
Figure 1: Generated samples with 20 steps inference from stable tetee--ifffiooo weweeiionnxlbbaaee bbsee-ae-... O optimized by Unified Feedback Learning (UniFL). The last three images of the third row are generated with 4 steps.
Abstract
Latent diffusion models (LDM) have revolutionized text-to-image generation, leading to the proliferation of various advanced models and diverse downstream applications. However, despite these significant advancements, current diffusion models still suffer from several limitations, including inferior visual quality, inade- quate aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present UniFL, a unified framework that leverages feedback learning enhance diffusion models comprehensively. UniFL stands out as universal, effective, and generalizable solution applicable various diffusion models, such SD1.5 and SDXL. Notably, UniFL consists of three key components: perceptual feedback learning, which enhances visual quality; decou- pled feedback lcarning, which improves aesthetic appeal: and adversarial feedback learning, which accelerates inference. In-depth experiments and extensive user studies validate the superior performance of our method in enhancing generation quality and inference acceleration. For instance, UniFL surpasses ImageRewar by 17% preference terms of generation quality and outperforms LCM and SDXL Turbo by 57% and 20% general preference with 4-step inference.
*Corresponding authors: wujie 10558 gmail.com liguanbin @ www.il.yys..edu.coo project lead t: Equal Contribution. Work done during an internship at ByleDance.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The emergence of diffusion models has led remarkable advances in the field of text-to-image (T2I) generation, marked by notable milestones like DALLE-3 [I] Imagen [2]. Midjourney [3] etc, elevating the generation quality of images to an unprecedented level. Particularly, the introduction of open-source image generation models, exemplified by latent diffusion model (LDM) 44. has inaugurated a transformative era of text-to-image generation. triggering numerous downstream applications such as T2I personalization 6
7 8), controllable generation [9] 10; [1] and text-to- video (T2V) generation [12 [3] 14]. Nevertheless, despite these advancements achieved thus far, current latent diffusion-based image generation models still exhibit certain limitations. Inferior visual quality: The generated images still suffer from poor visual quality and lack authenticity. Examples include characters with incomplete limbs or distorted body parts, well limited fidelity in terms of style representation. ii) Inadequate aesthetic appeal: The generated image tends to lack aesthetic appeal and often fails align with human preferences, especially in the abstract aesthetic concepts aspects such as color, lighting, atmosphere, etc. iii) Slow inference speed: The iterative denoising process employed by diffusion models led to inefficiencies during inference that significantly impede generation speed, thereby limiting the practicality of these models in various application scenarios. Recently, numerous works have endeavored to address the aforementioned challenges. For instance, RAPHAEL [15] resorts to the techniques of Mixture of [8] boost the generation performance via stacking the space MoE. and time MoE block. Works [19
Experts) [16 17
22 2.3] represented by ImageReward [23 propose incorporating human preference feedback to guide diffusion models toward aligning with human preferences. SDXL Turbo (24, PGD 255, and LCM (26 (2). on the other hand, targets on achieve inference acceleration through techniques like distillation and consistency models (28). However, these methods primarily concentrate on tackling individual problems through specialized designs, which poses significant challenge to the elegant integration of these techniques. For example, MoE significantly complicates the pipeline, making the acceleration method infeasible apply, and the consistency models (28) alter the denoising process of the diffusion model, making it arduous to directly apply the ReFL preference tuning framework proposed by IeageReward [23]. Therefore, natural question arises: Can we devise a more effective approach that comprehensively enhances diffusion models in terms of image quality, aesthetic appearance, and generation speed?
20 21
To tackle this issue, we present UniFL, a solution that offers a comprehensive improvement to latent diffusion models through unified feedback learning formulation. UniFL aims to boost the visual generation quality, enhance aesthetic attractiveness, and accelerate the inference process. To achieve these objectives, features three novel designs upon the unified formulation of feedback learning. Firstly, we introduce a pioneering perceptual feedback learning (PeFL) framework that effectively harnesses the extensive knowledge embedded within diverse existing perceptual models to provide more precise and targeted feedback on the potential visual defects of the generated results. Secondly, employ decoupled aesthetic feedback learning to boost the visual appeal, which breaks down the coarse aesthetic concept into distinct aspects such as color, atmosphere, and texture, simplifying the challenge of abstract aesthetic optimization. Furthermore, introduced to choose the more informative and diverse prompt to facilitate more efficient aesthetics preference learning. Lastly, UniFL develops adversarial feedback learning to achieve inference acceleration by incorporating the adversarial objective in feedback tuning. We instantiate UniFL with two-stage training pipeline and validate its effectiveness with SD1.5 and SDXL. yielding impressive improvements in generation quality and acceleration. Our contributions are summarized as follows:
active prompt selection strategy is also
Our proposed method, UniFL, introduces a unified framework of feedback learning to optimize the visual quality, aestheties, and inference speed of diffusion models. To the best of our knowledge, UniFL offers the first attempt to address both generation quality and speed simultaneously, offering a fresh perspective in the field.
New Insight:
Novelty and Pioneering: In our work, we shed light on the untapped potential of leveraging existing perceptual models in feedback learning for diffusion models. We highlight the sig- nificance of decoupled reward models and elucidate the underlying acceleration mechanism through adversarial training.
Through extensive experiments, we demonstrate the substantial im- provements achieved by UniFl across various types of diffusion models, including SD1.5 and SDXL, in terms of generation quality and inference acceleration.
High Effectiveness:
2

---

2 Related Works
Text-to-Image Diffusion Models. Text-to-image generation has gained unprecedented attention over other traditional tasks [29 30 31
32 33. Recently, diffusion models have gained substantial attention and emerged the de facto mainstream method for text-to-image generation, surpassing traditional image generative models like GAN
34 and VAE 355 Numerous related works have been proposed, including GLIDE [36), DALL-E2 M. Imagen 22 Cog View [37] etc.. Among these, Latent Diffusion Models (LDM) 4 extend the diffusion process to the latent space and significantly improve the training and inference efficiency of the diffusion models, opening the door diverse applications such as controllable generation ( TO], image editing [T] 38 39) and image personalization [5] 66] and so on. Even though. current text-to-image diffusion models still have limitations in inferior visual generation quality, deviations from human aesthetic preferences, and inefficient inference. The target of this work is to offer a comprehensive solution to address these issues.
Improvements on Text-to-Image Diffusion Models. Given the aforementioned limitations, re- searchers have proposed various methods to tackle these issues. Notably, 440 15
41 focuses on
improving generation quality through more advanced training strategies. Inspired by the success of re-
inforcement learning with human feedback (RLHF) 442 43 in the field of LLM, (20 2
44 23 45
explore the incorporation of human feedback to improve image aesthetic quality. On the other hand,
22 24 28 27 26 concentrate on acceleration techniques, such distillation and consistency mod- els (28) to achieve inference acceleration. While these methods have demonstrated their effectiveness in addressing specific challenges, their independent nature makes it challenging
combine them for comprehensive improvements. In contrast, our study unifies the objective of enhancing visual quality, aligning with human aesthetic preferences, and acceleration through the feedback learning framework.
3 Preliminaries
Latent Diffusion Model. Text-to-image latent diffusion models leverage diffusion modeling to generate high-quality images based on textual prompts, which generate images from Gaussian noise through a gradual denoising process. During pre-training, sampled image is first processed by pre-trained VAE encoder to derive its latent representation 2. Subsequently. random noise is injected into the latent representation through forward diffusion process, following a predefined schedule {3,)". This process can be formulated as 2 = Va,z + v1 a,c, where c E NO, 1) is the random noise with identical dimension to z, a. = I a and = 1 3r. To achieve the denoising process, UNet Ep is trained to predict the added noise in the forward diffusion process, conditioned on the noised latent and the text prompt Formally, the optimization objective of the UNet is:
((0)= E, Er,,,,, Ille -oa.clllleeeeeeee co(va,z + vI a,e, c, +
Reward Feedback Learning. Reward feedback learning (ReFL) 23 is a preference fine-tuning framework that aims to improve the diffusion model via human preference feedback. It consists of two phases: (1) Reward Model Training and (2) Preference Fine- fine-tuning. In the Reward Model Training phase, human preference data is collected to train human preference reward model, which serves as a proxy to provide human preferences. More specifically, considering two candidate generations, denoted as The (preferred generation) and z (unpreferred one), the loss function for training the human preference reward model rg can be formulated as follows:
(m) ((c.:,.)))>((og((( (Te(c, re(c, 21))]
(2)
where DD denotes the collected feedback data, a(.) represents the sigmoid function, and corresponds to the text prompt. The reward model r0 is optimized to produce reward score that aligns with human preferences. In the Preference Fine-tuning phase, ReFI. begins with an input prompt initializing random latent variable ZT. The latent variable then progressively denoised until reaching randomly selected timestep t. Then, the denoised image z/j is directly predicted from zt- The reward model obtained from the previous phase is applied to this denoised image, generating the expected preference score ra (c, z)). ReFL maximizes such preference scores to make the diffusion model generate images that align more closely with human preferences:
(r) Se-ptc) B (ateip(((ggellrrree c)]
Our method follows a similar learning framework ReFL. but devises several novel components to enable comprehensive improvements.
3

---

Bockward Gradient
Trainable
Flow
Stage-1: 1: Performance Enhancement
Stage-2: Inference Acceleration
peroptaars ZEcAEaICe
Decrindaake
derse
picheiapeekke
LDM(O)
roward
pati
LDM(O)
Extt
(rrage cepton)
Perception Feedbaak Learning
(iange saginal
Feslbeck Leuening
(cobc. peompp)
mwwd sootle
Gastac prosgpt)
Oykesalax
Fucs
Trear
anthe Actiction
Dassher
Figure 2: Overview of UniFL. We leverage a unified feedback learning framework to enhance the model performance and inference speed comprehensively. The training process of UniFL is divided into two stages, the first stage aims to improve visual quality and aesthetics, and the second stage speeds up model inference.
4 UniFL: Unified Feedback Learning
Our proposed method, UniFL. aims to improve the latent diffusion models in various aspects. including visual generation quality, human aesthetic quality, and inference efficiency. our method takes unified feedback learning perspective, offering comprehensive and streamlined solution, An overview of UniFL. is illustrated in Fig 2 In the following subsections, we delve into the details of three key components: perceptual feedback learning to enhance visual generation quality (section decoupled feedback learning to improve aesthetic appeal (section 4.2 and adversarial feedback learning to facilitate inference acceleration (section 4.3)
4.1 Perceptual Feedback Learning
Current diffusion models exhibit limitations in achieving high-fidelity visual generation, for example, object structure distortion. These limitations stem from the reliance on reconstruction loss(MSE loss) solely in the latent space, which lacks structural supervision on the high-level visual quality. To address this issue, we propose perceptual feedback learning (PeFL). Our key insight is that various visual perception models already embed rich visual priors, which can be exploited to provide feedback for visual generation and fine-tune the diffusion model. The complete PeFL process is summarized in Algorithm [] In contrast ReFL, which starts from a randomly initialized latent representation and only considers the text prompt as a condition, PeFL incorporates image content as an additional visual condition for perceptual guidance. Specifically, given a text-image pair. (c. z). we first select forward step Ta and inject noise into the ground truth image to obtain conditional latent zi- - ZT.- Subsequently, we randomly select denoising time step and denoising from TT., yielding 2T, - IT,- -> Tt. Next, we directly predict - 2/6- By incorporating the visual condition input, the denoised image is expected to restore the same high-level visual characteristics, such as object structure, and style, which existing perception models can capture. For instance, in the case of object structure, the instance segmentation model can serve as a valuable resource as it provides essential descriptions of object structure through instance masks. Consequently, the feedback on the generation of such visual characteristics on zo can be obtained by comparing it with the ground truth segmentation mask via:
Apter (0) = Eng Beg-D,aa (Gr, Linstance (m)()))) GT(aa))
where my is the instance segmentation model, GT(x)) is the ground truth instance segmentation mask and Linstance is the instance segmentation loss. Note that our PeFL differs from ReFL as indicated by the red font in Algorithm [ With the visual condition input and perception model, the diffusion model is allowed to get a detailed and focused feedback signal on a specific aspect, instead of the general quality feedback offered by ReFL. Moreover, the flexibility of PeFL allows us to leverage various existing visual perceptual models, more examples can be found in the Appendix A
4.2 Decoupled Feedback Learning
Decoupled Aesthetic Fine-tuning. Existing text-to-image diffusion models exhibit shortcomings in images that satisfy human aesthetic preferences. While PeFL prioritizes objective visual quality,

---

aesthetic quality is inherently subjective and abstract, requiring human aesthetic feedback to steer the generation process. Despite ImageReward's attempt to incorporate human aesthetic preferences through a reward model, its performance is hindered by oversimplified modeling that fails to capture the multidimensional nature of human aesthetic preferences. Generally, humans consider the aesthetic attractiveness of an image from various aspects, such as color, lighting, etc, and conflating these aspects without distinguishing during preference tuning would encounter optimization conflicts as evidenced in [46). To address this issue, we follow (23 to achieve aesthetic preference tuning but suggest decoupling the various aesthetic aspects when constructing preference reward models. Specif- ically, we decomposed the general aesthetic concept into representative dimensions and collected the corresponding annotated data, respectively. These dimensions include color, layout, lighting, and detail. Subsequently, we train separate aesthetic preference reward model for each annotated data according to 2 Finally, we leveraged these reward models for aesthetic preference tuning:
K
Laes (0)
Σ E Be
Eug Bng-p(atle [eeLU(a- - -ra(z6: c))l
(5)
rd is the aesthetic reward model on d dimension, d {color, layout, detail, lighting), Oud is the dimension-aware hinge coefficient, and K is the number of fine-grained aesthetic dimension. Active Prompt Selection. We observed that when using randomly selected prompts for aesthetic preference fine-tuning, the dif- fusion model tends to rapidly overfit the reward model due the limited seman- tic richness, leading to diminished effec- tiveness of the reward model. To address this issue, we further propose an active prompt selection strategy, which selects the most informative and diverse prompt from a prompt database. This selection process involves two key components: a semantic-baat based prompt filter and nearest neighbor prompt compression. By lever- aging these techniques, the overfitting can be greatly mitigated, achieving more effi- cient aesthetic reward fine-tuning. More details of this strategy are presented in the Appendix|B:. B.2
Algorithm 1 Perceptual Feedback Learning (PeFL.
Dataset: Captioned perceptual text-image dataset with D {(xx1, img, ...tt,,,,,,,,,, img, }}
2: Input: LDM with pre-trained parameters wp. perceptual
model m.. perceptual loss function o, loss weight À Initialization: The number of noise scheduler time steps
T, add noise timestep Ta. denoising time step t.
4:
for perceptual data point (txt;, img,) E D do
5:
+- Vaeenccimgg)) // From image to latent 2T. <- AddNoise((r)) // Add noise to latent
6:
7:
for j = .a...., 1+ do
8:
no grad: 2T-1 + LDM {
9:
end for
10:
with grad: 2--1 LDM (x+)
11:
To &- Jt-1 // Predict the denoised latent
12:
img, + VaeDec(ro) // From latent to image
13:
Epen <- (mn(ing))) ). (im,) // PeFL loss by per-
ceptual model
14: W + w, // Update LDM using PeFL loss
15:
end for
4.3
Adversarial Feedback Learning
The inherent iterative denoising process of diffusion models significantly hinders their inference speed. To address this limitation, we introduce adversarial feedback learning to reduce the denoising steps during inference. Specifically, to achieve inference acceleration, we exploit general reward model r-) to improve the generation quality of fewer denoising steps. However, as studied in [3], the samples under low inference steps tend to be too noisy to obtain the correct rewarding scores. To tackle this problem, rather than freeze the reward model during fine-tuning, we incorporate an extra adversarial optimization objective by treating (+ as a discriminator and update it together with the diffusion model. Concretely, we follow a similar way with PeFLL to take an image input and execute the diffusion and denoising consecutively. Afterward, in addition to maximizing the reward score of the denoised image, we also update the reward model in an adversarial manner. The optimization objective is formulated as:
C (0) = EeNpte) E Reb c)],
(6)
L (6) -E (20.tt Deman [-11,
(ra (trt [log
+ log(1 (ra(<())))
where 0 and @ are the parameters of the diffusion model and discriminator. With the adversarial objective, the reward model is always aligned with the distribution of the denoised images with various denoised steps, enabling the reward model to function well across all the timesteps. Note that
our method is distinct from the existing adversarial diffusion methods like SDXL-Turbo 224. These
5

---

methods take the adversarial distillation manner to accelerate the inference, which tends to require another LDM as the teacher model to realize distillation, incurring considerable memory costs. By contrast, we follow the reward feedback learning formulation, which integrates adversarial training with the reward tuning and achieves the adversarial reward feedback tuning via the lightweight reward model.
4.4 Training Objective
We employ a two-stage training pipeline to implement UniFL. The first stage focuses on improving generation quality, leveraging perceptual feedback learning and decoupled feedback learning boost visual fidelity and aesthetic appeal. In the second stage, we apply adversarial feedback learning to accelerate the diffusion inference speed. To prevent potential degradation, also include decoupled feedback learning to maintain aesthetics. The training objectives of each stage are summarized as follows:
L (0) Spen (0) + Laes (0); (0, (0) (((()) + Lara
5
Experiments
5.1 Implementation Details and Metrics
Dataset. We utilized the COCO2017 47 train split dataset with instance annotations and captions for structure optimization with PeFL. Additionally, we collected the human preference dataset for the decoupled aesthetic feedback learning from diverse aspects (such as color, layout, detail, and lighting). 100,000 prompts are selected for aesthetic optimization from DiffusionDB [48) via active prompt selection. During the adversarial feedback learning, we use data from the aesthetic subset of LAION [49 with image aesthetic scores above 5.
Training Setting. We utilize the SOLO [50] as the instance segmentation model. We utilize the DDIM 51 scheduler with total of 20 inference steps. Ta 10 and the optimization steps E [o. 5] during PeFL training. For adversarial feedback learning, we initialize the adversarial reward model with the weight of the aesthetic preference reward model of details. During adversarial training, the optimization step is set to E [0, 20) encompassing the entire diffusion process. Our training per stage costs around 200 A100 GPU hours.
Baseline Models. We choose representative text-to-image diffusion models with distinct gen- eration capacities to comprehensively evaluate the effectiveness UniFL., including (i) SD1.5 [4]: (ii) SDXL
44) Based on these models, we pick up several state-of-the-art methods(i.e. Im- Dreamshaper [52). and DPO [22] for generation quality enhancement, LCM (27).
ageReward (23
SDXL-Turbo 24 and SDXL-Lightning (53) for inference acceleration) to compare the effectiveness
of quality improvement and acceleration. All results of these methods are reimplemented with the official code provided by the authors.
Evaluation Metrics, We generate the 5K image with the prompt from the COCO2017 validation split to report the Fréchet Inception Distance (FID) [54] as the overall visual quality metric. We also report the CLIP score with ViT-B-32 (55) and the aesthetic score with LAION aesthetic predictor to evaluate the text-to-image alignment and aesthetic quality of the generated images, respectively. Given the subjective nature quality evaluations, we further conducted comprehensive user studies obtain more accurate evaluation.
5.2 Main Results
Quantitative Comparison. Tabl summarize the quantitative comparisons with competitive ap- proaches on SD1.5 and SDXL. Generally, UniFL exhibits consistent performance improvement on both architectures and surpasses the existing methods of focus on improving generation quality or acceleration. Specifically, for the generation quality, UniFL surpasses both DreamShaper (DS) and lewae (IR) across all metrics, where the former relies on high-quality training images while the latter exploits the human preference for fine-tuning. is also the case when compared with the recently proposed preference tuning method DPO. In terms of acceleration, UniFL also exhibits notable performance advantages, surpassing the LCM with the same 4-step inference on both SD1.5 and SDXL. Surprisingly, found that UniFL sometimes obtained even better aesthetic quality

---

SDAL regelonrt
55 0%
99 0%
12 P%
63.0%
16.0%
24.0%
5000
66.00 0%
Dhiy B3RGUCM
21.0% 36.0%
044
64
Figure 3: User study about UniFL and other methods with 10 users the generation of 500 prompts in generation quality (left) and inference acceleration (right).
SDAL-
SUXL TmaageReward
SDXL UniFF
SDXL LCM(4 steps)
SDXL Turbo(4 steps)
SDXL Unit' L(4 steps)
SDXI.
DPO
high of pandlo riding
The punde wizurd but and reading book.
bloadly cocknn
Figure 4: Qualitive comparison of the generation results of different methods based SDXL.
with fewer inference steps. For example, when applied to SD1.5, the aesthetic score is first boosted from 5.26 to 5.54 without acceleration, and then further improved to 5.88 after being optimized by adversarial feedback learning. This demonstrates the superiority of our method in acceleration. We also compared the two latest acceleration methods on SDXL, including the SDXL Turbo and SDXL Lightning. Although retaining the high text-to-image alignment, we found that the image generated by SDXL Turbo tends to lack fidelity, leading an inferior FID score, SDXL Lightning achieves the most balanced perfor- mance in all of these aspects and reaches im- pressive aesthetic quality in 4-step inference. However, UniFL still obtains slightly better per- formance on these metrics.
Model
Step
FIDJ
CLIP Scoret
Aes Score"
SDI.5-Base
20
37.99
0.308
5.26
SD1.5-1R
2X
20
32.31
0.312
5.37
SDI.5-PP 22
20
32.83
0.308
5.22
SD1.5-mmFL
20
31.14
0.318
5,54
SDI.5-Bus
42,91
0.279
5.16
SD1.5.L.MM
42.65
0.314
5.71
SDI.5DDS LCM 26
35.48
0.314
5.58
SD1.5-UmiPL.
33.54
0.316
5,88
SDXL-Base
25
27.92
0,321
5.65
SDXL-IR
2H
25
26.71
0.319
5.81
SDXI
52
25
28.53
0.321
5.65
User Study. We conducted comprehensive user study using SDXL to evaluate the effec- tiveness of our method in enhancing generation quality and acceleration. As illustrated in Fig 3 our method significantly improves the original SDXL in terms of generation quality with 68% preference rate and outperforms DreamShaper and DPO by 36% and 25% preference rate, re- spectively. Thanks to PeFL and decoupled aesthetic feedback learning, our method exhibits improve- ment even when compared to the competitive ImageReward, and is preferred by 17% additional people. In terms of accelcration, our method surpasses the widely used LCM by a substantial margin of 57% with 4-step inference. Even when compared the latest acceleration methods like SDXL- Turbo and SDXL-Lightning UniFL still demonstrates superiority and obtains more preference. This highlights the effectivvnnes of adversarial feedback learning in achieving acceleration.
SDXL-DPO DPO 22
25
35.30
0.325
5.64
SDXL-
125.89
0,256
5.18
SDXL-LCM
27.23
0.322
5.48
SDXL Turbo
24
30.43
0.325
5.60
SDXL SDXI- Lighting
53
28.48
0.323
5.66
SDIL-UniFT.
26.25
0.325
5.87
Table I: Quantitative comparison between our method and other meth ods SD1.5 and SDXI. architecture. The best performance high- lighted with bold font, and the sccond-best is undertined.
Qualitative Comparison. As shown in Fig UniFL achieves superior generation results compared with other methods. For example, when compared to ImageReward, UniFL generates images that exhibit a more coherent object structure (e.g., the horse), and more captivating aesthetic quality (e.g., the cocktail). Notably, even with fewer inference steps, UniFL consistently showcases higher generation quality, outperforming other methods. It is worth noting that SDXL.-Turbo, due to its modification of the diffusion hypothesis, tends to produce images with distinct style.
7

---

BT Image
Ganerated Imsaee Prodicted
40 1.5 Stracture
SIXT PoFt Structure
GT Image
GT
Gemertee Imegg
young girl playing frihhe in gransy surmundad trees.
close-p afa lady sing noees.
evacating ginurett
riding dlpphnntin
soddy on
persuz ridap 20 cazp.
boy jumping spaceship,
jumps plase.
(a)
Hayae Nijozaki style.
Figure 5: (a) Illustration of PeFL with instance segmentation model (SOLO). (b) Effect of PeFL on structure optimization.
Decouple
Decouple
UniFL (20-Step)
Adv (4-Step)
52
47.4%
UniFL
PeFl, (20.Step)
(20-Step)
71.9%
28.1%
A majestic bon stands proudly rock
UniFL (20-Stop)
wio Decouple (20-Step)
UniFL
w/o
(20-Step)
54.6%
45.4%
(20-Step)
Prefenence Rahath)
close-up of iady with sunglasses
Figure (a) Design components ablation of UniFL. (b) Visualization of decoupled and non-decoupled aesthetic feedback learning results.
5.3 Ablation Study
To validate the effectiveness of our design, we systematically remove one component at time and conduct study. The results summarized in Fig.6 (a). In the subsequent sections, will further analyze each component. More results are presented the Appendix.
Superiority of PeFL. As depicted in Fig 5 (a), PeFL leverages the instance segmentation model to capture the overall structure of the generated object effectively. By identifying structural defects, such the distorted limbs of the little girl, the broken elephant, and the missing skateboard, PeFL provides more precise feedback signals for diffusion models. Such fine- fine-grained flaws can not be recognized well with ReFL due to its global and coarse preference feedback, instead, the exploited professional visual perception provides more detailed and targeted feedback. As presented Fig 5 (b), the PeFL significantly boosts the object structure generation (e.g. the woman's glasses, ballet dancer's legs). It is also demonstrated by the notable performance drop (71.9% vs 28.1 28.1%) 1%) when disabling the PeFL.
Multiple Aspects Optimization with PeFL. PeFL exploits various perceptual models improve some particular visual aspects of the diffusion model and can easily be extended to multi-aspect optimization. As illustrated in Fig S the simultancous incorporation of two distinct optimization objectives (style and structure optimization) does not compromise the effectiveness of cach other. Take the prompt a baby Swan, graffiti as an example, integrating the style optimization via PeFL upon the base model successfully aligns the image with the target style. Further integrating the structure optimization objective preserves the intended style while enhancing the overall structural details (e.g. the feet of the Swan).
Necessity Decoupling Design. We conducted an experiment that finetuned the SD1.5 using same prompt set but global aesthetic reward model trained with all dimensions' collected aesthetic preference data. depicted Fig (b), the generated images are more harmonious and have an artistic atmosphere with the decoupled aesthetic reward tuning and are preferred by more 17%

---

8.0
SDXL
Feler
Foder
delik parciii Noap Air sincer in nefice shhnaa wilh lAVisw Mis portens
Rightng
Rpoed
Penal
Figure 7: Analysis on the ozd. Left: reward scores distribution on 5k validation preference image pairs with our final chosen values highlighted. Right: ablation on the 0gd on color and detail reward.
SDXL
StrecEREE
& PEELSON
SDXI
PARE
PAPT SISTE
BARE SInESS
Figure 8: Incorporating the style and structure optimization objectives simultancously with PeFL results in no effectiveness degeneration of each other.
individuals than the non-decoupled counterpart. This can be attributed to the ease of abstract aesthetic learning with the decoupling design. Moreover, it also can be found that aesthetic feedback learning with actively selected prompts leads to a higher preference rate (54.6% vs 45.4%) compared with the random prompts. Further analysis of the prompt selection can be found in the Appendix [.2
Selection of Hinge Coefficient Otd- We select the hinge coefficient for each aesthetic reward model based on their reward distributions on the validation set. As illustrated in Fig (left). there are clear margins in the reward scores between preferred and unpreferred samples. Moreover, such margin varies across these dimensions, emphasizing the necessity of the decoupled design. Empirically, we Oia to the average reward scores of the preferred samples encourage the diffusion model to prioritize generating samples with higher reward scores. Fig 7 (right) demonstrates that setting a small hinge for the "color" reward resulted in only minor improvement, while substantial coefficients led to image oversaturation. Optimal results were achieved by selecting a coefficient close to the average reward score of the preferred samples. A similar trend was observed for layout and lighting aesthetics from our experiments, except for the "detail" dimension. Interestingly, a slightly lower coefficient sufficed for satisfactory detail optimization, as a higher coefficient introduced more background noise. This could be attributed to the significant reward score difference between preferred and unpreferred samples, where a high coefficient could excessively guide the model toward the target reward dimension.
Analysis on Adversarial Feedback Learning. We analyzed the mechanism for the acceleration behind our adversarial feedback learning and found that (i) Adversarial training enables the reward model to provide guidance continuously. As shown in Fig 9 (a), the diffusion model offer suffers rapid overfitting when frozen reward models, known as reward hacks. By employing adversarial feedback learning, the trainable reward model (acting as the discriminator) can swiftly adapt to the distribution shift of the diffusion model output, significantly mitigating the wver-optimization phenomenon, and allowing the reward to provide effective guidance for a longer duration. (ii) Adversarial training expands the time step of feedback learning optimization. The adversarial objective poses strong constraint to force high-noise timesteps to generate clearer images, which allows the samples across all denoising steps to be rewarded properly. As presented in (ig objective while retaining the full optimization timesteps during rewarding, the reward model fails to provide effective guidance for samples under fewer denoising steps due to the high-level noise, which
(b), when disabling the adversarial

---

Training Steps
Training Steps
4-Step Inference
A majestic lion stands proudly rock, overlooking the vast African savannah Its golden mone is flying in the wind, bathed in the warm colors of the sunset
A young badger delicutely sniffing a yellow rose
street (a)
A close-up of lady with sunglasses
(b)
Figure 9: Analysis of the benefits of adversarial training. (a) It enables longer optimization time for the reward model. (b) enables the image under low denoising steps to be rewarded correctly. The red rectangle means incorporating the adversarial training objective.
UniF1
SDXL
LiniFI.
25 Stops
S Steps
Siess
Steps
Staps
25 Stepe
k Stepe
4
Steps
Steps
Steps
O
O
SOXI. 25 Sizpe
ICM 8 Steps
S0XI.
CCM 8 Steps
Skgge
2 Staps
|| Suppe
25 Steps
4 Suqpe
Stege
Stepe
NOXL-Turbe
SOAL
SDXL-Tarbo
25 Supe
Steps
Skope
2 Steps
I Supps
25 Stope
Skaps
Steps
Stepe
Supe
photoo of light futb in muber spare moviiig the golory with sailing Noot Jesole the right buil.
Drcov colorel Selrudsedic wouring plasses and black dener Staeling colcutas at Rachhand
Figure 10: Ablation on different inference steps of UniFL.
results in poor generation results. With these two benefits, adversarial feedback learning significantly improves the generation quality of samples in lower inference steps and achieves superior acceleration performance ultimately. Notably, as shown in Fig (a), the image generated with 4-step inference retains similar visual quality with 20-step inference (52.5% 47.4% after going through the second stage training, which demonstrates the superiority of UniFL in acceleration.
Ablation on Acceleration Steps. We examine the acceleration capacity of UniFL under various inference steps, ranging from to as illustrated Fi Generally, UniFL performs exceptionally well with 2 to 8 inference steps with superior text-to-image alignment and higher aesthetic quality. The LCM method is prone to generate blurred images when using fewer inference steps and requires more steps (c.g., 8 steps) to produce satisfied images. However, both UniFL and LCM struggle to generate high-fidelity images with just 1 1-step inference, exhibiting a noticeable gap compared to SDXL-TTurbo (e.g., the Labradoodle). which is intentionally designed and optimized for extremely low-step inference regime. Therefore, there still room for further exploration to enhance the acceleration capabilities of UniFL towards 1-step inference.
Conclusion
We propose UniFL, framework that enhances visual quality, aesthetic appeal, and inference effi- ciency latent diffusion models from the unified feedback learning perspective. By incorporating perceptual, decoupled, and adversarial feedback learning, UniFL. can be applied to various latent diffusion models, such as SD1.5 and SDXL, and exceeds existing methods terms of both generation quality enhancement and inference acceleration.
10

---

Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (NO. 62322608). in part by the CAAI-MindSpore Open Fund, developed on OpenI Community, in part by the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No. VRLAB2023A01).
References
[1] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical eext-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 2022.
[2] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic texl-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[3] Iulia Ture and Gaurav Nemade. Midjourney user prompts & generated images (250k), 2022.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[5] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-drive generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[6] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashmik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023.
[7] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, International Conference on Learning Representations (ICLR), 2022.
[8] Hu Ye, Jun Zhang. Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint (Rciv:2308.0672) 2023.
[9] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[10] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingho Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon. Yun Fu, and Ran Xu. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[11] Chenlin Meng, Yutong He, Yang Song. Jiaming Song, Jiajun Wa, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representaaionss (ICLR), 2022.
[12] Jay Zhangjie Wu, Yixiao Ge. Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[13] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. In European Conference on Computer Vision (ECCV), 2024.
[14] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations (ICLR), 2024.
[15] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text- to-image generation via large mixture of diffusion paths. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[16] Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoe Lc, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated -fixture-of-expeee layer. In International Conference on Learning Representations (ICLR), 2017.
[17] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Le, and James Laudon. Mixture-of-exae experts with expert choice routing. In Advances in Neural Information Processing Systems (NeurIPS), 2022.

---

[18] William Fedus. Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research (JMLR), 2022.
[19] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow. Rui Pan. Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research (TMLR), 2023.
[20] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[21] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid henchmark for evaluating human preferences text-to-image synthesis. arXiv 2023.
preprint arXiv:2306.0933
[22] Bram Wallace, Meihua Dang, Rafael Rafailov, Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2024.
[23] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems (NeurlPS), 2023.
[24] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint 2rXiv:2311.17042, 2023.
[25] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022.
[26] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang. Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint 2023.
arXiv:2311.0SS.
[27] Simian Luo, Yigin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resouution images with few-step inference. arXiv preprint arXiv:2310.03378, 2023.
[28] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning (ICML), 2023.
[29] Jiaming Li, Xiangru Lin, Wei Zhang. Xiao Tan, Yingying Li, Junyu Han, Errui Ding, Jingdong Wang. and Guanbin Li. Gradient- sampling for class imbalanced supervise object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[30] Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, and Guanbin Li. Learning background prompts to discover implicit knowledge for open vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
[31] Jiacheng Zhang, Xiangru Lin, Wei Zhang, Kuo Wang, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang. and Guanbin Li, Semi-detr: Semi-supervised object detection with detection transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ( (CVPR), 2023.
[32] Jiacheng Zhang. Xiangru Lin, Minyue Jiang. Yue Yu, Chenting Gong. Wei Zhang, Xiao Tan, Yingying Li, Errui Ding. and Guanbin Li. A multi- nuuti-granularity retrieval system for natural language-based vehicle retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022.
[33] Jiacheng Zhang, Jiaming Li, Xiangru Lin, Wei Zhang, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang. and Guanbin Li. Decoupled pseudo-labeling for semi-supervised monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[34] Ian J. Gondfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurlPS), 2014.
[35] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.60 2022.
[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning (ICML), 2022.
[37] Ming Ding, Zhuoyi Yang. Wenyi Hong. Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. and Jie Tang. Cogview: Mastering text-to-inat eext-to-imaee generation via transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
12

---

[38]
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022.
[39] Tim Brooks, Aleksander Holynski, and Alexci A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[40] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint X1v1:2307.019555 2023.
[41] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations (ICLR), 2023.
[42] Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishiin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[43] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk. Nelson Elhage. Zac Hatfield-Dodde
Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravee, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 005622, 2022.
[44] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference Learning Representations (ICLR), 2024.
[45] Miroslav Strupl, Francesco Faccio, Dylan R. Ashley, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. weight-weighted regression converges to global optimum. arXiv preprint arXiv:2107.0900 2022.
[46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Praijwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie- Anne Lachaux. Thibaut Lavril. Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kamhadur, Sharan Narang, Aurelien Rodriguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (i0i::2377.09888 2023.
[47] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. arXiv preprint arXiv: 14055 02.55.33222222 2015.
[48] Zijie J. Wang, Evan Montoya. David Munechika, Haoyang Yang. Benjamin Hoover, and Duen Horng Chau. Diffusiondb: large-scale prompt gallery dataset for text-to-image generative models. In Association for Computational Linquisticsbookiitte (ACL), 2023.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems (NeurIPS) Track Datasets and Benchmarks, 2022.
[50] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Solo: simple framework for instance segmentation. European Conference on Computer Vision (ECCV), 2020.
[51] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.
[52] Civitai. Dreamshaper v8. 2024.
[53] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightmng: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 130.2999... 2024.
[54] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
13

---

[55] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers image recognition scale. In International Conference on Learning Representations (ICLR), 2021.
[56] Karen Simonyan and Andrew Zisseman. Very deep convolutional networks for large-scale image recogni- tion. International Conference on Learning Representations (ICLR), 2015.
[57] Holger Cacsar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[58] Liang Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[59] Minghui Liao, Zhanyi Wan, Cong Yao, Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In The Association for the Advancement of Artificial Intelligence (AAAT), 2019.
[60] Shancheng Fang. Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[61] John Canny. A computational approach to edge detection. IEEE Transactions Pattenn Analysis and Machine Intelligence (TPAMI). 1986.
[62] Saining Xie and Zhuowen Tu. Holistically- edge detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2015.
[63] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Ven Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[64] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[65] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang. Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. In Advances in Neural Information Processing Systems (NeurIPS) Track Datasets and Benchmarks, 2023.
14

---

A Extend Details of Perceptual Feedback Learning
A.1 Additional Examples of PeFL
The proposed perceptual feedback learning (PeFL.) is highly flexible, allowing it to utilize different existing visual perception models to offer targeted visual quality feedback on particular aspects. To showcase the scalability of PeFL, we present two additional case studies where PeFL is employed to optimize style and layout generation.
i) Style: To effectively capture image style and provide feedback on style generation, we utilize the VGG-16 [56] model to encode image features and extract visual style using the well-established gram matrix in style transfer. Furthermore, we have curated a substantial dataset of approximately 150,000 high-quality artist-style text images. We leverage this dataset to conduct PeFl. for style optimization. The objective of the optimization can be formulated as follows:
ppell (0) Bzo-Dan (aaoiaootrGGaaaaaa |/raarra (V(z)))
Gram V (zoo))le..
(8)
where V is the VGG network, and Gram the calculation of the gram matrix. We validate the effect of PeFL in style optimization based on SD1.5 and SDXL. Note that due the newly introduced artist-style dataset, we compare our method with the DMs fine-tuned with the same style dataset via pre-train loss ensure fair comparison. As depicted in Fig IT the PeFL significantly boosts style generation (e.g. 'frescos", 'impasto' style), enabling the model to generate the image with a more aligned style compared with applied pre-train loss (MSE loss). We further conduct the quantitive experiment to evaluate the effectiveness of PeFL on style optimization. Specifically, we collect 90 prompts about style generation and generate 8 images for each prompt. Then, we manually statistic the rate of correctly responded generation to calculate the style response rate. As shown in Tab 2 is clear that the style PeFL. greatly boosts the style generation on both architectures thanks to superior style feedback provided by the VGG extracted feature, especially for SD1.5 with about 15% improvement. contrast, leveraging naive diffusion pre-train loss for fine-tuning with the same collected style dataset suffers limited improvement due to stylistic abstraction missing in latent space.
Style Response Rate
Model
SDL.5
30.55%
it Sianly Impanto
SDI. 5 Style Pretrain
35.25%
Style
45.14%
SDXL
66.67%
SDXL Style Pretrain
68.34 %
SDXL + Style PeFL
75.27%
A Agol,
Woreon sennnigg divse ricurion
Table 2: Quantitive performance of PeFL in style generation.
Figure 11: Style optimization of PeFL on SD1.5 and SDXL.
ii) Layout: Generally, the semantic segmentation map characterizes the overall layout of the image as shown 12 (a). Therefore, semantic segmentation models can serve as better layout feedback provider. Specifically, we utilize the visual semantic segmentation model to execute semantic segmentation on the denoised image si to capture the current generated layout and supervise it with the ground truth segmentation mask and calculate semantic segmentation loss as the feedback on the layout generation:
layout (0) = Beo-D, GG(t, C GT(zo)) pefi
where m, represents the semantic segmentation model, GT(zu) is the ground truth semantic seg- mentation annotation and the Leemantic is the semantic segmentation loss depending on the specific semantic segmentation model. We conducted an experiment on PeFL layout optimization based on SD1.5. Specifically, we utilize the COCO Stuff [57] with semantic segmentation annotation as the semantic layout dataset and DeepL.ab-V 58 the semantic segmentation model. The results are presented in Fig 12 (b). demonstrates that the PeFL significantly improves the layout of the generated image, for instance, the bear on the bed in diagonal layout. Note that here
15

---

focus on the objective layout generation that is explicitly mentioned in the prompts, for example, "stands", "overlooking', and 'sit at'. As a comparison, the layout reward model used in aesthetic feedback learning primarily emphasizes the subjective composition from the aesthetic angle, which may not described clearly by the textual prompt. We further conduct the user study evaluate the effectiveness of PeFI. with the semantic segmentation model quantitatively. As shown in Fig [3] (b), we are surprised to find that the image details also observed significant boost in addition to the improvement in the layout generation. This probably stems from the dense per-pixel feedback from the semantic segmentation objective.
Generated Emage Mark
GT Mark
Genrrated Image
Protrain Sex
A uvmon stundling alank wilh pick umbrelle her hand.
4 Roe stands proudly rock, overlooking vast African sovannah
Too people riding nwo grees fiold
ruening shoes door
man wouson standling foont of red white flog, holding large scisnors they are cuttiing the flag.
photo soft, plush fodly Diagonal compostioon
(a)
(b)
Figure 12: (a) The illustration of the PeFL on the layout optimization. The semantic segmentation
model captures the layout and rex-to-image misalignment between the ground truth image and the generated image (DeepLab- V3 [58) is taken the segmentation model). (b) The layout optimization effect of the PeFL. with semantic segmentation model on SD1.5.
Indeed, PeFL is an incredibly versatile framework that can exploit wide range of visual perceptual models, such as OCR models [59 60 and edge detection models [61 62] to boost the performance of LDMs. Furthermore,
actively delving into utilizing the visual foundation model, such as SAM 663] which holds promising potential in addressing various visual limitations observed in current diffusion models.
PoFUSOLO) ReFuMMall2Fomerr
8%
28.2%
Image
PeFLISOOI vs SD1
52.5%
47.5%
Image Layouts
58 4%
6%
20
Preference
Preference Rate (%)
(b)
Figure 13: (a) The study results the ablation of different instance segmentation models in PeFL during structure optimization. PeFL (SOLO): PeFL fine-tune SD1.5 with SOLO as the instance segmentation model. PcFL (Mask2Former): PeFL fine-tune SD1.5 with Mask2Former the instance segmentation model. (b) The user study results on the effect of PeFL layout optimization. Dark Green: SD1.5 with PeFL, Light Green: SDI.5 without PeFL.
A.2 Ablation on Visual Perceptual Model
PeFL utilizes various visual perceptual models to provide visual feedback in specific dimensions to improve the visual generation quality on particular aspects. Different visual perceptual models of certain dimension may have different impacts on the performance of PeFL. Taking the structure optimization of PeFI. as an example, we investigated the impact of the accuracy of instance segmen- tation models on PcFL performance. Naturally, the higher the precision of the instance segmentation,
16

---

girl riding an ostrich
robot kicking ball
paranoid android freaking jumping the oir because is surrounded colorfal Easter eggs
tennis with three yellow
SDXL
PeFL Structure(SOLO)
SDXL
PeFL Structure(SOLO)
Figure 14: Generalization of PeFL with SOLO. The generation of the concepts not included in COCO (e.g. ostrich, robot, cones) is also improved after PeFL. optimization.
the better the performance of structure optimization. To this end, we choose the Fask2Former (64] another representative instance segmentation model with state-of-the-e performance achieve structure optimization with PeFL. The results are shown in Fig (6 (a) and Fig (a). is intriguing to note that the utilization of higher precision instance segmentation model does not yield significantly improved results in terms of performance. We speculate it lies in the different architectures of the instance segmentation of these two models. In SOLO [50], the instance segmentation is formulated as pixel-wise classification, where each pixel will be responsible for a particular instance or the background. Such dense supervision fashion enables the feedback signal better cover the whole image during generation. In contrast, Mask2Former [64] takes the query-based instance segmentation paradigm, where only a sparse query is used to aggregate the instance-related feature and execute segmentation. This sparse nature of the query-based method makes the feedback insufficient and leads to inferior fine-tuning results. We leave further exploration how to choose the most appropriate visual perceptual model for feedback tuning to future work.
A.3
Generalization of PeFL with Close-set Perceptual models
We utilize the SOLO instance segmentation model trained with close-set dataset COCO for PeFL structure optimization. One may be concerned that this will lead to limited concepts that PeFL can optimize (i.e. only the COCO concept). However, the one hand, although we apply the SOLO instance segmentation model trained on the COCO dataset (80 categories), we observe that PeFL exhibits exceptional generalization capability and the generation performance of many concepts shown in the COCO dataset is also boosted significantly as shown in Fig [4] We believe LDM can be guided to learn general and reasonable structure generation via PeFL optimization. On the other hand, the proposed perceptual feedback learning is very flexible framework, and it is very straightforward to replace the close-set model SOLO with other open-set instance segmentation models such as Ground-SAMM to achieve further improvement for the concepts in the wild(c.g. concepts in LAION dataset).
B Extend Details of Decoupled Feedback Learning
B.1 Aesthetic Preference Data Collection
We break down the general and coarse aesthetic concept into more specific dimensions including color, layout, detail, and lighting to ease the challenge of aesthetic fine-tuning. We then collect the human preference dataset along each dimension. Specifically, we employ the SDXL. 40 as the base model and utilize the prompts from the MidJournyy [3] 65) input, generating two images for
17

---

Prefereed
Preferee
Preferred
Preferred nefice solle suinting Krnyer
4 foual shrligh
cossing of
of zurooud a Maackpeinte gous bask and whit colevig jugc scrilits, masay poster
nigg 1856
Caler Festbuck
Preferred
Preferred
Preferee
Preferred
myyol avenue ferllan, illaruted art Ans Sprocud, recliote
Lanso passowan tedagroms Aelognastir
MRignoning oen kastial
yellon, Bylisi aininaaiaiic Riing
bizzore weirdenne, surmunios
fluntng z1t bonzy ware yole gradhear has hnvkpoend foettal
Layoat Fredback Duta
Lighting Feedlbakk Data
Figure 15: Decoupled aesthetic feedback data examples. The preferred samples are highlighted with red rectangles.
each prompt. Subsequently, we enlist the expertise determine the superior image among the generated pair. Given the inherently subjective nature of judgment process, we have adopted a voting approach to ascertain the final preference results for each prompt. Finally, we curate 30,000, 32,000, 30,000, and 30,000 data pairs for the color, layout, detail, and lighting dimensions, respectively. Examples of the collected aesthetic feedback data of different dimensions are visually presented in Fig
professional annotators assess and
B.2 Active Prompt Selection
Prompt Selection Process. We introduce an active prompt selection strategy designed to choose the most informative and diverse prompts from vast prompt database. The comprehensive imple- mentation of this selection strategy is outlined in the Algorithm 2 Our strategy's primary objective to select prompts that offer maximum information and diversity. To accomplish this, we have devised two key components: the Semantic-based Prompt Filter and the Nearest Neighbor Prompt Compression. The semantic-based prompt filter is designed to assess the semantic relationship embedded within the prompts and eliminate prompts that lack substantial information. To accomplish this, we utilize an existing scene graph parse as a tool to parse grammatical components, such as the subjective and objective elements. The scene graph parser also generates various relationships associated with the subjective and objective, including attributes and actions. We then calculate the number of relationships for each subjective and objective and select the maximum number of relationships as the measurement of the information amount encoded in the prompt. A higher number of relationships indicates that the prompt contains more information. We filter out prompts that have fewer than 7i = relationships, which discard the meaningless prompt like 'ff O 0 Of to reduce the noise of the prompt set. Upon completing the filtration process, next objective is to select a predetermined number of prompts that exhibit maximum diversity. To achieve this, we adopt an iterative process achieve this objective. In each iteration, we randomly select a seed prompt and subsequently suppress its nearest neighbo[]promm prompts that have similarity greater than 72 = 0.8 as illustrated in Fig 16 (b). The next iteration commences with the remaining prompts, and we repeat this process until the similarity between the nearest neighbors of all prompts falls below the threshold 72. Finally, we randomly select the prompts, adhering to the fixed number required for preference fine-tuning.
Analysis of the Actively Selected Prompts. As illustrated in Fig (a) in the main paper, strategically chosen prompts yield superior performance in aesthetic feedback learning. To further comprehend the advantage of this design, we present the training loss curve in Fig (c), comparing the use of actively selected prompts versus random prompts. It clearly shows that the diffusion model rapidly overfits the guidance provided by the reward model when using the randomly selected prompts, ultimately resulting the loss of effectiveness of the reward model quickly. One contributing
18

---

Algorithm 2 Active prompt selection for decoupled aesthetic feedback learning
Input: Initial prompt database: D; number of desired prompts: N;; T1 and 72: relation and similarity threshold.
14:
for index 7, E do
15:
if not R[[.]] and 1; not in S then
16:
S f- 1; // append the selected prompt
Output: The selected prompt set: SP
17:
dist, inds Dists[[4), Inds[4]] // K- nearest neighbor similarities for index d, E inds do if dist[d,] > T2 then
1: P=
2: Semantic-bass
Prompt Filter
18:
3: for Pı E D do
19:
4: SR <- Semantic(arser((p))
20:
R|d,] = True
5: if [SR] тi then
21:
end if 22: end for
6: P s Pi choose informative prompt
7: end if
23: end if
8: end for
24: end for
9: # Nearest Neighbor Prompt Compression
25: SP RandomSelect(P, S, N) randomly
10: I < shufflc(ragge(ln() PI)))
select N diverse prompts according the re- tained index
<- False set the removed prompt array 12: S + Ø // set the selected prompt index Dist, Inds +- KNN(R, k) // K-nearest neigh- bor of each prompt
26:
return SP
factor to this phenomenon is the distribution of prompts for optimization. the prompts are too closely distributed, the reward model forced frequently provide reward signals on similar data points, leading to the diffusion model rapidly overfitting and collapsing within limited number of optimization steps. We statistic the average nearest embedding similarity (ANS) of the 100K prompts randomly selected from DiffusionDB 488 by calculating the cosine similarity between each prompt with its most similar prompt within the embedding space and taking the average over all the prompts. The ANS of the randomly selected prompts is approximately 0.89, which delivery highly redundant. As comparison, the prompts selected by our strategy exhibit considerable diversity with ANS of 0.73, enabling more balanced and broad reward calculation, which eases the over-fit significantly. Therefore, with the actively selected prompts, the diffusion model obtains more comprehensive feedback signal and can be optimized toward human preference more efficiently.
C Generalization Study
To further verify the generalization UniFL,, we performed downstream tasks including LoRA, and ControlNet Specifically, we selected several popular styles of LoRAs (7]. and several types of ControlNet 9 and inserted them into our models respectively perform corresponding tasks. As shown in Fig 17 our model demonstrates excellent capabilities in style adaptation and controllable generation.
D More Visualization Results
We present more visual comparison between different methods in Fig It demonstrates the superiority of UniFL in both the generation quality and the acceleration. In terms of generation quality, UniFL exhibits more details (e.g. the hands of the chimpanzee), more complete structure (e.g. dragon), and more aesthetic generation (e.g. the baby sloth and the giraffe) compared with DPO and Rewae. In terms of acceleration, the I.C tends to generate a blurred image, while the SDXL Turbo generates the image with an unpreferred style and layout. As comparison, UniFL still retains the high aesthetic detail and structure under the 4-step inference.
E Discussion and Limitations
Uni demonstrates promising results in generating high-quality images. However, there are several avenues for further improvement:
19

---

SDI.3 + Fell.MMaa2Former))
SDL.5 PeFL(SOL.O)
SD1.5
K-nearest Neighbors
A delicate porcelain teucup sits surface
adorned with intricate blue patterss.
(b)
0.175
Resthelic fime-fuee LOSS
0.150
Superhero, modeling in Venice, Italy, confudent smile.
0.125
0.0
0.000
An owl perches quietly on neisted branch deep within ancient forest.
2000
4000
6000
8000
10000
lteration
(a)
(c)
Figure 16: (a) The visual comparison between the PeFL structure optimization with different instance segmentation models. (b) Illustration of Nearest Neighbor Prompt Compression. (c) Training loss curve when utilizing different prompts for decoupled aesthetic feedback learning.
Large Visual Perception Models: We are actively investigating the utilization of advanced large visual perception models provide enhanced supervision.
Extreme Acceleration: While the current 1-step model's performance may relatively subpar, notable success we have achieved in 4-step inference suggests that UniFL holds significant potential for exploration one-step inference.
Streamlining into Single-stage Optimization: Exploring the possibility of simplifying our current two-stage optimization process into more streamlined single-stage approach is promising direction for further investigation.
F Broader Impact
The proposed framework, UniFL, has the potential to have significant broader impacts in the field of image generation and related downstream applications. The improved visual quality of image generation achieved through UniFL can enhance various applications that rely on generated images, including computer graphics, virtual reality, and content creation. This can lead to the creation of more realistic and visually appealing virtual environments, improved visual effects in movies and video games, and better-quality generated content for digital media. However, it is also important to consider the potential ethical implications and societal impacts advancements in image generation techniques. With the ability generate highly realistic images, there risk of misuse abuse, such as the creation of deepfake content for malicious purposes. Researchers, developers, and policymakers must be vigilant and consider the ethical implications of these advancements, promoting responsible use and raising awareness about the potential risks associated with synthetic media.
20

---

MaXin
Anriva Lineart
Noiseaffset
Pixe XL
Dressed animal
(UniFL SD1.5 LoRA)
(UniFL SDXL LoRA)
houve stande cur parhod by (UniFL SD1.5 ControlNet)
handsome male model with
Tskir RriherDD edmcudV2
ParchartXL
(UniFL SDXL 4-Step LoRA)
Figure 17: Both SDI.5 and SDXL still keep high adaptation ability after being enhanced by the UniFL, even after being accelerated and inference with fewer denoising steps.
SDXI- +
SDXL
SDXL UniFL
SDXL. LCM(4
SDXI Turbot4 steps)
SDXI. steps)
SDXI.
ImageReward
chimpaazee wearing and playing piano
dragon
lattin
fluffi baby sloth with knitted hat prying to figure out a laptop, clase up, highly detuiled, sudio Highting
17.
giraffe made turtle
old
Figure 18: More visual comparison with different methods. UniFL yields better prompt-aligned and visually appealing results than other methods. Our results are highlighted with red rectangles.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count toward the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided the appendix. If you answer [Yes] question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answerr, and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and the introduction we elaborate on the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
The limitation of this paper is well discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The details of our proposed PeFL and active prompt selection are clearly illustrated in the [ Algorithm respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details can be found in See 5.1
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All checkpoints and code to reproduce our results will be publicly available on our project page upon the acceptance of our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee////eeeubbiiiiin
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details can be found in S.c All checkpoints and code to reproduce our results would be publicly available on our project page long as our paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The evaluation metrics adopted in this paper do not measure accuracy but visual quality, aesthetic appeal, and text-to-image alignment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Implementation details can be found in See 5.1
25

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This paper definitely does not violate any Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
26

---

Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used this paper are explicitly mentioned.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com ddtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification; All checkpoints and code reproduce our results will be publicly available on our project page when our paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
27

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification:
This paper involved the user study for image quality evaluation in Sec 5.2 The
full text of the instructions given participants consists of only one sentence: "Given the two images below, which one do you prefer?"
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
This paper does not have potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "x9vwnKy0vw", "reviewer": "Reviewer_qtQN", "summary": "This paper proposes a framework to enhance visual quality, aesthetic appeal, and inference efficiency using various methods, including perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. Good experimental results are observed.", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The concept of perceptual feedback learning is promising, as it can leverage various pretrained expert models to improve learning.\nThe idea of decoupled feedback learning makes sense, as it allows the model to focus on fine-grained details in the images.", "weaknesses": "The author claims that previous works primarily focus on individual problems through specialized designs and proposes a unified framework. However, three different methods are designed to solve different problems. Why is it called unified?\n\nIn Line 145, it states that image content is incorporated as an additional condition for guidance. However, as shown in Figure 1, no extra inputs are added to the diffusion model.\n\nIn the experiment, only an instance segmentation model is used for perceptual feedback learning. Have the authors tried other types of models? It would be interesting to see some analysis on this aspect.\n\nFor decoupled feedback learning in Equation 5, why is a hinge loss used instead of a winner-loser loss as in Equation 4? What do the data annotations look like? How to choose the hyper-parameters of the hinge loss?\n\nWhat are the details of the semantic-based prompt filter and nearest neighbor prompt compression described in Line 189 for active prompt selection?\n\nIn Line 202, it states that samples with low inference steps tend to be too noisy to obtain correct rewarding scores in previous methods. How does the proposed adversarial feedback learning improve this and how does it accelerate inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about the design of perceptual feedback learning and decoupled feedback learning in our method. We would like to answer the proposed questions in the following:\n\n1. **Clarification on unified design**:  We claim our method is a unified design as all the modules are seamlessly integrated under the unified feedback learning framework. Specifically, under this framework, we developed different reward mechanisms (namely perceptual reward, decoupled reward, and adversarial reward), which are effectively combined to mitigate the problems of low fidelity, poor appeal, and inefficient inference of existing LDMs simultaneously at the first time via the **_unified rewarding then fine-tuning paradigm_**. In contrast, existing works exhibit high heterogeneity in the way(e.g. changing network structure, or altering the sampling scheduler) to address these defects of LDM, making it challenging for these methods to be well combined to achieve comprehensive improvement. \n2. **Additional image condition**: We emphasize that _the introduced additional condition of PeFL refers to the image content incorporated into the noise initialization_, instead of the extra control image input like in ControlNet. Specifically, we highlight the perceptual feedback learning(PeFL) with additional image input in Fig.1(left) via the **blue solid line**. As depicted in Fig.1, this image is first injected noise after encoded(the injection process is not illustrated in Fig.1 for clarity, but detailed in Algo.1), then sent to the diffusion model for denoising, the denoised image is then supervised by the perceptual label (instance/semantic seg mask) of the input image. As a comparison, below the input image for PeFL, the decoupled feedback learning starts with pure noise as indicated in the yellow solid line.\n3. **More instantiation of PeFL**: We have presented two more case studies of perceptual feedback learning with other types of perceptual models, including the semantic segmentation model(i.e. DeepLab-V3) and the style extraction model(i.e. VGG-16) in the Appendix attached to our main paper. The results(Fig.8 and Tab.2 for style, Fig.9 and Fig.10 for layout) demonstrate that PeFL can effectively exploit the prior image knowledge embedded in these expert models to boost the performance of style generation(increasing the style response rate) and semantic layout optimization(more preferred layout).\n4. **Loss formulation in Eq.5**: We choose the hinge loss instead of the winner-loser loss in Eq.5, as _it describes the reward-tuning procedure, wherein there are no winner and loser samples for pair-wise loss calculation_. Specifically, in this process, a sample is generated from the pure noise given a prompt and then scored via the already well-trained reward models. Eq.5 then encourages the generated image to gain a higher reward score, and the hinge coefficient helps to avoid the unbounded increasing reward, preventing the LDM overfit to the reward model. The winner-loser loss is only used in the reward model training stage, where the reward model is trained to increase the score of the preferred sample while lowering the unpreferred one, aligning with the human preference.\n5. **Preference data annotations**: We provide some examples of the collected annotated preference data in Sec.B.1(Fig.11) in the Appendix of our main paper.\n6. **More details of active prompt selection**: We provide more details of the semantic-based prompt filter and nearest neighbor prompt compression during active prompt selection in Sec. B.2 in the Appendix of the main paper. In short, the semantic-based prompt filter selects the prompts owning rich semantic meaning by the ruled-based filtering, while the nearest neighbor prompt compression is designed to suppress the redundant prompts by checking the semantic similarity in the embedding space to ensure diversity.\n7.  **Selection of hinge coefficient $\\alpha_d$**: Please refer to the Author Rebuttal part and Fig.4  of the global rebuttal PDF for more details.\n8. **Analysis on Adavasarial Feedback Learning**: We provide a detailed analysis of the effect of adversarial feedback learning on acceleration in our ablation study section (L316). In summary, incorporating adversarial feedback plays two roles: (1) With the diffusion model and the adversarial reward model updated adversarially, the diffusion model is not prone to overfitting. The synchronously updated discriminator forces the diffusion model to evolve continuously, enjoying the reward guidance for a longer duration; (2) The denoised images under the low inference steps are forced to be clearer via the strong adversarial objective and thus can be correctly rewarded by the aesthetic reward model. Given these two benefits, the generation quality of low-step images steadily improved via the reward guidance, achieving inference acceleration.\n\nWe hope our rebuttal can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lvOWiVfeUj", "reviewer": "Reviewer_nwEj", "summary": "The work introduces Unified Feedback Learning (UniFL), a unified framework to enhance diffusion models through feedback learning. It addresses three main challenges in diffusion models: visual quality, aesthetic appeal, and inference efficiency. UniFL comprises perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. \n- Perceptual Feedback Learning utilizes existing perceptual models, such as an instance segmentation model, to fine-tune diffusion models on a specific aspect. \n- Decoupled Feedback Learning decomposes the general aesthetic concept into color, layout, lighting, and detail. It fine-tunes the model by reward feedback learning in all these sub-categories. \n- Adversarial Feedback Learning exploits a general reward model as a discriminator to improve the generation quality of fewer denoising steps.\nExperiments demonstrate that UniFL improves the performance of diffusion models like SD1.5 and SDXL in terms of generation quality, aesthetic preference, and inference speed.", "questions": "- If users want to generate images containing concepts not shown in COCO dataset, but shown in LAION, it seems perceptual feedback learning part will not work. \n- If users want to fine-tune the diffusion model in several different perceptual aspects simultaneously, will the priors from different perceptual models interrupt each other? \n- Weaknesses above", "limitations": "Yes, the authors addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clear and easy to follow\n- The proposed method utilizes different priors from other perceptual models to improve diffusion models. \n- New aesthetic human feedback dataset is proposed.", "weaknesses": "- The overall method is some modification from ReFL.\n- The perceptual feedback learning part is limited by the selected models, which seem to only fine-tune concepts used to train these perceptual models. \n- The performance is not significantly superior to other methods according to Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our writing and contribution to the feedback dataset curation. We would like to answer the proposed questions in the following:\n\n1. **Comparison with ReFL**: There are significant differences between our method and ReFL. Specifically: 1) Given the input prompt, ReFL starts with pure noise to obtain the denoised image and then imposes the reward scoring. This limits their usage to the reward models pre-trained on the collected preference data for the reward guidance. In contrast, _we incorporate an extra image condition for noise initialization and use the perceptual label of this condition image for feedback supervision on the denoised image_. Such formulation allows us to leverage the prior knowledge of a wide range of existing perceptual models to perform feedback fine-tuning. 2) In ReFL, a single coarse-grained preference reward model is applied, which may encounter the rewarding conflict, resulting in insufficient fine-tuning. As a comparison,  _we develop multiple decoupled aesthetic reward models, which enable more effective fine-grained aesthetic preference alignment_. 3) A core finding exhibited in ReFL is that it can only be applied to later denoising time steps during fine-tuning with the reward model as the reward model cannot reward correctly in the early steps. By contrast,  _we introduce adversarial feedback supervision, which effectively removes this limitation on optimization steps, allowing us to apply feedback fine-tuning to any denoised time step_, thereby achieving model inference acceleration. Overall, these novel designs make our approach distinct from ReFL and yield a more flexible and effective feedback fine-tuning framework for LDM.\n2. **Fine-tune concepts limitation in PeFL**:  On the one hand, although we apply the SOLO instance segmentation model trained on the COCO dataset(80 categories), we observe that PeFL exhibits exceptional generalization capability and the generation performance of many concepts not shown in the COCO dataset is also boosted significantly as shown in Fig.2 of the global rebuttal PDF. We believe that the LDM can be guided to learn general and reasonable structure generation via PeFL optimization. On the other hand, the proposed perceptual feedback learning is a very general module, and it is very straightforward to replace the close-set model SOLO with other open-set instance segmentation models such as Ground-SAM to achieve further improvement for the concepts in the wild(e.g. concepts in LAION dataset).\n3. **About the performance in Tab.1**: We argue that our method demonstrates notable improvements compared to other approaches as outlined in Tab.1.  It is essential to emphasize that different metrics possess unique properties, and _absolute enhancement should not be the sole criterion for evaluation_. For instance, the inherent limitations in the fine text-to-image alignment of the CLIP model make it challenging to achieve significant advancements in CLIP scores as it cannot capture the nuanced content change. Consequently, even with tailored text-to-image alignment feedback fine-tuning, ImageReward achieves a mere 0.004 CLIP score improvement with SD1.5. Therefore, _it is more reasonable to asses the relative improvements_.  Specifically, UniFL obtains 2$\\times$ CLIP score improvement over SD1.5 base(0.005 _vs_ 0.01) than the second best method(i.e. SD15-DS), which showcases the superiority of our method. There is also a similar case of other metrics, in which UniFL obtains relatively more notable improvement upon the base model. Moreover, our method also displays an obvious advantage over other methods by the extensive user study. \n4. **About multiple perceptual aspects optimization with PeFL**:  We show the results of multi-aspect optimization on style and structure through PeFL in Fig.1 of the global rebuttal PDF. As can be seen, incorporating these two distinct objectives does not hurt the effectiveness of each other. Take the prompt ``a baby Swan, graffiti`` as an example, integrating the style optimization via PeFL upon the base model successfully aligns the image with the target style. Further integrating the structure optimization objective retains the correct style and exhibits more complete structural details(e.g. the feet of the Swan) simultaneously.\n\nWe hope our rebuttal can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqqWTXwsVy", "reviewer": "Reviewer_dHXQ", "summary": "Considering that current diffusion models still suffer from several limitations, this paper aims to propose a unified framework, UniFL to address the main existing challenges by applying feedback learning. To respectively solve the issues of visual distortion, poor aesthetic appeal, and inefficient inference, this paper demonstrates different sub-modules, namely Perceptual Feedback Learning, Decoupled Feedback Learning, and Adversarial Feedback Learning. By fully leveraging the ability of Perceptual Feedback Learning to fine-tune the diffusion during the two training stages, the proposed model aims to achieve both remarkable generation quality and speed simultaneously. Sufficient experiments, including extensive ablation studies for different proposed sub-modules, have proved the validation and effectiveness of the model’s design.", "questions": "Please see the weakness part.", "limitations": "Though there are still some blemishes part for some of the proposed results, generally the experiment part is well-organized, and filled with abundant ablation studies for the proposed modules.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea about a combination of feedback learning and diffusion models is well introduced with necessary background information and is sufficiently motivated. \n\nThe three main issues remaining for diffusion, namely inferior quality, lack of aesthetics, and slow inference speed are well addressed, keeping the following methodology sections’ logic pipeline clear and understandable. \n\nThe proof of the methodology part is adequately explained with abundant support of pipeline figures and pseudocode that help the reader get full access to the novelty insight. \n\nSufficient experiments over different domains, especially the wide range of ablation studies, have been carried out validly which demonstrate the effectiveness of all the proposed feedback learning modules, which enhances persuasiveness.", "weaknesses": "The proposed qualitative visualization comparison mainly focuses on justifying the overall generation style and the structure correctness. However, a main issue that may occur during the inference speed-up is detail loss. It will be more appreciated if the author can provide some further visual comparison results to demonstrate the model’s ability to keep the visualization results consistent with that adjective in the text prompt context.\n\nAlso, according to Figure 7, an unwelcomed issue can be found that the images generated by UniFL for text prompt A cream-colored labradoodle wearing glasses and a black beret teaching calculus at a blackboard mistakenly wears a black tie, which has never been mentioned in the text prompt. It shows an unexpected trend of overfitting and inconsistency. Further explanation about the cause of such results should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our well-motivated method, sufficient experiments, and good writing. We would like to answer the proposed questions in the following:\n1. **Visualization on T2I alignment after acceleration**:  We visualize the text-to-image(T2I) alignment performance of SDXL accelerated via our method on the widely used benchmark prompt set - DrawBench, and provide some results in Fig.3 of the global rebuttal PDF. It clearly demonstrates that, even with a reduced number of inference steps, our approach maintains superior text-image consistency under various circumstances after acceleration, including attribute binding, object counting, and counterfactual prompt inputs.\n2. **Explanation of the unexpected generation**: It should be noted that although it is not what the user expected,  there is nothing wrong with the model to generate this black tie, as the input prompt does not explicitly specify that a tie is undesired and all the stuff mentioned in the prompt(e.g. cream-colored labradoodle, glasses, black beret. etc) has already correctly generated. In other words, it is free for the model to generate additional items in addition to the input prompt specified. One possible reason for the model to generate the black tie in this case is the data bias residing in the training samples. That is, there are vast training examples of teachers lecturing at a blackboard, where most teachers in these examples all wear ties. Such co-occurrence bias makes the model easily generate an extra tie when comes to the character in front of the blackboard.  A possible solution is to input an additional negative prompt to suppress such undesired behavior.\n\nWe hope our rebuttal can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFIwE7XUUu", "reviewer": "Reviewer_af1S", "summary": "This paper introduces UniFL, a novel approach for improving diffusion models through unified feedback learning. The objective of UniFL is to enhance visual generation quality, preference aesthetics, and inference acceleration. To achieve this, the paper proposes three key components: perceptual feedback learning, decoupled aesthetic feedback learning, and adversarial feedback learning. UniFL is designed to be a two-stage training pipeline that can be applied to various models and yields impressive improvements in both generation quality and acceleration. The paper provides experimental results demonstrating the effectiveness of UniFL in terms of generation quality and acceleration. Additionally, the paper discusses the potential broader impacts and ethical implications of advancements in image generation techniques.", "questions": "1. Have the authors explored the impact of different aesthetics on generated images?\n2. Are the user studies enough to prove the effectiveness of each module in this field?\n3. Why not compare with SDXL-IR and SDXL-DPO in Ablation on Acceleration Steps?", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing of this article is commendable, with a well-structured format.\n2. The author conducted a large number of visualization experiments to illustrate the focus and effectiveness of the method, which is very attractive.\n3. The proposed method serves as a plug-and-play indeed improves the performance on both SD15 and SDXL.", "weaknesses": "1. There are some minor LaTeX formatting issues: for example, there should be a consistent space before parentheses in abbreviations, the quotes in the Appendix should be implemented using `’, and “our” should be “Our” in line 130. \n2. The values of hyper-parameters such as \\alpha_d are not explicitly stated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words about our good writing, sufficient experiments, and the effectiveness of our method. We would like to answer the proposed questions in the following:\n\n1. **Minor Typos**:  Thanks for the suggestion, we will modify these places in our revised version.\n2. **Selection of hinge coefficient $\\alpha_d$**: Please refer to the Author Rebuttal part and Fig.4 of the global rebuttal PDF for more details.\n3. **Impact of different aesthetics during generation**: The impact of different aesthetic reward models in the generated images can be clearly observed in Fig.3 in our main paper. As can be seen, with the integration of color aesthetic rewards, the colors of the images generated by UniFL are more harmonious and natural. This enhancement is particularly pronounced in the acceleration scenario, where LCM and Turbo exhibit color deterioration characterized by darker, grayer, and vaguer, contrasting with our method's maintenance of a lively color palette. The effects of the detail and layout reward model are also notable. For instance, when considering the prompt `A bloody may cocktail`,  the image generated by UniFL showcases intricate background details while maintaining a well-balanced layout with the foreground, resulting in a visually appealing composition. Furthermore, the lighting aesthetic reward ensures the lighting with an atmosphere in the image, as opposed to the flat center background lighting observed in alternative methods.\n4. **Effectiveness of User Study**: Due to the subjective nature of the image quality judgment, the most effective way to evaluate the performance of a model in the context of text-to-image generation so far is still the user evaluation. This is also the standard practice of various classical text-to-image works such as SDXL[1], SDXL-Turbo[2], etc. Moreover, to increase reliability, we involve a considerable number of users(10 users) in our quality evaluation study, which we believe can validate our conclusion relatively accurately.\n5. **Acceleration ablation with DPO/ImageReward**: We did not choose to compare the accelerated version of our method with the SDXL-IR and SDXL-DPO as _they are not the methods tailored for inference acceleration instead of generation quality optimization_. There is no few-steps optimized version of SDXL of ImageReward and DPO for fair comparison with our method of inference acceleration. Even though, the unaccelerated version SDXL optimized via our approach still displays more superior image quality than these methods as evidenced by our experiments.\n\nWe hope our rebuttal can address your concerns.\n\n[1] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis\n\n[2] SDXL-Turbo: Adversarial Diffusion Distillation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their constructive comments. We are delighted that the core contributions of our proposed method are regarded as promising and effective (R-qtQN, R-dHXQ, R-af1S), novel (R-dHXQ), the paper is well-motivated (R-dHXQ) and well-written (R-nwEj, R-dHXQ, R-af1S). All the comments will be addressed in the revised paper. We would like to answer some common questions in the following:

**R-qtQN, R-af1S: Selection of hinge coefficient $\alpha_d$**: We customized the hinge coefficient for each aesthetic reward model based on their reward distributions on the validation set. As illustrated in Fig.4(left) of the global rebuttal PDF,  there are clear margins in the reward scores between preferred and unpreferred samples. Moreover, such margin varies across these dimensions, which emphasizes the importance of the decoupled design. Empirically, we set the reward hinge coefficient to the average reward scores of the preferred samples to encourage the diffusion model to generate the sample with higher reward scores. Taking the reward model on color as an example, we ablate such coefficient selection. As depicted in Fig. 4(right) of the global rebuttal PDF, setting $\alpha_{color}$ too small resulted in marginal improvement due to limited guidance, while excessively large hinge coefficients led to over-saturation in images. Conversely, selecting a coefficient around the average reward score of the preferred samples yielded optimal results. A similar trend was observed in the layout and lighting aesthetic dimensions, except for the detail dimension. Notably, a slightly lower $\alpha_{detail}$ sufficed to achieve satisfactory results for the detail reward model, whereas a higher coefficient tended to introduce more background noise. This phenomenon is likely attributed to the substantial reward score margin between preferred and unpreferred samples for detail dimension, where a high coefficient could lead to overwhelming guidance toward the target reward dimension.

We hope our rebuttal can address the concerns about these questions.

Meta-review: The paper initially received mixed reviews. The reviewers appreciated the idea of introducing a unified feedback learning framework capable of integrating various priors from perceptual models to enhance different diffusion models, as well as the extensive experiments conducted. However, they expressed concerns about the limited novelty compared to previous work like ReFL, the unconvincing performance improvements and qualitative results, and the lack of explanations for some key design choices, implementation details, ablation studies, and user studies.


The rebuttal partially addressed the reviewers’ concerns by providing additional explanations, experimental results, and discussions. After the rebuttal, Reviewers dHXQ, nwEj, and af1S rated the paper as borderline accept or weak accept, while Reviewer qtQN maintained a borderline reject score. During the subsequent discussion, Reviewers dHXQ, nwEj, and af1S acknowledged that the rebuttal had addressed their concerns. However, Reviewer qtQN did not participate in the discussion and did not update their final recommendation.


The AC has read the paper, reviews, and rebuttal, and discussed with the reviewers at length. The AC found the arguments of Reviewers dHXQ, nwEj, and af1S to be the most convincing, noting that the concerns raised by Reviewer qtQN were either addressed in the original submission or adequately explained in the rebuttal. As a result, the AC weighed the paper's strengths more heavily than its current weaknesses.


The authors are encouraged to improve the camera-ready version by following the reviewers' recommendations, particularly by including the new experimental results provided in the rebuttal, offering a more in-depth discussion of the differences from prior work, and adding further explanation and analysis. Such revisions would greatly enhance the work and significantly increase its impact. This decision was discussed with and approved by the AC and SAC.
Decision: Accept (poster)