Title: UniFL: Improve Latent Diffusion Model via Unified Feedback Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Model

Feedback Learning

Acceleration
Abstract: Latent diffusion models (LDM) have revolutionized text-to-image generation, leading to the proliferation of various advanced models and diverse downstream applications. However, despite these significant advancements, current diffusion models still suffer from several limitations, including inferior visual quality, inadequate aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present **UniFL**, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL.
Notably, UniFL consists of three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which accelerates inference.
In-depth experiments and extensive user studies validate the superior performance of our method in enhancing generation quality and inference acceleration. For instance, UniFL surpasses ImageReward by 17\% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57\% and 20\% general preference with 4-step inference.


Peer reviews:
comments: SUMMARY:
This paper introduces UniFL, a novel approach for improving diffusion models through unified feedback learning. The objective of UniFL is to enhance visual generation quality, preference aesthetics, and inference acceleration. To achieve this, the paper proposes three key components: perceptual feedback learning, decoupled aesthetic feedback learning, and adversarial feedback learning. UniFL is designed to be a two-stage training pipeline that can be applied to various models and yields impressive improvements in both generation quality and acceleration. The paper provides experimental results demonstrating the effectiveness of UniFL in terms of generation quality and acceleration. Additionally, the paper discusses the potential broader impacts and ethical implications of advancements in image generation techniques.

STRENGTHS:
1. The writing of this article is commendable, with a well-structured format.
2. The author conducted a large number of visualization experiments to illustrate the focus and effectiveness of the method, which is very attractive.
3. The proposed method serves as a plug-and-play indeed improves the performance on both SD15 and SDXL.

WEAKNESSES:
1. There are some minor LaTeX formatting issues: for example, there should be a consistent space before parentheses in abbreviations, the quotes in the Appendix should be implemented using `’, and “our” should be “Our” in line 130. 
2. The values of hyper-parameters such as \alpha_d are not explicitly stated in the paper.

QUESTIONS:
1. Have the authors explored the impact of different aesthetics on generated images?
2. Are the user studies enough to prove the effectiveness of each module in this field?
3. Why not compare with SDXL-IR and SDXL-DPO in Ablation on Acceleration Steps?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsidering that current diffusion models still suffer from several limitations, this paper aims to propose a unified framework, UniFL to address the main existing challenges by applying feedback learning. To respectively solve the issues of visual distortion, poor aesthetic appeal, and inefficient inference, this paper demonstrates different sub-modules, namely Perceptual Feedback Learning, Decoupled Feedback Learning, and Adversarial Feedback Learning. By fully leveraging the ability of Perceptual Feedback Learning to fine-tune the diffusion during the two training stages, the proposed model aims to achieve both remarkable generation quality and speed simultaneously. Sufficient experiments, including extensive ablation studies for different proposed sub-modules, have proved the validation and effectiveness of the model’s design.\n\nSTRENGTHS:\nThe proposed idea about a combination of feedback learning and diffusion models is well introduced with necessary background information and is sufficiently motivated. \n\nThe three main issues remaining for diffusion, namely inferior quality, lack of aesthetics, and slow inference speed are well addressed, keeping the following methodology sections’ logic pipeline clear and understandable. \n\nThe proof of the methodology part is adequately explained with abundant support of pipeline figures and pseudocode that help the reader get full access to the novelty insight. \n\nSufficient experiments over different domains, especially the wide range of ablation studies, have been carried out validly which demonstrate the effectiveness of all the proposed feedback learning modules, which enhances persuasiveness.\n\nWEAKNESSES:\nThe proposed qualitative visualization comparison mainly focuses on justifying the overall generation style and the structure correctness. However, a main issue that may occur during the inference speed-up is detail loss. It will be more appreciated if the author can provide some further visual comparison results to demonstrate the model’s ability to keep the visualization results consistent with that adjective in the text prompt context.\n\nAlso, according to Figure 7, an unwelcomed issue can be found that the images generated by UniFL for text prompt A cream-colored labradoodle wearing glasses and a black beret teaching calculus at a blackboard mistakenly wears a black tie, which has never been mentioned in the text prompt. It shows an unexpected trend of overfitting and inconsistency. Further explanation about the cause of such results should be addressed.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nThough there are still some blemishes part for some of the proposed results, generally the experiment part is well-organized, and filled with abundant ablation studies for the proposed modules.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces Unified Feedback Learning (UniFL), a unified framework to enhance diffusion models through feedback learning. It addresses three main challenges in diffusion models: visual quality, aesthetic appeal, and inference efficiency. UniFL comprises perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. \n- Perceptual Feedback Learning utilizes existing perceptual models, such as an instance segmentation model, to fine-tune diffusion models on a specific aspect. \n- Decoupled Feedback Learning decomposes the general aesthetic concept into color, layout, lighting, and detail. It fine-tunes the model by reward feedback learning in all these sub-categories. \n- Adversarial Feedback Learning exploits a general reward model as a discriminator to improve the generation quality of fewer denoising steps.\nExperiments demonstrate that UniFL improves the performance of diffusion models like SD1.5 and SDXL in terms of generation quality, aesthetic preference, and inference speed.\n\nSTRENGTHS:\n- The paper is clear and easy to follow\n- The proposed method utilizes different priors from other perceptual models to improve diffusion models. \n- New aesthetic human feedback dataset is proposed.\n\nWEAKNESSES:\n- The overall method is some modification from ReFL.\n- The perceptual feedback learning part is limited by the selected models, which seem to only fine-tune concepts used to train these perceptual models. \n- The performance is not significantly superior to other methods according to Table 1.\n\nQUESTIONS:\n- If users want to generate images containing concepts not shown in COCO dataset, but shown in LAION, it seems perceptual feedback learning part will not work. \n- If users want to fine-tune the diffusion model in several different perceptual aspects simultaneously, will the priors from different perceptual models interrupt each other? \n- Weaknesses above\n\nLIMITATIONS:\nYes, the authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework to enhance visual quality, aesthetic appeal, and inference efficiency using various methods, including perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning. Good experimental results are observed.\n\nSTRENGTHS:\nThe concept of perceptual feedback learning is promising, as it can leverage various pretrained expert models to improve learning.\nThe idea of decoupled feedback learning makes sense, as it allows the model to focus on fine-grained details in the images.\n\nWEAKNESSES:\nThe author claims that previous works primarily focus on individual problems through specialized designs and proposes a unified framework. However, three different methods are designed to solve different problems. Why is it called unified?\n\nIn Line 145, it states that image content is incorporated as an additional condition for guidance. However, as shown in Figure 1, no extra inputs are added to the diffusion model.\n\nIn the experiment, only an instance segmentation model is used for perceptual feedback learning. Have the authors tried other types of models? It would be interesting to see some analysis on this aspect.\n\nFor decoupled feedback learning in Equation 5, why is a hinge loss used instead of a winner-loser loss as in Equation 4? What do the data annotations look like? How to choose the hyper-parameters of the hinge loss?\n\nWhat are the details of the semantic-based prompt filter and nearest neighbor prompt compression described in Line 189 for active prompt selection?\n\nIn Line 202, it states that samples with low inference steps tend to be too noisy to obtain correct rewarding scores in previous methods. How does the proposed adversarial feedback learning improve this and how does it accelerate inference?\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}