TreeReward: Improve Diffusion Model via Tree-Structured Feedback Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 04:24:31

Abstract: Recently, there has been significant progress in leveraging human feedback to enhance image generation, leading to the emergence of a rapidly evolving research area. However, current work faces several critical challenges: i) insufficient data quantity; and ii) rough feedback learning; To tackle these challenges, we present TreeReward, a novel multi-dimensional, fine-grained, and adaptive feedback learning framework that aims to improve both the semantic and aesthetic aspects of diffusion models. Specifically, To address the limitation of the fine-grained feedback data, we first design an efficient feedback data construction pipeline in an "AI + Expert" fashion, yielding about 2.2M high-quality feedback dataset encompassing six fine-grained dimensions. Built upon this, we introduce a tree-structure reward model to exploit the fine-grained feedback data efficiently and provide tailored optimization during feedback learning. Extensive experiments on both Stable Diffusion v1.5 (SD1.5) and Stable Diffusion XL (SDXL) demonstrate the effectiveness of our method in enhancing the general and fine-grained generation performance and the generalizability of downstream tasks.

Main paper: 
 introduction: 
 Reinforcement learning from human feedback (RLHF) [1, 18, 19] has recently made significant strides in enhancing large language models (LLM) [26, 32] , attracting substantial interest in the field. This technique, which incorporates human feedback, aims to improve the quality and safety of outputs from language models. Concurrently, similar methodologies are being adopted for diffusion-based image generation. In these applications [29, 30] , diffusion models are fine-tuned using human preference ratings, with reward functions designed to align the generated images more closely with human preferences. However, despite these developments, diffusion models that integrate human feedback learning still face several ongoing challenges. (i) Insufficient Feedback Data: Existing methods suffer from the limited preference data volume, especially the fine-grained preference feedback data, which may not sufficiently capture the diverse range of human preferences. Nonetheless, gathering large amounts of preference data is labor-intensive and costly. (ii) Coarse Feedback Learning: Due to the scarcity of tailored finegrained feedback data, the majority of existing datasets primarily focus on coarse feedback learning and fail to explore efficient methods for leveraging fine-grained preference fine-tuning to enhance the performance of diffusion. To address these challenges, we propose an effective fine-grained feedback-learning method to boost the performance of the diffusion model comprehensively. Specifically, to tackle the issue of lack of fine-grained feedback data, we first design an efficient feedback data construction pipeline for six fine-grained dimensions. According to the different characteristics of distinct fine-grained dimensions, it incorporated both the automatic feedback data generation and the human preference annotation to enable feedback data scale-up in a low-cost manner. With this pipeline, we scale up the feedback dataset to about 2.2M, which is the largest available dataset in the field of text-to-image generation. Based on this dataset, we further introduce a novel tree-structured reward model, namely TreeReward. It organizes the fine-grained feedback dimension hierarchically and utilizes the random sample ensemble training strategy to effectively integrate the scoring abilities of multiple fine-grained feedback into a single reward model. During reward feedback learning, it aggregates the reward scores from all the leaf nodes in an adaptive manner, offering the case-tailored feedback signal for optimization. Extensive experiments demonstrate the superiority of our method in enhancing the generation performance of both general quality and fine-grained dimensions. Furthermore, we validate the model's performance in downstream tasks, demonstrating its robustness and generalization. Our contributions are summarized as follows:• We design an efficient feedback data curation pipeline that combines automated feedback generation and manual annotation to efficiently collect fine-grained feedback data on various aspects such as style, content, light, structure, layout, and detail, finally yielding about 2.2M feedback dataset which is the largest dataset in the field. • We introduce an innovative feedback learning method based on a tree-structure reward, TreeReward, that enables multidimensional, fine-grained, and adaptive preference feedback learning, resulting in more comprehensive and effective reward tuning.• Extensive experiments on both SD1.5 and SDXL validate the effectiveness of our method, showcasing superior performance when compared to the state-of-the-art reward tuning method. Additionally, the experiment on the downstream tasks further validates the efficacy and generalization of our method. 
related works 2.1 text-to-image generation: 
 Text-to-image generative models, including auto-regressive [4, 6, 22] , GANs [7, 11, 12] and diffusion models [8, 17, 25, 33] , have become a prominent research area in various applications, attracting significant attention. Among the evolving methodologies, diffusion models have emerged as the de facto mainstream technique for textto-image synthesis due to their impressive generation capabilities. The widespread adoption of diffusion models can be attributed to their demonstrated effectiveness, as evidenced by pioneering works [3, 9, 15, 17, 21, 23] such as DALLE-2 [21] and Stable Diffusion [23] . However, despite diffusion models having great success in text-to-image synthesis, they still struggle to generate images that are well-aligned with the user preference within the text prompts. This paper addresses this limitation by directly incorporating finegrained human feedback into the training of diffusion models.Due to the inherent noise of the pre-trained dataset, there is often a gap between generative models' pre-training objectives and human intent. To mitigate this gap, human feedback learning [1, 5, 18, 19, 31] has been utilized to align model performance with human preference in the language domain. Inspired by these works, several works have endeavored to incorporate human feedback into the learning process of diffusion models to better understand human preferences. DDPO [2] adopts a reinforcement learning framework to align diffusion model generation with the supervision provided by an additional reward model. Approaches like HPS [28, 29] employ a separate reward model trained on curated human preference datasets to filter eligible preferred data for fine-tuning stable diffusion. Another approach, Reward Weighting [14] , utilizes reward-weighted likelihood as the optimization objective. ImageReward [30] proposes the ReFL training framework to fine-tune stable diffusion via a differentiable reward model. While these methods have shown success in human preference learning, they rely on a general reward model trained on coarse human preference datasets, limiting their ability to provide fine-grained preference evaluation. Additionally, they require training an extra reward model when preferences change. In this paper, we address these limitations by curating a fine-grained human preference dataset. Building upon this dataset, we propose a tree-structured reward model that offers more effective and flexible reward supervision for diffusion models. 
preliminary 3.1 text-to-image diffusion model: 
 Text-to-image models use diffusion modeling to create high-quality images from text prompts. The diffusion model produces data samples from Gaussian noise by gradually denoising process. During pre-training, a sampled image 𝑥 is first processed by a pre-trained VAE encoder to derive its latent representation 𝑧. Subsequently, random noise is injected into the latent representation through a forward diffusion process, following a predefined schedule {𝛽 𝑡 } 𝑇 . This process can be formulated as𝑧 𝑡 = √ 𝛼 𝑡 𝑧 + √ 1 -𝛼 𝑡 𝜖, where 𝜖 ∈ N (0, 1) is the random noise with identical dimension to 𝑧, 𝛼 𝑡 = 𝑡 𝑠=1 𝛼 𝑠 and 𝛼 𝑡 = 1 -𝛽 𝑡 . To achieve the denoising process, a UNet 𝜖 𝜃 is trained to predict the added noise in the forward diffusion process, conditioned on the noised latent and the text prompt 𝑐. Formally, the optimization objective of the UNet is:EQUATIONReward feedback learning(ReFL) [30] as an encoding mechanism for capturing human preferences. More specifically, considering two candidate generations, denoted as 𝑥 𝑤 (preferred generation) and 𝑥 𝑙 (unpreferred one), the loss function for training the human preference reward model 𝑟 𝜃 can be formulated as follows:EQUATIONwhere D denotes the collected feedback data, 𝜎 (•) represents the sigmoid function, and 𝑐 corresponds to the text prompt. The reward model 𝑟 𝜃 is optimized to produce a preference-aligned score that aligns with human preferences. In the Preference Fine-tuning phase, ReFL begins with an input prompt 𝑐, initializing a latent variable 𝑥 𝑇 at random. The latent variable is then progressively denoised until reaching a randomly selected timestep 𝑡. At this point, the denoised image 𝑥 ′ 0 is directly predicted from 𝑥 𝑡 . The reward model obtained from the previous phase is applied to this denoised image, generating the expected preference score 𝑟 𝜃 (𝑐, 𝑥 ′ 0 ). This preference score enables the fine-tuning of the diffusion model to align more closely with human preferences:EQUATION 
efficient feedback data scaling: 
 An essential challenge of feedback learning resides in the collection and construction of a high-quality dataset of human feedback.Although there are already some available feedback datasets, such as the ImageReward [30] and Pickascore [13] , these datasets often suffer from coarse feedback annotation and limited data volume.However, collecting large amounts of fine-grained human feedback data is time-consuming and expensive. To tackle this issue, we design an efficient pipeline to reduce the cost of feedback data construction via a series of automatic strategies. The core insight of our method is not all feedback data on each aspect requires accurate human annotation and we can combine the automatic data generation with manual human annotation to reduce the cost of the data construction. The comparison between our collected feedback data and the existing dataset is summarized in Tab.1. In total, we collect about 2M feedback data across six dimensions encompassing both semantic and aesthetic aspects with such pipeline, which will be elaborated in the following sections.The semantic alignment between the text prompt and the generated image is an important aspect of evaluating the performance of a text-to-image diffusion model. Rather than treating the semantic alignment as a whole [30] , we further break down the semantic alignment into style alignment and content alignment to aid the fine-grained feedback data collection. Style Alignment. To create a feedback dataset on style generation, we develop a contrastive data sample strategy to construct the style feedback data. As depicted in Fig. 2 , we initially collect a diverse set of approximately 500 commonly used target style words from the user prompts(e.g. prompt in the JourneyDB [20] ). Subsequently, given a prompt containing a particular target style word, we randomly substitute the target style word with another word from the vocabulary. Then, we generated two images with the original prompt and the replaced prompt. Thus, for the target style, the image generated by the target style word serves as a positive sample, whereas the image generated by the randomly substituted style word acts as a negative sample. For each image, we considered the original caption and the regenerated caption from the MLLM as feedback data for content alignment with the regenerated caption regarded as the preferred alignment as a more detailed and accurate description of the image.The aesthetic quality is another critical aspect of generation performance. Due to the inherent abstract and subjective, it is hard for the diffusion model to grab the aesthetic essence. Therefore, the current methods propose to exploit human preference to steer the diffusion in the right direction. However, the abstract aesthetic concept contains quality in various dimensions, such as color, lighting, layout, and details, and the existing methods only consider coarse-grained aesthetics and cannot refine each dimension of aesthetics, and also has the risk of optimization conflict during feedback tuning as analyzed in [27] . To address this limitation, we propose to decouple the aesthetic into these dimensions and introduce an AI-assisted feedback data construction strategy.Color, Lighting, Layout, and Detail. The primary challenge of fine-grained aesthetic feedback data curation lies in the lack of the image pair focusing on a particular aspect to annotate. The ordinary image pair generated by a text-to-image diffusion model of the same prompt tends to have a similar aesthetic quality, leading to hard judgment and requiring much more time to distinguish the reference sample. To ease the human annotation burden, we design two strategies to make the AI-assisted feedback data generation. Specifically, we utilize a text-to-image diffusion model to generate several images for a particular prompt and then ask the annotator to select the preferred and unpreferred sample. During this process, (i) To make the model to generate the image that focuses on a particular aspect, we manually curate a set of trigger words for each aesthetic dimension. For instance, in the lighting dimension, trigger words like "Soft lighting, " "Side lighting, " "rim lighting, " and "Moody lighting" are included. By incorporating these trigger words into the input prompt, we enhance the model's focus on specific aesthetic dimensions during the image generation. (ii) To ease the judgment process, we manually create the sample with aesthetic differences as depicted in Fig. 2 . On one hand, we utilize diffusion models with varying generative capabilities to generate the image pair. For example, we take the image generated by SDXL and its improved version Kindminsky as the candidate images for easier preference annotation. On the other hand, we take the images generated at different denoised timesteps to assist the fast preference annotation. With these two strategies, we can achieve efficient fine-grained aesthetic feedback data annotation, and reduce the annotation cost. 
tree-structured feedback learning: 
 Given the collected fine-grained feedback dataset, the next question is how to utilize these datasets efficiently. The most direct way is to utilize these datasets as a whole and train a simple reward model as the practice in [30] . However, this way overlooks the difference between these dimensions, leading to inefficient feedback learning. To this end, we introduce a TreeReward, an effective way to combine fine-grained feedback data and dynamically provide fine-grained and adaptive preference feedback for fine-tuning diffusion models. Fig. 3 illustrates the TreeReward training and preference fine-tuning process, which will be explained in the following sections.In general, TreeReward exhibits a hierarchical structure comprised of two internal nodes and six local leaf nodes. The role of the internal nodes is to determine which general aspect (such as semantic or aesthetic) to reward, while the leaf nodes are responsible for providing precise reward scores along fine-grained dimensions. In line with ImageReward [30] , we implement TreeReward with the BLIP model as the backbone, which employs ViT-L14 as the image encoder and a 12-layer transformer as the text encoder. For the internal node and leaf node of our TreeReward, we implement them with simple 2x and 4x MLPs, respectively. We take a random sample ensemble strategy to train our TreeReward. Specifically, we first randomly sampled a data point from the collected feedback dataset, which contains the reward dimension, reward data. Then, a path is chosen from the root of TreeReward to the leaf node according to the reward dimension. Formally, for the internal nodes,EQUATIONHere, 𝐺 (•) represents the predicted logits of the internal node, and 𝑔 is the target internal reward label determined by the source of the training data. This objective aims to let the model know which aspect to reward when given a text-image pair. For the leaf node, we expect the leaf node to output the correct reward score. We take a similar way with E.q.2 to optimize the leaf node along a particular reward dimension. Specifically, for the 𝑗-th reward local leaf node denoted as 𝑅 𝑗 𝜃 , the preference feedback reward data pair is represented as (𝑥 𝑤 , 𝑥 𝑙 ). Here, 𝑥 𝑤 corresponds to the preferred text-image pair, while 𝑥 𝑙 represents the non-preferred text-image Image Generation Figure 3: The overview of our TreeReward, a tree-structured reward model trained with vast high-quality fined-grained preference data to facilitate more effective feedback learning for text-to-image generative models.pair. The loss function for the leaf can be formulated as: EQUATIONIt is worth noting that we only optimize one particular path in the TreeReward according to the source of the feedback data for each time, and leave the parameters of other nodes unchanged. However, with the random sampling training data, the whole tree reward will be fully optimized via a path ensemble way.Aligned with the practice in ImageReward [30] , we adopt the direct preference fine-tuning fashion to harness the reward guidance offered by our TreeReward. However, rather than obtaining a coarse reward score, we utilize an adaptive reward score derived from our tree-structured reward model to fine-tune the diffusion model comprehensively.Specifically, given the generated image 𝑦 𝑖 and the corresponding prompt 𝑥 𝑖 , we calculate the reward score for the input text-image pair (𝑥 𝑖 , 𝑦 𝑖 ) hierarchically and adaptively. Starting from the root node, we first calculate the internal node logits and obtain the reward weights for the semantic alignment and aesthetic quality. Formally, we have:EQUATIONwhere 𝑟 𝑏 𝜃 (𝑥) with 𝑏 ∈ {semantic, aesthetic} represents the prediction output of the internal node, and 𝑤 𝑏 is the reward weight of along these two aspects. Next, we obtain all the fine-grained reward scores on the leaf node and compute the adaptive weight of each leaf node under the internal node:𝑤 𝑘 = 𝑒 𝑟 𝑘 (𝑥 𝑖 ,𝑦 𝑖 ) 𝑒 𝑟 𝑘 (𝑥 𝑖 ,𝑦 𝑖 ), where 𝑘 is the leaf nodes under a internal node and 𝑤 𝑘 is the corresponding weight. The final reward 𝑅 is obtained by combining the rewards hierarchically from the root to the leaf:EQUATIONwhere 𝑁 𝑏 is the number of reward leaf nodes under the branch 𝑏, and 𝑀 is the number of global reward branches. By combining the fine-grained rewards from all nodes, our model can adaptively focus on the reward dimensions that have not been well optimized yet, providing case-tailored preference feedback for the diffusion model via:EQUATIONFollowing [30] , we also incorporate the naive diffusion pre-train loss [30] as a regularization term:L 𝑝𝑟𝑒𝑡𝑟𝑎𝑖𝑛 = E (𝑦 𝑖 ,𝑥 𝑖 )∼𝐷 E E (𝑥 𝑖 ),𝑦 𝑖 ,𝜖∼N (0,1),𝑡 ∥𝜖 -𝜖 𝜃 (𝑧 𝑖 , 𝑡, 𝜏 𝜃 (𝑦 𝑖 )) ∥ 2 2 . (10)Therefore, the final training objective is:EQUATIONwhere 𝜆 is the loss weight and is set to 0.05 by default. 
experiments 6.1 implementation details: 
 Dataset and Training Setting. We utilize the collected fine-grained preference data to train our TreeReward. We randomly sample a prompt set from DiffusionDB following [30] for preference finetuning. We conduct experiments with Stable Diffusion v1.5 and Stable Diffusion XL base 1.0. Additionally, to validate the effectiveness and generalization of our method, we further utilize the JourneyDB [20] , a large-scale generated image dataset collected from Midjourney, to fine-tune the base SD1.5 to acquire an improved base diffusion model where we subsequently conduct the preference fine-tuning. Fine-Grained Evaluation. In addition to the general performance evaluation on the overall quality, we further constructed a finegrained evaluation benchmark to comprehensively evaluate the model performance on the fine-grained dimensions. Specifically, we request the ChatGPT to write several typical prompts for each finegrained dimension (i.e., content alignment, style alignment, color, lighting, detail, and layout). These prompts ensure that describe the picture that is most relevant to the corresponding fine-grained dimension. We further execute the manual check to filter out the invalid prompt. This finally results in 100 prompts for each dimension and 600 prompts in total. The examples of these prompts and details of the evaluation procedure are displayed in the supplementary material.Qualitative Results. We compared our method with ImageReward, the current state-of-the-art SD preference modeling method.It clearly shows that our method exhibits a superior preference for learning performance in both semantic alignment and aesthetic quality enhancement. As shown in Fig. 4 , our method exhibits superior overall visual quality. Take SD1.5 and the prompt of "A dolphin leaps through the waves, set against a backdrop of bright blues and teal hues" as an example, there are no dolphins in the image generated by SD1.5. The dolphin generated by ImageReward is either too small, or the waves are blurry. By contrast, both the waves and dolphins generated by TreeReward are rich in detail and highly realistic. As depicted in Fig. 5 , TreeReward also shows superiority in generating images with better visual quality in various fine-grained aspects. For example, only TreeReward generates the correct result for prompt "a horse without a rider", while both the base model and ImageReward generate the mismatched content (The riders). And for prompt "A mountain retreat's spa, zen-inspired, ... overlook forest views", both SD1.5 and ImageReward present unreasonable layouts for the tables and swimming pool (Too small tables and truncated swimming pool), while TreeReward displays the more aesthetic layout. Note that the ImageReward does not exhibit much improvement when applied to the improved stable diffusion base model which is fine-tuned with JourneyDB. As a comparison, Our TreeReward still delivers notable improvement, which demonstrates the superiority of our method.Quantitative results. To evaluate the performance of our method quantitatively, we conducted comparisons using CLIP scores and aesthetic scores, which provide metrics for semantic alignment and aesthetic quality, respectively. The results are presented in Tab. 2. It demonstrates that our method outperforms the baseline model in both semantic alignment and aesthetic quality and also surpasses the performance of ImageReward. For instance, on SD1.5, our method achieved a 2% improvement in semantic alignment compared to ImageReward, along with a 1.6% enhancement in aesthetic quality. These results indicate the superiority of our method in generating images that are not only visually appealing but also semantically aligned. It is worth emphasizing that the score achieved by the SD1.5 model with fine-tuning using journey-db data is higher than that of the SDXL model. This observation underscores the significance of utilizing high-quality fine-tuned data to enhance model performance. However, it is important to note that these metrics only provide a general overview. To obtain more detailed insights, we further conducted fine-grained human evaluation to validate the superiority of our method in generating high-quality images across diverse dimensions. Specifically, we take the original SD1.5 and JourneyDB-tuned SD1.5 as our baselines. Then, we evaluated the performance of the models optimized by ImageReward and TreeReward, respectively. Concretely, human raters were tasked with scoring the images generated by these two models ranging from -5 to 5 across various dimensions against the baseline models.The results of the fine-grained evaluation are presented in Tab.3. It is evident that our TreeReward feedback learning approach significantly outperforms the ImageReward across all the dimensions. Remarkably, our TreeReward approach demonstrates a notable enhancement of 1.5 points in the 'Color' dimension when compared to the ImageReward method on SD1.5. Moreover, when applied on JourneyDB-tuned SD1.5, our TreeReward approach showcases a significant improvement of 2.2 points in the 'Detail' dimension, surpassing the ImageReward model by a substantial margin. Such comparison is also visualized in Fig. 1 , clearly demonstrating that our method outperforms ImageReward across all dimensions.We have conducted a series of ablation experiments to showcase the key contributions of our method, specifically the fine-grained feedback data scaling and the design and tree-structured reward finetuning. These experiments encompass several settings: (i) "Data-Scale": We employ the same reward model as ImageReward but utilize our collected feedback data for training the reward model without distinguishing the different fine-grained dimensions. (ii) "Decouple": Instead of training a single reward model, we train separate reward models for each fine-grained dimension using our feedback data and utilize these models for preference fine-tuning simultaneously. (iii) "ImageReward": Preference fine-tuning using the reward model provided by ImageReward. (iv) "TreeReward": Preference fine-tuning using the reward model provided by our TreeReward approach. The detailed comparison between these settings is presented in Tab.4. As illustrated in Fig. 7 , incorporating more feedback data significantly enhances the performance of preference fine-tuning, resulting in an impressive increase of 18% compared to the naive ImageReward approach. This finding underscores the importance of gathering a larger quantity of high-quality feedback data, even without considering fine-grained distinctiveness. Building upon this, the decoupling of the reward model for different fine-grained dimensions leads to a further improvement (44% vs 32%). This demonstrates the necessity of the decoupled reward model design, which effectively eliminates potential conflicts in preference tuning as analyzed in [27] . However, training multiple reward models not only results in memory inefficiency but also achieves sub-optimal multiple reward fine-tuning. In comparison, our TreeReward approach leverages fine-grained feedback data hierarchically and rewards in an adaptive manner, offering greater flexibility and delivering superior performance compared to naive fine-tuning with multiple reward models (54% vs 35%). By incorporating these two improvements, our method ultimately achieves a 32% increase in user preference compared to ImageReward, highlighting the significant advantages of our approach.We conduct an extensive study to evaluate the generalization potential of our method in adapting to various downstream tasks, such as LORA [10] , DreamBooth [24] , Image-to-Image [16] , and Control-Net [34] . As illustrated in Fig. 6 , our model showcases remarkable compacity in style learning, IP preservation, reference generation, and controllable generation. 
conclusion: 
 We propose TreeReward, an effective method to boost the diffusion model across various fine-grained dimensions. It includes an efficient "AI + Expert" fine-grained feedback data construction pipeline, and a tree-structured reward model to achieve fine-grained, multidimension, and adaptive reward feedback learning. Extensive experiments on both SD1.5 and SDXL models demonstrate the superiority of our method in both boosting the general quality and fine-grained generation and at the same exhibiting excellent generalization.  
 
 Appendix:  


Decision: The manuscript has been reviewed by four experienced reviewers, all of whom, in the final rating, gave positive scores, indicating the manuscript indeed meets the bar of MM.

The AC therefore agrees with the consensus and recommends acceptance.

Please, however, do account for the comments in the final version.

Reviews: review: paper_summary: paper_summary: The paper presents TreeReward, an innovative feedback-learning framework designed to enhance diffusion models using a tree-structured reward system. It addresses challenges such as insufficient data quantity and coarse feedback learning. By developing a comprehensive feedback data construction pipeline and a hierarchical reward model, TreeReward aims to improve several aspects, including Content, Style, Color, Lighting, Detail, and Layout. Extensive experiments demonstrate significant improvements over existing models, highlighting the framework's potential in various downstream tasks.
strength_weakness: strength_weakness: 1. The authors have developed an efficient feedback data construction pipeline that combines automatic data generation with manual annotation. This approach has resulted in a dataset of 2.2 million high-quality feedback pairs, the largest in the field, which is a notable achievement.

2. The paper presents extensive experimental results on Stable Diffusion v1.5 and Stable Diffusion XL. The comparisons with the state-of-the-art ImageReward model are thorough and clearly demonstrate the superiority of TreeReward in both semantic alignment and aesthetic quality.

3. The proposed method includes more dimensions (=6) to evaluate the quality of generated images. By combining this with the tree-structured method, the performance of diffusion model can be effectively improved.
limitations: limitations: 1. The main limitation of this paper is the manually designed tree structure used in the method. The proposed aspects are coarse-grained, which might limit the model's ability to capture finer nuances in feedback data.

2. Although the authors address the issue of data scalability by combining automated and manual feedback data generation, the process of manually annotating large datasets is still labor-intensive and costly. The reliance on such a large dataset might pose challenges for replication and scalability in different contexts or with different datasets.

3. The paper could provide a more balanced view by discussing the limitations and potential drawbacks of the proposed approach in greater detail. Acknowledging these aspects would offer a more comprehensive perspective and guide future research directions.
paper_summary: paper_summary: In this work, the authors propose a method for text-to-image generation constrained by human feedback, following the recent success of similar training paradigms in the field of text generation. The manuscript presents two main issues in current solutions: that the current datasets are limited in sample size and that current feedback learning methods are crude. In order to tackle both problems, a semi-automated pipeline is proposed to assemble a new dataset with pair annotations much larger than existing datasets. By constructing their proposed dataset in 6 fine-grained dimensions, the authors then suggest a way to provide an adaptive preference of the different categories in a hierarchical fashion. A tree structure is essentially implemented in order to learn the most important preference when fine-tuning under a particular text prompt. Experiments and ablations on different versions of the Stable Diffusion model are performed in order to motivate the design and showcase its usefulness.
strength_weakness: strength_weakness: - The proposed dataset is a large step forward in terms of data scale compared to existing alternatives. In particular, the semi-automated strategy allows for easy extension of the dataset even further. It is also the only dataset to consider different types of preferences on the feedback type, which the authors refer to as dimensions. The annotation strategy is also a clever way of using MLLMs.

- TreeReward presents a simple way to separate the major feedback classes (semantics and aesthetics) in a learnable and dynamic fashion. This ensures that small modifications are more accurately reflected in the images produced by the fine-tuning procedure. The rewards create a sort of hierarchy in this sense, albeit not a very deep one. In some qualitative cases, TreeReward seems to produce much higher-fidelity samples.
limitations: limitations: - The experimental results of the proposed method are somewhat shaky. The results with TreeReward do not provide a major improvement compared to ImageReward, especially when considering SDXL (Table 1). Even when looking at qualitative examples, although both generated results are correct, there is no clear reason why one would prefer TreeReward-generated ones. A similar observation holds when looking at Table 3, where if one considers a rating scale of cardinality 10 (i.e., from -5 to 5), using ImageReward does not seem to provide much worse results on average. Related to the last statement, it would be beneficial to have the deviation of the average human rating in Table 3, especially since the number of human evaluators is not given. Qualitatively, while the benefits of TreeReward are visible on SD1.5, as mentioned previously, with SDXL, these improvements do not necessarily seem much better than the ImageReward counterpart. Indeed, in the main paper, the authors only present a visual comparison of each evaluation dimension. with SD1.5 and not with SDXL.

Minor:
- Some of the strategies for the dataset creation seem somewhat arbitrary. For example, the choice of having the diffusion model stop at different timesteps in order to generate positive-negative pairs is strange, as before reaching the final timestep, one is generating from the same mode, so the pair might not even be that different. Another aspect that deserves more detail should be presented in the vocabulary for the contrastive sample generation procedure since it has to be somewhat varied. As a minor point regarding this issue, shouldn't the timestep sequence in Fig. 2 be x_T -> x_{t2} -> x_{t1} -> x_0   

- The caption of Fig. 1, line 82 seems to contain a mistake "three two components".
paper_summary: paper_summary: This work proposed a reward learning model for enhancing text-to-image generation performance. Fine-gained datasets are collected considering different dimensions of aesthetics and semantics. Then the adaptive reward learning to proposed for diffusion-based text-image generation models.
strength_weakness: strength_weakness: The results show the advantage of the treereward compared to other reward learning approaches. The collected datasets would help the community a lot if they are released. The effectiveness is shown with SD1.5 and SDXL models.
limitations: limitations: One concern is that authors ought to compare different reward learning methods more fairly, with the same volume of data. Otherwise, it is hard to say which part, the data or the reward learning, benefits more in the final performance.
It is also interesting to demonstrate how much more training overhead induced by the reward learning. 
Regarding the claim "By combining the fine-grained rewards from all nodes, our model can adaptively focus on the reward dimensions that have not been well optimized yet,", I didn't see how equation 8 makes this possible. It seems like any leaf node getting more reward would be further rewarded by more chance. Does it cause any dominance  of some leaf nodes over others?
paper_summary: paper_summary: This paper presents TreeReward, a novel multi-dimensional, fine-grained, and adaptive feedback learning framework that aims to improve both the semantic and aesthetic aspects of diffusion models. Firstly, this framework design an efficient feedback data construction pipeline in an "AI + Expert" fashion, yielding about 2.2M high-quality feedback dataset encompassing six fine-grained dimensions. Then, by introducing a tree-structure reward model, this framework improves model's performance of feedback learning. The extensive experiments demonstrate their method's effectiveness in enhancing the general and fine-grained generation performance and the generalizability of downstream
tasks.
strength_weakness: strength_weakness: 1. This work designs an efficient feedback data curation pipeline that combines automated feedback generation and manual annotation to efficiently collect fine-grained feedback data on various aspects such as style, content, light, structure, layout, and detail, finally yielding about 2.2M feedback datase which is the largest dataset in the field.

2. This work introduces an innovative feedback learning method based on a tree-structure reward, TreeReward, that enables multi-dimensional, fine-grained, and adaptive preference feedback learning, resulting in more comprehensive and effective reward tuning.


3. Extensive experiments on both SD1.5 and SDXL validate the effectiveness of our method, showcasing superior performance when compared to the state-of-the-art reward tuning method. Additionally, the experiment on the downstream tasks further validates the efficacy and generalization of our method.
limitations: limitations: 1. Is the impact of manually designing trigger words on the final model results significant, as mentioned in line 404 of the paper ?

2. In the paper, a large amount of aesthetic difference data was generated. How is this aesthetic difference defined ? Is it solely based on personal perception ? In figure3, how to define "preferred" and "unpreferred"?

3. In line 489, is it $r_{\theta}^{i}$ and $D_{j}$ ? 
    In line 571, is it "and $\lambda$ is set to"?

4. I believe the sensitivity of the model's performance to lambda needs to be discussed.

5. I really like this data construction method, but I still have many questions about how to judge the quality of the image in this work. In the comparison results in Figure 5, I often feel that ImageReward is not necessarily worse than TreeReward. How can such subjective content be formally defined?