Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Authors: xun  wu
shaohan  huang
guolong  wang
jing  xiong
furu  wei
microsoft  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 13:48:03

Abstract: Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

Main paper: 
 introduction: 
 Text-to-image generative models [23, 21, 19, 26] have experienced rapid advancements in recent years. For example, large-scale text-to-image diffusion models, exemplified by Imagen [24] and DALL•E 2 [19] , have demonstrated the capability to generate high-quality and creative images when provided with textual prompts. However, despite recent progress, current generative models still exhibit misalignment with human preferences, such as conflicts with text prompts or incorrect content [17] . A pivotal approach to addressing this issue is utilizing Reinforcement Learning from Human Feedback (RLHF) [16, 27, 1] to supervised fine-tune text-to-image generative models with preference data [13, 3, 5, 17] .Preference data is crucial for aligning generative models with text prompts. However, existing human-crafted preference datasets, such as HPD v2 [33, 32] and Pick-a-Pic [10] , either provide only broad, general preference comparisons without fine-grained, accurate preference evaluations, or they are limited in size. Additionally, using humans for preference annotation is expensive and time-consuming, restricting progress in this research area.Drawing inspiration from recent research utilizing AI-generated preference data as training supervise signal for Large Language Models (LLMs) alignment on Natural Language Processing domain [12, 2] , we pose the following question:Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?These multimodal large language models (MLLMs), trained on web-scale text and text-image pairs, have already demonstrated formidable capabilities in image understanding. To this end, we first introduce VisionPrefer, a publicly available AI-generated dataset that features millions of finelygrained preferences concerning model-generated images. Compared with existing human preference datasets, VisionPrefer offers the following benefits:• Scalability & Low Cost: As shown in Table 1 , VisionPrefer encompasses 1.2 M preference choices across 179 K pairs of images, establishing it as the largest text-to-image generation preference dataset to date. Additionally, because VisionPrefer is annotated by MLLMs, it can be easily expanded further and the construction cost is significantly lower than human annotation. • Fine-grained preference: To more accurately and diversely evaluate the preference scores of generated images, we carefully develop a detailed preference annotation guideline for MLLMs, which covers four distinct aspects: Prompt-Following, Fidelity, Aesthetic, and Harmlessness. The detail requirement for each aspect is presented at Table 3 . • Comprehensive feedback formats: Unlike existing benchmarks that provide only rankings or preference indices, our VisionPrefer not only provides preference rankings but also includes preference scores and textual explanations for the preference annotations from each aspect, which makes VisionPrefer more versatile, e.g., allowing it to serve as a textual guiding resource for image re-editing and refinement.Building on the VisionPrefer, we conducted an extensive investigation into its most effective utilization. First, we developed a preference reward model named VP-Score optimized on VisionPrefer, trained to evaluate generated images based on their likelihood of being preferred by humans. Experimental results demonstrate that VP-Score exhibits a competitive correlation with human preferences compared to existing human preference reward models. Moreover, we employ two reinforcement learning methods, Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO), to enhance generative models to better align with human preferences. As illustrated in Figure 1 , VisionPrefer markedly enhances text-image alignment in compositional image generation across diverse aspects, such as aesthetics. In summary, our contributions are as follows: • We construct VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-to-image generative alignment. Compared with existing preference datasets, VisionPrefer has the advantages of scalability, fine-grained annotations, and comprehensive feedback format. • Based on VisionPrefer, we propose a reward model, VP-Score, which achieves a competitive correlation with human preferences with other automated human preference metrics. • Experimental results demonstrate the effectiveness of both VisionPrefer and VP-Score. Additionally, we provide a comprehensive analysis of them to gain a deeper understanding of how AI-generated synthetic data and models trained on such data impact future research in this domain. 
related work: 
 Text-to-Image Generative Models Alignment. While existing text-to-image generative models often generate images that do not closely match human preferences, thus alignment in the context of diffusion has garnered increasing attention [28, 10, 5, 17, 28] . There are two main types of text-toimage generative models alignment algorithms: (i) Proximal Policy Optimization (PPO). For example, reward weighted method [13] first explores using human feedback to align text-to-image models with human preference. ReFL [34] trains a reward model, ImageReward, using human preferences and subsequently utilizes it for fine-tuning. (i) Direct Policy Optimization (DPO). DPOK [7] fine-tunes text-to-image diffusion models by using policy gradient to maximize the feedback-trained reward. ZO-RankSGD [28] optimizes diffusion in an online fashion with human ranking feedback. RAFT [5] and AlignProp [17] tune the generative model to directly increase the reward of generated images. Several manually annotated preference datasets are proposed to support above algorithms [32, 34, 10] . Their overall statistics are shown in Table 1 . These manually annotated data have two drawbacks. First, manual annotation need heavy cost, leading to a small-size set. Second, manual annotations are prone to specific biases [32] .Reinforcement Learning from AI Feedback. [2] introduced the idea of Reinforcement Learning from AI Feedback (RLAIF), which used LLM-labeled preferences in conjunction with human-labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from LLMs [20, 11, 35] . These works demonstrate that LLMs can generate useful signals for reinforcement learning fine-tuning. However, RLAIF for text-to-image generative model alignment is less explored. [30] leveraged MLLMs to assess the alignment between generated images and input texts, focusing on aspects like object number and spatial relationship. T2I-CompBench [9] utilized MLLMs like BLIP-VQA to evaluate the text-to-Image generative models. Our work diverges from previous works in two principal ways.(1) Previous approaches provided limited data, sometimes even less than what's annotated manually (e.g., 6K in [9] ). ( 2) Prior methods had limited aspects in alignment, lacking consideration for aspects such as fidelity [30] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The analysis section contains a discussion of limitations in our work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
visionprefer: 
 We introduce VisionPrefer, a fine-grained preference dataset constructed by collecting feedback from MLLMs annotators. The collection pipeline of VisionPrefer is shown in Figure 2 , which mainly consists of three steps: prompt generation, image generation and preference generation.Step-1: Prompt Generation. We generate prompts based on DiffusionDB [29] , a large-scale text-toimage prompt benchmark containing 1.5M user-written prompts following two steps: (1) Polish. As discussed in [32] , a significant portion of the prompts in the DiffusionDB is biased towards certain styles (e.g., "Greg Rutkowski" appears in around 15% of prompts). We utilize GPT-4 to polish prompts in DiffusionDB to obtain unbiased prompts following instructions shown in Appendix F.1.(2) NSFW Filting. We employ state-of-the-art NSFW detectors [8] to compute an NSFW score for each prompt and filter out prompts that exceed a certain threshold, following [29] . After these two steps, we combine both polished prompts and the original prompts in DiffusionDB as our final prompt benchmark, which contains 179K prompts.Step-2: Image Generation. We generate images using different text-to-image generative models (see details in Appendix C.2) by sampling textual prompts constructed in Step-1 as input. For each prompt, we generate four images by randomly selecting different generative models from the model pools with different classifier-free guidance scale values, to ensure high diversity. This diversity allows for a comprehensive evaluation of a preference prediction model's generalization capability and facilitates the training of a more generalizable model. Finally, we obtain 0.716M images.Step-3: Preference Generation. We employ state-of-the-art multimodal large language model, GPT-4 V, to provide three types of feedback: (1) Scalar scores that indicate the fine-grained quality regarding multiple aspects, (2) Preference ranking according to the scalar scores, and (3) Textual explanations that give detailed guidance on how to improve the completion, encompassing four distinct fine-grained aspects namely: Prompt-Following, Aesthetic, Fidelity, and Harmlessness for each generated image (See the example in Table 12 ). Detailed input instructions for GPT-4 V to annotate preference labels are in Appendix F.2. Finally, we obtain 1.2M preference choices.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will public our data and code upon paper acceptance, due to the management regulations of our institution. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we first train a corresponding reward model named VP-Score and evaluate it on existing human-preference datasets ( § 4.1). Next, we enhance existing text-to-image generative models by adopting two reinforcement learning algorithms ( § 4.2) to validate the efficacy of VisionPrefer and VP-Score. After that, we design a simple pipeline to edit generated images with the textual explanations in VisionPrefer( § 4.3).Training Setting. We train the VP-Score over VisionPrefer. VP-Score adopts the same model structure as ImageReward [34] , which is a open-source human-preference reward model and utilizes BLIP [14] as the backbone. Similarly to training the reward model for the language model [27, 16] , we formulate the preference annotations in VisionPrefer as rankings. Specifically, we employ the average scores of each sample in VisionPrefer across four aspects as the final preference score, and then we have k images ranked generated by the same prompt T according to final preference score (the best to the worst are denoted asx 1 ≻ x 2 ≻ ... ≻ x k ).For each comparison, if x i is better and x j is worse, the loss function can be formulated as:EQUATIONwhere f θ (T, x) is a scalar value of reward model for prompt T and image x.Evaluation Results. We evaluate the preference prediction accuracy on the test sets among three human preference datasets: ImageRewardDB [34] , HPD v2 [32] and Pick-a-Pic [10] . Furthermore, DiffusionDB [29] ImageRewardDB [34] HPD v2 [32] Figure 4 : Win rates of generative models optimized with VP-Score compared to other reward models on three test benchmarks for PPO experiments. 'Tie' indicates instances where annotators think two images are of comparable quality.to better demonstrate the model's generalization performance, we computed the harmonic mean of accuracy across three sets for each model as an overall indicator of model performance. We use the CLIP score [18] , BLIP score [14] , Aesthetic score [25] , ImageReward [34] , HPS [33] , HPS v2 [32] and PickScore [10] as baselines to compare with the VP-Score.The results are presented at Table 2 . Our VP-Score demonstrates strong competitiveness compared to the current state-of-the-art reward models trained on human preference data. It achieves the second-best average performance among all preference reward models, following only HPS v2. Moreover, our model achieves optimal performance on the ImageRewardDB dataset, achieving a 0.6 performance gain compared to HPS v2. These results validate that leveraging fine-grained feedback provided by MLLMs enables learning a proficient human preference reward model.We aim to leverage the constructed preference dataset to align the performance of generative models more closely with human preferences. We utilize two popular reinforcement learning methods for finetuning: (1) Proximal Policy Optimization (PPO), where select ReFL [34] as our PPO implementation to adjust the generative model. (2) Direct Preference Optimization (DPO), which allows for direct model fine-tuning using preference data without a reward model, employing D3PO [36] as our DPO implementation. We use Stable Diffusion v1.5 [22] as the text-to-image generative model.Training Setting. For PPO experiments, we randomly sample 20,000 real user prompts from DiffusionDB [29] and 10,000 prompts in ImageRewardDB [34] as the training dataset. We compare VP-Score against five open-source reward models, including ImageReward [34] , PickScore [10] , and HPS v2 [32] , all trained on large-scale preference datasets (see Table 1 ). All models are fine-tuned with identical data and settings for consistency. For DPO experiments, we compare our VisionPrefer along with three open-source large-scale human-annotated preference datasets, ImageRewardDB [34] , HPD [33] and Pick-a-Pic [10] (see in Table 1 ). Notably, VisionPrefer scores are averaged across four aspects for fine-tuning, with an analysis on individual aspect scores presented in § 5. Both PPO and DPO use the same test benchmarks: 400 real user prompts from DiffusionDB [29] , 200 prompts from ImageRewardDB [34] and 400 prompts from HPD v2 [33] . Further details are in Appendix D.PPO Results. First, we visualize the evolution of various metrics as the model training steps increase when using our VP-Score as the reward function at Figure 3 . As training progresses, all metrics, including human preference metrics like HPS v2, show an increasing trend. This indicates consistency DiffusionDB [29] ImageRewardDB [34] HPD v2 [32] Figure 6 : Win rates of generative models optimized with VP-Score compared to other reward models on three test benchmarks for DPO experiments. 'Tie' indicates instances where annotators think two images are of comparable quality.between our VP-Score and other human preference metrics, demonstrating that VP-Score reliably aligns generative model outputs with human preferences.Then, we conduct a human preference study. Specifically, we use these fine-tuned generative models to generate 64 images for each prompt in evaluation dataset, followed by a top-3 selection by the corresponding reward models. Finally, ten human annotators rank these selected images. The results are presented at Figure 4 , detailed win count and win rates can be found in Table 4 . We observed that VP-Score fine-tuned generative model's Win+Tie ratio exceeds 50% when compared to all other models across all three test benchmarks, including some trained on large-scale human preference datasets like HPS v2. This suggests that, compared to other human preference reward model, VP-Score can serve as a reliable and competitive reward model for fine-tuning generative models to produce outputs closer to human preferences. This further underscores the effectiveness and competitiveness of VisionPrefer.The corresponding qualitative results shown at Figure 5 demonstrate that VisionPrefer fine-tuned generative model can generate images that are more aligned to text and with higher fidelity and avoid toxic contents. More qualitative results can be found in Figure 13 .DPO Results. We conduct a human preference study using the same procedure as PPO experiments at Figure 6 , detailed win count and win rates can be found in Table 5 . We found that the Win+Tie ratio of the generative model optimized on our VisionPrefer, when compared to the other three large-scale human datasets, exceeds 50%, substantiating the competitiveness of our VisionPrefer against humanannotated preference data. We show the qualitative results in Figure 7 . The results indicate that finetuning the generative model directly on our VisionPrefer using DPO yields performance comparable to that of fine-tuning the generative model on large-scale human-annotated preference dataset (e.g., Pick-a-Pick). Specifically, the generated results are more aligned with human preferences, exhibiting increased visual detail, better conformity to input prompts. More qualitative results can be found in Figure 14 . These experimental outcomes collectively affirm the efficacy of using preference data generated by MLLMs.Unlike existing preference datasets, our VisionPrefer not only provides preference rankings and scores for images but also includes corresponding textual explanations. This makes VisionPrefer The image captures the style of Van Gogh and an epic sky, but does not show Mona Lisa eating a burrito.Make the Mona Lisa eating a burrito.Original AfterBased on the explanations, please output one or more concise and effective editing instructions to address the issues raised in the explanations. We design an image editing pipeline as in Figure 8 , which contains two main steps: (1) Integrate the textual explanations from VisionPrefer into a specific prompt template and input it into LLMs (e.g., GPT-4), encouraging the LLMs to output one or more concise editing instructions to address the issues raised in the textual explanations. (2) Input these generated editing instructions into an image editing model (e.g., InstructPix2Pix [4] ), guiding it to edit images and address issues. This pipeline is simple in structure and can perform better by substituting more advanced LLMs and editing models.Training Setting. To explore the effectiveness of the textual explanations provided by VisionPrefer, we randomly selected 200 text-image pairs with scores below 3 (out of a maximum score of 5) as test cases. Following the outlined process, we performed image edits and evaluate the results before and after editing using existing reward models and conducted human studies with 10 participants.Results. The evaluation results for images before and after editing are summarized in Figure 9 . We found that both the reward models and human studies indicate a Win ratio > 50%, demonstrating that the edited images are superior to the original ones. This also validates the effectiveness of the textual explanations provided by VisionPrefer and the design of our pipeline. 
analysis: 
 Which MLLMs is the Best Annotator?The annotation of VisionPrefer heavily relies on GPT-4 V. Although many researchers pointed out that GPT-4 V capable of providing meticulous judgments and feedback [31, 6] , we still concern whether the GPT-4 V preferences are qualified. We then conduct a probing experiment by utilizing different MLLMs, GPT-4 V, Gemini-pro-V and LLaVA 1.6-34B, to provide their preference on two existing human-preference datasets (HPD [33] and ImageRewardDB [34] ). The corresponding pair-wise preference prediction accuracy is shown in Figure 10 (a) . We observed that the accuracy of GPT-4 V surpasses that of both LLaVA 1.6-34B and Gemini-pro-V on both datasets, achieving accuracy rates exceeding or approaching 70%, and LLaVA 1.6-34B notably scoring significantly lower than the former two. According to previous research [6, 32] , the agreement rate between qualified human annotators is also around 70% (65.3% for ImageRewardDB and 78.1% for HPD). Therefore, the probing experiment validates that GPT-4 V can be a well human-aligned annotator, thus ensure the quality and reliability of our VisionPrefer.To further validate the efficacy on preference annotation ability of GPT-4 V, we utilize Gemini-pro-V and LLaVA 1.6-34B to collect similar amount of data (1.2 M pair-wise preference choices) following the same collection pipeline described in Section 3. Then we train the corresponding reward model on these two datasets and show the performance in Figure 10 (b). As we can see, consistent with the aforementioned conclusion, the testing accuracy of the reward model trained on data annotated by GPT-4 V exhibits the highest performance, followed by Gemini-pro-V. This demonstrates that GPT-4 V is currently the most proficient annotator for text-to-image generation.Encouraging GPT-4 V(ision) for Enhanced Annotations.• Prompt Manner. As described in Section 3, during the construction of VisionPrefer, we encourage GPT-4 V to directly output scores for various aspects (e.g., aesthetic) of each image (denoted as score feedback). Another straightforward prompting manner (denoted as rank feedback) is encourage GPT-4 V to directly provide a ranking of images (α and β) in a certain aspect (i.e., α ≻ β, β ≻ α, or α = β). It is interesting to explore which prompting manner is best suit for AI Annotators. We randomly sampled 1,000 samples from ImageRewardDB and 500 samples from HPD, and utilized the two aforementioned prompt manners to ascertain GPT-4 V's annotations. The results are presented at Figure 10 (c ), we observe that in both datasets, the accuracy achieved using score feedback is higher than that achieved using rank feedback. • Temperature τ . Temperature τ is a hyperparameter used in multimodal large language models (e.g., GPT-4 V) to control the randomness and creativity of the generated results. A lower value of the temperature parameter will lead to a more predictable and deterministic output, while a higher value will produce a more random and surprising output. We investigate the influence of the variation in τ on both the accuracy of annotation and annotation consistency (where the same input yields identical annotation results). The results are shown in Figure 10 (d), we observe a decrease in accuracy as τ increases, indicating that lower values of τ should be set when conducting preference annotations. Furthermore, as τ increases, annotation consistency continues to decline, which is sensible because more randomness in the results leads to different preference outcomes for identical inputs over time.Fine-Grained Feedback Leads to Better Results.• Better Reward Modeling. In our previous experiments, we used the average score of each sample across four evaluation aspects as the final preference score for modeling reward or optimizing generation models. Here, we explore the impact of separately modeling the four evaluation aspects. We first train four reward models on these four different aspect in VisionPrefer, namely VP-Score-P, VP-Score-A, VP-Score-F and VP-Score-H, respectively. The corresponding preference accuracy are presented at Figure 10(e) . We can observe that the accuracy of reward models individually trained using a single aspect preference data is consistently lower than VP-Score, which validates the effectiveness of our approach in designing four evaluation aspects to model the preference level. • Better Prompt-Following. We found fine-grained preference data enables our fine-tuned model to generate images that better adhere to the input prompt. For instance, as shown in Figure 11 , the top-3 sampled images generated from our fine-tuned model all satisfy the "holding" prompt requirement, whereas only HPS v2 among baseline models achieves this. • More Aesthetically Pleasing. Fine-grained data enhances the visual appeal and vividness of images generated by our model. As shown in Figure 12 , our results exhibit enhanced luminosity, dynamic sensation, and increased detail, aligning more closely with human aesthetic preferences. • Enhance Image Safety.Using unsafe prompts from [36] , we generated 1K images and assessed safety using the Diffusion library's NSFW detector. The NSFW ratio for models fine-tuned with VP-Score (4.4%) was substantially lower compared to HPS v2 (21.1%) and PickScore (22.3%) fine-tuned models, indicating our preference scoring's effectiveness in reducing harmful content generation.More related details and ablation studies about the effectiveness of fine-grained feedback can be found in Appendix B. 
conclusion: 
 In this paper, we explore utilizing MLLMs to construct a large-scale high-quality feedback dataset, Vi-sionPrefer, for diffusion models alignment and refining. Costly experiments conducted across various experimental settings have validated the efficacy of VisionPrefer. This also represents a comprehensive and substantial endeavor by RLAIF in the realm of visual generative models, demonstrating the effectiveness of utilizing AI-synthesized data for aligning visual generative models.We organize our appendix as follows:• In Section A, we provide an ablation study about the impact of training dataset sizes and different reward model backbones. • In Section B, we additional ablation studies to validate the efficacy of fine-grained feedback.• In Section C, we present the statistical analysis of VisionPrefer, revealing that preference labels annotated by MLLMs demonstrate characteristics akin to those found in humanannotated preference datasets. • In Section D, we provide training details of the experiments discussed in the main text.• In Section E, we analyze the annotation costs of constructing VisionPrefer and demonstrate that employing MLLMs as annotators not only achieves results aligned with human annotators but also significantly reduces labor resources and time costs. • In Section F, we provide the corresponding prompt templates used in employing GPT-4 to polish existing prompts and employing GPT-4 V to generate preference annotations.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our contribution does not include new datasets or pre-trained models that pose a risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Code that we derive from earlier work is properly licensed and referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: Our experiment solely involves measurement and does not entail behavioral manipulation; therefore, we did not apply for IRB approval. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We conducted ablation experiments to explore the scalability of VisionPrefer and the impact of different backbones on the reward model's performance.• Scalability. To investigate the impact of training dataset sizes on VP-Score performance, we conduct comparative experiments (see Figure 15 ). Results show that increasing training data enhances VP-Score's prediction accuracy. This indicates that models trained on our VisionPrefer exhibit strong performance scalability, implying that more training data leads to further performance improvements.In the future, we plan to further increase the volume of data in our VisionPrefer and explore whether models trained on our dataset can outperform all these trained on human-annotated datasets. This endeavor holds significant promise and interest. • Reward Model Backbone. VP-Score adopts BLIP [14] as the backbone, which may raise curiosity about how well BLIP compares to CLIP [18] . We employed these two models as the backbone for our reward model and explored their effectiveness on our VisionPrefer. The results are summarized in Table 6 , where we observed that the performance of BLIP surpassed that of CLIP and this conclusion aligns with the findings on human preference datasets [34] . CLIP [18] BLIP [14] ImageRewardDB [34] 65.9 66.3 HPD v2 [32] 79.1 79.4 Pick-a-Pic [10] 67.3 67.1 Average 70.3 70.5In this section, we present additional ablation studies and qualitative results to validate the efficacy of fine-grained feedback design.B.1 Better Prompt-Following.To further substantiate that "Prompt-Following" rating labels in VisionPrefer can enhance the ability of fine-tuned models to generate images that more accurately align with the input prompts, we removed the "Prompt-Following" rating labels from the VisionPrefer, preserving labels for the other three aspects, and train a reward model, which we named VP-Score † . An additional human preference study was conducted on the DiffusionDB, specifically focusing on the aspect of Prompt-Following.The outcomes, presented in Table 7 , indicate two key findings: Firstly, VP-Score exhibits competitive performance in comparison to HPS v2. Secondly, the efficacy of VP-Score † experiences a notable decline with the omission of Prompt-Following rating labels. These results decisively confirm the critical role of "Prompt-Following" rating labels in enhancing the model's proficiency in adhering to prompts, thereby facilitating the generation of images that more precisely reflect the provided descriptions.Besides, we provide additional visualization results in Figure 17 to validate that generative models guided by VP-Score are capable of producing images that more closely adhere to the descriptions provided in the prompts.B.2 More Aesthetically Pleasing.Similar to the last section, we removed the "Aesthetic" rating labels from the VisionPrefer and trained a reward model named VP-Score ♣ . Subsequent to this, we embarked on an additional human preference study utilizing the DiffusionDB, with a singular focus on the dimension of "Aesthetics". The findings, elucidated in Table 8 , revealed that VisionPrefer achieved the best performance, while the exclusion of aesthetic labels markedly diminished the operational efficiency of VP-Score ♣ . This phenomenon starkly highlights the integral value of aesthetic rating labels.Further, we showcase additional visual outcomes in Figure 18 . Our observations indicate that generative models refined under the auspices of VP-Score manifest the capacity to engender imagery replete with more vibrant detail and sophisticated interplays of light and shadow.To ascertain the impact of "Fidelity" rating labels, we excised these labels from the VisionPrefer and subsequently trained a reward model VP-Score ♢ . A human preference study, concentrated solely on the "Fidelity" aspect, is documented in Table 9 . This study utilized the "anything" prompts delineated in [36] , encompassing 442 prompts, as the evaluation benchmark. The outcomes illustrate that generative models guided by our VP-Score manifest competitive performance. In contrast, VP-Score ♢ exhibits a discernible performance decrement relative to VP-Score.Additionally, the visualization results showcased in Figure 19 demonstrate that models optimized under the guidance of VP-Score excel in producing images with diminished distortion, e.g., less distortion of human hands. Note that image distortion, particularly the deformation of hands and limbs, is a common issue with diffusion generative models. Our "Fidelity" assessment can only mitigate, not eliminate, this phenomenon. Therefore, we look forward to the development of more robust techniques to address this drawback.Similarly, we removed the "Harmlessness" labels from the VisionPrefer and trained a corresponding reward model named VP-Score ♠ . Then we employ unsafe prompts provided in [36] to generate 1K images and utilize the built-in NSFW detector in the diffusion library 1 to quantify the frequency of generating harmful content. Detailed results is presented in Figure 16 . We find that VP-Score ♠ , trained without the "Harmlessness rating" labels, exhibited a significant increase in the NSFW ratio compared to the original VP-Score (4.4% to 20.2%). This further underscores the importance of "Harmlessness" labels. In this section, we provide more details of VisionPrefer. We encourage readers to delve into Section C.3, where we analyze the characteristics of preferences generated by GPT-4 V. This analysis reveals that the generated preferences exhibit properties remarkably similar to those of humanannotated preferences. Such findings serve to demonstrate the capability of MLLMs to closely align with human judgment and preferences in the context of text-to-image generation.C.1 Prompts.A key step in VisionPrefer contruction pipeline is utilizing GPT-4 to polish the existing prompt benchmarks. This process is designed to reduce potential biases and inconsistencies in user-generated terminology. We present examples of original prompts alongside their polished counterparts at Table 10 , and quantitatively illustrates the frequency distribution of certain stylistic words and conflicting prompts at Figure 20 . Our analysis reveals that the post-polish prompts not only align more closely with conventional expression norms but also demonstrate a significant reduction in the use of stylistically charged and specific words, such as platform and artist names. Moreover, the occurrence of prompts with conflicting information witnessed a marked decrease post-cleanup. As a result, these polished prompts are better suited for use as training data, meticulously crafted to minimize bias and enhance the model's robustness and generalization abilities. Neon gorilla skull in a cyberpunk style.highly detailed digital painting, black male anthro -lynx, human with head of lynx, with hair like fabio, facial scar, hairy masculine gigachad, muscular, wearing kilt and gold armbands, fur texture, lounging on bed aboard the nostromo, trending on artstation, romance novel.A digital painting depicts a black male anthropomorphic lynx with long hair, a facial scar, and a muscular build, wearing a kilt and gold armbands, lounging on a bed. 11 . For illustrative purposes, Figure 21 showcases representative images produced by each of these models. All preferences in VisionPrefer were generated using GPT-4 V API. As stated in the main text, we designed four evaluation aspects to assess the quality of each data entry. For each aspect, we individually invoked the GPT-4 V API to generate the corresponding preferences, the corresponding prompts regarding to these four aspects can be found depicted in Section F.2. Please refer to Table 12 to see some annotation examples.We illustrate the distribution of GPT-4 V's ratings across four distinct aspects within VisionPrefer in Figure 22 . Our analysis discerns a relatively even distribution of ratings for the Prompt-Following aspect, where the allocation of ratings from 1 to 5 is almost uniform. Conversely, in the domains of Fidelity and Harmlessness, GPT-4 V exhibits a propensity towards assigning the highest rating of 5 to a predominant share of the samples. This pattern suggests that the majority of generated images are free from substantial distortions and objectionable content.VisionPrefer offers a unique opportunity to leverage GPT-4 V' preferences for unbiased analysis. A critical step in the construction of VisionPrefer is the random application of class-free guidance values (from 3 to 12), aiming to enhance the generalization of VisionPrefer. We analyze the impact of changing the class-free guidance values of Stable Diffusion XL on its performance. For each guidance value, we compute the win ratio, representing the percentage of judgments where its use led to a preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling a detailed analysis of which classifier-free guidance scales are more effective. The results are presented at Figure 23 (a), and we find higher guidance value always lead to a higher win rate, e.g., a guidance value of 9 usually yields preferred images when compared to a guidance value of 3. This conclusion is in well agreement with the conclusions obtained in the human-annotated preference dataset [10] .Additionally, by comparing preferences for images generated by different generative models in response to identical prompts, we are able to identify the model that is more favorably rated by GPT-4 V. For instance, considering judgments in which one image was generated by Dreamlike Photoreal 2.05 and the other by Stable Diffusion 2.1, we can evaluate which model is more performant. As shown in Figure 23 (b), we find that GPT-4 V typically exhibits a preference for Dreamlike Photoreal 2.05 over Stable Diffusion 2.1. This preference aligns with the characteristics of human-annotated preference data [10] , demonstrating a consistency between the MLLMs generated preferences and human judgment.Table 12 : Example of annotations in VisionPrefer. In VisionPrefer, each data item includes a prompt, four images generated based on that prompt, along with the preference rating for each image across four different aspects, and their corresponding rationales.Prompt: minimalist summertime architecture by atey ghailan ( ( and edward hopper ) ).Prompt-Following Aesthetic Fidelity HarmlessnessRating: 5 Rationale: The image showcases a minimalist architectural style with a focus on simple geometric forms and warm colors that can be associated with summer. The artwork resembles the style of Atey Ghailan, aligning well with the input text. Rating: 4 Rationale: The architecture is minimalist and the image has a clear, crisp style similar to Edward Hopper's work. However, the image lacks the warm summertime lighting, which slightly deviates from the input text.Rationale:This image shows a simple depiction of a building's façade with no elements that could be deemed unsafe or inappropriate.Rating: 3 Rationale: The image displays architecture with a slightly more complex design, deviating from strict minimalism. The warm lighting suggests summertime, and there are hints of both Atey Ghailan and Edward Hopper's styles in the color palette and depiction of sunlight, though the alignment is not as strong as in other images.Rating: 3 Rationale: The image is in focus with adequate lighting, but the colors are somewhat dull. The composition is decent, capturing the essence of the architecture, but it lacks a bit of creativity to elevate it further.Rating: 4 Rationale: The image shows a street scene with architecture that could be interpreted as minimalist. The lighting suggests summertime. The style is slightly more detailed than what is typically seen in Atey Ghailan's work, hence the rating.The image presents an impressionistic view of a street and buildings, without any offensive or sensitive material.This image strongly aligns with the input text, featuring a minimalist architectural scene with a clear influence of Edward Hopper's style in the composition and use of light. The simplicity and color choice also reflect the summer theme and Atey Ghailan's artistic tendencies.Rationale: The image is sharp and well-exposed, with a good contrast between the cool tones of the building and the warm red of the figure. The composition is thoughtful, with the window framing the subject, creating an intimate and introspective mood.Rating: 5 Rationale: The image showcases a very minimalist interior scene with a clear influence from Edward Hopper in the composition and lighting. It aligns well with the summertime architecture theme. No errors are present.This image shows a stylized representation of a person sitting by a window. The implied nudity and the person's pose may be considered suggestive, making it moderately safe. It may not be suitable for children but does not contain explicit sexual content or severe violenceFollowing [34] , we load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of VP-Score, and initialize MLP head according to N (0, 1/(d model + 1)) decaying the learning rate with a cosine schedule. To avoid the overfitting of reward model during training phase and reach up to the best preference accuracy, VP-Score is fixed 70% of transformer layers and is trained on 4 × 32 GB NVIDIA V100 GPUs, with a per-GPU batch size of 16.PPO. Following the setting in ReFL [34] , we fine-tuned all text-to-image generative models employing the PNDM noise scheduler and half-precision computation on an array of 8 × 32GB NVIDIA V100 GPUs. The process utilized a learning rate of 1 × 10 -5 and a total batch size of 64 (32 for pre-training and 32 for ReFL).DPO. Following the setting in [36] , we conducted a total of 400 epochs during the training process, utilizing a learning rate of 3 × 10 -5 and the Adam optimizer, alongside half-precision computation. This was conducted on a configuration comprising 8 × 32GB NVIDIA V100 GPUs.One of the primary motivations for utilizing MLLMs as annotators is their ability to significantly reduce the cost of data construction compared to human annotators. Take construction process of the two largest existing human preference datasets as an example, during the construction process of Pick-a-Pic [10] , approximately 6,394 web users participated in tagging images with their preferences. For the development of HPD v2 [32] , a total of 57 high-quality annotation experts were employed and trained to construct preference labels. Moreover, these annotators were required to meticulously adhere to the annotation standards provided by the system throughout the labeling process. Thus, it is evident that obtaining large-scale humans preference annotation is time-consuming, resourceintensive, and laborious, which hinders the progress of related research.In contrast, employing MLLMs for annotation can effectively overcome these limitations. Utilizing the construction process of VisionPrefer as an example, each invocation of the GPT-4 V API is capable of tagging four images with preference labels pertaining to a specific aspect (e.g., promptfollowing aspect), meaning a single API call can generate C 2 4 preference ranking results in that aspect. Throughout the construction of VisionPrefer, each GPT-4 V API can accommodate approximately 10,000 requests per day, thus generating around 60,000 preference ranking results in a given aspect per day. In the specific construction process, we employed two APIs for parallel annotation, with the total annotation process taking approximately 15 days. This efficiency and cost-effectiveness are significantly superior to using human expert annotations. Moreover, despite the minimal cost and high efficiency, the reliability and quality of the preference labels provided by MLLMs are not compromised. Preference Instruction for Prompt-Following Aesthetic: Your role is to evaluate the aesthetic quality score of given images ("Images") generated by the corresponding text ("Input"). The four images given are independent, and should be evaluated separately and step by step. Note that the rating has nothing to do with image input order. Scoring: Rating outputs 1 to 5:1. Bad: Extremely blurry, underexposed with significant noise, indiscernible subjects, and chaotic composition. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/huggingface/diffusershttps://huggingface.co/runwayml/stable-diffusion-v1-5https://huggingface.co/stabilityai/stable-diffusion-2-1https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0 


Decision: Summary of Review Process:
- 3 reviews.
- Mixed scores: 4,6,7
- Authors submitted rebuttals/responses
- 1 out of 3 reviewers did not engage in discussion


Meta Review:
There was consensus from all reviewers regarding the novelty of the approach, clarity of writing, and experimental findings of the paper.  The following is a summary of questions and rebuttal:
- Reviewer AdFm who did not participate in the discussion gave an "Accept" (7) rating with two weaknesses mentioned: (1) efficiency/practicality concerns due to slow convergence (2) inconsistent improvement across datasets. The reviewer also pointed out two limitations. 
    - The authors have added some clarification for (1) and have acknowledged that improving the convergence speed should be a future research direction. 
    - The authors will be adding a note about bias in annotations to the Limitations sections.
    - The authors have provided more insight into the failures of the VisionPrefer model along with additional experiments for combining human and AI annotations.
- Reviewer 8ytx mentioned two weaknesses/requests: (1) provide quantitative metrics and (2) move details about the dataset to the main paper
    - authors have responded to (1) by stating that some quantitative metrics are in Table 2 of the main text and Tab 4,5 in Appendix. Authors also performed additional measurements and compared FID values, to be included in the final version
- Reviewer zspv mentioned 4 weaknesses/questions. Q1 was about novelty, Q2 was about the focus of the paper, Q3 was about experimental setup, and Q4 was about training a bigger model.  
    - authors and reviewers discussed these concerns but there was no change in the rating as there was a disagreement on the  focus and scope of the paper

Overall, the AC believes this to be an interesting work on leveraging MLLMs to create a preference dataset and training a reward model to guide the tuning of text-to-image generative models. The AC's recommendation is to carefully integrate suggestions from reviewers and additional experiments/evidence/clarifications that emerged from the discussion, into the final version.

Reviews: review: paper_summary: paper_summary: The paper presents a new AI-generated dataset aimed at enhancing text-to-image generative models by aligning them more closely with human preferences. The data is annotated by multimodal large language models (MLLMs) and captures detailed preferences across multiple dimensions like prompt-following, aesthetic, fidelity, and harmlessness. The dataset facilitates the training of a reward model called VP-Score, which approximates human preference predictions.
strength_weakness: strength_weakness: 1. the overall novelty is a bit limited as it heavily relies on existing large models to generate the dataset and to train additional models (more in the questions). 

2. the paper seems to have a diverse set of focuses. For example, the section regarding editing images does not seem to correlate well with the main proposed method, thus seem irrelevant.
questions: questions: 1. The biggest question would be the relevance of the proposed method. The authors only showed the empirical strength of the proposed method with a randomly sampled subset of the images. In this case, the true value of the proposed method can hardly be validated. I would highly recommend the authors to repeat the experiments with the standard large-scale training regime. I understand the computation requirement will be huge, but there does not seem to have an alternative path to show the power of the proposed method. 

2. Similarly as above, a big question would be whether the proposed method is only due to the power of GPT-4V, which is a larger model, to show the true value of the method, the authors might have to use the method and GPT-4V to train a bigger model.
limitations: limitations: The paper has an analysis section and ablation study in the appendix. An explicit limitation section can further help readers appreciate the paper.
paper_summary: paper_summary: The paper introduces VisionPrefer, a large-scale, fine-grained preference dataset constructed using multimodal large language models (MLLMs) as annotators. VisionPrefer aims to improve the alignment of text-to-image generative models with human preferences by providing detailed feedback on generated images. The dataset is created through a three-step process: generating and polishing prompts, generating images using various models, and obtaining preference annotations from GPT-4 V, including scalar scores, rankings, and textual explanations.
strength_weakness: strength_weakness: Convergence Speed of VP-Score Figure 3 indicates that VP-Score converges significantly slower than other baselines. This slower convergence raises questions about the efficiency and practicality of using VP-Score in real-world applications. The authors should investigate and explain the reasons behind this slower convergence. Potential factors could include the complexity of the fine-grained annotations or the training dynamics of the reward model. Providing insights into this issue and suggesting potential optimizations to improve convergence speed would enhance the utility of VP-Score.


Inconsistent Improvement Across Datasets
Despite the larger scale and richer annotations of VisionPrefer, the dataset does not achieve consistent improvements across all evaluated datasets. This inconsistency suggests that VisionPrefer may not fully capture human preferences, indicating potential limitations in the design of the annotations or the preference modeling approach. The authors should analyze why VisionPrefer fails to deliver uniform performance gains and identify aspects where it falls short. This analysis could involve examining the nature of the datasets where VisionPrefer underperforms and exploring ways to refine the annotation process to better align with human preferences. Additionally, providing more detailed comparisons and breakdowns of performance across different datasets would offer valuable insights into the strengths and weaknesses of VisionPrefer.
questions: questions: see weaknesses
limitations: limitations: Bias in Annotations: The use of AI-generated annotations (e.g., GPT-4 V) could propagate existing biases, leading to biased outputs from generative models. The authors should discuss mechanisms to detect and mitigate such biases.

Combining AI and Human Annotations: Explore the benefits of combining AI and human annotations to enhance the diversity and generalizability of preference data, ensuring the model aligns more closely with a broad range of human preferences.
paper_summary: paper_summary: This paper introduces VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-to-image generative alignment.  VisionPrefer offers advantages in scalability, fine-grained annotations, and a comprehensive feedback format compared with existing preference datasets. The authors further propose a reward model, VP-Score, which exhibits competitive correlation with human preferences by utilizing VisionPrefer. The experimental results underscore the effectiveness of both VisionPrefer and VP-Score.
strength_weakness: strength_weakness: 1. This paper demonstrates that fine-grained feedback from MLLM help to yield more human-preferred images in terms of prompt-following, esthetic, fidelity and harmlessness with qualitative results.  It is better if the authors could provide some quantitative metrics.

2. A significant contribution of this paper lies in the dataset. However, examples of annotations in VisionPrefer can only be found in the appendix. It would be more intuitive if these were included in the main paper.
questions: questions: Please see the wearknesses.
limitations: limitations: The authors have discusses the limitations in the paper.

Comments: title: Response to Reviewer zspv
Replies to rebuttal
Reviewer zspv: please respond to the authors' rebuttal
comment: Dear Reviewer zspv:

Thank you for your detailed feedback. We believe there are some misunderstandings that we need to clarify:

1. First, we want to clarify that the two types of models involved in this paper (MLLMs, specifically GPT-4V, and text-to-image generative models) are not the same type of models. The text-to-image generative model (simplified as T2I) takes text as input and produces images, while the language-based MLLMs take both text and images as input and output text. **Therefore, they are not the same kind of models as large and small models, and this differs from distillation**, which simply transfers the capabilities of a large model to a smaller model for the same task. **To consider this as distillation, MLLMs would need to provide reliable T2I preference labels. But is it certain that they are more reliable and stronger preference providers?** Although MLLMs are powerful tools proven to offer various functions (such as image captioning, QA, etc.), **there has been no prior research exploring or proving that MLLMs can produce reliable preference labels. Our research is the first to address this concern and demonstrate this point (which is the core motivation of our study, see line 39)**. Therefore, we used GPT-4V to create VisionPrefer and conducted a series of extensive experiments to show that MLLMs are indeed reliable preference providers. This is one of the main contributions and value propositions of our paper.
2. **We want to emphasize that our approach and the research motivation of this paper are independent of and unrelated to the size of the T2I models.** Preference labels reflect human preferences for different images generated from the same prompt. We have demonstrated that, for current-scale T2I models, the preference labels provided by MLLMs are effective (sections 4) and align with those provided by humans. Current-scale T2I models can further optimize their generation results by learning from these preference labels to better align with human preferences. **If very large-scale T2I models (perhaps on the same scale as GPT-4V) learn little from these preference labels, this would occur with both GPT-4V-generated and human-annotated preference labels.** This does not affect the value of our research; rather, **it raises the question of whether aligning very large-scale T2I models is necessary, which is an interesting research topic in its own right** (though currently difficult to verify).
3. **We selected the SD v1.5 model following previous research settings in the T2I alignment domain [1-5].** Your suggestion to align larger T2I models to verify the effect is an interesting proposal that could enrich our research. However, we believe that the existing timeline is insufficient to support further experimental feedback. We plan to conduct experiments in this area in the future.
4. Lastly, we want to reiterate that the research motivation of this paper is to explore "Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?" (line 39). We have emphasized the research motivation of our paper in many places throughout the text (e.g., lines 20, 39, 200, 240, 288, etc.). To avoid any remaining misunderstandings, we will make sure to revise the paper to highlight our motivation more clearly in the modified version.

---

Additionally, we would like to reiterate the broader value of our work, which extends beyond just VisionPrefer and VP-Score:

1. Since human annotation of preference labels is expensive and time-consuming, proving the reliability of MLLMs in generating human preference annotations allows humans to easily scale existing human preference datasets, thereby achieving better text-to-image generative alignment.
2. It further supports the notion that AI-generated synthetic data is a reliable way to create augmented datasets and provides a large dataset for further research related to AI-generated synthetic data.
3. It shows that existing AI models may exhibit capabilities similar to those of human experts in some interesting areas, e.g., image aesthetics, which is exciting and worthy of further exploration.

---

**If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)**

Best regards,

All Authors

**Reference**

[1] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." NeurIPS 2024.

[2] Fan, Ying, et al. "Reinforcement learning for fine-tuning text-to-image diffusion models." NeurIPS 2024.

[3] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." CVPR 2024.

[4] Yang, Kai, et al. "Using human feedback to fine-tune diffusion models without any reward model." CVPR 2024.

[5] Clark, Kevin, et al. "Directly fine-tuning diffusion models on differentiable rewards." ICLR 2024.
Dear authors, 

Thank you for offering a comprehensive rebuttal in a polite manner despite my review is the only one showing negative polarity. I understand you have concerns about the review; sharing the same concerns, I have re-read the paper, which is why it takes time for me to respond. 

First of all, by the question raised "Unfortunately, we did not fully understand your specific suggestion regarding 'training a bigger model to show the true value of the method.'" I was referring to the experiment in section 4.2, which I believe is to use the constructed dataset to show that generative models can benefit from the dataset to align better (per line 157). In this case, it seems to me that the authors are forming a loop of construct a dataset with a bigger generative model and then show that it can help train a better smaller model. This is where I ask the question about the "true value" of the method yet to be validated. 

I understand the above requirement might be difficult to implement but I believe it will be essential for the study. For example, a simple logic loophole is that the bigger model will of course be a better aligned one and help the smaller model to perform better, it's like the conventional knowledge distillation research that we can get better small model with the help of a big model, this does not mean the same techniques will help improve the bigger model. 

On the other hand, the authors might want to respond that "getting a better big model" is not the goal of this research, this research is about investigating how GPT-4V level models can offer better-aligned data to train smaller model in the beginning. In this case, I agree the paper is better justified, but in this case, it seems to me the paper will need some re-writing work, e.g., line 19 sends out a much stronger message than this. 

Again, I thank the reviewers for offering a professional and polite rebuttal, and it takes me a while to respond because the current situation makes me feel like I need to re-read the paper, but it seems some of the concerns are still there. There is still a chance that I misread some parts, and I'm here for the authors to enlighten on these parts.
Dear reviewer,

thanks for your participation in the NeurIPS peer review process.  We are waiting for your response to the rebuttal.  You gave a reject rating (3), with 2 weaknesses and 2 questions.

Is the response from authors satisfactory?
- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Dear reviewer,

thanks for your participation in the NeurIPS peer review process.  Thanks for indicating that you are leaning towards accepting the paper.  Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review? 
- If yes, are you planning to increase your score?  
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Thans for the authors' response. I tend to keep my original rating for acceptance.