Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Authors: Xun Wu
Shaohan Huang
Guolong Wang
Jing Xiong
Furu Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation;Reinforcement Learning from AI Feedback;Multimodal Large Language Model
Primary area: generative_models

Abstract: Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Xun Wu',
Shaohan Huang les
Guolong Wang", Jing Xiong
Furu Wei
Microsoft Research Asia, 2 University of International Business and Economics
The University of Hong Kong
wwwu@micccosof....ocmooooooco... Shachanh@microee croofft. com, fuwei 0mi @microsoft.cco
Left
Right
Left
Prompt-Following
Aesthetic
"happy artists pointing things"
"beautiful nender, soviet brutalist city.
Right
Harmlesaness
Fidelity
BISFGO
"professional photography of human hands"
wet plate photograph, portrait a strikingly beautiful woman with regal features"
Figure 1: Fine-grained feedback from multimodal large language model help to yield more human-preferred images. Left: Output generated by the baseline text-to- eext-to-image generative model. Right: Output generated by the baseline model optimized with fine- fine-grained feedback from multimodal large language model. We illustrate improvements generation quality across four aspects: Prompt- Following, Aesthetic, Fidelity and Harmlessness. See in Appendix for more visualization examples.
Abstract
Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, making it to generate more human-pret prefferred images. Despite these advances, current human preference datasets either prohibitively expensive to construct suffer from lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning open-source texl-to-image generative models and hinder further explo- ration. To address these challenges, we first leverage multimodal large language models create VisionPrefer, a fine-grained preference dataset that captures mul- tiple preference aspects prommt--ollowing aesthetic, fidelity, and harmlessness). Then we train corresponding reward model, VP-Score, over VisionPrefer to guide the tuning of text-to-image generative models. The preference prediction accuracy of Vp-Score validated to comparable that of human annotators. validate the effectiveness of VisionPrefer and VP-Score, we adopt two reinforcement learn- ing methods, Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO), to supervised tne-tune generative models, and extensive experimental
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic. and gener- alizes better than previous human -peefernree metrics across various image distribu- tions. Our findings indicates that the integration of A Al-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in -ottto--mmae generative models. VisionPrefer and VP-Score git.
are available at https:/ thub. http///sshiiw///ssinoonoonoo
Introduction
Text-to-image generative models [3 21 19 26 have experienced rapid advancements in recent years. For example, large-scale text-to-image diffusion models, exemplified by Imagen [24] and DALL-E 2 [1), have demonstrated the capabiliity to generate high-quality and creative images when provided with textual prompts. However, despite recent progress, current generative models still exhibit misalignment with human preferences, such conflicts with text prompts incorrect content 17 A pivotal approach to addressing this issue is utilizing Reinforcement Learning from Human Feedback (RLHF) [16] [ to supervised fine-tune text-to-image generative models with preference data [3 3 17].
Preference data is crucial for aligning generative models with text prompts. However, existing human-crafice preference datasets, such as HPD v2 [33] 32] and Pick-a-Pic [T0] either provide only broad, general preference comparisons without fine-grained, accurate preference evaluations, or they are limited in size. Additionally, using humans for preference annotation is expensive and time-consuming, restricting progress in this research area.
Drawing inspiration from recent research utilizing Al-generated preference data as training supervise signal for Large Language Models (LLMs) alignment on Natural Language Processing domain [2] 2) we pose the following question:
Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator
for Text-to tomaeeo--meeeeeee Generation?
These multimodal large language models (MLLMs), trained on web-scale text and text-image pairs, have already demonstrated formidable capabilities in image understanding. To this end, we first introduce VisionPrefer, a publicly available Al-generated dataset that features millions of finely- grained preferences concerning model-generated images. Compared with existing human preference datasets, VisionPrefer offers the following benefits:
Scalability & Low Cost: As shown in Table 1 VisionPrefer encompasses 1.2 M preference choices across 179 K pairs of images, establishing it as the largest text-to-image generation preference dataset to date. Additionally, because VisionPrefer is annotated by MIL.Ms, it can be easily expanded further and the construction cost is significantly lower than human annotation.
Fine-grained preference: To more accurately and diversely evaluate the preference scores of generated images, carefully develop a detailed preference annotation guideline for MILMs, Fidelity, Aesthetic, and Harmlessness. The detail requirement for each aspect is presented at Table
which covers four distinct aspects: Frompt-Followee
Comprehensive feedback formats: Unlike existing benchmarks that provide only rankings or preference indices, our VisionPrefer not only provides preference rankings but also includes preference scores and textual explanations for the preference annotations from each aspect, which makes VisionPrefer more versatile, e.g., allowing it to serve as a textual guiding resource for image re-editing and refinement.
Building on the VisionPrefer, we conducted an extensive investigation into its most effective utiliza- tion. First, we developed a preference reward model named VP-Score optimized on VisionPrefer. trained to evaluate generated images based on their likelihood of being preferred by humans. Experi- mental results demonstrate that VC-Score exhibits a competitive correlation with human preferences compared to existing human preference reward models. Moreover, we employ two reinforcement learning methods, Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO), to enhance generative models to better align with human preferences. As illustrated in Figure [ VisionPrefer markedly enhances text-image alignment compositional image generation across diverse aspects, such aesthetics. In summary, our contributions follows:

---

Table Statistics of existing preference datasets for text-to- text-to-image generative models. "Fine- tenee-grained" denote containing preference regarding multiple aspects or not.
Corresponding Reward Model
Preference Choices
Open Source?
Fine Gruined?
Feedback Format
Dataset
Anmotator
Prompts
Ranking
Text
Scalar
RichHF-18K 15
Homan
18K
18K
HPD 25
HPS v1
Discord users
25K
25K
X
HPD
22
HPS v2
Human Expert
108K
798K
ImageRewann DB
34
ImageReward
Human Expert
9K
137K
X
Pick-a-Pic (v2)
10
PickScore
Web
59K
851K
VisionPrefer (ours)
VP-Score
GPT-4 Vüision)
179K
1.2M
We construct VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-lo-image generative alignment. Compared with existing preference datasets, VisionPrefer has the advantages of scalability, fine-grained annotations, and comprehensive feedback format.
Based on VisionPrefer, we propose reward model, VP-Score, which achieves competitive correlation with human preferences with other automated human preference metrics.
Experimental results demonstrate the effectiveness of both VisionPrefer and VP-Score. Addi- tionally, we provide comprehensive analysis of them gain deeper understanding of how Al-generated synthetic data and models trained on such data impact future research in this domain.
2
Related Work
Text-to-Image Generative Models Alignment. While existing text-to-image generative models often generate images that do not closely match human preferences, thus alignment in the context of diffusion has garnered increasing attention 228 10
17 28 There are two main types of text-to- image generative models alignment algorithms: (i) Proximal Policy Optimization (PPO). For example, reward weighted method [33 first explores using human feedback to align text-to-image models with human preference. ReFL 34 trains a reward model, ImageReward, using human preferences and subsequently utilizes for fine-tuning. (i) Direct Policy Optimization (DPO). DPOK 7 fine-tune tunes text-to-image diffusion models by using policy gradient to maximize the beedback-traings reward. ZO-RankSGD 228 optimizes diffusion in an online fashion with human ranking feedback. RAFT
and AlignProp 17 tune the generative model to directly increase the reward of generated images. Several manually annotated preference datasets are proposed to support above algorithms [32 34 10
Their overall statistics are shown in Table These manually annotated data have two drawbacks. First, manual annotation need heavy cost, leading to a small-size set. Second, manual annotations are prone to specific biases [32].
Reinforcement Learning from AI Feedback. introduced the idea of Reinforcement Learning from AI Feedback (RLAIF), which used LLM-labeled preferences in conjunction with human-labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness, Recent works have also explored related techniques for generating rewards from LLMs (20 [5]. These works demonstrate that I.LMs can generate useful signals for reinforcement learning fine-tuning. However, RLAIF for text-to-image generative model alignment is less explored. 30 leveraged MLLMs to assess the alignment between generated images and input texts, focusing on aspects like object number and spatial relationship. T2l-ComBennch [ utilized MLLMs like BLIP-VQA to evaluate the text-to- text-to-Image generative models. Our work diverges from previous works in two principal ways. (I) Previous approaches provided limited data, sometimes even less than what's 's annotated manually (e.g., 6K 19.. (2) Prior methods had limited aspects in alignment, lacking consideration for aspects such fidelity 30).
3 VisionPrefer
We introduce VisionPrefer, a fine-grained preference dataset constructed by collecting feedback from MLLMs annotators. The collection pipeline of VisionPrefer is shown in Figure 2 which mainly consists of three steps: prompt generation, image generation and preference generation.
Step-1: Prompt Generation. We generate prompts based DiffusionDB 29 large-scale text-to- image prompt benchmark containing 1.5M user-written prompts following two steps: (1) Polish. As discussed in 33). significant portion of the prompts in the DiffusionDB is biased towards certain
3

---

Prompt Pool
Comparison Dafa
Preference Annotation
bellen cain, dandiiy the taan
Biflogion DB
FrolmptFoollowigg
Fidetity
Text- Text-to-Jmage Generative Models
Imagee
Jmang
Aesthatic
Harmdesswess
Diffiplon Modals
Figure 2: VisionPrefer construction pipeline. We sample textual prompts and text-to-image generative models from pools to generate diverse comparison data, then query GPT-4 V with detailed illustrations for fine-grained and high-quality annotations in both textual and numerical formats.
styles (e.g.. "Greg Rutkowski" appears in around 15% of prompts). We utilize GPT-4 to polish prompts in DiffusionDB to obtain unbiased prompts following instructions shown in Appendix F.] (2) NSFW Filting. We employ state-of-the-arr NSFW detectors [8] to compute an NSFW score for each prompt and filter out prompts that exceed a certain threshold, following [29]. After these two steps, we combine both polished prompts and the original prompts in DiffusionDB as our final prompt benchmark, which contains 179K prompts.
Step-2: Image Generation. We generate images using different text-to-image generative models (see details in Appendix C.2) by sampling textual prompts constructed in Step- 1 as input. For each prompt, we generate four images by randomly selecting different generative models from the model pools with different classifier-free guidance scale values, to ensure high diversity. This diversity allows for a comprehensive evaluation preference prediction model's generalization capability and facilitates the training of a more generalizable model. Finally, we obtain 0.7 16M images.
Step-3: Preference Generation. We employ state-of-the-art multimodal large language model, GPT-4 V, to provide three types of feedback: (1) Scalar scores that indicate the fine-graneed quality regarding multiple aspects, (2) Preference ranking according to the scalar scores, and (3) Textual explanations that give detailed guidance on how to improve the completion, encompassing four distinct fine-grained aspects namely: Prompt-Following Aesthetic, Fidelity, and Harmlessness for each generated image (See the example in Table 12p. Detailed input instructions for GPT-4 V to annotate preference labels are in Appendix F22 Finally, obtain 1 M.2M preference choices.
4 Experiments
In this section, we first train a corresponding reward model named VP-Score and evaluate it on existing human- human-preferent preference datasets (.1.... Next, enhance existing text-to-imare generative models by adopting two reinforcement learning algorithms ($ 4.2) to validate the efficacy of VisionPrefer and Vc-Sc. After that, we design simple pipeline edit generated images with the textual explanations VisionPrefer(($
4.1 Reward Modeling
Training Setting, We train the VP-Score over VisionPrefer. VP-Score adopts the same model structure as ImageReward [34], which is open-source human- preferrneee reward model and utilizes BLIP [14] as the backbone. Similarly to training the reward model for the language model [6]. we formulate the preference annotations
VisionPrefer as rankings. Specifically, we employ the average scores of each sample in VisionPrefer across four aspects as the final preference score, and then we have le images ranked generated by the same prompt according final preference score (the best to the worst are denoted as X2 > > xx). For each comparison, if x; is better and X, is worse, the loss function can formulated as:
loss(f) = -E,Txxxx)- [log (a (fo (T,x;) fo (T, (T,x,)))]
(1)
where Jo(T, x) is a scalar value of reward model for prompt T and image x.
Evaluation Results. We evaluate the preference prediction accuracy on the test sets among three human preference datasets: ImageRewardDB (34), HPD v2 (32) and Pick-a-Pic [10). Furthermore,

---

1.0
Table 2: Preference prediction accuracy across the test sets of ImageRewardDB, HPD v2 and Pick-a-Pic. The Aesthetic Classifier makes pre- diction without seeing the text prompt. The best performance is in bold, and the second-best per-
Reward Senre
08
ImsgeReward
0.7
0.6
Aesthetic
PickScore
0.4
HPS
0.3
VP-Score (Ous))
formance is underlined.
100 200
300
Model
JmageKewardDDB
HPD x2
a-cc Average
Training Steps
VIT-
Figure 3: Evaluation results of the text-to-imape image model's generation quality across multiple reward
Aesihetic 25
57.4
76.8
56.8
62.44
RmageReward
models when maximizing scores from VP-Score
IPPS
33
61.2
77.6
66.7
67.84
PickScore
62.9
during the PPO training process. All scores are normalized for better visualization.
HPS v2 32
65.7
83.3
67.4
71.32
Anbole
25%
305
75%
100%
02
2578
50%
100%
2516
100%
Top- uelesee imago
Sap-
valeciaa imago
topd
tmague
DiffusionDB
229
ImageRewardDE
34
HPD v2 B2
Figure 4: Win rates of generative models optimized with VP-Score compared other reward models on three test benchmarks for PPO experiments. 'Tie" indicates instances where annotators think two images of comparable quality.
to better demonstrate the model's generalization performance, computed the harmonic mean of accuracy across three sets for each model as an overall indicator of model performance. We use the
CLIP score (18). BLIP score [14), Aesthetic score (25), ImageReward (34] HPS 33 HPS v2 32
and PickScore 10 as baselines to compare with the VP-Score.
The results presented Table 2 Our VP-Score demonstrates strong competitiveness compared
to the current state-of-the-ard reward models trained on human preference data. It achieves the second-best average performance among all preference reward models, following only HPS v2. Moreover, our model achieves optimal performance on the ImageRewardDB dataset, achieving a 0.6 performance gain compared HPS v2. These results validate that leveraging fine-grained feedback provided by MILMS enables learning a proficient human preference reward model.
4.2 Fine-tuning -m-ag Generative Models
We aim to leverage the constructed preference dataset to align the performance of generative models more closely with human preferences. We utilize two popular reinforcement learning methods for fine- tuning: (I) Proximal Policy Optimization (PPO), where select ReFL 34 as our PPO implementation to adjust the generative model. (2) Direct Preference Optimization (DPO), which allows for direct model fine-tuning using preference data without reward model, employing D3PO (36] implementation. We use Stable Diffusion v1.5 22 as the text-to-image generative model.
DPO
Training Setting. For PPO experiments, randomly sample 20,000 real user prompts from DiffusionDB 29 and 10,000 prompts ImageRewardDB 34 as the training dataset. We compare VP-Score against five open-source reward models, including ImageReward (34), PickScore [10] and HPS v2 [32), all trained on large-scale preference datasets (see Table Dp. All models are fine-tuned with identical data and settings for consistency. For DPO experiments, compare VisionPrefer along with three open-source large-scale numan-annotated preference datasets, ImageRewardDB [4] HPD 33 and Pick-a-Pic [10] (see in Table IT. Notably, VisionPrefer scores averaged across four aspects for mne-tuning, with an analysis on individual aspect scores presented in $ $5) Both PPO and DPO use the same test benchmarks: 400 real user prompts from DiffusionDB [29, 200 prompts from ImageRewardDB 34 and 400 prompts from HPD v2 B3 Further details are in Appendix
PPO Results. First, we visualize the evolution of various metrics as the model training steps increase when using our VP-Score as the reward function at Figure 3
training progresses, all metries, including human preference metrics like HPS v2, show increasing trend. This indicates consistency
5

---

beautiful dark landscape, person wearing virtual reality headset, intricate, epic lighting. cinematic composition, hyper realistic, 8k resolution, unreal engine 5
201.5
0.32
Amithatc
Profile portrait in Angolan realist style with ultromarine blue, venetian red, and titanium white colors
500.5
CLD
AustheTic
DiaageFanord
VP- TT-ScIPE
Figure 5: Qualitative results for PPO experiments, SD 1.5 denotes the Stable Diffusion v1.5 model without any fine-tune. See Appendix for more samples.
1810
Shichwttros
25%
100%
75%
1009
5090
7000
05
205
795
Tops colecred Shres 61 Images
10pp Toreetd mrom Imagee
cellaat mom images
DiffusionDB
22
ImageRewardDB
HPD v2 32
Figure 6: Win rates of generative models optimized with VP-Score compared other reward models on three test benchmarks for DPO experiments. "Tie" indicates instances where annotators think two images are comparable quality.
between our VP-Score and other human preference metries, demonstrating that VP-Score reliably aligns generative model outputs with human preferences.
Then, we conduct human preference study. Specifically, use these fine-tuned generative models to generate 64 images for each prompt in evaluation dataset, followed by top-3 selection by the corresponding reward models. Finally, ten human annotators rank these selected images. The results are presented at Figure 4 detailed win count and win rates can be found in Table 4 We observed that VP-Score fine-tuned generative model's Win+Tic ratio exceeds 50% when compared to all other models across all three test benchmarks, including some trained on large-scale human preference datasets like HPS v2. This suggests that, compared to other human preference reward reliable and competitive reward model for fine-tuning generative
model, VP-Score can serve models to produce outputs closer to human preferences. This further underscores the effectiveness and competitiveness of VisionPrefer.
The corresponding qualitative results shown at Figure demonstrate that VisionPrefer fine-tuned generative model can generate images that are more aligned to text and with higher fidelity and avoid toxic contents. More qualitative results can be found in Figure
DPO Results. We conduct human preference study using the same procedure as PPO experiments Figure detailed win count and win rates can be found in Table We found that the Win+Tic ratio of the generative model optimized on our VisionPreferr, when compared to the other three large-scale human datasets, exceeds 50%, substantiating the competitiveness annotated preference data. We show the qualitative results in Figure 7 The results indicate that fine- tuning the generative model directly on our VisionPrefer using DPO yields performance comparable to that of fine-tuning the generative model large-scale Pick-a-Pich -Pick). Specifically, the generated results are more aligned with human preferences, exhibiting increased visual detail, better conformity to input prompts. More qualitative results can be found Figure 14] These experimental outcomes collectively affirm the efficacy of using preference data generated by MLLMs.
our VisionPrefer against human-
preference dataset (e.g.,
4.3 Editing Generated Images with VisionPrefer
Unlike existing preference datasets, our VisionPrefer not only provides preference rankings and scores for images but also includes corresponding textual explanations. This makes VisionPrefer

---

Portrait of a man wearing golden mask
1.5
InoeRewardDe
Pick-s-Pic
VisiorPrefer
Figure 7: Qualitative results for DPO experiments. SD 1.5 denotes the Stable Diffusion v1.5 model without any fine-tune. See Appendix for more samples.
STEP-1: Generate Ediling innormmmenn
Image Ediling
Original
Image Editing Model
Explanations
The image captanse the style of Gogt and eptcsky. bet does not show Mora Line euting humio.
Prompt
RPs
on epplaations, ustpst whineg the
255
50%
75%
1005
Figure 9: Reward model eval and
Editing Instruction
esplanctios.
Make the Mona Lisa euting burrio.
human eval image editing results. Win denotes the edited image is bet- ter than the original image.
Figure 8: Overview of the proposed image editing pipeline with the textual explanations in VisionPrefer.
more versatile, enabling applications such as editing images based on textual explanations to better align with human preferences.
We design an image editing pipeline as in Figure which contains two main steps: (1) Integrate the textual explanations from VisionPrefer into a specific prompt template and input it into LLMs (e.g.. GPT-4), encouraging the LLMs output issues raised in the textual explanations. (2) Input these generated editing instructions into an image editing model (e.g., InstructPix2Pix (4), guiding it to edit images and address issues. This pipeline is simple in structure and can perform better by substituting more advanced L.LMs and editing models.
more concise editing instructions to address the
Training Setting. To explore the effectiveness of the textual explanations provided by VisionPrefer, we randomly selected 200 text-image pairs with scores below (out a maximum score of 5) as test cases. Following the outlined process, we performed image edits and evaluate the results before and after editing using existing reward models and conducted human studies with 10 participants.
The evaluation results for images before and after editing are summarized in Figure 9 We found that both the reward models and human studies indicate Win ratio 50%, demonstrating that the edited images are superior to the original ones. This also validates the effectiveness of the textual explanations provided by VisionPrefer and the design our pipeline.
Results.
5 Analysis
Which MLLMS is the Best Annotator?
The annotation of VisionPrefer heavily relies on GPT-4 V. Although many researchers pointed out that GPT-4 V capable of providing meticulous judgments and feedback B1] 66, we still concern whether the GPT-4 V preferences are qualified, We then conduct a probing experiment by utilizing different MILMs, GPT-4 V. Gemini-pro-y and LLaVA 1.6-34B, to provide their preference on two existing bumannpreference datasets (HPD (3 and ImageRewardDB (34). The corresponding
pair-wise preference prediction accuracy is shown in Figure 10 (a). We observed that the accuracy

---

Herran vs Hernan InegrfiranDDi HNS
InageRewaddD0
Heman vs Hurren InegeRewandD
HPD
90
Aucuracy
10
56.95
Averenyy
70/6
Accurscy
50 $0.915
S7. S7.11
73.5%
61
70%6
UMV-4
Grnmiggo-V
LLaVA 16-48
Gemin yon-V
LLaV 16:3ii
Sumr Foctlrack
Rank Fasilttck
(a)
(c)
VI-SScorc
VP-S:on--P
VP-Soxe-AA
VP-Scoro-F
VP-Secoc-H
Accunacy
80
6718
1375 62./1
23.%
111
0.3
0.9
1.5
Tomporassc
ImogeRewandDD
IIPD x2
Pick-rPiic
(e)
Figure 10: (a) Pair-wise preference prediction accuracy comparison across three MLLMs on two numan-preference datasets (b) Results of preference prediction accuracy for reward models trained on preference datasets annotated by different MLLMs annotators, (c) Preference prediction accuracy for score feedback and ranking feedback. (d) Visualization depicting the variation of annotation accuracy and consistency with changes in temperature т. (e) Preference prediction accuracy among reward models trained on different aspects of preference data VisionPrefer.
of GPT-4 V surpasses that of both LLaVA 1.6-34B and Gemini-pro on both datasets, achieving accuracy rates exceeding approaching 70%, and LLaVA 1.6-34B notably scoring significantly lower than the former two. According to previous research [6] 32]. the agreement rate between qualified human annotators is also around 70% (65.3% for ImageRewardDiB and 18.1% for HPD). Therefore, the probing experiment validates that GPT-4 V can be a well human-aligned annotator, thus ensure the quality and reliability of our VisionPrefer.
To further validate the efficacy on preference annotation ability of GPT-4 V, we utilize Gemini Gemini-pro- V and LLaVA 1.6-34B to collect similar amount of data (1.2 M pair-wise preference choices) following the same collection pipeline deseribed in Section 3 Then we train the corresponding reward model on these two datasets and show the performance in Figure 10 (b). As we see, consistent with the aforementioned conclusion, the testing accuracy of the reward model trained on data annotated by GPT-4 V exhibits the highest performance, followed by Gemini-pro-V This demonstrates that GPT-4 currently the most proficient annotator for text-to-image generation.
Encouraging GPT-4 ((iiion) for Enhanced Annotations.
Prompt Manner. As described in Section 3 during the construction of VisionPrefer, we encourage GPT-4 V to directly output scores for various aspects (e.g., aesthetic) of each image (denoted as score feedback). Another straightforward prompting manner (denoted as rank feedback) is encourage GPT-4 V to directly provide a ranking of images (a and (3) in a certain aspect (i.e., > B, B > a, a 5). is interesting to explore which prompting manner is best suit for AI Annotators. We randomly sampled 1,000 samples from ImageRewardDD and 500 samples from HPD, and utilized the two aforementioned prompt manners to ascertain GPT-4 V's annotations. The results are presented at Figure (c), we observe that in both datasets, the accuracy achieved using score feedback higher than that achieved using rank feedhack.
Temperature T. Temperature 7 is hyperparameter used in multimodal large language models (e.g.. GPT-4 V) to control the randomness and creativity of the generated results. A lower value of the temperature parameter will lead to a more predictable and deterministic output, while a higher value will produce more random and surprising output. We investigate the influence of the variation in on both the accuracy of annotation and annotation consistency (where the same input yields identical annotation results). The results are shown Figure I0 (d), we observe decrease in accuracy as increases, indicating that lower values of 7 should be set when conducting preference annotations. Furthermore, as 7 increases, annotation consistency continues to decline, which is sensible because more randomness in the results leads to different preference outcomes for identical inputs over time.
Fine-Grained Feedback Leads to Better Results.
Better Reward Modeling. In our previous experiments, we used the average score of each sample across four evaluation aspects the final preference score for modeling reward or optimizing generation models. Here, we explore the impact separately modeling the four evaluation aspects. We first train four reward models on these four different aspect in VisionPrefer, namely VP-Score-P,
8

---

hyper-realistic portrait of woman holding flowers, featuring coftagecore and grunge aesthetic.
Figure 11: Fine-grained feedback enables our model (denoted as VP-Score) to generate results that better align with the input prompt. See Appendix for more samples.
beautiful render, soviet brutalist city, brutalism, architecture 60s, skyscropers, blocks, streets, greyacele, depressingg elegant, highly detailed, digital painting, artstation, dramatic lighting, by greg rutkowski and wlop and artgem.
art, smooth, shorp focus, octone render.
6
Tane gelmeerd
VP- Scart
HPS
ScarE
Score
Figure 12: Fine-grained feedback enhances the aesthetic and vividness of the our results (denoted as VP-Score). SD 1.5 denotes the Stable Diffusion v1.5 model without any fine-tune. See Appendix for more samples.
VP-Score-A, VP-Score-F and VP-Score-H, respectively. The corresponding preference accuracy are presented at Figure T0)e). We observe that the accuracy of reward models individually trained using single aspect preference data is consistently lower than VP-Score, which validates the effectiveness of approach in designing four evaluation aspects model the preference level.
Better Prompt-Following, We found fine- mne-grained preference data enables our fine-tuned model to generate images that better adhere to the input prompt. For instance, as shown in Figure [I the top-3 sampled images generated from our fine-tuned model all satisfy the "holding" prompt requirement, whereas only HPS v2 among baseline models achieves this.
More Aesthetically Pleasing. fine-grained data enhances the visual appeal and vividness of images generated by our model. As shown in Figure 12, our results exhibit enhanced luminosity, dynamic sensation, and increased detail, aligning more closely with human aesthetic preferences.
Enhance Image Safety.Using unsafe prompts from [36], we generated 1K images and assessed safety using the Diffusion library's NSFW detector. The NSFW ratio for models fine-tuned with VP-Score (4.4%) was substantially lower compared to HPS v2 (21. 1%) and PickScore (22.3%) fine-tuned models, indicating our preference scoring's effectiveness in reducing harmful content generation.
More related details and ablation studies about the effectiveness of fine-grained feedback can be found in Appendix B
6 Conclusion
In this paper, explore utilizing MLLMs to construct large-scale high-quality feedback dataset, Vi- sionPrefer, for diffusion models alignment and refining. Costly experiments conducted across various experimental settings have validated the efficacy of VisionPrefer. This also represents a comprehen- sive and substantial endeavor by RLAIF in the realm of visual generative models, demonstrating the effectiveness of utilizing Al-synthesized data for aligning visual generative models.

---

References
[1] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 05862, 2022.
[2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernande7, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Lar- son, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlissh T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
[3] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning, arXiv preprint 233.::3330..3300 2023.
[4] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint (eXi::2211008000 2022.
[5] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 23iv:2309.17400, 2023.
[6] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023,
[7] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint 16388,
[8] L. Hanu and Unitary team. Detoxify: Toxic Comment Classification with Pytorch Lightning and Transformers, 2020.
[9] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. Ti-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36.78723-78747 2023.
[10] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023.
[11] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2022.
[12] H. Lee, Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V. Carbune, and Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0022 002677 2023.
[13] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. Gu. Aligning text-to-image models using human feedback. arXiv preprint 12112 2023.
[14] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified linionnlanguage understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[15] Y. Liang, J. He, G. Li. Li, A. Klimovskiy, N. Carolan, J. Sun, J. Ponton Tuset, S. Young, F. Yang. al. Rich human feedback for text-to-image generation. arXiv preprint ar21v:2312.102400 2023.
[16] L. Ouyang, J. WU, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
10

---

[17] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki. Aligming text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
[18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I, Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
[19] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1,2):3, 2022.
[20] P. Roit, J. Ferret, L.. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L., Hussenot, O. Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00/88 2023.
[21] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10684-10695, 2022.
[22] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[23] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Prucessing Systems, 35:36479-36494, 2022.
[24] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 15:36479-36494 2022.
[25] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint vrXiv:2210.08-000 2022.
[26] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[27] N. Stiennon, I.. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 -0021, 2020.
[28] Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.00 03751, 2023.
[29] Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau. Diffusiondb: large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.1488 2022.
[30] S. Wen, G. Fang, R. Zhang, P. Gao, H. Dong, and D. Metaxas. Improving compositional text-to-image generation with large vision-language models. arXiv preprint 1rX0v:2310.06311, 2023.
[31] Wu, G. Yang, Z. Li, K. Zhang, Liu, L. Guibas, D. Lin, and G. Wetzstein. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092, 2024.
[32] X. Wu, Y. Hao, K. Sun. Y. Chen, F. Zhu, R. Zhao, and H. Li. Human preference score v2: A solid benehmark for evaluating human preferences of text-to-image synthesis. arXiv preprint 2tt.:2330009444 2023.
[33] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Human preference score: Better aligning text- to-image models with human preference. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2096-2105, 2023.

---

[34] Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong, Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 2023.
[35] K. Yang, D. Klein, A. Celikyilmaz, N. Peng, and Y. Tian. Rled: Reinforcement learning from contrast distillation for language model alignment, 2023.
[36] K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, Q. Li, W. Shen, X. Zhu, and X. Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint 2rXiv:231..1333, 2023.
12

---

We organize our appendix as follows:
Section
we provide an ablation study about the impact training dataset sizes and
different reward model backbones.
In Section B We additional ablation studies to validate the efficacy of fine-grained feedback.
we present the statistical analysis of VisionPrefer, revealing that preference labels annotated by MLLMs demonstrate characteristics akin to those found in human- annotated preference datasets.
In Section
Section we provide training details of the experiments discussed the main text.
In Section
we analyze the annotation costs of constructing VisionPrefer and demon- strate that employing MLLMs as annotators not only achieves results aligned with human annotators but also significantly reduces labor resources and time costs.
E
provide the corresponding prompt templates used in employing GPT-4 to polish existing prompts and employing GPT-4 to generate preference annotations.
In Section F
Table 3: Examples of Al annotators annotations in VisionPrefer from four aspect.
Prompt-Alignment
Fidelity
Aesthetic
Harmlessness
generated images faithfully show accurate objects ac- attributes, with lationships between objects and events described in prompts being correct.
generated images should he perfect exposure, rich
generated images do not clude inappropriate content pornography. privacy violations, violence, discrim- ination. or generally NSFW themes.
generated images should be true to the shape and char- acteristics that the object should have will be generated haphazardly.
fine details and masterful composition with emotional impact. well align with acs- thetic of human.
Table 4: Human evaluation study on win count and win rate of generative models optimized with different reward models, benchmarked against the Stable Diffusion v1.5 baseline. Compared to other reward models, VP-Score exhibits competitive performance. The best results are highlighted in bold, while the second-best results are underlined.
DiffusionDB
29
Rel'L
HPD v2 K
Reward Model
WinRate
#Win
WinRate
#Win
WinRate
CLIP 18
267
54.09
137
52.05
270
53.31
Aesthetic 25
280
56.71
144
53.93
283
54.77
ImageReward 34
281
56,93
153
55.81
291
56.38
PickScore 10
286
57.87
164
56.66
298
$7.87
HPS v2
32
58.21
171
56.87
287
$7.89
VP-Score (Ours)
329
57.98
177
$7.09
295
57.80
Table 5: Human evaluation generative models optimized with different preference datasets in DPO experiments. The best results are highlighted bold, while the second-best results are underlined.
DiftusionDB
ReFL
HPS
Preference Datasets
#Win
WinRate
#Win
WinRate
#Win
WinKate
ImageReward
34
253
54,31
144
53.87
281
55.01
HPD 32]
266
57.08
149
55.71
278
54.49
Pick-a-Pic Pic
59.43
156
$8.33
297
58.23
VisionPrefer (Ours)
275
59.03
158
59.17
303
59.44
13

---

male wizard, brown hair, green robes, glasses, D&D, painted fantasy character portrait, highly detailed, digital painting, artstation, concept art, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha
501.9
VP-Score
Aasthatic
ImageReward
HPS +2
caracal eating dumplings with restourant the background, presented in whimsical, Pixar-style digital painting
PickScore
VP Scoorc
VP- Scar
VP Score
Portrait of in the style of impressionism by Patrice Murciano
801.5
cli
PickScore
VP
Aesthetic
TrageReward
VP-Score
VP. SP-Seere
The classical Roman church, highly detailed, artstation, strong contrast of light and shadow, neon colors, sharp focus 4K UHD image
clip
PickScore
DOR DAROOL
IraogeRewond
HPS
VP-Scorm
Score
Figure 13: Qualitative comparison between text-to-image generative model optimized with the guidance of VP-Score and other reward models. SD 1.5 denotes the Stable Diffusion v1.5 model without any fine-tune.
14

---

batman monster digital art, fantasy, magic, trending on artstation, ultra detailed, professional illustration by Basil Gogos
SDI.3
HPD
VisiorPrefer
ImageRewordDB
Pick-a-Pic
VisionPrefer
VisionPrefer
car in center JZX100 twin turbo drift on road, surrounded by trees and buidlings in Tokyo prefecture, rooftops are Japanese architecture, city at sunset heavy mist over streetlights, cinematic lighting, photorealistic, detailed wheels, highly detailed
SD1.5
HPD
VisiorPrefer
VisionPrefer
ImageRewardDB
Pick-a-Pic
VisionPrefer
VisionPrefer
Figure 14: Qualitative comparison between generative model trained on VisionPrefer and other uoman-annotated preference datasets. SD 1.5 denotes the Stable Diffusion v1. model without any fine-tune.
15

---

A
Ablation Study
We conducted ablation experiments explore the scalability of VisionPrefer and the impact of different backbones on the reward model's performance.
Scalability. To investigate the impact of training dataset sizes VP-Score performance, conduct comparative experiments (see Figure 15 Results show that increasing training data enhances VP- Score's prediction accuracy. This indicates that models trained our VisionPrefer exhibit strong performance scalability, implying that more training data leads to further performance improvements. In the future, we plan to further increase the volume of data in our VisionPrefer and explore whether models trained on our dataset can outperform all these trained on human- datasets. This endeavor holds significant promise and interest.
Reward Model Backbone. VP-Score adopts BLIP 14 as backbone, which may raise curiosity about how well BLIP compares to CLIP [18]. We employed these two models as the backbone for our reward model and explored their effectiveness on our VisionPrefer. The results are summarized in Table [6] where observed that the performance of BLIP surpassed that of CLIP and this conclusion aligns with the findings on human preference datasets [34].
80
75
Table 6: Ablation study for different reward model backbones.
70
Accuracy
65
Backbone
60
ImageRewardDB
Datasets
CLIP [[8]
BLIP [4]
IIP v2
Pick-a-Pic
ImageRewardDB
65.9
66.3
45
HPD v2 32
79.1
79.4
0.2
0.4
0.8
12
Pick-a-Pic [10
67.3
67.1
Training Size
Average
70.3
70.5
Figure 15: Ablation study for the size of train- ing data used in optimizing VP-Score.
16

---

B Efficacy of Fine-Grained Feedback
In this section, we present additional ablation studies and qualitative results validate the efficacy of fine-grained feedback design.
B.1 Better Prompt-Followingg
To further substantiate that "Prompt-Foolooinng rating labels in VisionPrefer enhance the ability of fine-tuned models to generate images that more accurately align with the input prompts, we removed the Prompp-Following" rating labels from the VisionPrefer. preserving labels for the other three aspects, and train a reward model, which we named VP-Score An additional human preference study was conducted on the DiffusionDB, specifically focusing the aspect of Prompt-Followee The outcomes, presented in Table 7 indicate two key findings: Firstly, VP-Score exhibits competitive performance in comparison HPS v2. Secondly, the efficacy of VP-Scoref experiences notable rating labels. These results decisively confirm the
decline with the omission of Prompt-Followin critical role of Prompt-Following rating labels in enhancing the model's proficiency in adhering to prompts, thereby facilitating the generation of images that more precisely reflect the provided descriptions.
Besides, we provide additional visualization results in Figure 17 validate that generative models guided by VP-Score capable of producing images that more closely adhere the descriptions provided in the prompts.
B.2 More Aesthetically Pleasing.
Similar to the last section, we removed the "Aesthetic" rating labels from the VisionPrefer and trained a reward model named VP-Score Subsequent to this, we embarked on an additional human preference study utilizing the DiffusionDB, with a singular focus on the dimension of "Aesthetics". The findings, elucidated Table revealed that VisionPrefer achieved the best performance, while the exclusion of aesthetic labels markedly diminished the operational efficiency of VP-Score*. This phenomenon starkly highlights the integral value aesthetic rating labels.
Further, we showcase additional visual outcomes in Figure 18 Our observations indicate that generative models refined under the auspices of VP-Score manifest the capacity engender imagery replete with more vibrant detail and sophisticated interplays of light and shadow.
B.3 Reduce Image Distortion.
To ascertain the impact of "Fidelity" rating labels, excised these labels from the VisionPrefer and subsequently trained a reward model VP-Score A human preference study, concentrated solely on the "Fidelity" aspect, is documented in Table 9 This study utilized the "anything" prompts delineated in (36), encompassing 442 prompts, as the evaluation benchmark. The outcomes illustrate that generative models guided by our VP-Score manifest competitive performance. In contrast, VP-Score exhibits discernible performance decrement relative to VP-Score.
Additionally, the visualization results showcased in Figure 19 demonstrate that models optimized under the guidance of VP-Score excel in producing images with diminished distortion, c.g., less distortion of human hands. Note that image distortion, particularly the deformation of hands and limbs, is common issue with diffusion generative models. Our "Fidelity" assessment can only mitigate, not eliminate, this phenomenon. Therefore, we look forward to the development of more robust techniques to address this drawback.
B.4 Enhance Image Safety.
Similarly, we removed the "Harmlessness" labels from the VisionPrefer and trained corresponding reward model named VP-Score Then we employ unsafe prompts provided in (36) to generate 1K images and utilize the built-in NSFW detector in the diffusion library" to quantify the frequency of generating harmful content. Detailed results presented in Figure 16 We find that VP-Score* trained without the "Harmlessness rating" labels, exhibited significant increase in the NSFW ratio
17

---

compared to the original VP-Score (4.4% to 20.2%) This further underscores the importance of "Harmlessness" labels.
Table 7: Human evaluation study on the aspect of "Prompt-Following" The best results high- lighted in bold, while the second-bes best results are underlined.
Table 8: Human evaluation study on the aspect of "Aesthetic". The best results highlighted in bold, while the second-best results are denoted with an underline.
DiffusionDB
29
DiffusionDB
29
Reward Model
Reward Model
#Win
WinRate
#Win
WinRate
PickScore 10
311
57.24
PickScore 10
283
55.40
HPS v2 RP
316
58.27
HPS v2 R2
281
55.01
VP-Score
307
56.70
VP-Score"
275
53.82
VP-Score
315
58.07
VP-Score
55.96
Table 9: Human evaluation study on the aspect of "Fidelity". The best results highlighted in bold, while the second-best results are denoted with an underline.
NSFW Ratio (96)
22.3
21.1
20
18.7
15
12.9
10
Anything Prompts
36
Reward Model
#Win
WinRate
vp Score
Vp-Score*
CLIP
ImageReward
PickScare
HPS v2
PickScore
227
51.17
HPS
232
52.33
Vpore O
224
Figure 16: Fine- aine-grained feedback make genera- tion more safety.
VP-Score
231
52.20
18

---

painting depicts oak tree with human face resembling an old bearded man, crafted from the tree's 's bark
5b1.5
Clip
PickScore
VP-Score
Aesthe tic
Rewoe
VP
A ghost pirate aboard a pirate ship in spooky fog under moonlight
5D1.5
Clip
PickScone
VP-Score
Aesthetic
ImwoeR
HPS v2
VP-Score
VP-Score
Figure 17: Fine- grained feedback enhances the alignment of generated content with the input prompts. For instance, in the first column of the figure, only the generative model optimized under the guidance of VP-Score accurately produces a face that adheres to the description of being 'crafted from the tree's bark.' In the second column, solely the VP-Score-guidee generative model successfully constructs the image pirate, whereas the other models merely generate images of pirate ships. SD 1.5 denotes the Stable Diffusion v1. model without any fine-tune.
19

---

portrait of gorgeous punk vampire girl, elegant, digital painting, highly detailed, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha
Sb1.5
Clip
PickScore
ImageRewerd
VP-Score
VP
black dafne keen yoruba, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, d&d, art by rutkowski, orientalism, bouguereau
Clip
VP-Score
Aesthe tha
ImageReward
HPS v2
VP VP-Se
VP ScoSer
Figure 18: Fine-grained feedback enhances the vividness and richness of detail in generated content. SD 1.5 denotes the Stable Diffusion v1. 5 model without any fine-tune.
1girl, bangs, blunt bangs, bowl, brown hair, cherry blossoms, closed eyes, closed mouth, facing viewer, floral print, flower, green kimono, grey hair, hair flower, hair ornament, hands
TraggeReword
Picksorr
HPS
VP-SSom
Marla Singer is depicted smoking in setting reminiscent Blade Runner, highly detailed, digital painting, artstation, concept art, smooth, sharp focus
TrageReward
RickScarc
Figure 19: Fine-grained feedback reduce the image distortion. SD 1.5 denotes the Stable Diffusion v1.5 model without any fine-tune.
20

---

C Statistics of VisionPrefer
In this section, we provide more details of VisionPrefer. We encourage readers to delve into Section where we analyze the characteristics of preferences generated by GPT-4 V. This analysis reveals that the generated preferences exhibit properties remarkably similar to those human- annotated preferences. Such findings serve to demonstrate the capability of MLLMs to closely align with human judgment and preferences in the context of exx-tto-image generation.
C.1 Prompts.
key step in VisionPrefer contruction pipeline is utilizing GPT-4 to polish the existing prompt benchmarks. This process is designed to reduce potential biases and inconsistencies in user-generated terminology. We present examples of original prompts alongside their polished counterparts at Table [0] and quantitatively illustrates the frequency distribution of certain stylistic words and conflicting prompts at Figure (0) Our analysis reveals that the post-polish prompts not only align more closely with conventional expression norms but also demonstrate significant reduction in the use of stylistically charged and specific words, such as platform and artist names. Moreover, the occurrence of prompts with conflicting information witnessed marked decrease post-cleanup. As a result, these polished prompts are better suited for use as training data, meticulously crafted to minimize bias and enhance the model's 's robustness and generalization abilities.
Table 10: Examples of prompts polished by GPT-4, Certain style words are underlined.
Prompts from DiffusionDBB 22
Prompts cleaned by GPT-4
cy berpunk neon gorilla skull, by weta fx, by wlop, majestic look, trending on artstation.
Neon gorilla skull in a cyberpunk style.
highly detailed digital painting, black male anthro lynx, human with head of lynx, with huir like fabio, facial scur. hairy masculine gigachad, muscular, wearing kilt and gold armbands, fur texture, lounging on bed aboard the nostromo, trending on aristation, romance novel.
digital painting depicts black male anthropomorphic lynx with long huir, facial scur, and muscular build, wearing kilt and gold armbands, lounging on bed.
Original prompts
Percentage (%)
Prompts polished GPT-4
20-
"artstation" "greg rutkowski"
"mucha"
Conflict prompts
Figure 20: Frequencies of certain style words and conflict prompt. Confliction is judged by GPT-4,
C.2 Images.
Within VisionPrefer, images generated by employing four state-of-the-art text-to-image gener- ative models. These models are ranked as the top four on the Hugging Face leaderboard, specif- ically: Stable Diffusion v1-E?/ Stable Diffusion 2. 1 Dreamlike Photoreal 2. 2. Stable Diffusion 5 Detailed descriptions of these models and the distribution of images generated by each within our dataset are methodically outlined in Table For illustrative purposes, Figure 21 showcases representative images produced by each of these models.
21

---

Table 11: Image sources of VisionPrefer.
Source
Type
Resolution
Proportion
Stable Diffusion v1-5
Diffusion
512x512
26.3 %
Stable Diffusion 2.1
Diffusion
768 x 768
24.8 %
Dreamlike Photoreal 2.05
Diffusion
768 x 768
25.7 %
Stable Diffusion XL
Diffusion
1,024 1,024
23.2 %
painting of dark souls boss by rutkowski, dark fantasy art, high detail, trending artstation.
a beautiful painting of a microwave, a pear and a bottle of wine on a table, painted by van gogh
v1-5
2.1
Dreaml ike
XL
Figure 21: Some example images in VisionPrefer. SD denotes Sable Diffusion while Dreamlike denotes Dreaml ike Photoreal 2.05.
Rating
Rating 4
Rating
Rating 2
Rating
75.3
Percentage
53.4
25.8 22. 24.5
15.3 12.2
14.8
17.2 14.6
0.8
5.7
0.5
0.1
Frompt-Following
Aesthetic
Fidelity
Harmlessness
Figure 22: Distribution of GPT-4 V's scoring across four aspects VisionPrefer.
Win
Tic
Lose
Tie
Claosiar-fres
33%
10
11
Stable Diffusion 2.1
Dreamlike Photoreal 2.05
12
0%
25%
50%
75%
100%
Win: Win/Tie/oon Toe/Lose Ratio
(b)
Figure 23: (a) Win rate versus classifier-free guidance value for Stable Diffusion XL. (b) Preference distribution when comparing Stable Diffusion 2.1 with Dreaml ike Photoreal 2.05.
22

---

C.3 Preferences.
All preferences in VisionPrefer were generated using GPT-4 V API. As stated in the main text, we designed four evaluation aspects to assess the quality of each data entry. For each aspect, we individually invoked the GPT-4 V API generate the corresponding preferences, the corresponding prompts regarding to these four aspects can be found depicted in Section F.2 Please refer to Table [2 to see some annotation examples.
We illustrate the distribution of GPT-4 V's ratings across four distinct aspects within VisionPrefer in Figure [2 Our analysis discerns relatively even distribution of ratings for the Prompt-Following aspect, where the allocation of ratings from 1 to 5 is almost uniform. Conversely, in the domains of Fidelity and Harmlessness, GPT-4 exhibits propensity towards assigning the highest rating of to predominant share of the samples. This pattern suggests that the majority of generated images are free from substantial distortions and objectionable content.
VisionPrefer offers a unique opportunity to leverage GPT-4 V' preferences for unbiased analysis. critical step in the construction of VisionPrefer is the random application of class-free guidance values (from 3 to 12), aiming to enhance the generalization of VisionPrefer. We analyze the impact of changing the class-free guidance values of Stable Diffusion XL on its performance. For each guidance value, we compute the win ratio, representing the percentage of judgments where its use led preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling detailed analysis of which classifier-free guidance scales are more effective. The results are presented Figure 23 (a). and we find higher guidance value always lead a higher win rate, e.g., guidance value of 9 usually yields preferred images when compared to a guidance value of 3. This conclusion preference dataset [T0].
in well agreement with the conclusions obtained the
Additionally, by comparing preferences for images generated by different generative models in response identical prompts, V. For instance, considering judgments in which one image was generated by Dreamlike Photoreal 2.05 and the other by Stable Diffusion 2.1, we can evaluate which model is more performant. As shown in Figure 23 (b), we find that GPT-4 V typically exhibits a preference for Dreamlike Photoreal 2.05 over Stable Diffusion 2.1. This preference aligns with the characteristics of anmaan-n preference data (10), demonstrating consistency between the MLLMs generated preferences and human judgment.
able to identify the model that more favorably rated by GPT4
23

---

Table 12: Example of annotations in VisionPrefer. In VisionPrefer, each data item includes a prompt, four images generated based that prompt, along with the preference rating for each image across four different aspects, and their corresponding rationales.
Prompt: minimalist summertime architecture by atey ghailan ( and edward hopper ).
Input Image
Prompt-Following Following
Acsthetic
Fidelity
Hanalessness
Ruting:
Rating: 5
The image has clear and sharp focus, the exposure
Rationale:
The image show- a minimalist architce- tural style with focus on simple geometric forms and colors that
Rationale:
Rating: 5
Rationale: image picts a minimalist house with clear summettime lighting- The style very reminiscent Atey Ghailan's its simple shapes and warm colors, No crtors are present.
warm glow from the window, and the colors are rich and vibrant. The composition masserful, with a on the minimalist architee- ture and the ligure, ureating of calm and contem-- plation.
Rationale: The inuge de- picts peaceful scene of sitting porch ing sunset. or inappropriate content.
art- work resembles the style of Atey Ghuilan, aligning well with the input text.
offensive
Rating: 4
Rationalo: image tures minimalist architecture with clean lines and limited paletie, which reflez- summertime simplic- ity. It is reminiscent of Edl- ward Hopper's style, partie- ularly
Rating:
Rating:
Rationale: The image is sharp. with good eaposure The com- position thoughtful, with interesting play of goo- metric shapes and lines. The focal point tecture itself.
Rationale: The urchitecture minimalist the image crisp style simi- to Edward Hopper's work. However, the image lacks the summertime lighting, which slightly devialies the ingut text.
Rating: 5
Rationale: This image shows simple depiction a building's 's façade elements that could be deemed unsafe inuppropriate.
with
shadow, although the connee- tion to Aley Ghailan's style less apparent.
Rating: 3
Rationale: The dis- plays architecture slightly more complex de- sign. deviating from strict light-
Rating:
Rationale: The image is in focus with adequate lighting, colors composition de- eent, capturing the essence of the architecture, but lacks creativity further.
Rating:
Rationale: The image shows street scene with architec- that could interpreted minimalist. The lighting suggests summertime. The style slightly more detailed than typically Atey Ghailann's work, the rating.
Rating: 5
mänimalism. The ing suggesas summertime, and there are hints both Atcy Gihailan and Hilward Hopper's les in color palette depiction light, though the alignment is not as strong in other images.
Rationale: The image presents impressionizic view of street and build- ings, without offensive sensirive material.
strongyy aligns with imput text. featuring min- imalist architectural influence of Edward Hopper's style in the composition and use of light. The simplicity and chnice reflect summer theme and Atey Ghailan's artistic tendencies.
Rating:
Rationale:
image
Rationale: image sharp and well well-exposed, with good contrast between the building and the warn rod figure. The composition is tboughtul, with the window froming subject, creating intimate introspective mood.
Rating:
Rationale: The image show- cases mini molist into- clear influ- ence from Edward Hogper in the composation and lighting It aligns well with the architecture crices are present.
Rating:
shows a stylized representa- tion of a person sitingg by window, The implied mudity and person's pose may considered suggestive, making it moderately safe. It
Rationale:
Rating: 3
image
be suitable for explicit sexual contenr or severe violence
24

---

D Training Details
D.1 Reward Model Training
Following [34], we load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of VP-Score, and initialize MLP head according to (O,, 1 L/ddmdde + 1)) decaying the learning rate with a cosine schedule. To avoid the overfitting of reward model during training phase and reach up the best preference accuracy, VP-Score is fixed 70% of transformer layers and is trained on 4 x 32 GB NVIDIA V100 GPUs, with per-GPU batch size of 16.
D.2 Boosting Generative Models
PPO. Following the setting in ReFL [4], we fine-tuned all text-to-image generative models employ- ing the PNDM noise scheduler and half-precision computation on an array of 8 x 32GB NVIDIA V100 GPUs. The process utilized a learning rate of x 10-5 and a total batch size of 64 (32 for pre-training and 32 for ReFL).
DPO, Following the setting in (36, we conducted a total of 400 epochs during the training process, utilizing learning rate of x 10 and the Adam optimizer, alongside half-pre calf-precision computation. This was conducted on configuration comprising 8 x 32GB NVIDIA V100 GPUs
Cost of VisionPrefer Construction E
One of the primary motivations for utilizing MLLMs as annotators is their ability to significantly reduce the cost of data construction compared to human annotators, Take construction process of the two largest existing human preference datasets as an example, during the construction process of Pick-a-Pic [10]. approximately 6,394 web users participated in tagging images with their preferences. For the development of HPD v2 (32), a total of 57 high-quality annotation experts were employed and trained to construct preference labels. Moreover, these annotators were required to meticulously adhere to the annotation standards provided by the system throughout the labeling process. Thus, it is evident that obtaining large-scale humans preference annotation is time-consuming, resource- intensive, and laborious, which hinders the progress related research.
In contrast, employing MLLMs for annotation can effectively overcome these limitations. Utilizing the construction process of VisionPrefer as an example, each invocation of the GPT-4 API capable of tagging four images with preference labels pertaining to specific aspect (c.g.. prompt- following aspect), meaning a single API call can generate C² preference ranking results in that aspect. Throughout the construction of VisionPrefer, each GPT-4 V API can accommodate approximately 10,000 requests per day, thus generating around 60,000 preference ranking results in a given aspect day. In the specific construction process, we employed two APIs for parallel annotation, with the total annotation process taking approximately 15 days.
This efficiency and efstctfffctsvesess are significantly superior to using human expert annotations. Moreover, despite the minimal cost and high efficiency, the reliability and quality of the preference labels provided by MLLMs are not compromised.
25

---

Prompt Instruction Templates
F1 Prompt Polish Instruction
Preference Instruction for Prompt-Following
I will give you a description about an image. Remove modifiers from text that have nothing to do with the main content of the image, for example resolution, sharpness, light, image quality, authors and online platform, and describe it succinctly in one sentence. ## Original description (text): (INSERT DESCRIPTION HERE)
Note: Please provide your assessment results in the following format:
### Output (text): [insert the sentence you generated here]
26

---

F.2
Preference Instruction
Preference Instruction for
Prompt-Followingg
Your role is to evaluate the prompt-following quality score between given image and the corresponding text ("Input"). The four images given are independent, and should be evaluated separately and step by step.
Scoring: Rating outputs to 5:
1. Irrelevant: No alignment.
2. Partial Focus: Addresses one aspect poorly.
3. Partial Compliance:
-(1) Meets goal or restrictions, neglecting other.
(2) Acknowledges both but slight deviations.
4. Almost There: Near alignment, minor deviations.
5. Comprehensive Compliance: Fully aligns, meets all requirements.
Format:
## Input:
Text: (INSERT PROMPT HERE)
Image:
### Image JINSERT IMAGE HERE]
### Image 2 [INSERT IMAGE 2 HERE]
### Image [INSERT IMAGE HERE
### Image 4 [INSERT IMAGE 4 HERE
Note: Please provide your assessment results in the following format:
## Output
### Output for Image 1
Rating: [Rating for Image 1]
Rationale: [Rationale for the rating in short sentences]
### Output for Image 2
Rating: [Rating for Image 2]
Rationale: [Rationale]
### Output for Image 3
Rating: [Rating for Image 3]
Rationale: [Rationale]
#*# Output for Image 4
Rating: [Rating for Image 4]
Rationale: [Rationale]
27

---

Preference Instruction for Prompt-Following
Aesthetie:
Your role is to evaluate the aesthetic quality score of given images ("Images") generated by the corresponding text ("Input"). The four images given are independent, and should be evaluated separately and step by step. Note that the rating has nothing to do with image input order.
Scoring:
Rating outputs to 5:
1. Bad: Extremely blurry, underexposed with significant noise, indiscernbbee subjects, and chaotic composition.
2. Poor:
Noticcable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects.
3. Fair: focus with adequate lighting, dull colors, decent composition but lacks creativity.
4. Good: focal point.
Sharp. good exposure, vibrant colors, thoughtful composition with clear
5. Excellent: Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact.
Format:
## Input:
Text: (INSERT PROMPT HERE)
Image:
### Image JINSERT IMAGE HERE
### Image 2 |INSERT IMAGE 2 HERE
### Image [INSERT IMAGE HERE
### Image 4 [INSERT IMAGE 4 HERE]
Note: Please provide your assessment results in the following format:
## Output
### Output for Image 1
Rating: [Rating for Image 1]
Rationale: [Rationale for the rating in short sentences]
### Output for Image 2
Rating: [Rating for Image 2]
Rationale: [Rationale]
### Output for Image 3
Rating: [Rating for Image 3]
Rationale: [Rationale]
### Output for Image 4
Rating: [Rating for Image 4]
Rationale: [Rationale]
28

---

Preference Instruction for Prompt-Following
Fidelity:
Your role is to evaluate the fidelity score of given images ("Images") generated by the corresponding text ("Input"). Higher fidelity score denote the image should true to the shape and characteristics that the object should have and will not be generated haphazardly. Here are some counter-examples of low fidelity images:
"Spider-Man" should only have two legs, but generating an image with more legs is incorrect.
2. "Unicom" should only have one horn, but generating an image with multiple horms is incorrect.
3. Generating more less than five fingers for human hand is incorrect, as each human hand has only five fingers.
Fidelity Rating: Rating outputs to 5:
1. No fidelity: There are lots of obvious shape and characteristics error in the image, and seriously affects the degree of beauty coordination of the image.
2. Insignificant fidelity: There are lots of obvious shape and characteristics obvious error in the image, but the overall image coordinated and acceptable.
3. Partial fidelity: There are some obvious mistakes (less than 2) but no harm, the overall image is coordinated.
4. Severe fidelity: There are a little inconspicuous obvious shape and characteristics mistakes, the full image is coordinated and beautiful.
5. Completely fidelity: No image errors, beautiful and coordinated images.
# Format:
## Input:
Text: (INSERT PROMPT HERE]
Image:
### Image |INSERT IMAGE HERE]
### Image 2 JINSERT IMAGE HERE
### Image 3 |INSERT IMAGE HERE
### Image 4 [INSERT IMAGE HERE
Note [IMPORTANT]] The four images given are independent, and should be evalu- ated the fidelity score of each image separately and step by step. Note that the rating has nothing to do with image input order. Please provide your assessment results the following format:
## Output
### Output for Image 1
Rating: [Rating for Image 1]
Rationale: [Rationale for the rating in short sentences]
### Output for Image 2
Rating: [Rating for Image 2]
Rationale: [Rationale]
### Output for Image 3
Rating: [Rating for Image 3]
Rationale: [Rationale]
### Output for Image 4
Rating: [Rating for Image 4]
Rationale: [Rationale]
29

---

Preference Instruction for Prompt-Follosee
Harmlessness:
Your role is to evaluate the harmlessness score of given images ("Images") generated by the corresponding text ("Input"). Higher harmlessness score denotes images do not include inappropriate content such as pornography, privacy violations, violence, discrimination, or generally not safe for work (NSFW) themes. The four images given are independent, and should be evaluated separately and step by step. Note that the rating has nothing to do with image input order.
Harmlessness Rating: Rating outputs 5:
1. Unsafe:
Not suitable for general audiences, includes explicit sexual content, real violence, teror imagery, or privacy violations.
2. Potentially Sensitive: NSFW, with explicit violence, suggestive themes, graphic medical content, but no pomography or severe violence.
Moderately Safe: Suitable for adults, may not suit children. Includes moderate
violence, implied nudity, and political social commentary. No explicit sexual content or severe violence.
4. Generally Safe:
Safe for most, with mild thematic elements. Includes non-sexual nudity in art or history, minimal violence, and mild satire.
5. Highly Safe: Images are free from any offensive, sensitive, or potentially harmful content, with no traces of violence, nudity, or any form of hate speech. They are suitable for all.
Format:
## Input:
Text: [INSERT PROMPT HERE]
Image:
### Image JINSERT IMAGE HERE]
### Image JINSERT IMAGE HERE
### Image 3 JINSERT IMAGE HERE
### Image 4 [INSERT IMAGE HERE
Note [IMPORTANT]] Please provide your assessment results the following for- mat:
## Output
### Output for Image 1
Rating: [Rating for Image 1]
Rationale: [Rationale for the rating in short sentences]
### Output for Image 2
Rating: [Rating for Image 2]
Rationale: [Rationale] ### Output for Image 3
Rating: [Rating for Image 3]
Rationale: [Rationale]
#*# Output for Image 4
Rating: [Rating for Image 4]
Rationale: [Rationale]
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper includes our motivation, preference data construction details, ex- perimental settings, quantitative experimental results, and qualitative visual examples that reflect and justify the claims our abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The analysis section contains discussion of limitations our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer:
[Yes]
Justification:
We state the reference before illustrating formulations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed description of the dataset construction process, imple- mentation for training the reward model, and offer a multitude of visualization examples.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
32

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will public our data and code upon paper acceptance, due to the manage- ment regulations of our institution.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/Coniic/ Soaeleese/CooeSSSmmmsiiinoffee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full training and testing details in $ and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would take too long with our available computational resources to repeat all experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the name of GPU we used for experiments in Setup. We include extra cost in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All relevant human questions and relevant datasets have been checked for privacy compliance prior to experiments and submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work supports the alignment of generative models with human values, and its societal impact is discussed in the introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our contribution does not include new datasets pre-trained models that pose a risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Code that we derive from earlier work is properly licensed and referenced.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed illustration of new data benchmark and visualized exam-
ples.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
We only recruited participants for user experiments to validate the effective- ness of our model, where they were asked to choose from generated images. No human participants were involved in the dataset construction model training process.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiment solely involves measurement and does not entail behavioral manipulation; therefore, we did not apply for IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "sVk0nLFZ3I", "reviewer": "Reviewer_zspv", "summary": "The paper presents a new AI-generated dataset aimed at enhancing text-to-image generative models by aligning them more closely with human preferences. The data is annotated by multimodal large language models (MLLMs) and captures detailed preferences across multiple dimensions like prompt-following, aesthetic, fidelity, and harmlessness. The dataset facilitates the training of a reward model called VP-Score, which approximates human preference predictions.", "questions": "1. The biggest question would be the relevance of the proposed method. The authors only showed the empirical strength of the proposed method with a randomly sampled subset of the images. In this case, the true value of the proposed method can hardly be validated. I would highly recommend the authors to repeat the experiments with the standard large-scale training regime. I understand the computation requirement will be huge, but there does not seem to have an alternative path to show the power of the proposed method. \n\n2. Similarly as above, a big question would be whether the proposed method is only due to the power of GPT-4V, which is a larger model, to show the true value of the method, the authors might have to use the method and GPT-4V to train a bigger model.", "limitations": "The paper has an analysis section and ablation study in the appendix. An explicit limitation section can further help readers appreciate the paper.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. it introduces a detailed dataset for text-to-image generation preferences. \n\n2. the research has involved human study to validate the results, showing the strength of the results. \n\n3. the experiments section has multiple elements, including results allowing editing of the generated images.", "weaknesses": "1. the overall novelty is a bit limited as it heavily relies on existing large models to generate the dataset and to train additional models (more in the questions). \n\n2. the paper seems to have a diverse set of focuses. For example, the section regarding editing images does not seem to correlate well with the main proposed method, thus seem irrelevant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address your questions point by point below:\n\n---\n>**Q1**: the overall novelty is a bit limited as it heavily relies on existing large models to generate the dataset and to train additional models.\n\n**A1**: In fact, the main motivation for our research is to explore: **Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?**  (as noted in line 38 of the main text). **We aim to demonstrate that MLLMs can generate reliable human preference annotations for text-to-image generation results**. To this end, we explored the performance of several state-of-the-art MLLMs (GPT-4V, Gemini-pro-V, and LLaVA 1.6) in generating human preference annotations (see Figure 10a in the main text). Using the best-performing GPT-4V, we constructed VisionPrefer, the largest fine-grained preference dataset to date, and developed the reward model VP-Score derived from VisionPrefer. We validated the effectiveness of VisionPrefer and VP-Score under PPO and DPO experimental settings.\n\nWe believe our research has the following significance: \n\n* Since human annotation of preference labels is expensive and time-consuming, proving the reliability of MLLMs in generating human preference annotations allows humans to easily scale existing human preference datasets, thereby achieving better text-to-image generative alignment.\n* It further supports the notion that AI-generated synthetic data is a reliable way to create augmented datasets and provides a large dataset for further research related to AI-generated synthetic data.\n* It shows that existing AI models may exhibit capabilities similar to those of human experts in some interesting areas, e.g., image aesthetics, which is exciting and worthy of further exploration.\n\n---\n>**Q2**: the paper seems to have a diverse set of focuses. For example, the section regarding editing images does not seem to correlate well with the main proposed method, thus seem irrelevant.\n\n\n**A2**: We included the section on editing images to demonstrate that using MLLMs for preference annotation offers more than just aligning generative models to produce images that match human preferences, similar to existing human-annotated preference datasets. **A key advantage of MLLMs is their ability to provide detailed textual feedback for each text-to-image sample, which can guide further image editing to produce images that better align with human preferences.** This is another critical advantage of MLLMs beyond their ability to construct reliable preference labels quickly and cost-effectively. In this section, to further validate this advantage, we designed a simple pipeline (see in the Figure 8 in the maintext) using MLLM-generated textual feedback for image editing. Therefore, this section does not indicate a lack of focus in our paper but rather highlights the advantages of our approach.\n\n---\n>**Q3**: The authors only showed the empirical strength of the proposed method with a randomly sampled subset of the images. I recommend the authors to repeat the experiments with the standard large-scale training regime.\n\n**A3**: We demonstrated the effectiveness of our conclusions and methods in two ways:\n\n* For the accuracy of the reward model's preference prediction (see Table 2 in the main text), we computed quantitative metrics on three public test sets without any random components, ensuring reliable results.\n* To validate the model alignment under PPO and DPO settings, we followed existing research [1]. We generated 64 images for all aligned generative models and selected the top 3 images using the corresponding reward model. We then invited **ten human annotators** to evaluate these top 3 images. **This process was repeated on three test sets for both PPO and DPO settings**, with a workload and rigor that exceeds previous studies [1] (only three annotators on one test set). Therefore, we believe the qualitative and quantitative results presented in this paper substantiate the validity of our conclusions and methods.\n\nUnfortunately, we did not fully understand your suggestion to \"repeat the experiments with the standard large-scale training regime.\" We look forward to further discussions during the diffusion stage and welcome more specific feedback to conduct additional experiments.\n\n**Reference**\n\n[1] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" NeurIPS 2024.\n\n---\n>**Q4**: To show the true value of the method, the authors might have to use the method and GPT-4V to train a bigger model.\n\n**A4**: The primary motivation of our research is to explore: **Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?** We aim to demonstrate that MLLMs can generate reliable human preference annotations for text-to-image generation results. Proving that MLLMs can reliably generate these annotations offers many benefits, such as constructing preference labels quickly and cost-effectively.\n\nTherefore, we naturally rely on the powerful capabilities of MLLMs (such as GPT-4V), as stronger models may produce higher-quality preference annotations, which is a desirable outcome. Unfortunately, we did not fully understand your specific suggestion regarding \"training a bigger model to show the true value of the method.\" We look forward to further discussions with you during the diffusion stage and are eager to conduct additional experiments based on these more detailed discussions.\n\n---\n>**Q5**: Require for an explicit limitation section.\n\n**A5**: Thank you for your suggestion. We will include an explicit limitation section in the latest version, summarized as follows: Due to resource constraints, although our VisionPrefer is the largest fine-grained preference dataset for text-to-image generative alignment, we believe it can be further scaled to achieve better alignment performance. We plan to expand it further in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IYLMDuuq4Q", "reviewer": "Reviewer_AdFm", "summary": "The paper introduces VisionPrefer, a large-scale, fine-grained preference dataset constructed using multimodal large language models (MLLMs) as annotators. VisionPrefer aims to improve the alignment of text-to-image generative models with human preferences by providing detailed feedback on generated images. The dataset is created through a three-step process: generating and polishing prompts, generating images using various models, and obtaining preference annotations from GPT-4 V, including scalar scores, rankings, and textual explanations.", "questions": "see weaknesses", "limitations": "Bias in Annotations: The use of AI-generated annotations (e.g., GPT-4 V) could propagate existing biases, leading to biased outputs from generative models. The authors should discuss mechanisms to detect and mitigate such biases.\n\nCombining AI and Human Annotations: Explore the benefits of combining AI and human annotations to enhance the diversity and generalizability of preference data, ensuring the model aligns more closely with a broad range of human preferences.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "VisionPrefer introduces an innovative approach to generating a fine-grained preference dataset by leveraging multimodal large language models (MLLMs) as annotators. The combination of prompt generation, diverse image generation, and detailed preference annotations is novel. The use of GPT-4 V to provide multi-faceted feedback, including scalar scores, rankings, and textual explanations, distinguishes VisionPrefer from existing preference datasets that rely solely on human annotations. This approach not only automates the annotation process but also introduces a level of detail and consistency that is challenging to achieve with human annotators alone.\n\nThe paper is well-organized and clearly written. The structure follows a logical flow, from the introduction and motivation to the methodology, experiments, and analysis.\n\nVisionPrefer has significant implications for the field of text-to-image generation. The idea that using MLLM to guide the alignment ofimage generation models is interesting.", "weaknesses": "Convergence Speed of VP-Score Figure 3 indicates that VP-Score converges significantly slower than other baselines. This slower convergence raises questions about the efficiency and practicality of using VP-Score in real-world applications. The authors should investigate and explain the reasons behind this slower convergence. Potential factors could include the complexity of the fine-grained annotations or the training dynamics of the reward model. Providing insights into this issue and suggesting potential optimizations to improve convergence speed would enhance the utility of VP-Score.\n\n\nInconsistent Improvement Across Datasets\nDespite the larger scale and richer annotations of VisionPrefer, the dataset does not achieve consistent improvements across all evaluated datasets. This inconsistency suggests that VisionPrefer may not fully capture human preferences, indicating potential limitations in the design of the annotations or the preference modeling approach. The authors should analyze why VisionPrefer fails to deliver uniform performance gains and identify aspects where it falls short. This analysis could involve examining the nature of the datasets where VisionPrefer underperforms and exploring ways to refine the annotation process to better align with human preferences. Additionally, providing more detailed comparisons and breakdowns of performance across different datasets would offer valuable insights into the strengths and weaknesses of VisionPrefer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed, helpful feedback. We address your feedback point by point below.\n\n---\n>**Q1**: Figure 3 indicates that VP-Score converges slower than others. \n\n**A1**: In fact, **Figure 3 does not represent a comparison of the convergence speeds of VP-Score and other baselines**. Instead, it illustrates how the human preference quality of a generative model, which uses VP-Score as a reward model for alignment, changes during the fine-tuning process (as noted in lines 173-177 of the main text). This is assessed by various reward models such as ImageReward and HPS v2.\n\nAs shown in Figure 3, the quality of human preferences in the generative model increases as the fine-tuning steps progress. This improvement is reflected in the increasing reward score of models such as HPS v2, which are trained on human preference data. **This indicates that VP-Score can serve as a reliable reward model for aligning generative models, further proving the reliability of using MLLMs to generate human preference feedback.**\n\nAdditionally, inspired by your feedback, we observed that the growth in metrics for ImageReward, PickScore, and Aesthetic is not very pronounced. We hypothesize that these three models may struggle to capture variations in the quality of human preferences in images. In contrast, HPS v2, currently the most accurate and unbiased model for reflecting human preference quality, exhibits a clear upward trend. **This suggests that our VP-Score aligns better with HPS v2, further validating the effectiveness of VP-Score.**\n\nFurthermore, we appreciate your suggestion. All authors agree that the convergence speed of using human preference reward models for aligning generative models is an intriguing research question, and we plan to explore this in future work.\n\n---\n>**Q2**: The authors should analyze why VisionPrefer fails to deliver uniform performance gains and identify aspects where it falls short. \n\n**A2**: We first analyzed the differences between GPT-4V and human annotations at a fine-grained level: we randomly selected 1,000 samples from VisionPrefer and invited six human experts to rate them on four dimensions (Prompt-Following, Aesthetic, Fidelity, and Harmlessness). The correlation between human experts' preference judgments and GPT-4V's judgments is shown in Table A below. We found that GPT-4V's preference annotations align more closely with human experts in the areas of Prompt-Following and Fidelity (the same phenomenon can be observed in Figure 10 e in the main text). **This suggests that GPT-4V is better aligned with human preferences in these two aspects**. This finding might indicate that we can reduce the weight of Aesthetic and Harmlessness scores when constructing VisionPrefer to achieve more accurate preference annotations.\n\nFurthermore, **we hypothesize that using the average score across the four aspects as the final preference score in VisionPrefer might differ from human preference evaluations**. Humans may prioritize certain aspects more heavily, such as Prompt-Following and Aesthetic, which could be more significant than Harmlessness. Supporting this hypothesis, Table 2 in the main text shows that VP-Score achieves the highest accuracy on ImageRewardDB but not on the other two datasets. ImageRewardDB uses an averaged score across multiple aspects for human preference evaluation, similar to our VisionPrefer construction, while Pick-a-Pic and HPD involve direct human preference scoring without fine-grained evaluation across multiple aspects.\n\nBased on these insights, we envision two possible improvements for GPT-4V annotations: (1) Require GPT-4V to provide a confidence estimate for each aspect-specific preference score, using a weighted average of the four aspects as the final sample score. (2) Modify GPT-4V’s annotation process to evaluate all four aspects simultaneously for each text-image sample, outputting a final score rather than an average.\n\nIn summary, your question is very intriguing, and we will continue to explore and experiment with these ideas in future research.\n\n***Table A.***\n\n|Prompt-Following|Aesthetic|Fidelity|Harmlessness|\n|-|-|-|-|\n|94.1%|73.7%|92.0%|79.3%|\n\n---\n>**Q3**: Bias in Annotations.\n\n**A3**: All authors consider your suggestion both important and intriguing. In response, we have reviewed and searched for related research [1-4], including analyses of biases in AI-generated annotations and methods for detecting them. We will focus on these issues in future work to further optimize our VisionPrefer and VP-Score. Additionally, we will include your suggestion in the **Limitations** section to enrich the paper.\n\n**Reference**\n\n[1] Fang, Xiao, et al. \"Bias of AI-generated content: an examination of news produced by large language models.\".\n\n[2] Fan, Zhiting, et al. \"BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs.\".\n\n[3] Morehouse, Kirsten, et al. \"Bias Transmission in Large Language Models: Evidence from Gender-Occupation Bias in GPT-4.\".\n\n[4] Hajikhani, Arash, and Carolyn Cole. \"A critical review of large language models: Sensitivity, bias, and the path toward specialized ai.\".\n\n---\n>**Q4**: Combining AI and Human Annotations.\n\n**A4**: Thanks. In response, we conducted the following experiment: we combined our VisionPrefer dataset (AI annotations) with a randomly selected subset from the Pick-a-Pic dataset (human annotations). We then retrained a new reward model, named VP-Score+x, where x represents the percentage of Pick-a-Pic data used, while keeping the training steps constant. The preference prediction accuracy scores are shown in Table B below.  **We found that mixing AI and human annotations further improves the reward model's accuracy in reflecting human preferences**.\n\n***Table B.***\n\n|Model|ImageRewardDB|HPDv2|Pick-a-Pic|\n|-|-|-|-|\n|PickScore|62.9|79.8|70.5|\n|HPSv2|65.7|83.3|67.4|\n|VP-Score|66.3|79.4|67.1|\n|VP-Score+10|66.7|79.7|69.0|\n|VP-Score+20|66.9|80.2|69.5|\n|VP-Score+30|**67.1**|**80.3**|**70.7**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uvpc4xDuNl", "reviewer": "Reviewer_8ytx", "summary": "This paper introduces VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-to-image generative alignment.  VisionPrefer offers advantages in scalability, fine-grained annotations, and a comprehensive feedback format compared with existing preference datasets. The authors further propose a reward model, VP-Score, which exhibits competitive correlation with human preferences by utilizing VisionPrefer. The experimental results underscore the effectiveness of both VisionPrefer and VP-Score.", "questions": "Please see the wearknesses.", "limitations": "The authors have discusses the limitations in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.  The idea that utilizing a MLLM as a human-aligned preference annotator for text-to-image generation sounds reasonable.\n\n2. The introduced dataset VisionPrefer contains multiple preference aspects including prompt-following, aesthetic, fidelity, and harmlessness, which can contribute to generating images that are more aligned with human preferences.\n\n3. The fine-tuned text-to-image generation model achieves enhanced performance in various aspects from fine-grained feedback.\n\n4. The content of this paper is substantial, offering a wealth of experiments and analyses.", "weaknesses": "1. This paper demonstrates that fine-grained feedback from MLLM help to yield more human-preferred images in terms of prompt-following, esthetic, fidelity and harmlessness with qualitative results.  It is better if the authors could provide some quantitative metrics.\n\n2. A significant contribution of this paper lies in the dataset. However, examples of annotations in VisionPrefer can only be found in the appendix. It would be more intuitive if these were included in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed, helpful feedback. We address your feedback point by point below.\n\n---\n\n> **Q1**: It is better if the authors could provide some quantitative metrics.\n\n**A1**: Thank you very much for your insightful feedback :). Firstly, in **Table 2 of the main text**, we provide a comparison of the preference prediction accuracy scores of our reward model VP-Score, with other state-of-the-art reward models. Additionally, we present the numerical results for win count & win rate under PPO and DPO experiments in **Tables 4 and 5 in the appendix**. These quantitative results demonstrate the effectiveness of using MLLMs to generate preference feedback.\n\nMoreover, in response to your valuable suggestion, we conducted additional measurements. We compared the FID values (a widely recognized metric for evaluating the quality of images generated by models) under PPO and DPO experimental settings for generative model optimized with our data and model against those optimized with other models. As shown in **Tables A and B**, we found that the FID scores of ours are significantly lower than those of the other comparison groups, indicating the effectiveness of using MLLMs-generated feedback to align the generation model.\n\nWe will include these findings in the revised version of our paper to enrich the content. If you have other quantitative metrics of interest, please feel free to let us know :).\n\n***Table A.** FID score of generative models optimized with VP-Score compared to other reward models for PPO experiments.*\n|                     | FID Score $\\downarrow$ |\n| ------------------- | ---------------------- |\n| CLIP                | 8.32                   |\n| Aesthetic           | 8.17                   |\n| ImageReward         | 8.44                   |\n| PickScore           | 7.62                   |\n| HPS v2              | 7.51                   |\n| **VP-Score (Ours)** | **7.44**               |\n\n***Table B.** FID score of generative models optimized with VisionPrefer compared to other human preference datasets for DPO experiments.*\n|                         | FID Score $\\downarrow$ |\n| ----------------------- | ---------------------- |\n| ImageRewardDB           | 8.97                   |\n| HPD                     | 7.70                   |\n| Pick-a-Pic              | 6.62                   |\n| **VisionPrefer (Ours)** | **6.43**               |\n\n---\n\n> **Q2**: A significant contribution of this paper lies in the dataset. However, examples of annotations in VisionPrefer can only be found in the appendix. It would be more intuitive if these were included in the main paper.\n\n**A2**: Considering your insightful recommendations, in the revised paper, we will move the examples of annotations in VisionPrefer to the main body to make the central focus of the article clearer and more reader-friendly :).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of Review Process:
- 3 reviews.
- Mixed scores: 4,6,7
- Authors submitted rebuttals/responses
- 1 out of 3 reviewers did not engage in discussion


Meta Review:
There was consensus from all reviewers regarding the novelty of the approach, clarity of writing, and experimental findings of the paper.  The following is a summary of questions and rebuttal:
- Reviewer AdFm who did not participate in the discussion gave an "Accept" (7) rating with two weaknesses mentioned: (1) efficiency/practicality concerns due to slow convergence (2) inconsistent improvement across datasets. The reviewer also pointed out two limitations. 
    - The authors have added some clarification for (1) and have acknowledged that improving the convergence speed should be a future research direction. 
    - The authors will be adding a note about bias in annotations to the Limitations sections.
    - The authors have provided more insight into the failures of the VisionPrefer model along with additional experiments for combining human and AI annotations.
- Reviewer 8ytx mentioned two weaknesses/requests: (1) provide quantitative metrics and (2) move details about the dataset to the main paper
    - authors have responded to (1) by stating that some quantitative metrics are in Table 2 of the main text and Tab 4,5 in Appendix. Authors also performed additional measurements and compared FID values, to be included in the final version
- Reviewer zspv mentioned 4 weaknesses/questions. Q1 was about novelty, Q2 was about the focus of the paper, Q3 was about experimental setup, and Q4 was about training a bigger model.  
    - authors and reviewers discussed these concerns but there was no change in the rating as there was a disagreement on the  focus and scope of the paper

Overall, the AC believes this to be an interesting work on leveraging MLLMs to create a preference dataset and training a reward model to guide the tuning of text-to-image generative models. The AC's recommendation is to carefully integrate suggestions from reviewers and additional experiments/evidence/clarifications that emerged from the discussion, into the final version.
Decision: Accept (poster)