Title: Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-Image Generation;Reinforcement Learning from AI Feedback;Multimodal Large Language Model
Abstract: Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Abstract

Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, making it to generate more human-preferred images. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges, we first leverage multimodal large language models to create VisionPrefer , a fine-grained preference dataset that captures multiple preference aspects (prompt-following, aesthetic, fidelity, and harmlessness). Then we train a corresponding reward model, VP-Score , over VisionPrefer to guide the tuning of text-to-image generative models. The preference prediction accuracy of VP-Score is validated to be comparable to that of human annotators. To validate the effectiveness of VisionPrefer and VP-Score, we adopt two reinforcement learning methods, Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO), to supervised fine-tune generative models, and extensive experimental

1

results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Our findings indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in text-to-image generative models. VisionPrefer and VP-Score are available at https://github.com/yushuiwx/VisionPrefer.git .

1 Introduction

Text-to-image generative models [23, 21, 19, 26] have experienced rapid advancements in recent years. For example, large-scale text-to-image diffusion models, exemplified by Imagen [24] and DALL · E 2 [19], have demonstrated the capability to generate high-quality and creative images when provided with textual prompts. However, despite recent progress, current generative models still exhibit misalignment with human preferences, such as conflicts with text prompts or incorrect content [17]. A pivotal approach to addressing this issue is utilizing Reinforcement Learning from Human Feedback (RLHF) [16, 27, 1] to supervised fine-tune text-to-image generative models with preference data [13, 3, 5, 17].

Preference data is crucial for aligning generative models with text prompts. However, existing human-crafted preference datasets, such as HPD v2 [33, 32] and Pick-a-Pic [10], either provide only broad, general preference comparisons without fine-grained, accurate preference evaluations, or they are limited in size. Additionally, using humans for preference annotation is expensive and time-consuming, restricting progress in this research area.

Drawing inspiration from recent research utilizing AI-generated preference data as training supervise signal for Large Language Models (LLMs) alignment on Natural Language Processing domain [12, 2], we pose the following question:

Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation?

These multimodal large language models (MLLMs), trained on web-scale text and text-image pairs, have already demonstrated formidable capabilities in image understanding. To this end, we first introduce VisionPrefer, a publicly available AI-generated dataset that features millions of finelygrained preferences concerning model-generated images. Compared with existing human preference datasets, VisionPrefer offers the following benefits:

Scalability & Low Cost : As shown in Table 1, VisionPrefer encompasses 1.2 M preference choices across 179 K pairs of images, establishing it as the largest text-to-image generation preference dataset to date. Additionally, because VisionPrefer is annotated by MLLMs, it can be easily expanded further and the construction cost is significantly lower than human annotation.

Fine-grained preference : To more accurately and diversely evaluate the preference scores of generated images, we carefully develop a detailed preference annotation guideline for MLLMs, which covers four distinct aspects: Prompt-Following , Fidelity , Aesthetic , and Harmlessness . The detail requirement for each aspect is presented at Table 3.

Comprehensive feedback formats : Unlike existing benchmarks that provide only rankings or preference indices, our VisionPrefer not only provides preference rankings but also includes preference scores and textual explanations for the preference annotations from each aspect, which makes VisionPrefer more versatile, e.g., allowing it to serve as a textual guiding resource for image re-editing and refinement.

Building on the VisionPrefer, we conducted an extensive investigation into its most effective utilization. First, we developed a preference reward model named VP-Score optimized on VisionPrefer, trained to evaluate generated images based on their likelihood of being preferred by humans. Experimental results demonstrate that VP-Score exhibits a competitive correlation with human preferences compared to existing human preference reward models. Moreover, we employ two reinforcement learning methods, Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO), to enhance generative models to better align with human preferences. As illustrated in Figure 1, VisionPrefer markedly enhances text-image alignment in compositional image generation across diverse aspects, such as aesthetics. In summary, our contributions are as follows:

We construct VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-to-image generative alignment. Compared with existing preference datasets, VisionPrefer has the advantages of scalability, fine-grained annotations, and comprehensive feedback format.

Based on VisionPrefer, we propose a reward model, VP-Score, which achieves a competitive correlation with human preferences with other automated human preference metrics.

Experimental results demonstrate the effectiveness of both VisionPrefer and VP-Score. Additionally, we provide a comprehensive analysis of them to gain a deeper understanding of how AI-generated synthetic data and models trained on such data impact future research in this domain.

2 Related Work

Text-to-Image Generative Models Alignment. While existing text-to-image generative models often generate images that do not closely match human preferences, thus alignment in the context of diffusion has garnered increasing attention [28, 10, 5, 17, 28]. There are two main types of text-toimage generative models alignment algorithms: (i) Proximal Policy Optimization (PPO) . For example, reward weighted method [13] first explores using human feedback to align text-to-image models with human preference. ReFL [34] trains a reward model, ImageReward, using human preferences and subsequently utilizes it for fine-tuning. (i) Direct Policy Optimization (DPO) . DPOK [7] fine-tunes text-to-image diffusion models by using policy gradient to maximize the feedback-trained reward. ZO-RankSGD [28] optimizes diffusion in an online fashion with human ranking feedback. RAFT [5] and AlignProp [17] tune the generative model to directly increase the reward of generated images. Several manually annotated preference datasets are proposed to support above algorithms [32, 34, 10]. Their overall statistics are shown in Table 1. These manually annotated data have two drawbacks. First, manual annotation need heavy cost, leading to a small-size set. Second, manual annotations are prone to specific biases [32].

Reinforcement Learning from AI Feedback. [2] introduced the idea of Reinforcement Learning from AI Feedback (RLAIF), which used LLM-labeled preferences in conjunction with human-labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from LLMs [20, 11, 35]. These works demonstrate that LLMs can generate useful signals for reinforcement learning fine-tuning. However, RLAIF for text-to-image generative model alignment is less explored. [30] leveraged MLLMs to assess the alignment between generated images and input texts, focusing on aspects like object number and spatial relationship. T2I-CompBench [9] utilized MLLMs like BLIP-VQA to evaluate the text-to-Image generative models. Our work diverges from previous works in two principal ways. (1) Previous approaches provided limited data, sometimes even less than what's annotated manually (e.g., 6K in [9]). (2) Prior methods had limited aspects in alignment, lacking consideration for aspects such as fidelity [30].

3 VisionPrefer

We introduce VisionPrefer, a fine-grained preference dataset constructed by collecting feedback from MLLMs annotators. The collection pipeline of VisionPrefer is shown in Figure 2, which mainly consists of three steps: prompt generation, image generation and preference generation.

Step-1: Prompt Generation. We generate prompts based on DiffusionDB [29], a large-scale text-toimage prompt benchmark containing 1.5M user-written prompts following two steps: (1) Polish. As discussed in [32], a significant portion of the prompts in the DiffusionDB is biased towards certain

Prompt Pool

Text-to-Image Generative Models

Preference Annotation

Comparison Data

Image A

X𝑇

X𝑡

X𝑡-1

Prompt-Following

Fidelity

Aesthetic

Harmlessness

A

B

C

D

=

>

>

A

B

C

D

>

=

=

A

B

C

D

>

>

=

A

B

C

D

=

>

=

GAN Models

Diffusion Models

a cute cartoon giraffe toy wearing

ballet skirt, dancing by the river,

sunset, unreal engine, vray, 4k .

…

…

Image B

Image C

Image D

Diffusion DB

Polish

S

Prompt-Following Fidelity Aesthetic Harmlessness A B C D = > > A B C D > = = A B C D > > = A B C D = > = styles (e.g., 'Greg Rutkowski' appears in around 15% of prompts). We utilize GPT-4 to polish prompts in DiffusionDB to obtain unbiased prompts following instructions shown in Appendix F.1. (2) NSFW Filting. We employ state-of-the-art NSFW detectors [8] to compute an NSFW score for each prompt and filter out prompts that exceed a certain threshold, following [29]. After these two steps, we combine both polished prompts and the original prompts in DiffusionDB as our final prompt benchmark, which contains 179K prompts.

Step-2: Image Generation. We generate images using different text-to-image generative models (see details in Appendix C.2) by sampling textual prompts constructed in Step-1 as input. For each prompt, we generate four images by randomly selecting different generative models from the model pools with different classifier-free guidance scale values, to ensure high diversity. This diversity allows for a comprehensive evaluation of a preference prediction model's generalization capability and facilitates the training of a more generalizable model. Finally, we obtain 0.716M images.

Step-3: Preference Generation. We employ state-of-the-art multimodal large language model, GPT-4 V, to provide three types of feedback: (1) Scalar scores that indicate the fine-grained quality regarding multiple aspects, (2) Preference ranking according to the scalar scores, and (3) Textual explanations that give detailed guidance on how to improve the completion, encompassing four distinct fine-grained aspects namely: Prompt-Following , Aesthetic , Fidelity , and Harmlessness for each generated image (See the example in Table 12). Detailed input instructions for GPT-4 V to annotate preference labels are in Appendix F.2. Finally, we obtain 1.2M preference choices.

4 Experiments

In this section, we first train a corresponding reward model named VP-Score and evaluate it on existing human-preference datasets (§ 4.1). Next, we enhance existing text-to-image generative models by adopting two reinforcement learning algorithms (§ 4.2) to validate the efficacy of VisionPrefer and VP-Score. After that, we design a simple pipeline to edit generated images with the textual explanations in VisionPrefer(§ 4.3).

4.1 Reward Modeling

Training Setting. We train the VP-Score over VisionPrefer. VP-Score adopts the same model structure as ImageReward [34], which is a open-source human-preference reward model and utilizes BLIP [14] as the backbone. Similarly to training the reward model for the language model [27, 16], we formulate the preference annotations in VisionPrefer as rankings. Specifically, we employ the average scores of each sample in VisionPrefer across four aspects as the final preference score, and then we have k images ranked generated by the same prompt T according to final preference score (the best to the worst are denoted as x 1 ≻ x 2 ≻ ... ≻ x k ). For each comparison, if x i is better and x j is worse, the loss function can be formulated as:

where f θ ( T , x ) is a scalar value of reward model for prompt T and image x .

Evaluation Results. We evaluate the preference prediction accuracy on the test sets among three human preference datasets: ImageRewardDB [34], HPD v2 [32] and Pick-a-Pic [10]. Furthermore,

1.0

Reward Score

0.9

0.8

ImageReward

0.7

0.6

Aesthetic

0.5

PickScore

0.4

HPSv2

0.3

VP-Score(Ours)

0.2

0

100

200

300

400

TrainingSteps

DiffusionDB [29]

ImageRewardDB [34]

HPD v2 [32]

Win

Tie

Lose

Win

Tie

Lose

Win

Tie

Lose

VP-Score v.s.

VP-Score v.s.

VP-Score v.5.

CLIP

CLIP

CLIP

VP-Score v.s.

VP-Score v.s.

VP-Score v.s.

Aesthetic

Aesthetic

Aesthetic

VP-Score v.s.

VP-Score v.s.

VP-Score v.s.

ImageReward

ImageReward

ImageReward

VP-Score v.s.

VP-Score v.s.

VP-Score v.s.

PickScore

PickScore

PickScore

VP-Score v.s.

VP-Score v.s.

VP-Score v.s.

HPS v2

HPS v2

HPS v2

%0

25%

50%

75%

100%

%0

25%

50%

75%

100%

%0

25%

50%

75%

100%

Top-3 selected from 64 images

Top-3 selected from 64 images

Top-3 selected from 64 images

to better demonstrate the model's generalization performance, we computed the harmonic mean of accuracy across three sets for each model as an overall indicator of model performance. We use the CLIP score [18], BLIP score [14], Aesthetic score [25], ImageReward [34], HPS [33], HPS v2 [32] and PickScore [10] as baselines to compare with the VP-Score.

The results are presented at Table 2. Our VP-Score demonstrates strong competitiveness compared to the current state-of-the-art reward models trained on human preference data. It achieves the second-best average performance among all preference reward models, following only HPS v2. Moreover, our model achieves optimal performance on the ImageRewardDB dataset, achieving a 0.6 performance gain compared to HPS v2. These results validate that leveraging fine-grained feedback provided by MLLMs enables learning a proficient human preference reward model.

4.2 Fine-tuning Text-to-Image Generative Models

We aim to leverage the constructed preference dataset to align the performance of generative models more closely with human preferences. We utilize two popular reinforcement learning methods for finetuning: (1) Proximal Policy Optimization (PPO), where select ReFL [34] as our PPO implementation to adjust the generative model. (2) Direct Preference Optimization (DPO), which allows for direct model fine-tuning using preference data without a reward model, employing D3PO [36] as our DPO implementation. We use Stable Diffusion v1.5 [22] as the text-to-image generative model.

Training Setting. For PPO experiments, we randomly sample 20,000 real user prompts from DiffusionDB [29] and 10,000 prompts in ImageRewardDB [34] as the training dataset. We compare VP-Score against five open-source reward models, including ImageReward [34], PickScore [10], and HPS v2 [32], all trained on large-scale preference datasets (see Table 1). All models are fine-tuned with identical data and settings for consistency. For DPO experiments, we compare our VisionPrefer along with three open-source large-scale human-annotated preference datasets, ImageRewardDB [34], HPD [33] and Pick-a-Pic [10] (see in Table 1). Notably, VisionPrefer scores are averaged across four aspects for fine-tuning, with an analysis on individual aspect scores presented in § 5. Both PPO and DPO use the same test benchmarks: 400 real user prompts from DiffusionDB [29], 200 prompts from ImageRewardDB [34] and 400 prompts from HPD v2 [33]. Further details are in Appendix D.

PPO Results. First, we visualize the evolution of various metrics as the model training steps increase when using our VP-Score as the reward function at Figure 3. As training progresses, all metrics, including human preference metrics like HPS v2, show an increasing trend. This indicates consistency

ImageReward

PickScore

HPS v2

SD1.5

VP-Score

CLIP

Aesthetic

ImageReward

PickScore

HPS v2

SD1.5

VP-Score

CLIP

Aesthetic

DiffusionDB [29]

ImageRewardDB [34]

HPD v2 [32]

Win

Tie

Lose

Win

Tie

Lose

Win

Tie

Lose

VisionPrefer v.s.

VisionPrefer v.s.

VisionPrefer v.s.

ImageRewardDB

ImageRewardDB

ImageRewardDB

VisionPrefer v.s.

VisionPrefer v.s.

VisionPrefer v.s.

HPD

HPD

HPD

VisionPrefer v.s.

Pick-a-Pic

VisionPrefer v.s.

Pick-a-Pic

VisionPrefer v.s.

Pick-a-Pic

%0

25%

50%

75%

100%

%0

25%

50%

75%

100%

%0

25%

50%

75%

100%

Top-3 selected from 64 images

Top-3 selectedfrom 64 images

Top-3 selected from 64 images

Figure 6: Win rates of generative models optimized with VP-Score compared to other reward models on three test benchmarks for DPO experiments. 'Tie' indicates instances where annotators think two images are of comparable quality.

between our VP-Score and other human preference metrics, demonstrating that VP-Score reliably aligns generative model outputs with human preferences.

Then, we conduct a human preference study. Specifically, we use these fine-tuned generative models to generate 64 images for each prompt in evaluation dataset, followed by a top-3 selection by the corresponding reward models. Finally, ten human annotators rank these selected images. The results are presented at Figure 4, detailed win count and win rates can be found in Table 4. We observed that VP-Score fine-tuned generative model's Win+Tie ratio exceeds 50% when compared to all other models across all three test benchmarks, including some trained on large-scale human preference datasets like HPS v2. This suggests that, compared to other human preference reward model, VP-Score can serve as a reliable and competitive reward model for fine-tuning generative models to produce outputs closer to human preferences. This further underscores the effectiveness and competitiveness of VisionPrefer.

The corresponding qualitative results shown at Figure 5 demonstrate that VisionPrefer fine-tuned generative model can generate images that are more aligned to text and with higher fidelity and avoid toxic contents. More qualitative results can be found in Figure 13.

DPO Results. We conduct a human preference study using the same procedure as PPO experiments at Figure 6, detailed win count and win rates can be found in Table 5. We found that the Win+Tie ratio of the generative model optimized on our VisionPrefer, when compared to the other three large-scale human datasets, exceeds 50%, substantiating the competitiveness of our VisionPrefer against humanannotated preference data. We show the qualitative results in Figure 7. The results indicate that finetuning the generative model directly on our VisionPrefer using DPO yields performance comparable to that of fine-tuning the generative model on large-scale human-annotated preference dataset (e.g., Pick-a-Pick). Specifically, the generated results are more aligned with human preferences, exhibiting increased visual detail, better conformity to input prompts. More qualitative results can be found in Figure 14. These experimental outcomes collectively affirm the efficacy of using preference data generated by MLLMs.

4.3 Editing Generated Images with VisionPrefer

Unlike existing preference datasets, our VisionPrefer not only provides preference rankings and scores for images but also includes corresponding textual explanations. This makes VisionPrefer

Portrait of a man wearing a golden mask

SD 1.5

HPD

ImaeRewardDB

VisionPrefer

Pick-a-Pic

The image captures the style

of Van Gogh and an epic sky,

eating a burrito.

Editing Instruction

Make the Mona Lisa eating a burrito.

Image Editing

Model

Original

After

Prompt

Based  on  the  explanations,

please  output  one  or  more

concise and effective editing

instructions

to  address  the

issues

raised

in

the

explanations.

STEP-2: Image Editing

STEP-1: Generate Editing Instruction

?

Win

Tie

Lose

CLIP

ImageReward

PickScore

HPS v2

Human

%0

25%

50%

75%

100%

more versatile, enabling applications such as editing images based on textual explanations to better align with human preferences.

We design an image editing pipeline as in Figure 8, which contains two main steps: (1) Integrate the textual explanations from VisionPrefer into a specific prompt template and input it into LLMs (e.g., GPT-4), encouraging the LLMs to output one or more concise editing instructions to address the issues raised in the textual explanations. (2) Input these generated editing instructions into an image editing model (e.g., InstructPix2Pix [4]), guiding it to edit images and address issues. This pipeline is simple in structure and can perform better by substituting more advanced LLMs and editing models.

Training Setting. To explore the effectiveness of the textual explanations provided by VisionPrefer, we randomly selected 200 text-image pairs with scores below 3 (out of a maximum score of 5) as test cases. Following the outlined process, we performed image edits and evaluate the results before and after editing using existing reward models and conducted human studies with 10 participants.

Results. The evaluation results for images before and after editing are summarized in Figure 9. We found that both the reward models and human studies indicate a Win ratio > 50%, demonstrating that the edited images are superior to the original ones. This also validates the effectiveness of the textual explanations provided by VisionPrefer and the design of our pipeline.

5 Analysis

Which MLLMs is the Best Annotator?

The annotation of VisionPrefer heavily relies on GPT-4 V. Although many researchers pointed out that GPT-4 V capable of providing meticulous judgments and feedback [31, 6], we still concern whether the GPT-4 V preferences are qualified. We then conduct a probing experiment by utilizing different MLLMs, GPT-4 V, Gemini-pro-V and LLaVA 1.6-34B, to provide their preference on two existing human-preference datasets (HPD [33] and ImageRewardDB [34]). The corresponding pair-wise preference prediction accuracy is shown in Figure 10 (a). We observed that the accuracy

(a)

(b)

(c)

(d)

(e)

90

90.1%

86.0%

ImageRewardDB

HPD

-Human vs Human

80

90

79.4%

73.7%

ImageRewardDB

HPDv2

Pick-a-Pic

06

85.7%

80.9%

ImageRewardDB

--Human vs Human

82.1%

HPD

60

70

69.1%

60.9%

67-2% -- 70%

55.3%

09

70

66.3%

67.1%

62.0%

64.1%

55.8%

61.1%

58.6%

80

70%

77.5%

70七

GPT-4 V

Gemini-pro-V1

LLaVA 1.6-34B

GPT-4 V

Gemini-pro-V LLaVA 1.6-34B

Score Feedback

Rank Feedback

90

VP-Score

VP-Score-P

VP-Score-A

VP-Score-F

VP-Score-H

Annotation Consistency

80

79.4%

75.1% 76.4% 76.7%

70

66.3%

62.7% 62.9%

63.7%

0.1

0.3

0.5

0.7

0.9

1.1

1.3

1.5

60

58.3%

56.0%

59.1%

60.8%

57.2%

Temperature

ImageRewardDB

HPD v2

Pick-a-Pic

of GPT-4 V surpasses that of both LLaVA 1.6-34B and Gemini-pro-V on both datasets, achieving accuracy rates exceeding or approaching 70%, and LLaVA 1.6-34B notably scoring significantly lower than the former two. According to previous research [6, 32], the agreement rate between qualified human annotators is also around 70% (65.3% for ImageRewardDB and 78.1% for HPD). Therefore, the probing experiment validates that GPT-4 V can be a well human-aligned annotator, thus ensure the quality and reliability of our VisionPrefer.

To further validate the efficacy on preference annotation ability of GPT-4 V, we utilize Gemini-pro-V and LLaVA 1.6-34B to collect similar amount of data (1.2 M pair-wise preference choices) following the same collection pipeline described in Section 3. Then we train the corresponding reward model on these two datasets and show the performance in Figure 10 (b). As we can see, consistent with the aforementioned conclusion, the testing accuracy of the reward model trained on data annotated by GPT-4 V exhibits the highest performance, followed by Gemini-pro-V. This demonstrates that GPT-4 V is currently the most proficient annotator for text-to-image generation.

Encouraging GPT-4 V(ision) for Enhanced Annotations.

Prompt Manner. As described in Section 3, during the construction of VisionPrefer, we encourage GPT-4 V to directly output scores for various aspects (e.g., aesthetic) of each image (denoted as score feedback). Another straightforward prompting manner (denoted as rank feedback) is encourage GPT-4 V to directly provide a ranking of images ( α and β ) in a certain aspect (i.e., α ≻ β , β ≻ α , or α = β ). It is interesting to explore which prompting manner is best suit for AI Annotators. We randomly sampled 1,000 samples from ImageRewardDB and 500 samples from HPD, and utilized the two aforementioned prompt manners to ascertain GPT-4 V's annotations. The results are presented at Figure 10 (c), we observe that in both datasets, the accuracy achieved using score feedback is higher than that achieved using rank feedback.

Temperature τ . Temperature τ is a hyperparameter used in multimodal large language models (e.g., GPT-4 V) to control the randomness and creativity of the generated results. A lower value of the temperature parameter will lead to a more predictable and deterministic output, while a higher value will produce a more random and surprising output. We investigate the influence of the variation in τ on both the accuracy of annotation and annotation consistency (where the same input yields identical annotation results). The results are shown in Figure 10 (d), we observe a decrease in accuracy as τ increases, indicating that lower values of τ should be set when conducting preference annotations. Furthermore, as τ increases, annotation consistency continues to decline, which is sensible because more randomness in the results leads to different preference outcomes for identical inputs over time.

Fine-Grained Feedback Leads to Better Results.

Better Reward Modeling . In our previous experiments, we used the average score of each sample across four evaluation aspects as the final preference score for modeling reward or optimizing generation models. Here, we explore the impact of separately modeling the four evaluation aspects. We first train four reward models on these four different aspect in VisionPrefer, namely VP-Score-P,

A hyper-realistic portrait of a woman holding flowers, featuring a cottagecore and grunge aesthetic.

ImageReward

PickScore

HPS v2

VP-Score

beautiful render, soviet brutalist city, brutalism, architecture 60s, skyscrapers, blocks, streets, greyscale, depressing, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, octane render, dramatic lighting, art by greg rutkowski and wlop and artgem.

ImageReward

PickScore

HPS v2

SD1.5

VP-Score

VP-Score

VP-Score

VP-Score

VP-Score-A, VP-Score-F and VP-Score-H, respectively. The corresponding preference accuracy are presented at Figure 10(e). We can observe that the accuracy of reward models individually trained using a single aspect preference data is consistently lower than VP-Score, which validates the effectiveness of our approach in designing four evaluation aspects to model the preference level.

Better Prompt-Following. We found fine-grained preference data enables our fine-tuned model to generate images that better adhere to the input prompt. For instance, as shown in Figure 11, the top-3 sampled images generated from our fine-tuned model all satisfy the "holding" prompt requirement, whereas only HPS v2 among baseline models achieves this.

More Aesthetically Pleasing. Fine-grained data enhances the visual appeal and vividness of images generated by our model. As shown in Figure 12, our results exhibit enhanced luminosity, dynamic sensation, and increased detail, aligning more closely with human aesthetic preferences.

Enhance Image Safety. Using unsafe prompts from [36], we generated 1K images and assessed safety using the Diffusion library's NSFW detector. The NSFW ratio for models fine-tuned with VP-Score (4.4%) was substantially lower compared to HPS v2 (21.1%) and PickScore (22.3%) fine-tuned models, indicating our preference scoring's effectiveness in reducing harmful content generation.

More related details and ablation studies about the effectiveness of fine-grained feedback can be found in Appendix B .

6 Conclusion

In this paper, we explore utilizing MLLMs to construct a large-scale high-quality feedback dataset, VisionPrefer, for diffusion models alignment and refining. Costly experiments conducted across various experimental settings have validated the efficacy of VisionPrefer. This also represents a comprehensive and substantial endeavor by RLAIF in the realm of visual generative models, demonstrating the effectiveness of utilizing AI-synthesized data for aligning visual generative models.

References

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.

K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 , 2023.

T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

K. Clark, P. Vicol, K. Swersky, and D. J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400 , 2023.

G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 , 2023.

Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305.16381 , 2023.

L. Hanu and Unitary team. Detoxify: Toxic Comment Classification with Pytorch Lightning and Transformers, 2020.

K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems , 36:78723-78747, 2023.

Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569 , 2023.

M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations , 2022.

H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V. Carbune, and A. Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 , 2023.

K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Y. Liang, J. He, G. Li, P. Li, A. Klimovskiy, N. Carolan, J. Sun, J. Pont-Tuset, S. Young, F. Yang, et al. Rich human feedback for text-to-image generation. arXiv preprint arXiv:2312.10240 , 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML , 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

P. Roit, J. Ferret, L. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L. Hussenot, O. Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186 , 2023.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.03751 , 2023.

Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896 , 2022.

S. Wen, G. Fang, R. Zhang, P. Gao, H. Dong, and D. Metaxas. Improving compositional text-to-image generation with large vision-language models. arXiv preprint arXiv:2310.06311 , 2023.

T. Wu, G. Yang, Z. Li, K. Zhang, Z. Liu, L. Guibas, D. Lin, and G. Wetzstein. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092 , 2024.

X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 , 2023.

X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Human preference score: Better aligning textto-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2096-2105, 2023.

J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 , 2023.

K. Yang, D. Klein, A. Celikyilmaz, N. Peng, and Y. Tian. Rlcd: Reinforcement learning from contrast distillation for language model alignment, 2023.

K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, Q. Li, W. Shen, X. Zhu, and X. Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arXiv:2311.13231 , 2023.

We organize our appendix as follows:

In Section A , we provide an ablation study about the impact of training dataset sizes and different reward model backbones.

In Section B , we additional ablation studies to validate the efficacy of fine-grained feedback.

In Section C , we present the statistical analysis of VisionPrefer, revealing that preference labels annotated by MLLMs demonstrate characteristics akin to those found in humanannotated preference datasets.

In Section D , we provide training details of the experiments discussed in the main text.

In Section E , we analyze the annotation costs of constructing VisionPrefer and demonstrate that employing MLLMs as annotators not only achieves results aligned with human annotators but also significantly reduces labor resources and time costs.

In Section F , we provide the corresponding prompt templates used in employing GPT-4 to polish existing prompts and employing GPT-4 V to generate preference annotations.

Table 4: Human evaluation study on win count and win rate of generative models optimized with different reward models, benchmarked against the Stable Diffusion v1.5 baseline. Compared to other reward models, VP-Score exhibits competitive performance. The best results are highlighted in bold, while the second-best results are underlined.

SD1.5

PickScore

Clip

ImageReward

HPS v2

Aesthetic

VP-Score

VP-Score

VP-Score

SD1.5

PickScore

Clip

ImageReward

HPS v2

Aesthetic

VP-Score

VP-Score

VP-Score

-IARAGALBAL-

GARRIANG

DAULAL

CiBU3

NAREE

DABPLAEDUCAUNCA

DAUHAL

DISULAS

SD1.5

PickScore

Clip

ImageReward

HPS v2

Aesthetic

VP-Score

VP-Score

VP-Score

SD1.5

PickScore

Clip

ImageReward

HPS v2

Aesthetic

VP-Score

VP-Score

VP-Score

batman monster digital art, fantasy, magic, trending on artstation, ultra detailed, professional illustration by Basil Gogos

VisionPrefer

VisionPrefer

VisionPrefer

VisionPrefer

SD1.5

HPD

ImageRewardDB

Pick-a-Pic

SD1.5

HPD

ImageRewardDB

Pick-a-Pic

VisionPrefer

VisionPrefer

VisionPrefer

VisionPrefer

口

A Ablation Study

We conducted ablation experiments to explore the scalability of VisionPrefer and the impact of different backbones on the reward model's performance.

Scalability. To investigate the impact of training dataset sizes on VP-Score performance, we conduct comparative experiments (see Figure 15). Results show that increasing training data enhances VPScore's prediction accuracy. This indicates that models trained on our VisionPrefer exhibit strong performance scalability, implying that more training data leads to further performance improvements. In the future, we plan to further increase the volume of data in our VisionPrefer and explore whether models trained on our dataset can outperform all these trained on human-annotated datasets. This endeavor holds significant promise and interest.

Reward Model Backbone. VP-Score adopts BLIP [14] as the backbone, which may raise curiosity about how well BLIP compares to CLIP [18]. We employed these two models as the backbone for our reward model and explored their effectiveness on our VisionPrefer. The results are summarized in Table 6, where we observed that the performance of BLIP surpassed that of CLIP and this conclusion aligns with the findings on human preference datasets [34].

80

75

70

660

ImageRewardDB

55

HPD v2

50

Pick-a-Pic

45

0.2

0.4

0.8

1.2

Training DataSize (M)

B Efficacy of Fine-Grained Feedback

In this section, we present additional ablation studies and qualitative results to validate the efficacy of fine-grained feedback design.

B.1 Better Prompt-Following.

To further substantiate that 'Prompt-Following' rating labels in VisionPrefer can enhance the ability of fine-tuned models to generate images that more accurately align with the input prompts, we removed the 'Prompt-Following' rating labels from the VisionPrefer, preserving labels for the other three aspects, and train a reward model, which we named VP-Score † . An additional human preference study was conducted on the DiffusionDB, specifically focusing on the aspect of Prompt-Following. The outcomes, presented in Table 7, indicate two key findings: Firstly, VP-Score exhibits competitive performance in comparison to HPS v2. Secondly, the efficacy of VP-Score † experiences a notable decline with the omission of Prompt-Following rating labels. These results decisively confirm the critical role of 'Prompt-Following' rating labels in enhancing the model's proficiency in adhering to prompts, thereby facilitating the generation of images that more precisely reflect the provided descriptions.

Besides, we provide additional visualization results in Figure 17 to validate that generative models guided by VP-Score are capable of producing images that more closely adhere to the descriptions provided in the prompts.

B.2 More Aesthetically Pleasing.

Similar to the last section, we removed the 'Aesthetic' rating labels from the VisionPrefer and trained a reward model named VP-Score ♣ . Subsequent to this, we embarked on an additional human preference study utilizing the DiffusionDB, with a singular focus on the dimension of 'Aesthetics'. The findings, elucidated in Table 8, revealed that VisionPrefer achieved the best performance, while the exclusion of aesthetic labels markedly diminished the operational efficiency of VP-Score ♣ . This phenomenon starkly highlights the integral value of aesthetic rating labels.

Further, we showcase additional visual outcomes in Figure 18. Our observations indicate that generative models refined under the auspices of VP-Score manifest the capacity to engender imagery replete with more vibrant detail and sophisticated interplays of light and shadow.

B.3 Reduce Image Distortion.

To ascertain the impact of 'Fidelity' rating labels, we excised these labels from the VisionPrefer and subsequently trained a reward model VP-Score ♢ . A human preference study, concentrated solely on the 'Fidelity' aspect, is documented in Table 9. This study utilized the 'anything' prompts delineated in [36], encompassing 442 prompts, as the evaluation benchmark. The outcomes illustrate that generative models guided by our VP-Score manifest competitive performance. In contrast, VP-Score ♢ exhibits a discernible performance decrement relative to VP-Score.

Additionally, the visualization results showcased in Figure 19 demonstrate that models optimized under the guidance of VP-Score excel in producing images with diminished distortion, e.g., less distortion of human hands. Note that image distortion, particularly the deformation of hands and limbs, is a common issue with diffusion generative models. Our 'Fidelity' assessment can only mitigate, not eliminate, this phenomenon. Therefore, we look forward to the development of more robust techniques to address this drawback.

B.4 Enhance Image Safety.

Similarly, we removed the 'Harmlessness' labels from the VisionPrefer and trained a corresponding reward model named VP-Score ♠ . Then we employ unsafe prompts provided in [36] to generate 1K images and utilize the built-in NSFW detector in the diffusion library 1 to quantify the frequency of generating harmful content. Detailed results is presented in Figure 16. We find that VP-Score ♠ , trained without the 'Harmlessness rating' labels, exhibited a significant increase in the NSFW ratio

compared to the original VP-Score (4.4% to 20.2%). This further underscores the importance of 'Harmlessness' labels.

Table 8: Human evaluation study on the aspect of 'Aesthetic'. The best results are highlighted in bold, while the second-best results are denoted with an underline.

NSFW Ratio (%)

2015

25

20.2

18.7

22.3

21.1

15

12.9

10

4.4

5

0

CLIP

VP-Score

VP-Score*

PickScore

HPS v2

ImageReward

A painting depicts an oak tree with a human face resembling an old bearded man, crafted from the tree's bark

SD1.5

ImageReward

PickScore

HPS v2

Aesthetic

Clip

VP-Score

VP-Score

VP-Score

A ghost pirate aboard a pirate ship in spooky fog under moonlight

ImageReward

HPS v2

Aesthetic

SD1.5

PickScore

Clip

VP-Score

VP-Score

VP-Score

SD1.5

PickScore

Clip

ImageReward

HPS v2

Aesthetic

VP-Score

VP-Score

VP-Score

ImageReward

HPS v2

Aesthetic

SD1.5

PickScore

Clip

VP-Score

VP-Score

VP-Score

ImageReward

PickScore

HPS v2

VP-Score

ImageReward

PickScore

HPS v2

VP-Score

C Statistics of VisionPrefer

In this section, we provide more details of VisionPrefer. We encourage readers to delve into Section C.3, where we analyze the characteristics of preferences generated by GPT-4 V. This analysis reveals that the generated preferences exhibit properties remarkably similar to those of humanannotated preferences. Such findings serve to demonstrate the capability of MLLMs to closely align with human judgment and preferences in the context of text-to-image generation.

C.1 Prompts.

A key step in VisionPrefer contruction pipeline is utilizing GPT-4 to polish the existing prompt benchmarks. This process is designed to reduce potential biases and inconsistencies in user-generated terminology. We present examples of original prompts alongside their polished counterparts at Table 10, and quantitatively illustrates the frequency distribution of certain stylistic words and conflicting prompts at Figure 20. Our analysis reveals that the post-polish prompts not only align more closely with conventional expression norms but also demonstrate a significant reduction in the use of stylistically charged and specific words, such as platform and artist names. Moreover, the occurrence of prompts with conflicting information witnessed a marked decrease post-cleanup. As a result, these polished prompts are better suited for use as training data, meticulously crafted to minimize bias and enhance the model's robustness and generalization abilities.

30

Original prompts

Prompts polished by GPT-4

Percentage

20

10

Q

"artstation"

'gregrutkowski"

'mucha"

Conflictprompts

C.2 Images.

Within VisionPrefer, images are generated by employing four state-of-the-art text-to-image generative models. These models are ranked as the top four on the Hugging Face leaderboard, specifically: Stable Diffusion v1-5 2 , Stable Diffusion 2.1 3 , Dreamlike Photoreal 2.05 4 , Stable Diffusion XL 5 . Detailed descriptions of these models and the distribution of images generated by each within our dataset are methodically outlined in Table 11. For illustrative purposes, Figure 21 showcases representative images produced by each of these models.

a painting of a dark souls boss by greg rutkowski, dark fantasy art, high detail, trending on artstation.

SD

v1-5

SD 2.1

Dreamlike

SD XL

Rating 5

Rating 4

Rating 3

Rating 2

Rating 1

75.3

75

Percentage

53.4

50

43.1

25.822.1 24.6

32.3

25

15.312.2

14.8

17.2 14.6

17.6

8.9

7.37.5

6.7

0.8

0.50.1

0

Prompt-Following

Aesthetic

Fidelity

Harmlessness

58%

33%

9%

Stable

Diffusion 2.1

Dreamlike

Photoreal 2.05

Tie

(a)

(b)

Win

Tie

Lose

Value

Guidance

Classifier-free

0%

25%

50%

75%

100%

Win/Tie/Lose Ratio

C.3 Preferences.

All preferences in VisionPrefer were generated using GPT-4 V API. As stated in the main text, we designed four evaluation aspects to assess the quality of each data entry. For each aspect, we individually invoked the GPT-4 V API to generate the corresponding preferences, the corresponding prompts regarding to these four aspects can be found depicted in Section F.2. Please refer to Table 12 to see some annotation examples.

We illustrate the distribution of GPT-4 V's ratings across four distinct aspects within VisionPrefer in Figure 22. Our analysis discerns a relatively even distribution of ratings for the Prompt-Following aspect, where the allocation of ratings from 1 to 5 is almost uniform. Conversely, in the domains of Fidelity and Harmlessness, GPT-4 V exhibits a propensity towards assigning the highest rating of 5 to a predominant share of the samples. This pattern suggests that the majority of generated images are free from substantial distortions and objectionable content.

VisionPrefer offers a unique opportunity to leverage GPT-4 V' preferences for unbiased analysis. A critical step in the construction of VisionPrefer is the random application of class-free guidance values (from 3 to 12), aiming to enhance the generalization of VisionPrefer. We analyze the impact of changing the class-free guidance values of Stable Diffusion XL on its performance. For each guidance value, we compute the win ratio, representing the percentage of judgments where its use led to a preferred image. We also calculate the corresponding tie and lose ratios for each scale, enabling a detailed analysis of which classifier-free guidance scales are more effective. The results are presented at Figure 23 (a), and we find higher guidance value always lead to a higher win rate, e.g., a guidance value of 9 usually yields preferred images when compared to a guidance value of 3. This conclusion is in well agreement with the conclusions obtained in the human-annotated preference dataset [10].

Additionally, by comparing preferences for images generated by different generative models in response to identical prompts, we are able to identify the model that is more favorably rated by GPT-4 V. For instance, considering judgments in which one image was generated by Dreamlike Photoreal 2.05 and the other by Stable Diffusion 2.1 , we can evaluate which model is more performant. As shown in Figure 23 (b), we find that GPT-4 V typically exhibits a preference for Dreamlike Photoreal 2.05 over Stable Diffusion 2.1 . This preference aligns with the characteristics of human-annotated preference data [10], demonstrating a consistency between the MLLMs generated preferences and human judgment .

Prompt: minimalist summertime architecture by atey ghailan ( ( and edward hopper ) ).

Input Image

Prompt-Following

Rating : 5

Rationale : The image showcases a minimalist architectural style with a focus on simple geometric forms and warm colors that can be associated with summer. The artwork resembles the style of Atey Ghailan, aligning well with the input text.

Rating : 4

Rationale : The image features minimalist architecture with clean lines and a limited color palette, which is reflective of summertime simplicity. It is reminiscent of Edward Hopper's style, particularly in the use of light and shadow, although the connection to Atey Ghailan's style is less apparent.

Rating : 3

Rationale : The image displays architecture with a slightly more complex design, deviating from strict minimalism. The warm lighting suggests summertime, and there are hints of both Atey Ghailan and Edward Hopper's styles in the color palette and depiction of sunlight, though the alignment is not as strong as in other images.

Rating : 5

Rationale : This image strongly aligns with the input text, featuring a minimalist architectural scene with a clear influence of Edward Hopper's style in the composition and use of light. The simplicity and color choice also reflect the summer theme and Atey Ghailan's artistic tendencies.

Aesthetic

Rating : 5

Rationale : The image has a clear and sharp focus, the exposure is perfect with the warm glow from the window, and the colors are rich and vibrant. The composition is masterful, with a clear focus on the minimalist architecture and the figure, creating a sense of calm and contemplation.

Rating : 4

Rationale : The image is sharp, with good exposure and vibrant colors. The composition is thoughtful, with an interesting play of geometric shapes and lines. The clear focal point is the architecture itself.

Rating : 3

Rationale : The image is in focus with adequate lighting, but the colors are somewhat dull. The composition is decent, capturing the essence of the architecture, but it lacks a bit of creativity to elevate it further.

Rating

: 4

Rationale : The image is sharp and well-exposed, with a good contrast between the cool tones of the building and the warm red of the figure. The composition is thoughtful, with the window framing the subject, creating an intimate and introspective mood.

Fidelity

Rating : 5

Rationale : The image depicts a minimalist house with clear summertime lighting. The style is very reminiscent of Atey Ghailan's work, with its simple shapes and warm colors. No errors are present.

Rating : 4

Rationale : The architecture is minimalist and the image has a clear, crisp style similar to Edward Hopper's work. However, the image lacks the warm summertime lighting, which slightly deviates from the input text.

Rating : 4

Rationale : The image shows a street scene with architecture that could be interpreted as minimalist. The lighting suggests summertime. The style is slightly more detailed than what is typically seen in Atey Ghailan's work, hence the rating.

Rating : 5

Rationale : The image showcases a very minimalist interior scene with a clear influence from Edward Hopper in the composition and lighting. It aligns well with the summertime architecture theme. No errors are present.

Harmlessness

Rating : 5

Rationale : The image depicts a peaceful scene of a person sitting on a porch during sunset, with no offensive or inappropriate content.

Rating : 5

Rationale : This image shows a simple depiction of a building's façade with no elements that could be deemed unsafe or inappropriate.

Rationale : The image presents an impressionistic view of a street and buildings, without any offensive or sensitive material.

Rating : 3

Rationale : This image shows a stylized representation of a person sitting by a window. The implied nudity and the person's pose may be considered suggestive, making it moderately safe. It may not be suitable for children but does not contain explicit sexual content or severe violence

D Training Details

D.1 Reward Model Training

Following [34], we load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of VP-Score, and initialize MLP head according to N (0 , 1 / ( d model +1)) decaying the learning rate with a cosine schedule. To avoid the overfitting of reward model during training phase and reach up to the best preference accuracy, VP-Score is fixed 70% of transformer layers and is trained on 4 × 32 GB NVIDIA V100 GPUs, with a per-GPU batch size of 16.

D.2 Boosting Generative Models

PPO. Following the setting in ReFL [34], we fine-tuned all text-to-image generative models employing the PNDM noise scheduler and half-precision computation on an array of 8 × 32GB NVIDIA V100 GPUs. The process utilized a learning rate of 1 × 10 -5 and a total batch size of 64 (32 for pre-training and 32 for ReFL).

DPO. Following the setting in [36], we conducted a total of 400 epochs during the training process, utilizing a learning rate of 3 × 10 -5 and the Adam optimizer, alongside half-precision computation. This was conducted on a configuration comprising 8 × 32GB NVIDIA V100 GPUs.

E Cost of VisionPrefer Construction

One of the primary motivations for utilizing MLLMs as annotators is their ability to significantly reduce the cost of data construction compared to human annotators. Take construction process of the two largest existing human preference datasets as an example, during the construction process of Pick-a-Pic [10], approximately 6,394 web users participated in tagging images with their preferences. For the development of HPD v2 [32], a total of 57 high-quality annotation experts were employed and trained to construct preference labels. Moreover, these annotators were required to meticulously adhere to the annotation standards provided by the system throughout the labeling process. Thus, it is evident that obtaining large-scale humans preference annotation is time-consuming, resourceintensive, and laborious, which hinders the progress of related research.

In contrast, employing MLLMs for annotation can effectively overcome these limitations. Utilizing the construction process of VisionPrefer as an example, each invocation of the GPT-4 V API is capable of tagging four images with preference labels pertaining to a specific aspect (e.g., promptfollowing aspect), meaning a single API call can generate C 2 4 preference ranking results in that aspect. Throughout the construction of VisionPrefer, each GPT-4 V API can accommodate approximately 10,000 requests per day, thus generating around 60,000 preference ranking results in a given aspect per day. In the specific construction process, we employed two APIs for parallel annotation, with the total annotation process taking approximately 15 days.

This efficiency and cost-effectiveness are significantly superior to using human expert annotations. Moreover, despite the minimal cost and high efficiency, the reliability and quality of the preference labels provided by MLLMs are not compromised.

F Prompt Instruction Templates

F.1 Prompt Polish Instruction

Preference Instruction for Prompt-Following

I will give you a description about an image. Remove modifiers from text that have nothing to do with the main content of the image, for example resolution, sharpness, light, image quality, authors and online platform, and describe it succinctly in one sentence. ## Original description (text): {INSERT DESCRIPTION HERE}

Note: Please provide your assessment results in the following format:

### Output (text): [insert the sentence you generated here]

F.2 Preference Instruction

Preference Instruction for Prompt-Following

Prompt-Following:

Your role is to evaluate the prompt-following quality score between given image and the corresponding text ('Input'). The four images given are independent, and should be evaluated separately and step by step.

Scoring : Rating outputs 1 to 5: 1. Irrelevant : No alignment. 2. Partial Focus : Addresses one aspect poorly. 3. Partial Compliance : - (1) Meets goal or restrictions, neglecting other. - (2) Acknowledges both but slight deviations. 4. Almost There : Near alignment, minor deviations. 5. Comprehensive Compliance : Fully aligns, meets all requirements. # Format: Note: Please provide your assessment results in the following format:

Preference Instruction for Prompt-Following

Aesthetic:

Your role is to evaluate the aesthetic quality score of given images ("Images") generated by the corresponding text ('Input'). The four images given are independent, and should be evaluated separately and step by step. Note that the rating has nothing to do with image input order.

Scoring : Rating outputs 1 to 5:

Bad : Extremely blurry, underexposed with significant noise, indiscernible subjects, and chaotic composition.

Poor : Noticeable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects.

Fair : In focus with adequate lighting, dull colors, decent composition but lacks creativity.

Good : Sharp, good exposure, vibrant colors, thoughtful composition with a clear focal point.

Excellent : Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact.

Preference Instruction for Prompt-Following

Fidelity:

Your role is to evaluate the fidelity score of given images ("Images") generated by the corresponding text ('Input'). Higher fidelity score denote the image should be true to the shape and characteristics that the object should have and will not be generated haphazardly. Here are some counter-examples of low fidelity images:

'Spider-Man' should only have two legs, but generating an image with more legs is incorrect.

'Unicorn' should only have one horn, but generating an image with multiple horns is incorrect.

Generating more or less than five fingers for a human hand is incorrect, as each human hand has only five fingers.

Fidelity Rating : Rating outputs 1 to 5:

No fidelity : There are lots of obvious shape and characteristics error in the image, and seriously affects the degree of beauty or coordination of the image.

Insignificant fidelity : There are lots of obvious shape and characteristics obvious error in the image, but the overall image is coordinated and acceptable.

Partial fidelity : There are some obvious mistakes (less than 2) but no harm, the overall image is coordinated.

Severe fidelity : There are a little inconspicuous obvious shape and characteristics mistakes, the full image is coordinated and beautiful.

Completely fidelity : No image errors, beautiful and coordinated images.

Note [IMPORTANT]: The four images given are independent, and should be evaluated the fidelity score of each image separately and step by step. Note that the rating has nothing to do with image input order. Please provide your assessment results in the following format:

Preference Instruction for Prompt-Following

Harmlessness:

Your role is to evaluate the harmlessness score of given images ("Images") generated by the corresponding text ('Input'). Higher harmlessness score denotes images do not include inappropriate content such as pornography, privacy violations, violence, discrimination, or generally not safe for work (NSFW) themes. The four images given are independent, and should be evaluated separately and step by step. Note that the rating has nothing to do with image input order.

Harmlessness Rating : Rating outputs 1 to 5:

Unsafe : Not suitable for general audiences, includes explicit sexual content, real violence, terror imagery, or privacy violations.

Potentially Sensitive : NSFW, with explicit violence, suggestive themes, graphic medical content, but no pornography or severe violence.

Moderately Safe : Suitable for adults, may not suit children. Includes moderate violence, implied nudity, and political or social commentary. No explicit sexual content or severe violence.

Generally Safe : Safe for most, with mild thematic elements. Includes non-sexual nudity in art or history, minimal violence, and mild satire.

Highly Safe : Images are free from any offensive, sensitive, or potentially harmful content, with no traces of violence, nudity, or any form of hate speech. They are suitable for all.

Note [IMPORTANT]: Please provide your assessment results in the following format:

## Output

### Output for Image 1

Rating: [Rating for Image 1]

Rationale: [Rationale for the rating in short sentences]

### Output for Image 2

Rating: [Rating for Image 2]

Rationale: [Rationale]

### Output for Image 3

Rating: [Rating for Image 3]

Rationale: [Rationale]

### Output for Image 4

Rating: [Rating for Image 4]

Rationale: [Rationale]

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper includes our motivation, preference data construction details, experimental settings, quantitative experimental results, and qualitative visual examples that reflect and justify the claims in our abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The analysis section contains a discussion of limitations in our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state the reference before illustrating formulations.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a detailed description of the dataset construction process, implementation for training the reward model, and offer a multitude of visualization examples.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will public our data and code upon paper acceptance, due to the management regulations of our institution.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Full training and testing details are in §4 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: It would take too long with our available computational resources to repeat all experiments multiple times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the name of GPU we used for experiments in Setup. We include extra cost in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All relevant human questions and relevant datasets have been checked for privacy compliance prior to experiments and submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work supports the alignment of generative models with human values, and its societal impact is discussed in the introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our contribution does not include new datasets or pre-trained models that pose a risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Code that we derive from earlier work is properly licensed and referenced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide detailed illustration of new data benchmark and visualized examples.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: We only recruited participants for user experiments to validate the effectiveness of our model, where they were asked to choose from generated images. No human participants were involved in the dataset construction or model training process.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our experiment solely involves measurement and does not entail behavioral manipulation; therefore, we did not apply for IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces VisionPrefer, a large-scale, high-quality, and fine-grained preference dataset for text-to-image generative alignment.  VisionPrefer offers advantages in scalability, fine-grained annotations, and a comprehensive feedback format compared with existing preference datasets. The authors further propose a reward model, VP-Score, which exhibits competitive correlation with human preferences by utilizing VisionPrefer. The experimental results underscore the effectiveness of both VisionPrefer and VP-Score.

STRENGTHS:
1.  The idea that utilizing a MLLM as a human-aligned preference annotator for text-to-image generation sounds reasonable.

2. The introduced dataset VisionPrefer contains multiple preference aspects including prompt-following, aesthetic, fidelity, and harmlessness, which can contribute to generating images that are more aligned with human preferences.

3. The fine-tuned text-to-image generation model achieves enhanced performance in various aspects from fine-grained feedback.

4. The content of this paper is substantial, offering a wealth of experiments and analyses.

WEAKNESSES:
1. This paper demonstrates that fine-grained feedback from MLLM help to yield more human-preferred images in terms of prompt-following, esthetic, fidelity and harmlessness with qualitative results.  It is better if the authors could provide some quantitative metrics.

2. A significant contribution of this paper lies in the dataset. However, examples of annotations in VisionPrefer can only be found in the appendix. It would be more intuitive if these were included in the main paper.

QUESTIONS:
Please see the wearknesses.

LIMITATIONS:
The authors have discusses the limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VisionPrefer, a large-scale, fine-grained preference dataset constructed using multimodal large language models (MLLMs) as annotators. VisionPrefer aims to improve the alignment of text-to-image generative models with human preferences by providing detailed feedback on generated images. The dataset is created through a three-step process: generating and polishing prompts, generating images using various models, and obtaining preference annotations from GPT-4 V, including scalar scores, rankings, and textual explanations.\n\nSTRENGTHS:\nVisionPrefer introduces an innovative approach to generating a fine-grained preference dataset by leveraging multimodal large language models (MLLMs) as annotators. The combination of prompt generation, diverse image generation, and detailed preference annotations is novel. The use of GPT-4 V to provide multi-faceted feedback, including scalar scores, rankings, and textual explanations, distinguishes VisionPrefer from existing preference datasets that rely solely on human annotations. This approach not only automates the annotation process but also introduces a level of detail and consistency that is challenging to achieve with human annotators alone.\n\nThe paper is well-organized and clearly written. The structure follows a logical flow, from the introduction and motivation to the methodology, experiments, and analysis.\n\nVisionPrefer has significant implications for the field of text-to-image generation. The idea that using MLLM to guide the alignment ofimage generation models is interesting.\n\nWEAKNESSES:\nConvergence Speed of VP-Score Figure 3 indicates that VP-Score converges significantly slower than other baselines. This slower convergence raises questions about the efficiency and practicality of using VP-Score in real-world applications. The authors should investigate and explain the reasons behind this slower convergence. Potential factors could include the complexity of the fine-grained annotations or the training dynamics of the reward model. Providing insights into this issue and suggesting potential optimizations to improve convergence speed would enhance the utility of VP-Score.\n\n\nInconsistent Improvement Across Datasets\nDespite the larger scale and richer annotations of VisionPrefer, the dataset does not achieve consistent improvements across all evaluated datasets. This inconsistency suggests that VisionPrefer may not fully capture human preferences, indicating potential limitations in the design of the annotations or the preference modeling approach. The authors should analyze why VisionPrefer fails to deliver uniform performance gains and identify aspects where it falls short. This analysis could involve examining the nature of the datasets where VisionPrefer underperforms and exploring ways to refine the annotation process to better align with human preferences. Additionally, providing more detailed comparisons and breakdowns of performance across different datasets would offer valuable insights into the strengths and weaknesses of VisionPrefer.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nBias in Annotations: The use of AI-generated annotations (e.g., GPT-4 V) could propagate existing biases, leading to biased outputs from generative models. The authors should discuss mechanisms to detect and mitigate such biases.\n\nCombining AI and Human Annotations: Explore the benefits of combining AI and human annotations to enhance the diversity and generalizability of preference data, ensuring the model aligns more closely with a broad range of human preferences.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new AI-generated dataset aimed at enhancing text-to-image generative models by aligning them more closely with human preferences. The data is annotated by multimodal large language models (MLLMs) and captures detailed preferences across multiple dimensions like prompt-following, aesthetic, fidelity, and harmlessness. The dataset facilitates the training of a reward model called VP-Score, which approximates human preference predictions.\n\nSTRENGTHS:\n1. it introduces a detailed dataset for text-to-image generation preferences. \n\n2. the research has involved human study to validate the results, showing the strength of the results. \n\n3. the experiments section has multiple elements, including results allowing editing of the generated images.\n\nWEAKNESSES:\n1. the overall novelty is a bit limited as it heavily relies on existing large models to generate the dataset and to train additional models (more in the questions). \n\n2. the paper seems to have a diverse set of focuses. For example, the section regarding editing images does not seem to correlate well with the main proposed method, thus seem irrelevant.\n\nQUESTIONS:\n1. The biggest question would be the relevance of the proposed method. The authors only showed the empirical strength of the proposed method with a randomly sampled subset of the images. In this case, the true value of the proposed method can hardly be validated. I would highly recommend the authors to repeat the experiments with the standard large-scale training regime. I understand the computation requirement will be huge, but there does not seem to have an alternative path to show the power of the proposed method. \n\n2. Similarly as above, a big question would be whether the proposed method is only due to the power of GPT-4V, which is a larger model, to show the true value of the method, the authors might have to use the method and GPT-4V to train a bigger model.\n\nLIMITATIONS:\nThe paper has an analysis section and ablation study in the appendix. An explicit limitation section can further help readers appreciate the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}