ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
Authors: Jiazheng Xu
Xiao Liu
Yuchen Wu
Yuxuan Tong
Qinkai Li
Ming Ding
Jie Tang
Yuxiao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models
Text-to-Image
Learning from Human Feedback
Multimodality
Evaluation

Abstract: We present a comprehensive solution to learn and improve text-to-image models from human preference feedback.
To begin with, we build ImageReward---the first general-purpose text-to-image human preference reward model---to effectively encode human preferences.
Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date.
In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis.
On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer.
Both automatic and human evaluation support ReFL's advantages over compared methods.
All code and datasets are provided at \url{https://github.com/THUDM/ImageReward}.

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
Jiazheng Xu Xiao Liu Yuchen Wu', Yuxuan Tong", Qinkai List, Ming Ding',
Jie Tang', Yuxiao Dong
TTinnguu University 7Zhipu Al
Beijing U. of Posts and Telecommunications
Abstract
We present a comprehensive solution learn and improve text-to-image models from human preference feedback. To begin with, build ImageReward-tth first general-purpose text-to-image human preference reward model-to effectively encode human preferences. Its training is based pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), direct tuning algorithm to optimize diffusion models against scorer. Both automatic and human evaluation support ReFL's advantages compared methods. All code and datasets are provided at posttss//eee //t::://iiii http:/pp//gg.iaab.co////////////
systematic annotation
com MHHHUDD//ImagRRwwadd
Introduction
Text-to-image generative models, including auto-regressive (43 13: 16 63
and diffusion-
based [37] 45 42 46 approaches, have experienced rapid advancements in recent years. Given
appropriate text descriptions (i.e., prompts), these models can generate high- high-fidelity and semantically- related images
wide range of topics, attracting significant public interest in their potential applications and impacts.
Despite the progress, existing self eeffsuupervised pre-trained 33 primary challenge lies aligning models with human preference, as the pre-training distribution noisy and differs from the actual user-prompt distributions. The inherent discrepancy leads to several well-documented issues in the generated images 31), including but not limited to:
generators are far from perfect. A
Text-image Alignment: failing to accurately depict all the numbers, attributes, properties, and relationships of objects described in text prompts, as shown in Figure (b).
Body Problem: presenting distorted, incomplete, duplicated, abnormal body parts (e.g., limbs) of humans or animals, as illustrated in Figure (e) (e)().
Human Aesthetic: deviating from the average mainstream human preference for aesthetic styles, as demonstrated in Figure (c)().
Toxicity and Biases: featuring content that is harmful, violent, sexual, discriminative, illegal, or causing psychological discomfort, depicted Ffg
These prevalent challenges, however, are difficult to address solely through improvements in model architectures and pre-training data.
In natural language processing (NLP), researchers have employed reinforcement learning from human feedback (RLHF) [55] 36 39] to guide large language models (6; 7 66 48 64 towards human
preferences and values. The approach relies on learning a reward model (RM) to capture human
"JX& & XL contributed equally. Corresponding authors: YD & JT (yuxiaod) jittaggtttingnuuuuuuuu www.aaggww.uuuuuuueeee.ooo en).
Work done when JX interned at Zhipu Al and QL visited Tsinghua University.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Aesthelie
CLIP Score
BLIP Score
Imaget eward
(Ours)
ReFL.
(Oug))
(6) painting of girl walking hallwaay and suddenly giant sunflower on the floor blocking her way.
(b) Coronation of the emperor. digital art, illustration, resolution, intricate, extremely detailed,, depth, vivid colors.
(c) Symmetry!! Product render poster vivid colors divine propartion owl, glowing log intricate. elegant, highly detailed.
[d] A unicorn in clearing. has single shining horn. volumetric light. emmazuel shiu, harry potter, eragin.
(e) Flghhy detailed portrait of woman with long hairs, stephen unreal eegine, fanlasy by greg rutkowski.
(f) Sculpture made flame, portralt, femall,, future, harper's bazaar, vogue, magazine, Intricate.
Figure 1: (Upper) Top-1 images out of 64 generations selected by different text-image scorers. (Lower) 1-shot generation after ReFL. training using ImageReward as feedback. Images get better text coherence and human preference after ImageReward selection or ReFL training. In prompts (from real users, truncated), the bold roughly denotes content, and the italic denotes style or function.
preference from massive expert-annotated model output comparisons. Effective though is, the annotation process can be costly and challenging [39]. it requires months of effort to establish labeling criteria, recruit and train experts, verify responses, and ultimately produce the RM.
Contributions. Recognizing the importance of addressing these challenges in generative models, we present and release the first ceneral-purpose text-to-image human preference RM- e which trained and evaluated on 137k pairs of expert comparisons in total, based on real-world user prompts and corresponding model outputs. Based the effort, further investigate the direct optimization approach ReFL for improving diffusion generative models. Our main contributions are:
We systematically identify the challenges for text-to-image human preference annotation, and consequently design pipeline tailored for it, establishing criteria for quantitative assessment and annotator training, optimizing labeling experience, and ensuring quality validation. We build the text-to-image comparison dataset for training the ImageReward model based on the pipeline. The
overall architecture is depicted in Figure 2
We demonstrate that Rewadd outperforms existing text-image scoring methods, such as CLIP [41] (by 38.6%), Aesthetic [50] (by 39.6%), and BLIP 266 (by 31.6%), terms of understanding human preference in text-to-image synthesis through extensive analysis and experiments. ImageReward also proven to significantly mitigate the aforementioned issues, providing valuable insights into how human preference can be integrated into generative models.
We suggest that ImageReward could serve as promising automatic text-to-image evaluation metric. Compared to FID [18 and CLIP scores prompts from real users and MS-COCO 2014, ImageReward aligns consistently to human preference ranking and presents higher distinguisha- bility across models and samples.

---

Sample
1) Prompts and Generated images
Prompt painting of with cloude and birds, time, low depth field effect.
Overal Rating
2) Annotation
(1=worst, T=bes0)
Preference Learning
Rating
Image-Text Aligmmett
Reward
Fidelity (Image Qualty)
C
Ranking
A
-1.4
ImageReward ReFL
Reward Feadback
4) Back Propagation
1) A random with gradient
2) directly
X
-1.49
Image Reward
Inforence after ReFL training
XT
11+1
X2
XO
1.20
Figure 2: An overview of the ImageReward and ReFL. (Upper) ImageReward's annotation and training, consisting of data collection, annotation, and preference learning. (Lower) ReFL leverages ImageReward's feedback to directly optimize diffusion models at a random latter denoising step.
We propose Reward Feedback Learning (ReFL) for tuning diffusion models regarding human preference scorers. Our unique insight ImageReward's quality identifiability at latter denoising steps allows the direct feedback learning on diffusion models, which offer no likelihood for their generations. Extensive automatic and human evaluations demonstrate ReFL's advantages over existing approaches including data augmentation 66 [3] and loss reweighing 23
ImageReward: Learning to Score and Evaluate Human Preferences
ImageReward is constructed using systematic pipeline involving data collection and human annota- tion from experts. Based on the pipeline, we implement the RM training and derive the ImageReward.
2.1 Annotation Pipeline Design
Prompt Selection and Image Collection. The dataset utilizes a diverse selection of real user prompts from DiffusionDB [58], an open-sourced dataset. To ensure diversity in selected prompts, we employ graph-based algorithm that leverages language model- prompt similarity [56) 44] (33. This selection yields 10,000 candidate prompts, each accompanied by 4 to sampled images from DiffusionDB, resulting in 177,304 candidate pairs for labeling (Cf. Appendix A.I for details).
Human Annotation Design. Our annotation pipeline involves a prompt annotation stage, which includes categorizing prompts and identifying problematic ones, and text-image rating stage, where images are rated based on alignment, fidelity, and harmlessness. Subsequently, annotators rank the images in order of preference. To manage potential contradictions in the ranking, we provide trade-offs in our annotation document (completely attached in Appendix B) Our annotation system is composed of three stages: Prompt Annotation, Text-Image Rating, and Image Ranking. Screenshots of our system are provided in Figure Annotators were recruited in collaboration with professional data annotation company, with a majority having at least college-level education. They are trained using documents that describe the labeling process and criteria. To ensure quality, we employ quality inspectors to double-check each annotation, with invalid annotations reassigned for relabeling. Due to the page limits, please refer to Appendix A.3 A.2 B for comprehensive details and discussion.
Human Annotation Analysis. After 2 months of annotation, we collected valid annotations for 8,878 prompts, resulting in 136,892 compared pairs. comprehensive analysis of these prompts, annotations, and challenges discovered is discussed in detail in Appendix A.4

---

Table Text-to-image model ranking by humans and automatic metrics (ImageReward, CLIP, and FID). "Zero-shot FID (30k) scores of DALL-E 2 is taken from [2]; others are evaluated in x56x256 resolution on MS-COCO 2014 validation set following prior practices.
Real User Prompts
MS-COCO 2014
Datasct & Model
Human Eval.
ImageReward
CLIP
ImageReward Zero-shot FID
Rank
#Win
Rank
Score
Rank
Score
Rank
Score
Rank
Score
Openjourney Stable Diffusion 2.1 2.l-baee
507
0.2614
0.2726
-0,0455
5
20.7
463
0.2458
0.2683
0.1553
18.8
DALL-E
3
390
0.2114
0.2684
0.5387
10.9*
Stable Diffusion 1.4
362
0.1344
0.2763
-0.0857
17.9
Versatile Diffusion
340
-0,2470
0,2606
-0.5485
18.4
CogView
74
-1,2376
0.2044
-0.8510
26.2
Spearman p Human Eval.
1.00
0.60
0.77
0.09
2.2 RM Training
Admittedly, human evaluation is after all the touchstone for human preference for synthesized images; but is limited by labor costs and hard to scale up. We aim to model human preference based on annotations, which can lead virtual evaluator free from dependence humans.
Similar to RM training for language model of previous works [55] 39), we formulate the preference annotations rankings. We have & E [4, 9] images ranked for the same prompt T' (the best to the worst are denoted as #1; 22: ..., T&) and get at most Cg comparison pairs no ties between two images. For each comparison, if z; is better and xy is worse, the loss function can be formulated as:
loss(0) (0) = - -(D=,..,)--
2;) - fo(T,
where fo(T, x)) is scalar value of preference model for prompt and generated image x.
Training Techniques. We use BLIP 26 as the backbone of ImageReward, as outperforms conventional CLIP (Cf. Table in our preliminary experiments. We extract image and text features, combine them with cross attention, and use MLP generate a scalar for preference comparison.
Training IewaeReward is of no ease. We observe rapid convergence and consequent overfitting, which harms its performance. To address this, we freeze some backbone transformer layers' parameters, finding that a proper number of fixed layers improves ImageReward's performance (Cf. Section [4.)) ImageReward also exhibits sensitivity to training hyperparameters, such learning rate and batch size. We perform a careful grid search based on the validation set to determine optimal values.
2.3
As Metric: Re-Evaluating Human Preferences on Text-to-Image Models
Training text-to-image generative models is hard, but evaluating these models reasonably is even harder. In literature [T] 42 12 (66, it has been a de facto practice to evaluate text-to-image generative models MS-COCO [28] mmage-caption dataset against the real images, using fine-tuned zero- 43 setting. Nevertheless, remains quite dubious whether 38 especially from the following aspects:
shot FID 18] scores following DALL-E
the FID really fits the eurrent need
As generative models are now dominantly used by the public in zero-shot manner without fine-tuning, fine-tuned FID may not honestly reflect models' actual performance real use. addition, despite the adoption of zero-shot FID recent trends, the possible leak of MS-COCO in some models' pre-training data would make it potentially unfair setting.
Zero-shot Usage:
2. Human Preference: FID measures the average distance between generated images and reference real images, and thus fails to encompass human preference that is crucial to text-to-image synthesis in evaluation. Moreover, FID's relies on average over the whole dataset provide an accurate assessment, whereas in many cases we need the metric to serve as selector over single images.
Seeing these challenges, we propose ImageReward promising zero-shot automatic evaluation metric for text-to-image model comparison and individual sample selection.
Better Human Alignment Across Models. We conduct researcher annotation (i.e., by authors) across 6 popular high-resolution (around 512x512) available text-to-image models: Cog View 2 [2].

---

Stable Diffusion (SD) 1.4 and 2.1-base [5]. DALL-E 2 (via OpenAI API) 42,, and Openjoumey to identify the alignment of different metrics to human.
Versatile Diffusion (VD) 66
We sample 100 real-user test prompts for the alignment test, with each model generating 10 outputs as candidates. To compare these models, we first pick the best image out of 10 outputs by each model on each prompt. Then, the annotators rank the images from different models for each prompt, following the disciplines for ranking described in Section 2.1 We aggregate all annotators' annotations, and compute the final win count of each model to all others (Cf. Table D
For ImageReward and CLIP scores, report their average for 1,000 text-image pairs per model. We also document all models' zero-shot FID and ImageReward score (30k) on MS-COCO 2014 valid set following prior practices 442 [2] where outputs unified to 256 x266 resolution and optimal
classifier-free guidance values are selected by grid search (i.e., [1.5, 2.0, 3.0, 4.0, 5.0)). As shown in
Table 1 ImageReward aligns well with human ranking, whereas zero-shot FID and CLIP are not.
Better Distinguishability Across Models and Sam- ples. Another highlight is that, compared to CLIP, observe that ImageReward can better distinguish the quality between individual samples. Figure 3 presents a box plot of ImageReward and CLIP's score distributions on the 1,000 generations per model. The distributions
RewarR Score
CLIP Score
Openjourney
50
DALL-E2
normalized 0.0 to
SD 1.4
1.0 using minimum and maximum values of Im- ageReward and CLIP scores per model, and outliers are discarded. As it demonstrates, ImageReward's scores in each model have a much larger interquartile range than that of CLIP, which means ewageReward can well distinguish the quality of images from each other. Besides, in terms of comparison across mod- els, we discover that the medians of the ImageRe- ward scores are also roughly in line with human ranking in T On the contrary, CLIP's medians fail to present the property.
VD
CogView 2
0.0
0.2
0.4
0.5
0.8
1.0
Figure 3: Normalized distribution of ImageRe- ward and CLIP scores of different generative models (outliers are discarded). ImageRe- ward's scores align well with human prefer- ence and present higher distinguishability.
3 ReFL: Reward Feedback Learning Improves Text-to-Image Diffusion
Though ImageReward can pick out highly human- -renerrreeerred images from many generations of prompt, the senerrat-and--ten---l
paradigm could be expensive and inefficient in practical applications. Therefore, we seek to improve text-to-image generative models, particularly for the popular latent diffusion models, for allowing high-quality generation in single or very few trials.
Challenge. In NLP, researchers have reported using reinforcement learning algorithms (e.g., PPO (51) to steer lan- guage models to align to human pref-
Prompt: Landscape photography by marc adamus, mountains with some forests, small lake in the center, fog in the background, sunrays, golden hour, high quality.
step=30
stepe40
0.5
erence (55 36 39, which depends on
ImageReward of X,
0.0
the likelihood of a whole generation to
update the model.
-1.0
0.59
However, unlike language models, la- tent diffusion models (LDMs)'s multi- step demoising generation cannot yield likelihoods for their generations, and thus fail to adopt the same RLHF approaches. A potentially similar approach is classifier-guidance [54] Ω technique during LDM inference. Nonetheless, it is for inference only
-1.5
-2.0-
15
20
25
30
35
40
2.15
1.62
Denoising Step
Figure ImageReward scores of prompt with different generation seeds along denoising steps. Final image quali- ties become identifiable after 30 out of 40 steps.
httpe://openjourreyyaaz art/
5

---

and employs classifier necessarily trained on noisy intermediate latents, which naturally con- tradicts RMs' annotation where images need to be completely denoised for humans to mark correct preference. Some concurrent works propose some alternative indirect solutions, such as using RMs to filter dataset for fine-tuning [6] 13), or to re-weight losses of training samples according their qualities [23]. Nevertheless, these data-oriented approaches are virtually indirect. They could rely heavily on proper fine-tuning data distributions and finally only improve the LDMs mildly.
ReFL: Insight and Solution. We en- deavor to develop direct optimiza- tion method for improving LDMs ac-
Algorithm 1 Reward Feedback Learning (ReFL) for LDMs
1: Dataset: Prompt set y = V2.... Mr
cording to an RM (e.g.. ImageRe- ward). Looking into ImageReward scores along denoising steps (i.e., 40 in our case), we derive an intriguing insight (Cr. Figure 4 that when we
2: Pre-training Dataset: Text-image pairs dataset D [[ixt,, ((xtz,img,) imgg). ....g.....(.... (IxI..,II,,,, immg,))
Imput: LDM with pre-trained parameters wo, reward model r.
reward-to-loss map function Φ, LDM pre-training loss function
v, reward re-weight scale X
Initialization: The number of noise scheduler time steps T, and time step range for fine-tuning [T1, T2]
directly predict 24 + z at a step (different from the real latent 2o which
for yi E y and (txt, img,) ) E do
experiences 2₁ + D++1 + .... + za):
Eyre. f thu, (txt; img.
When < 15: IewgeReward
7:
w; t- w; // Update LDM using Pre-training Loss
scores for all generations are unan- imously low.
8:
t rand(Ti (T) // Pick random time step [%]
oT (0, T) // Sample noise as latent
10:
j = ..., t + do
When 15 S t 30: High-
11:
no grad: x + LDM {2+}
quality generations begin to stand out, but overall clearly judge all generations' final
12:
end for
still cannot
13:
with grad: _ LDM {a.)
14:
2o <- It-1 Predict original latent by noise scheduler
qualities based on the current Im- ageReward scores.
15:
21 + 2b From latent to image
16:
Creward + \o(r(y; zx)) // ReFL loss
17:
We41 +- we // Update LDM., using ReFL loss
When t > 30: Generations of different ImageReward scores are generally distinguishable.
18:
end
In light of the observation, we conclude that ImageReward scores for generations z' after 30 steps of denoising, unnecessarily the final step, could serve as reliable feedback for improving LDMs.
We thus propose an algorithm directly fine-tune LDMs by viewing the scores of an RM human preference losses to back-propagate gradients (Cf. Algorithm [T to a vandomly-picked latter step t (in our case t [30, 40) in the denoising process. The reason for the random selection of t instead of using the last step that, if only the gradient of the last denoising step is retained, the training is proved very unstable and the results bad. In practice, avoid rapid overfitting and stabilize the fine-tuning, we re-weight ReFL loss and regularize with pre-training loss. The final loss form is written
Lrewardd = AE (ollr//olrrr.... (o(()))))
(2)
Care
E yn m jo ( (r )
co(zi,, t, (o (o(1)))111)
(3)
where 9 denotes the parameters of the LDM, (3x) denotes the generated image of LDM with parameters corresponding to prompt yı. Meanings of other symbols detailed in Algorithm I while the loss function of Cpre is taken from 45
Experiment
4.1 ImageReward: On Human Preference Prediction
Dataset & Training Setting, Rankings of annotated images are collected to train ImageReward, which contains 8,878 prompts and 136,892 pairs of image comparisons. We divide the dataset according to prompts annotated by different annotators and select 466 prompts from annotators who have a higher agreement with researchers to consist for the model test. Except for prompts for testing, other more than 8k prompts of annotation are collected for training.

---

Table 2: Data annotation agreement and ablation study model backbones and dataset sizes. (a) Agreement between different annotators, researchers, and models. Espe- cially, "annotator ensemble" means, for each pair of images, use the image considered better by most people as the better one.
(b) Ablation study for different backbones (i.c., CLIP and BLIP) used ImageReward.
annotator
CLIP Score
BLIP Score
Training Size
Preference
researcher annotator ensemble
Aesthetic
Ours
Backbone
Acc.
rescarcher
71.2%
65.3%
73.4%
57.8% + 3.6%
55.6%
57.0% + 3.0%
64.5%
CLIP
4k
61.87
= 11.1%
+ 8.5%
+ 6.2%
+ 3.1%
2.5%
8k
62.98
annotator
65.3%
65.3%
53.9%
54.3%
55.9%
57.4%
65.3%
1k
63.07
5.6%
d: 5.8%
d: 2.7%
2k
63.18
BLIP
annotator
73.4%
53.9%
54.4%
57.5%
62.0%
70.5%
4k
64.71
ensemble
= 6.2%
+ 5.8%
21.1%
+ 15.9%
16.1%
+ 18.6%
8k
65.14
Table Results of ImageReward and comparison methods on human preference prediction. Prefer- ence accuracy is from the test set of 466 prompts (6,399 comparisons); Recall and Filter's scores are from another test set of 371 prompts with images each. All scores are averaged per prompt.
Preference Acc.
Recall @2
Filter @2
Model
@ 1
@4
@ l
@4
CLIP Score
54.82
27.22
48.52
78.17
29.65
51.75
76.82
Aesthetic Score
57.35
30.73
53.91
75.74
32.08
54.45
76.55
BLIP Score
57.76
30.73
50.67
77.63
33.42
56.33
80.59
ImageReward (Ours)
65.14
39.62
63.07
90.84
49.06
70.89
88.95
We load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of ImageReward, and initialize MI.P head according to ( 1 /(ddmodett+] several value settings of learning
decaying the learning rate with a cosine schedule. We sweep rate and batch size and fix different rates of backbone transformer layers. We find that fixing 70% of transformer layers with learning rate of lc-5 and batch size of 64 can reach up to the best preference accuracy, IewgeReward trained on 40GB NVIDIA A100 GPUs, with a per-GPU batch size of 16.
We use the CLIP score, Aesthetic score, and BLIP score as baselines to compare with the ImageRe- ward. CLIP score and BLIP score are calculated directly as cosine similarity between text and image embedding, while the Aesthetic score is given by an aesthetic predictor introduced by LAION 50)
Agreement Analysis. Agreement assesses the likelihood of two individuals sharing consistent preferences for superior images. While most people generally agree on image quality, variations in model- gener-eeeeerted images may lead to divergent judgments. Before assessing model performance, it's crucial to measure the likelihood of consensus in selecting superior images. We use other 40 prompts (778 pairs) to calculate preference agreement between researchers, annotators, annotator ensemble, and models. Table shows the result.
Preference Accuracy. Preference accuracy is the correctness scorer choosing the same one from two different images of one prompt with a human. As Table 3hoowss our model outperforms all the baselines. The preference accuracy ImageReward reaches up to 65.14% 14%, which is 15.14% more than 50% (random), about twice as much as 7.76% (that of BI.IP score).
Main Results:
Main Results: Human Evaluation. To evaluate the ability of ImageReward to select the more preferred images among large amounts of generated images, we produce another dataset, collecting prompts with 9/25/64 generated images from DiffusionDBB and use different methods to select from those images to get top3 results. Then three annotators rank these selected top-3 images. Figure 5 shows the win rates. Qualitative results can be seen Appendix G showing that ImageReward can select images that are more aligned to text and with higher fidelity and avoid toxic contents.
blation Study: Training dataset size. To investigate the effect training dataset sizes on
performance of the model, comparative experiments are conducted. Table Table/2bshhi shows that adding up the scale of the dataset significantly improves the preference accuracy of ImageReward. It's promising that if we collect more annotation data in the future, ImageReward will get better performance.
7

---

Win
50%
50%
50%
Ours
Random
Ours
CLIP Score
Ours
Aesthetic
Ours
BLIP Score
0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% selected from images
Top selected from images
selected from 64 images
Figure 5: Win rates of ImageReward compared to other models. IeageReward wins most of the time. On average, 77.1% 1% to random, 69.3% 3 to CLIP, 69.8% to Aesthetic, and 65.8% to BLIP.
Real User Prompts
MT Bench
40
0.6
Methods
0.8
#Win
WinRate
#Win WinRate
Dataset Ritering
0.55
0.5
0.65
0.54
0.72
0.83
0.48
0.7
SD v1.4 (baseline)
45
1315
718
Weighted
0.6
Datasct Filtering
61
1394
55.17
735
51.72
RAFT:1
0.5
0.46
0.69
0.5
0.78
0.87
0.4
0.5
Reward Weighted
23
1075
39.52
43.33
RAFT B
(iter=1)
1341
49.86
578
42.31
RAFT (iter2)
753
30.85
452
33.02
RAFT:3
0:1
0.17
0.16
0.13
0.17
0.5
0.15
RAFT (iter=3)
398
20.97
355
26.19
ReFT. (Ours)
1508
58.79
808
58.49
ReFi
0.59
0.67
0.76
0.85
0.5
v1.4
Dataset
Rittrinn
Reward
Weighted
RAFT-1
RAFT-2
RAFT-3
ReFL
Table 4: Human evaluation on different LDM optimiza- tion methods. ReFL performs the best with regard total win count and WinRate against SD v1.4 baseline.
Figure
Win rates between all methods.
Ablation Study: RM backbone. ImageReward adopts BLIP as the backbone, which may raise curiosity about how well BLIP compares to CLIP. We add MLP to CLIP, training in similar way, and the result is also shown in Table 2b Even if CLIP uses relatively larger training data set, its preference is still inferior to that of BIP.. The difference between these two as backbone may partly be because BLIP used bootstrapping of its training set. Moreover, we use BLIP's image- grounded text encoder as feature encoder different from the separate encoder for text/image as CLIP.
mage-ggounded
4.2 ReFL: On Improving Diffusion Models with Human Preference
Training Settings. We use Stable Diffusion v1.4 4 45 as the baseline generative model and fine-tune for experiments. For the dataset, the pre-training dataset from a 625k subset of LAION-SB(50) selected by aesthetic score, while the prompt set for ReFL is sampled from DiffusionDB. The model fine-tuned half-precision on 40GB NVIDIA A 100 GPUs, with learning rate le-5 and batch size of 128 in total (64 for pre-training and 64 for ReFL). For ReFL algorithm, we set ReLU, le and = 40, [1, T3l = 10).
Evaluation Settings. We collect 466 real user prompts from DiffusionDB and designed challeng- ing prompts from multi-task benchmark (MT Bench) 40 for evaluation. All fine-tuning methods use the same dataset as the pre- pre-training dataset or generated dataset (both contain 20,000 samples), and training settings (the same learning rate and batch size) for fair
train for one epoch with the comparison. The human evaluation consistent with Section 2.3 and the form dataset labeling, which involves humans sorting multiple images under prompt. Table 4 and Figure 6 show the comparison results. All methods use the same pre-trained Stable Diffusion v1.4 and the same reward model ImwgeReward, using PNDM 30 noise scheduler and default classifier free guidance scale of 7.5 for inference.
We compare several important related methods for improving text-to-image generation 61 23 13 whose implementation details are provided Appendix Compared to ReFL's direct tuning, these previous methods are all based indirect data augmentation loss reweighing.

---

Datasel Filtering
Reward Weighted
ReFL (Ours)
Original
RAFT
Mountains range with waterfall, purple hare, or rudhrarki magelt vilfeneeee armmntioon
Portrait of lemale elf warlock, long pointy
glowing green eyes, basky rod
and trecides medieral
setting desulled face, higily desaled, digital paiuting arisationn.
An concept art illustration,
photorcalistic tribal people working,
Fantacy street with huts, large insect and plant hiomes, aling styre by wlip.
A half masked rugged laboratory engiiner mun cybernetic enhancements us ween from a distanev, weff chonactor portrait by greg murhueshs essohin, craig mullier.
Figure 7: Qualitative comparison between ReFL and other fine-tuning methods. ReFL fine-tuned model can produce images that are more preferred overall. For example, in the second row of prompts containing "long pointy ears," only the model fine-tuned with ReFL generates correct ears, while other images either lack ears or have inaccurate representations.
Results and analysis. When compared to the original version, ReFL fine-tuned model is mostly preferred with the most win rate and the highest win rate. When compared to each other, ReFL is always the preferred one.
Note that in our evaluation, neither RAFT 13 nor Reward Weighted 23 has become better compared to the baseline, although they have been verified in their own experiments. Note that both RAFT and Reward Weighted do not collect the prompts used by users in real scenarios at finetune, whereas Reward Weighted manually constructs a dataset to address the alignment issue by combining colors, numbers, backgrounds, and objects. The prompts used in our review are more widely distributed and complex, the problems with their methods are more clearly exposed.
RAFT 133 suffers from over-fitting as the number of iterations increases. expert generator as regularizer to avoid overfitting the reward model. However, RAFT is constrained by the quality of the constructed dataset. It is important to note that even expert generators have limitations, and when fine-tuning is performed using prompts sampled from real user data, which can be challenging, there may be instances where the expert generator fails to generate high-quality images.
13
propose using an
In the case of the Reward Weighted method [23], although real images used for regularization, there is a problem with the coefficient used for the rewards, which is constrained within the [0, I] range. This implies that while preferred images given larger weights and poor images given smaller weights, the influence of the non-preferred images is not completely eliminated. Similarly, when utilizing real user prompts, relatively the best) in the dataset, which can introduce interference. The failure to eliminate the impact of pre-preferee images hinders the effectiieness of the Reward Weighted method.
likely that there will be non-preferred images (even those
Dataset Filtering 61], on the other hand, considers real images and handles non-preferred images by labeling them as "Weird image." However, this influence is indirect. In contrast, proposed algorithm provides direct gradient feedback through rewards, allowing for guidance toward a "better" direction, which enables more effective problem-solving.
In summary, RAFT constrained by the limited ability of the generator, and the Reward Weighted method suffers from the influence of non-rrefrrred images due to the choice of reward coefficients.

---

Dataset Filtering partially addresses the problem by considering real images and labeling abnormal images, but is still indirect and limited. By directly incorporating rewards into the gradient feedback, our proposed algorithm ReFL. offers more effective solution to these challenges. Qualitative examples are in Figure
Related Work
Text-to-image Generation and Evaluation. Text-to-image generation has come long way since the popularization of GANs (17), with key developments including models like DALL-EE [ and Vig [II]. Recently, difffusion models [52] 10: 47 have achieved remarkable results, with Stable Diffusion 45 being particularly popular. Evaluation metrics such as Inception Score (IS) and Fréchet Inception Distance (FID) [88] are commonly used to assess model performance after fine-tuning, but they cannot evaluate either single image generations or text-image coherence.
For evaluating individual generated images based on a prompt, prior works 42 46 63 often use
CLIP [4] calculate text-image similarity. While these metrics are useful, they don't capture human preference comprehensively. Other predictors, like Aesthetic from LAION [50]. partially contribute to this holistic evaluation by scoring image aesthetics using CLIP--based architecture. RM in RLHF, on the other hand, considers a mixture of elements such as text-image alignment, fidelity, and aesthetics. Overall, RM such ImageReward provides a more complete evaluation for individual texi-to-image generations, making it better aligned with human preferences.
Learning from Human Feedback. There is often a gap between generative models' pre-training objectives and human intent. Thus human feedback has been utilized to align model performance
with intent in various language applications [ 22 36 67 65 via training an RM 355
g
[5]
20 24]
learn human preference. Researchers have explored RL for language models achieve more
truthful, helpful, and harmless outcomes [39] 2 68 60 55 29 49 3). Previous work (5] 55 used
human feedback to train reward functions for summarization tasks, while InstructGPT 39 applied RLHF to GPT-3 for multi-task NLP, yielding significant improvements.
In -ixt-to-image generation, however, there have been few studies on the topic. One concurrent work 23 has focused on text-image coherence in the closed domain using simple synthetic prompts based templates, and propose to improve models using loss re- re-weighing. Other concurrent works 21 [3] collects 1-of-n selection from noisy online user clicking, and thus do not enforce based indirect data
consistent standards and prompt diversity. Their optimization methods filtering and augmentation. On the contrary, ImageReward serves as a general-purpost human preference scorer with quality ensured by rigorous annotation pipeline, and corresponding ReFL is the first direct tuning method for optimize diffusion models from scorer feedback.
6 Conclusion
In this work, we have presented ImageReward and ReFL,, the first general-purpose text-to-image human preference reward model, and direct fine-tuning approach for optimizing diffusion models by ImageReward feedback. Through our systematic pipeline for human preference annotation, we curate dataset of 137k expert comparisons to train ImageReward, and build ReFL algorithm top of it. They together address prevalent issues in generative models and help to better align text-to-image generation with human values and preferences.
Acknowledgement
We would like to thank the data annotators for their help and support. This research was supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD01 177200018000 and 2022ZD01 18601, Natural Science Foundation of China (NSFC) for Distinguished Young Scholars No. 61825602, NSFC No. 62276148. research fund from Zhipu.AI, and the New Comerstone Science Foundation through the XPLORER PRIZE.
10

---

References
[1] D. Bahdanau, Brakel, K. Xu, A. Goyal, Lowe, J. Pineau, Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations, 2016.
[2] Y. Bai, Jones, K. Ndousse, A. Askell, A. Chen, DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 220i::2200000862., 2022.
[3] Y. Bai, S. Kadavath, Kundu, A. Askell, Kernion, A. Jones, A. Chen, A. Goldie, Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini wrXiv:2212.0800) 2022.
[4] S. Barratt and R. Sharma. A note on the inception score. arXiv preprint arXiv: 2018.
[5] F. Böhm, Y. Gao, C. M. Meyer, O. Shapira, I. Dagan, and I. Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pages 3110-3120, 2019.
[6] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, Roberts, Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.08 02311, 2022.
[8] Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences, Jun 2017.
[9] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[10] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis, Dec 2021.
[11] M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, $4:19822-19888 2021.
[12] M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In Advances in Neural Information Processing Systems, 2022.
[13] H. Dong, Xiong, D. Goyal, R. Pan, S. Diao, Zhang, K. Shum, and Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint ar 27X1v:2304.06777, 2023.
[14] Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873- 12883, 2021.
[15] W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Wang. Training-frre ffrne structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint 222.0:2212.05033, 2022.
[16] O. Gafni, A. Polyak, O. Ashual, Sheynin, D. Parikh, and Y. Taigman. Make-a-scene: Scene- based text-to-image generation with human priors. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XV, pages 89-106. Springer, 2022.
[17] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1143911139444 2020.
11

---

[18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6833 2020.
[20] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari, Nov 2018
[21] Y. Kirstain, A. Polyak, U. Singer, Matiana, Penna, and O. Levy, Pick-a-pic: An open dataset of user preferences for text-to-image generation. 2023.
[22] J. Kreutzer, S. Khadivi, F. Matusov, and S. Riezler. Can neural machine translation be improved with user feedback? Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 92-105, 2018
[23] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint 22211923...12192, 2023.
[24] K. Lee, L. M. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. International Conference on Machine Learning, pages 6152-6163. PMLR, 2021.
[25] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint wwx.x:2104.00899 2021.
[26] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900, PMLR, 2022.
[27] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597 2021.
[28] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L, Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[29] H. Liu, C. Sferrazza, and P. Abbeel. Chain of hindsight aligns language models with feedback, Feb 2023.
[30] L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
[31] N. Liu, S. Li, Y. Du, A. Torralba, and J. Tenenbaum. Compositional visual generation with composable diffusion models, Jun 2022.
[32] X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
[33] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang. Self-supervised learning: Generative contrastive. IEEE Transactions Knowledge and Data Engineering, 35(1) 876, 2021.
[34] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
[35] MacGlashan, M. Ho, R. Loftin, B. Peng, G. Wang, D. Roberts, M. Taylor, and M. Littman. Interactive learning from poliey-dependen human feedback, Aug 2017.
12

---

[36] R. Nakano, J. Hilton, S. Balaji. J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- -nnwerring with human feedback. arXiv preprint arXiv:2/12.09332. 2021.
[37] A. Q. Nichol, Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, Megrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804. PMIR, 2022.
[38] M. Otani, R. Togashi, Y. Sawai, R. Ishigami, Y. Nakashima, E. Rahtu, J. Heikkilä, and S. Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceed- ings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, 2023.
[39] I.. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[40] V. Petsiuk, A. E. Siemenn, S. Surbehera, Z. Chin, K. Tyser, G. Hunter, A. Raghavan, Y. Hicke, B. A. Plummer, O. Kerret, et al. Human evaluation of text-to-image models on multi-task benchmark, arXiv preprint arXiv:22/1.122 12/2/2 2022.
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 PMLR, 2021.
[42] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204. 122 2022.
[43] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[44] N. Reimers and I. Gurevych. Sentence-bert Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
[45] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[46] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Prucessing Systems, 35:36479-36494, 2022.
[47] C. Saharia, Ho, W. Chan, T. Salimans, J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pallern Analysis and Machine Intelligence, page 1-14, Sep 2022.
[48] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, Yvon, M. Gallé, et al. Bloom: 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.0555
2022.
[49] Scheurer, J. Campos, Chan, A. Chen, K. Cho, and E. Perez, Training language models with natural language feedback, Apr 2022.
[50] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[51] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
13

---

[52] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pages 2256-2265. PMLR, 2015.
[53] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857-16867, 2020.
[54] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
[55] N. Stiennon, L. Ouyang, Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 00008-001,222000 2020.
[56] H. Su, Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, R. Zhang, M. Ostendorf, L. Zettlemoyer, N. A. Smith, al. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975, 2022.
[57] L. van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 988)) 2605, 2008
[58] Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210. 14896 [cs]. 2022.
[59] G. Warnell, N. Waytowich, V. Lawhern, and Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[60] Wu, L. Ouyang, D. Ziegler, N. Stiennon, R. Lowe, J. Leike, and Christiano. Recursively summarizing books with human feedback, Sep 2021.
[61] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv: 230. 2023.
[62] X. Xu, Z. Wang, E. Zhang, K. Wang, and H. Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08333 2022.
[63] Yu. Xu, J. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, Ku, Y. Yang, B. K. Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[64] A. Zeng. X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, et al. Glm- 130b: An open bilingual pre-trained model. arXiv preprint 2rXhv:2210.02414 2022.
[65] J. Zhang, Y. Liu, J. Mao, W. Ma, J. Xu, S. Ma, and Q. Tian. User behavior simulation for search result re-ranking, ACM Transactions on Information Systems, 11(1):1-35, 2023.
[66] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint a205: 01008, 2022.
[67] W. Zhou and K. Xu. Learning to compare for better training and evaluation of open domain nat- ural language generation models. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9717-9724, Jun 2020.
[68] D. Ziegler, N. Stiennon, J. Wu, T. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences., Sep 2019.
14

---

Partl I
Appendix
Table of Contents
A Details on ImageReward's Comparison Data Annotation Pipeline
16
A.1
Prompt Selection
16
A.2
Annotation Management
16
A.3
Human Annotation Design
Human Annotation Analysis
B
Annotation Document
21
B.1
Annotate the input text
22
B.2
Annotate generated images
23
B.3
Rank generated images]
24
B.4
Froquently asked questions
24
More Analysis on Image Reward's Performance and Properties
24
C.I
Average Scores of the Highest/Loowe Lowest Ranked Images
24
2 Recall/Filter the Best/Worst Image
25
D Comparison between Image and Other Reward Models
26
D.I Human Evaluation
26
D.2 Train Set Distribution
27
E
Implementation Details Related LDM Optimization Methods
27
F More Results of ReFL
28
F1 Demonstrations of ReFL
28
F2 ReFL compares other fine fine-tuning methods
28
Additional Results of Image Compared to Other Typical Image Scorers
28
Limitations
33
Broader Impact
33
Reproducibility
33
15

---

A Details on ImageReward's Comparison Data Annotation Pipeline
A.1 Prompt Selection
Training human preference RM requires diverse prompt distribution that could cover and represent users' authentic usage. DiffusionDB has 1.8M prompts which are far beyond the number we plan to annotate. To ensure the diversity and representativeness of topic distribution in selected prompts, we adopt the similar method introduced in (56) for selection. During the graph-based selection, every sample, which is prompt in our case, is represented by vector calculated by Sentence- -ERT [44] Then graph is constructed with represented samples vertices and every vertex is connected to k nearest neighbors, where & a syper-parameter the algorithm and k 150 is found to perform well. The distance between two vertices is the cosine similarity between vertex representations. With the graph constructed, the score is calculated for every vertex, which is related the number of neighbors that have not yet been selected. Vertex selection is based on calculated scores, which are calculated repeatedly after every selection until the required number is reached.
Note that the complexity of the algorithm is of the squared order of the number of samples. For computational feasibility efficiency, grouped all prompts for 100 with about 20k prompts set. We use the method to select 100 prompts among every set and get total of 10k prompts for annotation.
A.2 Annotation Management
We cooperate with a professional annotation company complete professional annotation. Our process of hiring annotators strictly complies with labor laws and other laws and regulations, and we pay annotators wages at legal market prices.
Before annotators are hired, they first learn the annotation documents and examples provided by researchers. Then, they are asked take test and we caleulate their agreement with researchers and the annotator ensemble. Those who get low agreement scores would not be employed for the annotation. To ensure quality, hire quality inspectors double-check each annotation, and those invalid ones will be assigned to other annotators for relabeling.
In the final list of annotating experts, 95.8% of experts have finished at least college-level education. Although a thousand readers have a thousand Hamlets, the rating and ranking of generated images can reach consensus, especially when associated with objective criteria and social ethics. We write and compile documents describing the labeling process and quality (Cf. Appendix B). which serve the standard for training annotators. For scoring and ranking, we design the criteria for giving different scores/copparioon and offer specific examples.
A.3 Human Annotation Design
Although an individual can easily identify his or her preference for a pair of images, a group of people can hardly reach consensual criteria over massive number of comparisons in pragmatic annotation. In this section, we discuss our efforts spanning months to design and build an effective yet simple-to-use pipeline for collecting human preference in text-to-image generation.
Prompt Annotation. Prompt annotation includes prompt categorization and problem identification. We adopt prompt category schema from Parti [63] and require our annotators to decide the category for each prompt. The category information helps to better understand problems and per-category features in the later investigation.
In addition, some prompts are problematic and need pre-annotation identification. For example, some are identified as ambiguous and unclear (e.g.. "a brand new medium". "low quality", etc). Others may contain different kinds of toxic content, such as pornographic, violent, and discriminatory words, although they have been filtered DiffusionDB processing. Therefore, we design several checkboxes concerning these latent issues for annotators in the pipeline (Cf. Appendix B
Text- Image Rating, Before diving into ranking model outputs, we also design an annotation stage for each text-image pair to identify its properties and potential problems. From an overall perspective, we take into account the following measurements: alignment, fidelity, and harmlessness.
16

---

Alignment: which requires generated images faithfully show accurate objects of accurate at- tributes, with relationships between objects and events described prompts being correct.
Fidelity: which concentrates on the quality of images, and especially whether objects in generated images are realistic, aesthetically pleasing, and with emrrr of the image itself.
Harmlessness: which means images should not have toxic, illegal, and biased content, or cause psychological discomfort.
The criteria correspond to other binary checkboxes dedicated to image problem identification and three seven-level quantitative measures concerning 1) Overall Rating, 2) Image-Text Alignment, and 3) Fidelity (Cf. Figure 8 (a)).
Image Ranking. After rating each text-image pair, annotators will finally come to the ranking stage, where they express their preference by ranking a series of generated images conditioned on a prompt from best to worst. The ranking generally follows the criteria mentioned in Image Rating.
However, it is common that sometimes these criteria contradict each other in the ranking given certain comparisons. We identify some common contradictions observed in the preliminary test, and specify the trade-offs one should adopt on our annotation document (Cf. Appendix B). For example, in comparison, if an image is more aligned to prompt but also more toxic, the less toxic one should outweigh it since we regard toxicity as a more unacceptable property.
Annotation System Design. Considering the criteria above, our annotation system consists of three stages: Prompt Annotation, Text-Image, and Image Ranking. The screenshots of our system are shown in Figure g The procedures for annotators to go through a prompt are as follows:
Label the checkboxes and enter the category for the text prompt.
2. Annotate each image one by one. Rate the image from aspects of alignment, fidelity, and overall satisfaction using a seven-point Likert scale. If the generated image has certain issues such as body problems or psychological discomfort, point them out.
3. Rank all images generated from the same prompt. There are 5 slots that be filled, the first slot corresponds to the best one among images, and the last slot placed for the worst one. Ties are allowed when two images hard to discriminate for which one is better, but one slot allows two images at most to enforce distinguishing different qualities.
A.4 Human Annotation Analysis
Among 10k prompts selected for annotation, after the expert annotation mentioned in Section 2.1 finally collected 8,878 pieces of valid prompts, which comprise a total of 136,892 compared pairs.
Prompt categories distribution. As we men- tioned before, have required the annotators to classify the prompt before scoring the images. According to the prompt classification standard of Parti (63), we divided all prompts into 12 cate- gories: Abstract, Animals, Artifacts, Arts, Food, Illustrations, Indoor Scenes, Outdoor Scenes, People, Plants, Vehicles, and World Knowledge. The distribution of the prompts in our annotation data is shown in Figure g9] As we can see, the distribution is diverse yet representative. Most prompts fall into common topics such as People (3,360), Arts (2,407), Outdoor Scenes (1,189), Artifacts (831), and Animals (413). Yet, rare categories such Plants, Illustrations, and World Knowledge are also considered the prompt selection.
Quodoor Scones
1.20%
Plants
4,55%
9.65%
Jota 27.11%
Othera
(..88%
ri 0.50%
1,40%
37.59%
Sndoor Scancs
Figure 9: Prompt distribution in annotation data, which comprises 12 categories and 8,878 prompts.
Average score distribution different prompt categories. We have scored the images in three dimensions including text-image alignment, fidelity, and overall satisfaction. The average scores of each category are shown in Figure 10 Across the three scoring aspects, scores for each category present roughly the same pattern. We find that generated images of the Abstract prompts get the lowest scores. We speculate that Stable Diffusion does not comprehend abstract and vague prompts well, which often lack the description of concrete objects. Besides, notice that more low-quality
17

---

Prompt
painting of an ocean with clouds and birds, day time, low depth field effect
Pleaso enter phrases from that you think are important but not reflected in generated image (separated by commas)
Overall Rating (1=worst, 7=best) O
Image-Tex Text Alignment
Fidelity (Image quality)
Does the image have any of the following issues?
Obvious 'repeatad generation' rasulting in unreality
Existence of body problem
Too blurry to see objects
Causes psychological discomfort
Output contains sexual content
Output contains violent content
Output contains content that defames certain groups
(a) Stage Text-Image Rating, Three seven-point Likert scales are rated for text-image alignment, fidelity, and overall quality. Several options are to be checked and filled to identify problems.
Prompt
pointing oceen with clouds and birds, day time, depth effect
Ranking outputs (1=best, 5=worst)
(b) Stage 2: Image Ranking, There are 4-9 generated images to be ranked by being dragged into 5 slots below that represent the different levels preference.
Figure 8: Screenshots of our annotation system. Annotators first annotate each text-image pair and then finish ranking all images conditioned on the prompt.
prompts exist in the Abstract category than in others, which may also affect the performance of the text-to-image generation. Images that get higher scores are in the categories of Plants, Outdoor Scenes, Indoor Scenes, and Plants, whose prompts are usually describing landscapes, non-living objects, and other common concrete things.
Problem distribution different prompt categories. addition to the scores, to understand common problems presented in generated images is of great importance. We have required the annotators identify seven problems of the image, including unrealisticness caused by repeated generation, body problems, fuzziness, toxicity, pornographic content, violence. We report the
18

---

(3360) Abstract (94)
Illusrations Animals (613)
Vehides (2.30)
(305) Indoor Scones (382)
(65)
World knowledge (44)
Artfacts (833)
Outdoor Scenes (1189)
5.5
6.0
5.95.9
5.35.
5.4-
1.33
5.3
5.4
5.8
5,75.7
Nwersge Soore
5.2
5.11
5.0
5.8.0500
5.15.2
5.2
5.0
5.15
4.9
5:15
5.15.1
5.15
5.4
5.6
5.45
5.3
5.35444
5.5
5.2
4.6
5.0
(a) Overal Salisfaction
Text image Alignment
(c) Fidelity
Figure 10: Average scores of each category. Each image is scored in three aspects of text-image alignment, fidelity, and overall satisfaction. For each category, calculated the average scores of corresponding images in these three aspects. Moreover, the average scores of all images are shown as the horizontal dotted line in the figure. The number in the legend shows the number of prompts in each category.
People (3360)
Arts (2400))
Illutration: (38)
Vehides (2.30)
Food (105)
Plants (G5)
Abstract
World Knowledge
Animals
Artifacts (831)
Indeor (182)
Scenes
5,0
2.0
1.8
Percentage Ss
1.5
21.14
21.0
1.2
14.2
2-
2.2.22
0.2
0.8
1.3
1.1
1.8
12.512.0
0.5
0.2
0.3
0.6
Repeated Generation
Body Problem
(c) Fuzzy Image
0.2
Porcentage %
0.7
0.15
0.52
1.
0.4
0.6
0.2
0.0
0.0
0.00
Toxic
(e) Pornographic
Figure 11: Problem frequency in each category. For each image, annotators are required to identify whether there are problems including unrealistic caused by repeated generation, body problems, fuzzy images, toxic, pornograhhcc. violent, or violating protected groups. For images in each category, we calculated the frequency of all these seven problems. Especially, no images are found violating protected groups, so omit this figure. Moreover, the frequency of these problems among all images shown as the horizontal dotted line.
frequency of each problem in each category in Figure 11 It shows that the most severe one lies in the body problem, whose average frequency among all prompt categories is 21. 14%. The problem appears most frequently in the categories of People and Arts. The Animals, Abstract, and Artifacts categories take second place. Body problems may indicate a lack of knowledge precise body and limb structures. And this may also explain why the People category get the lowest fidelity score.
The second severe problem is repeated generation with an average frequency of 3.16%. The problem mostly appears in the categories of Word Knowledge, People, Arts, and Vehicles. In contrast, we observe little of this problem occurring in the categories of Indoor Scenes, Food, Abstract, Outdoor Scenes, and Illustrations, which usually have loose quantity requirements.
Another important problem is fuzzy images, which mostly found in the category of Abstract, and then in the category of Animal and Arts. It may further imply that the text-to-image model may also perform poorly when encountering prompts that are too simple (like "a cat"), prompts that are unreal (like "an anthropomorphic duck in a blue shirt in the style of zootopia").
Besides the three problems that mentioned above, toxie, poroographic, and violent content is also found in some images due to related descriptions in their prompts (like "monster peering out of a cave, dark lighting, horror, realistic"). This indicates that the current text-to-image model cannot identify these problems in prompts and consequently cannot avoid them in a generation.
19

---

"Function" words distribution. When an- alyzing the prompts, we find interesting phenomenon that many prompts not only de- scribe the content and style but also contain some "function" words, like "8k" and "highly detailed", trying to improve the quality of gen- erated images. Therefore, we decide to un- derstand how the existence of these function phrases influences the performance of the text- to-image model. To study the question, we first fine-tuned token-clat classificaation model based on BERT, which can classify words or phrases in the prompt into three categories of Content, Style, and Function. For each prompt, we used classification model classify the words and phrases in the prompt, and then we calcu- lated the proportion of function phrases. We evenly divide the proportion from 0% to 100% into five buckets. Considering the huge number of prompts without function phrases, we assign them single group. The distribution of prompts is shown Figure 12 Most prompts do not contain any function phrases, and very few prompts contain more than 60% function words.
(a) Overall Salisfaction
(c) Image Fidelity
(b) Text maxt- Alignment
6000
5.8
5224
5000
5.6
Average Scoree
4000
Count
2000
5.2
1000
686
253
71
5.0
0.2-0.4 Proportion
Figure 12: We divided phrases prompts into three categories: content, style, and function. For each prompt, we calculated the proportion of function phrases, and we divided all prompts into six categories according to the proportion. The number of prompts and average scores in each category are marked.
Prompts distribution & average score.
Average score distribution of different proportions of "function" phrases. For each category, we calculate the average scores of text-image alignment, fidelity, and overall satisfaction again, and the result is shown in Figure [2 As it indicates, when the proportion of function phrases is low, the prompt itself mainly contains the description of concrete content, and the generated pictures get relatively low scores. As the proportion of function phrases increases, the three scores generally grow. The increasing trend reflects that the existence of proper function phrases does improve the text-image alignment, fidelity, and overall satisfaction of images to a certain extent.
Purcentage 1%
+
4.49
3.76
20
21.53
21.72
21.40
0.8
0.81
0.80
3.18
2.35
15-
14.86
12.3
0.6
2.
10
0.4
(a) Repeated Generation
(b) Body Problem
(c) Fuzzy Image
2.5
0.15
2.18
0.73
0.78
2.0-
0.66
Percentoge
1.5
...3333.
1.48
1.47
1.37
0,53
0,09
0.09
1.09
0.4-
0.41
0.05
0.42
0.2
0.0
(d) Toxic Problem
(e) Pornographic
(f) Violence Problem
Figure 13: Problem frequency in each category (divided by function phrases). Like Figure we calculated the frequency of all image problems in each category as well as the frequency among all generated images. The legend shows the six categories of prompts, divided by the proportion of function phrases, and the number in the legend shows there is how many prompts in each category.
Problem distribution different proportions of "function" words. The frequency of image problems is shown in Figure 13 The proportion of function words also influences the problem distributions. With the increase of function phrases, the frequency of the repeated generation problem shows trend of first increasing and then decreasing in the range of 0% to 80%, and then increasing again. As function words increase in prompts, they initially cause issues, then help, but ultimately, when dominating, they decrease the model's understanding and increase problems.
The body and fuzzy problems exhibit similar trends. the body problem, high proportion of function phrases can cause the main object to disappear, leading general decrease in frequency.
20

---

The fuzzy image problem, mainly tied to image quality, is less influenced by the model's 's understanding of prompts, frequency also declines as function phrases increase. Other issues like toxic, pornographic, and violent content mainly stem from the prompts themselves, so there is no strong correlation with the proportion of function phrases.
B
Annotation Document
You will be given a number of prompts and there are several generated images according to the prompt.
Your annotation requirement is to evaluate these outputs to ensure that they have high quality in terms of image-text alignment, fidelity, and harmlessness.
What we mean by "image-text alignment" is that the generated image should match the text entered by the user. Some examples of inconsistent image generation are:
The subject described the text does not appear the image generated, for example, cat dressed as Napoleon Bonaparte" generates an image without the word "cat".
The object properties generated in the image are different from the text description, for example, generating an image of "a little girl sitting in front of sewing machine" with many little girls is incorrect.
What mean by "fidelity" is that the generated image should be true the shape and characteristics of the object, and not generated haphazardly. Some examples low-fidelity images are:
"Spider-Mar" should only have two legs, generating an image with more legs incorrect.
"Unicorn" should only have one horn, generating an image with multiple horns is incorrect.
People eat noodles with utensils instead of grabbing them with their hands, generating an image of someone eating noodles with their hands is incorrect.
What we mean by "harmlessness" is that the generated images should not cause physical, psychologi- cal, or social harm to people; damage or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary for human well-being. Some examples of harmful images are:
Images that are pornographic, violent, prejudicial or even denigrating specific groups are harmful. Images that cause psychological discomfort when seen are harmful.
Evaluating the output of the model may involve making trade-offss between these criteria. These trade-offs will depend on the task. When making these trade-offs, use the following guidelines to help choose between outputs.
1. For most tasks, fidelity and harmlessness are more important than image-text alignment. So, in most cases, the image having a higher fidelity and harmlessness rated higher than an output that is more image-text alignment.
2. However, if: one output image clearly matches the text better than the other; is only slightly lacking the requirements of truthfulness and harmlessness; the content does not fall into "sensitive areas" (e.g.), the body of the person generating it cannot go wrong, etc.); then the more consistent results are rated higher.
3. When selecting outputs that are having equal image-text alignment but are harmful in different ways, then ask: Which output is most likely to cause harm to the users (the person most affected by the task the real world), then the corresponding output should ranked lower. If this is not clear from the task, then mark these outputs as tied.
4. There is kind of low fidelity due to repeated generation, which we consider to be less low fidelity, but if there are more realistic images with about the same degree of image-text alignment, the images with more fidelity are at least notch higher.
21

---

Guidelines for deciding boundary cases: Which generated images would you prefer to receive from AI painters?
Ultimately, making these trade-offs can be challenging, and you should use your best judgment. We give three specific examples of trade-offs in Figure 14
Promptt:
Prompt:
Prompe
Bezutfil portal ol tomake mocel hakil hodar with intricsse color crushed flvd serylo on the ahouldors, Headdrioss
High qua iy nooclass cull portakk of car, dressed es Napolson Bonaperts, very desa ldd, 1890, catross
Micholnnglee statue Davic oriving a blue Roils Royce comneritae ablo, right Cacsara Pulaco Vogao
Output
Gannrating imagea numan body with amora importa papt dutail such the lace and ayes would obbicuslyy cussse paychological disco miror in the word. detail of color crusieed oodeedddd flud rininn acry the anoukdass about the shoulcers, but the gorerated effect eyis image likely couse
Outout la unnall iskl the car's head looks uni slund, Howova, output is clearty a bettor match than outout proloood.
Both curpul pars and laod match only "Michdarged aagteo' statue of Dipio" and "A right at Cassurs ico Las
subject "cal". wen
Vogus background but "Crving Buth Rayes
"he
gane uio potentia sum inforaaion about status of Dovid, such as the fact that the cctuila of the stauue are uniorgi-ing. Since than these ourput more
Figure 14: Examples of how to trade-off when selecting images.
For each input text, the annotation will consist of the following three parts.
1. Annotate the input text. You will describe the input with several characteristics, including whether the input text is harmful, etc.
2. Annotate AI model output. For each output of an Al model, you will annotate the output images based on several different dimensional questions, including giving score of 1-7, determining whether the image contains pornographic/violent content, etc.
3. Rank AI model output. You will rank the output images from best to worst, including ties.
B.1 Annotate the input text
Based on the input text, annotate the following questions.
Label (drop-down menu)
What is the main object of this text description? Choose the label that seems most appropri-
ate.
The user's intent is unclear(yes/no)
If it is really unclear what the user is asking the Al system to do, then mark "yes" here. This may be because the input text is really confusing, or there seems be no instruction. If the input text is a little unclear, but you can still understand what the text means, leave this box unchecked.
One way think about it is: If you are given this command, can you understand what the user is asking and give a reasonable corresponding image? If the answer is "no", then mark the box "yes".
(Multiple choice) Does the input text have any of the following issues?
Contains sexual content
Contains violent content
Contains content that defames certain groups: By" defame we mean unfair criticism. Does the output portray members of protected class in a derogatory manner, or superior/in- ferior others? See Appendix B for a list of protected classes (including race, religion, etc.).
22

---

Contains Personally Identifiable Information (PII): Does this input text contain information that can be used to identify person?
There are no such problems
B.2 Annotate generated images
For each Al model output, please give the annotation of the following questions. Overall Rating (Likert scale 1-7)
How good is the generated image of this text (i.e., has an image-text alignment, high fidelity, and harmlessness.)? How happy would you be you gave an Al assistant this text and received this image result?
1 Outpptt is terrible, it is completely useless and/or can cause real damage.
7 The output is almost perfect and I cannot think of way make it better.
Image-text alignment (Likert scale 1-7)
How well do the generated images match the given text?
= The picture and the text do not match all, it does not contain the object described in the text at all, and the picture does not match at all.
7 = The image and text match almost perfectly, including all objects described in the text, and all object properties are also consistent with the text.
Fidelity (Likert scale 1-7)
The output image should be true to the shape and characteristics that the object should have and will be generated haphazardly. Some counter-exand exampess of low-fidelity images.
Spider-Man" should only have two legs, but generating an image with more legs is incorrect.
"Unicorn" should only have one horn, but generating an image with multiple horns is incorrect.
People eat noodles with utensils instead of grabbing them with their hands, but generat- ing an image of someone eating noodles with their hands is incorrect.
There is obvious error in the above image, and seriously affects the degree of beauty or coordination of the picture
4 There are obvious mistakes but no harm, the tires in the picture below are obviously separated from the car body is not right, but the overall picture coordinated.
7 = No image errors, beautiful and coordinated pictures
(Multiple choice) Does the image have any of the following issues?
Obvious 'repeated generation' resulting in unreality: If the output image repeatedly generates certain images, such as "a little girl's buttocks directly under another little girl's head" or "a horse's buttocks with new body continues be attached new body", then this is problem.
Existence of body problems: including apparently incoherent abnormally shaped body parts humans and animals, etc.
Too blurred to see objects
Causes psychological discomfort: If you feel discomfort when you see this picture, you can mark this question and answer it according to how you feel.
Output contains sexual content
The output contains violent content
23

---

Q: Many of the prompts use words from Western culture and various artists' styles, what should I do Output contains content that defames certain groups: By "defame", we mean unfair crit- icism. Does the output portray members of protected class in a demeaning way, or as superior/inferior to others? A list of classes are protected (including race, color, religion, national origin, sex, age, physical or mental disability, etc.).
There are no such problems
B.3 Rank generated images
Given all Al model output images together, rank them from good to bad, including relationships. The best output the one that is the most image-text alignment, high fidelity, and harmless for the given text. If the images are close in quality, do not hesitate to mark them as tied. Attention: Your ranking order must be consistent with the previous scoring for each picture, otherwise it cannot be submitted.
B.4 Frequently asked questions
Should the exposure of some special primary and secondary sexual characteristics be classified as sexual content? Nudity of breasts, for example, is common in some sculptures and paintings and may need to be generated, but is also generally considered inappropriate, especially in the Chinese cultural environment.
Q:
A: the text requires artistic creation then nudity is allowed, in the case of real people it is considered violation of the rules for sex-related content.
if I do not understand?
A: Although the Chinese results of machine translation are attached, you can directly use the search engine to search the image for reference when you encounter words you do not understand. If you think some of the vocabulary semantics do not understand does not affect the scoring, you can also keep part of the unknown semantics to do the scoring of the image as a whole.
Q: The input contains command, but it is confusing/obscure What should do?
A: You may encounter the following tasks.
The task seems confusing
You do not feel you know exactly what it means to do this task well
There are two possible plausible explanations for this task
In these cases, we again encourage you to use your best judgment to infer the intent of the user submitting this text and judge the output accordingly.
Q:
When should skip mission?
A: There is an option to skip a task
You are uncomfortable with the task, e.g. it involves gore, horror, pornography, etc.
You do not think you do the task well, e.g. it requires expertise you do not have, or is very confusing, or requires specific life experience, etc. (Note: there is limit to the number of skips)
C More Analysis on ImageReward's Performance and Properties
C.1 Average Scores of the Highest/Lowest Ranked Images
When evaluating several images, we are also concerned about which one is the best worst, and whether the preference model can pick. Figure 15 shows scores of the highest-ranked and lowest- ranked images picked by different methods. For image fidelity, the Aesthetic score performs better than CLIP/BLIP score, which is trivial because image fidelity is more about aesthetics. ImageReward still performs quite better than the Aesthetic score, indicating that human preference for image fidelity is far more complex than aesthetics. It's interesting that the lowest-ranker images' average score
24

---

of Aesthetic is lower than that of CLIP/BLIP scores. which may be because that images with too low quality may affect human judgment about whether the object drawn corresponds to certain text. Overall, our IeageReward model performs the best. Among the highest-ranked images, the average score of images picked by our models gets the highest score, while the average score among lowest-ranked images gets the lowest score. Our ImageReward model maximizes the difference between superior and inferior images.
The lighest-ranked Image
The lowest-ranked image
4.85
5.31
5.05
5,54
5.1 5.47
Ours
BLIP Score
Aesthetic
CLIP Score
4.0
4.5
5.0
5.5
4.0
4.5
5.0
5.5
4.0
4.5
5.0
5.5
Overall Satisfaction
Text-image Alignment
Image Fidelity
Figure 15: Average scores for the highest and lowest ranked images of each model. Models are expected to rank the image with the highest human rating the top rank and the one with the lowest rating last. In each figure, two dashed lines denote the mean scores of the four models' scoring the last images and the top images, respectively.
C.2 Recall/Filter the Best/Worst Image
To further evaluate the models' ability select the best image while filtering out the worst image, we collect 371 other prompts with 8 images per prompt and require annotators to select the best and worst among 8 images. Then use different methods to rank these 8 images and calculate the rate they recall the best one or filter the worst one human annotated when selecting 1/2/4 images. These statistics also shown in Table Figure 16 shows the bucket distribution of the best worst image humans selected when being ranked by different methods, our model significantly has the largest proportion to pick precisely and the minimum ratio rank incorrectly.
Ours
BLIP Score
Aesthetic
CLP Score
40
39.6
40
40
40
Best %9
30
30-
30.7
30:
30.7
30
27.2
23.5
23.2
20
20
19.0
20
20
36.4
10
10.8
10
10288
10
11.
1.29..2
5.9
7.5
10
13.2
8.4
1.3
2.4
0.5
2.7
Bucket
Worst 59
40-
32.1
29.6
22,9
22.4
22.
20
0.5
1.9
5:1
12.9
20
5.1
7.3
20
3.5 5.7
4.6
9.7
20
1.9
4.9
8.1
8.6
10.0
14.3
2
1
4 5
Bucket
Bucket
Bucket
Bucket
Figure 16: Bucket distribution the best and the worst images in the human annotation. We collect prompts(excep) those for training) each with 8 images, among which annotators pick the best/worst one. Then different methods are applied to rank these images, where buckets 1-8 correspond to ranks 1-8. The figure shows the distribution of human-annotater best/worst images through these model-ranked buckets.
Interpolation Analysis Between Different Scorers When humans evaluate images, the selection process contains multiple elements such as fidelity, image-text alignment, harmlessness, etc. We are curious about the performance of combination of different models. We test interpolation among
25

---

Table Results ImageReward and other reward models on human preference evaluation. Prefer- ence accuracy is calculated on the test set of 466 prompts (6,399 comparison pairs in total); Recall and Filter's scores are evaluated on another test set of 371 prompts with 8 images per prompt. All scores are averaged per prompt.
Preference Acc.
Recall @2
Filter @2
Model
@ 1
@4
@ 1
@44
HPS
60.79
39.89
58.76
83.29
47.17
65.50
84.10
PickScore
62.78
38.27
63.07
84.10
46,36
65.77
84.91
IeageReward (Ours)
65.14
39.62
63.07
90.84
49.06
70.89
88.95
CLIP score, Aesthetic score, and our ImageReward model, their accuracy on the test set can seen
in Figure 17
Ours mixed with CLIP
CLIP mixed with Aesthetic
Aesthetic mixed with Ours
65.4
Accuracy %
65
65
65
60
60
58.9
60
55
55
55
50
50
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 17: Accuracy of interpolation between different models. Interpolation between CLIP score and Aesthetic score can get higher accuracy, but still much lower than ImageReward. When ImageReward interpolated with CLIP score or Aesthetic score, it reaches only bit better performance.
D Comparison between ImageReward and Other Reward Models
Besides ImageReward, other reward models aimed alignment with human preferences have also emerged recently, such as HPS [61] and PickScore (21). To facilitate a comprehensive evaluation of these reward models, we have undertaken series of analyses, yielding the following results.
Table 6: Comparison between different reward models. "# Win" is counted from all comparisons in the human evaluation, while "WinRate" is calculated from comparisons against the baseline.
Real User Prompts
Multi-task Benchmark 40
Methods
Human Eval.
Image
Human Eval.
Image
Win
WinRate
Reward
Win
WinRate
Reward
SD v1.4 (baseline)
399
0.1058
459
0.1859
HPS
572
67.24
0.6274
662
69.15
0.6788
Bo64
PickScore
620
72.16
0.7033
773
72.73
0.7579
ImageReward (Ours)
676
73.33
1.3374
824
74.42
1.4098
HPS
428
52.86
0.4749
426
52.86
0.4646
ReFL
PickScore
472
56.91
0,4618
454
55.09
0.4908
lmageReward (Ours)
512
58.38
0.6072
492
58.67
0.4822
D.1 Human Evaluation
Human evaluation requires annotators to rank all the images generated from the same prompt in different datasets, and the comparisons were analyzed in [3d where "Bo64" means "the Best of 64 (Images)", i.e., every image was assigned the highest reward by the corresponding reward model out pool of 64 images, and "ReFI." means that every image was generated using a model tuned through ReFL with the respective reward model.
26

---

Perplexity 5, Num of Neighbors 15
ImageReward/Train
300
Pick-a-Pic/Train
200
100
-100
-20
-300
-300
-200
-100
100
200
300
Figure 18: Prompts in train sets PickScore and ImageReward visualized by t-SNE.
D.2 Train Set Distribution
We employed (- to visualize the prompt distribution of training of PickScore and ImageReward. Specifically, we initiated the process by randomly selecting prompts from PickScore's training set, consisting of 583,747 comparisons, match the size of our own training set prompts (8,000). Subsequently, we utilized CLIP to extract the features from these two prompt sets. Finally, we used t-SNE with several parameter configs to visualize the feature vectors as two-dimensional scatter plots, which can be seen in Figure 18
We observed that training set is slightly more evenly distributed compared the training set of PickScore.
E Implementation Details of Related LDM Optimization Methods
Optimization methods that are now available that use human feedback to fine-tune LDM can be basically divided into categories, one is acquiring new datasets [3), and the other is changing the coefficients of loss function (23. We have selected three methods as baseline models. To have fair comparison, all methods use half- crefipreecsion on 8 40GB NVIDIA A100 GPUs, keeping training settings the same (such as a learning rate of le-5). If a pre-treeiii pre-trained dataset is required, all fine-tuning methods use the same subset of LAION-AES.
Dataset Filtering. 66l use a reward model to filter the dataset. Specifically, they use the reward model to score multiple images for the same prompt, and then select the images with the highest or lowest scores. Those images with the highest scores consist of new dataset, and those with the lowest scores are paired with prompt prefix (they choose "Weird image.") to indicate that the image relatively non-prefeen preffrrerddd. These two newly constructed datasets of model- generrted images are then used to fine-tune the LDM, together with the pre-training dataset. After constructing the dataset, there is no longer different from the normal fine-tuning process (the reward model is not used again). We replicated this fine-tuning process with RewaeReward as described in [6]] Specifically, the constructed dataset contains 20,000 pre-training samples and 20,000 filtered samples from DiffusionDB (10,000 preferred samples and 10,000 non-preferred samples).
27

---

Reward Weighted. [23] also added a dataset of model- -gedeerte images to the pre-trained dataset, but they used the reward model for the coefficients of the loss function instead of using the reward model when constructing the dataset. Specifically, their loss function is shown in 4
() [2,s)) (ro(z, 2) (=|=))) + (x.)))) (-og((p
(4)
where denotes parameters of pre-trained LDM, & denotes parameters the reward model, and penalty parameter. During fine-tuning, the dataset includes 20,000 pre-training samples and 20,000 generated samples (2,000 prompts from DiffusionDB, 10 generated images per prompt). [23),
the reward is constrained within the range [0, 11, while model follows approximately normal distribution with a mean of and variance of Therefore, in order to replicate their methodology, need to map the scores to the range [0. 1]. We adopted the approach of min-max normalization, selecting the maximum and minimum scores from the sample and calculating the corresponding mapping values. Additionally, we set the penalty coefficient 0.5.
RAFT. [13 proposes fine-tuning method by constructing dataset of generated images with higher rewards. The process consists of three steps: data collection of generated images, data ranking, and model fine-tuning, which be repeatedly performed. In every iteration, generate 100,000 images (10,000 prompts, 10 images per prompt) and use ImageReward to rank generated images, getting 10,000 selected images to fine-tune LDM.
F
More Results of ReFL
F1
Demonstrations of ReFL
Figure 19
shows the insight of ImageReward score during denoising.
1.0
(77.50%, 0.80)
ImageReward
-0.5
Original Reward
0.8
Spearman
Trained Reward
-1.0
0.6
0.4
-1.5
0.2
-2.0
77.50%
20
40
60
80
100
0
20
40
60
80
100
Denoising Step Rate / %
Denoising Step Rate / %
Figure 19: (Left) Correlation between Spearman p and denoising step rate. (Right) Correlation between ImageReward score and denoising step rate.
F.2 ReFL compares other fine-tuning methods
Figure 20 shows the comparison be- tween ReFL and other fine-tuning methods. ReFL gets the highest win rate compared any other method.
Win
Tie
Lose
50%
Ours v.s.
RAFT (iteration=3)
Ours v.s. RAFT (Iteration=2)
More qualitative examples of ReFL are provided in Figure 21
22
Ours v.S. Reward Weighted
G Additional Results of ImageReward Compared to Other Typical Image Scorers
RAFT literation=1)
v.s. Dataset Filtering
0%
20%
40%
60%
80%
100%
More qualitative examples of ImageRe- ward can be seen in Figure 23 26
Figure 20: Win rate between different fine-tuning methods.
28

---

Dataset Filtering
Reward Weighted
ReFL (Ours)
Original
RAFT
Spaceship waves, 3d, unreal engine octane render, detailed, cinematografic. cinema d. nvidia ray tracing graphics,
artstation, greg rutkowski style, style rene magrine.
Medieval old king, rpg hearthstone, fantasy, elegant, highly desailed, digital painting, artstation, concept art, sharp focus, illustration, global illumination.
Epic painting portrait photography beautiful goddess wearing white silk blouse with golden glow background, drawing, Intricate. symmetrical front, by amy leibowirz, wlop.
Small red wooden cottage by the lake, lanterns on the porch, smoke coming out of the chimney, dusk, birch trees, tranquility, two swans swimming on the lake.
Portrait a man who looks like bulldog, oil on canvos by william sidney mont, 1.8 8.8.3 3, trending on art station.
Figure 21: More examples for comparing fine-tuning methods.
29

---

Dataset Filtering
Reward Weighted
ReFL (Ours)
Original
RAFT
serene landscape with singular building near lake sumset, anime style, Sk, low saturation, high quafity, high desail, cartoon.
realistic illustration, young man with gray skin, short white hair, intricate, with dark clothes, elegant, highdy detailed, digital painting, aristation.
DAW 00
Dsir photo still house fire under water the bottom of the occan, 85 F.
Portrait flower goddess, full body, cute detailed face, bay leafes, tendrils, intricate, elegant, highly detuiled, digital painting, aristation, concept art, smooth, sharp focus, illustration.
Bat eagle lizard hybrid.
Figure 22: More examples for comparing fine-tuning methods.
30

---

Random
Aesheric
CLIP Score
BLII Soore
Ours
An anthopomorphic fox man wearing long walking glacier, hands chameter illustration by Aaron Miller, Greg Rutkowaki, thomas kinkade, Howard Pyle.
A portrait puinting of rachel lane / sabrina perditu weeks meadams de boer hyhhid oil painting, gentle es prpession, smiling, elegani clothing, scenic backgnound.
Hyperrealimm close mythhlogiccal portrait bcautiful madiieval woman's shatteree face portially made of coma color flowers style of classicism using fibonacci golden ralio.
Figure 23: Qualitative comparison with previous typical methods. Each method selects the top images based on corresponding scores/rewards. Prompts are sampled from DiffusionDB except for annotated dataset, which has more than 64 generated images be picked.
Random
Acsthetic
CLIP Score
BI TP Score
Ours
humunodd german shepherd
Beautiful dark landscape, an netronaut, in the style heeple and Mike Winkelmam,, intricate, spic lighting, cinematic composition, hyper realistic, resolution.
dressed
Key visual portrait anthropomophic anthro wolf fursona, in jacket, with hambooee eyes, oflicial modem anime
sitting, and watching a soccerr mutch in his house television, be has his knee and did, hanson, alexi mitsev.
Figure 24: Qualitative comparison ImageReward to typical image scoring methods.
31

---

Random
Aeshetic
CLIP Score
BLII Score
Ours
Pink hand, weuring cyberponk intricate stroetwear, beautifull detailed portrait, intricate complexity, ilya kuvshinov, cell shoded, kk, concept wlop, kuvshinov.
Hyperrealism close-up my thologicaal portraii of huge number lavender flowers merged with female, turquoise palette pole skin, wearing fuchsia robe, style classicism.
A young smiling, long shiny breeze brown medium skin tone, light cute freckles, smiling softly, wearing clothing.
face, green eyes,
Figure 25: Qualitative comparison ImageReward typical image scoring methods.
Randem
Acstheric
CL.P Soore
BLIP Score
Ours
Highly detailed portrait woman long hairs, stephen bliss, unreal engine. fantasy art by greg nurkowski.
Sculpture flame, portrait, female, future, rorch, lire, harper's hazzar, vogue, fashion magazine, intricate.
wlop render highly detailed besutiful mystic portrait of phantom priestest Yemale ape with whirling golaxy around.
Figure 26: Qualitative comparison of ImageReward typical image scoring methods.
32

---

H Limitations
In this section, we discuss some limitations we realize during the development of ImageReward.
Annotation scale, diversity, and quality. Although our annotation data has reached up about 9k prompts and 137K pairs of expert comparisons, the larger scale of the annotation dataset is still needed for better RM training. In addition, our current prompts are all sampled from DiffusionDB, which is an abundant collection of human real use but still exists some bias. Despite these prompts may close many real cases, biases exist since the real application when people use the rext-to-image model are far beyond trying strange prompts. It's worth exploring more diverse prompts distribution to meet the more abundant need of humans. Last but not least, our annotation uses a single-person annotation plus quality control strategy for each prompt annotation, but multi person fitting annotation may achieve better annotation consistency and is worth trying in the future.
RM training techniques. As we mentioned in Section 2.2 overfitting dose affects the RM training, and fixing part of transformer layers helps lot. Nevertheless, we speculate that more advanced techniques (e.g., parameter- effficieteef tuning (2 25 32)) could be helpful for the problem. On the other hand, since BLIP improves over CLIP substantially in ImageReward training, we also expect stronger and larger text-image backbone model may contribute additional gains.
Using RM to improve generative models. Though we have proposed ReFl. as an effective method to utilize human preference scorers' feedback to optimize LDMs, it remains an approximation of original RLHF algorithms and could be improved fundamentally. It is necessary develop corresponding unbiased and efficient feedback learning algorithms with solid theory groundings to allow better human alignment.
Broader Impact
The aim of this paper is to introduce human preference feedback to improve text-to-image generation, which will help image generation to better match the needs of human life and to conform to social norms. tine-tuning the model with human feedback helps to avoid researchers from over-relying on various types of data with copyright issues for training, and can instead directly improve performance with reward model feedback. A downside is that the preferences of single reward model representative of the multiplicity of human aesthetics, and we can address this by training a variety of reward models and limiting the use of individual reward models. We believe that these benefits outweigh the drawbacks.
J Reproducibility
We have made substantial efforts to guarantee the reproducibility our assessments. The code and detailed information for the ImageReward model and ReFL. algorithm are openly accessible in our repository (Cf. Abstract). This availability covers the entire training and evaluation processes.
Training. For specifics regarding the objective function and dataset of ImageReward, please refer to the hyperparameters and cluster configurations in Section 2.2 Detailed information concerning the model architecture, hyperparameter settings, and experimental setups can be found in Section 4.
Evaluation. We have organized all evaluations, including text-to-image model ranking and human preference prediction results of ImageReward, into bash scripts that can be executed with a single command in the code repository. Further details regarding ImageReward as a metric can found in while details regarding ImageReward as a reward model are provided in Section 4.1
Section 2.3
ReFL. The ReFL. algorithm has also been organized into mne-command-to-ru bash scripts in our code repository. Detailed insights into ReFL. can be located in Section 4.2
33


Reviews:
[{"review_id": "XmwLheaWaq", "reviewer": "Reviewer_kP49", "summary": "This paper aims to improve text-to-image (T2I) from human preference feedback. They first collect a human rating dataset to train their human preference model, ImageReward. With the reward model, they further optimize a pre-trained T2I via the proposed Reward Feedback Learning (ReFL). The experimental results indicate that their ImageReward is more robust than the widely-used CLIP-Score, and ReFL-optimized T2I also performs better than baselines.", "questions": "Please see the Weakness", "limitations": "Since their pipeline relies on the collected labels, the human annotation can contain ethnic issues, which they also discuss in Appendix B.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ This paper is well-written and easy to follow.\n+ The collected dataset is valuable for the V+L community as well as the trained ImageReward, which can help various visual generation tasks (not only T2I).\n+ The proposed ReFL pipeline can keep improving the T2I model. Both automatic metrics and human evaluation support the superior performance of their framework. \n+ They provide lots of qualitative examples and detailed discussion in the supplementary.", "weaknesses": "+ The novelty can be an issue since the human preference-trained reward model and feedback learning are already introduced in large language modeling (LLM). It looks like they just apply the same pipeline from LLM to T2I.\n+ It is not easy to collect large-scale human preferences for reward model training. Is there a more efficient way to build ImageReward instead of fully relying on human annotations?\n+ A detailed analysis of ImageReward should be considered. For example, how many human preference pairs can lead to how well ImageReward and then how well the final optimized T2I model is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments, we will explain your concerns point by point.\n\n### Weakness 1: Novelty of ImageReward & ReFL\n\nDespite drawing inspirations from ChatGPT, our work introduces several crucial novelties for T2I scenarios:\n1. We are pioneering in the introduction of RLHF to the Text-to-Image (T2I) domain, making the incorporation of human preference novel within T2I.\n2. ImageReward serves as a pioneering reward model that surpasses previously popular models like CLIP and Aesthetic in its superior understanding of human preference. Furthermore, ImageReward proves to be a better T2I evaluation metric in alignment with human preferences than FID and CLIPScore.\n3. We have released the earliest human preference T2I dataset and corresponding annotating documents, consisting of 137k pairs of expert comparisons.\n4. Our ReFL algorithm represents the first instance of directly fine-tuning Latent Diffusion Models (LDMs) using human preference gradients, achieving superior results compared to previous algorithms. \n\nIt's important to note that we distinguish ourselves from RLHF for Large Language Models (LLMs) in the following significant ways:\n1. T2I faces distinctive challenges (aligning models with human preference), requiring careful analysis, as discussed in our introduction. The solutions to these challenges diverge from those in LLMs.\n2. ImageReward operates across both image and text modalities, unlike LLMs that exclusively utilize language models. Our training process involves intricate parameter tuning and strategies, not amenable to casual training.\n3. Our ReFL algorithm is distinctly founded on analyses of the diffusion model, differing substantially from Transformer-based LLMs. ReFL is also not an RL-based algorithm as those for RLHF on LLMs.\n\n### Weakness 2: More efficient ways for human preference annotation collecting\n\nThis is an excellent suggestion, as we wholeheartedly agree that collecting a large-scale human-annotated dataset can be challenging. To address this data collection challenge, we think there might be two potential approaches that are worth exploring: \n1. With the advancement of Visual Large Language Models (LLMs) such as MiniGPT-4, LLaVA, VisualGLM, etc., it might be possible to leverage Visual LLMs for data annotation. However, the effectiveness of this approach remains unverified and requires further research and understanding. Additionally, it might involve interactions between different LLMs. \n2. On the other hand, another promising approach is to deploy the T2I model as a usable product, similar to Midjourney, where human preference data can be naturally collected during users' interactions with the system.\n\nNevertheless, building expert annotated dataset and training high-quality image RMs at first is crucial for identifying problems in T2I generation and calibrating automatically created preference data, as which could be noisy and inaccurate.\n\n### Weakness 3: Detailed analysis of ImageReward \n\n> A detailed analysis of ImageReward should be considered. For example, how many human preference pairs can lead to how well ImageReward and then how well the final optimized T2I model is.\n\nThank you for your suggestion! In fact, we have provided detailed training specifics of ImageReward in section 4.1, along with result analysis and an ablation study. For instance, the impact of the number of preference pairs, as you mentioned, is addressed in line 245-248 (Ablation Study: Training dataset size), and the numerical results are presented in Table 2b.\n\n### Ethics Concern\n\nOur work is built upon the foundation of academic research in the field of Text-to-Image (T2I) and involves the exploration of human feedback. Throughout this process, we have considered and taken measures to address potential ethical concerns associated with T2I. Our aim has been to reduce and mitigate existing ethical challenges, rather than amplify them. Specifically, we have taken the following steps to contribute to the reduction of ethical concerns in T2I:\n1. Starting with the data annotation phase, we have prioritized ethical considerations (details can be found in Appendix B Annotation Document):\n   - We required annotators to identify ethical issues present in generated images, including but not limited to toxicity, pornographic content, or violence.\n   - During the scoring and ranking phase, annotators were instructed to assess the harmlessness of images, giving poorer evaluations to images with harmful content.\n2. In Appendix A, we analyzed various ethical concerns highlighted by annotators in the labeled dataset, conducting both qualitative and quantitative analyses (see Appendix A.4). These insights can serve as a basis for future research efforts. We strictly adhered to human rights and legal considerations during annotation (see Appendix A.2).\n3. Our ImageReward model, during experimental evaluations, demonstrates improved handling of toxic content and related issues, thereby better avoiding ethical problems present in baseline models.\n4. The T2I models obtained through our ReFL algorithm are more adept at addressing pre-existing ethical issues compared to models before fine-tuning.\nWe have made substantial efforts to address ethical considerations throughout our research and have sought to contribute positively to the field's ethical discourse by actively mitigating potential issues.\n"}, {"review_id": "rOwsFnk2BO", "reviewer": "Reviewer_7vWQ", "summary": "The paper introduces ImageReward, a new dataset of human preference over generated images given a text prompt. Human preference for images is rated across three dimensionalities: text alignment, image fidelity, and harmlessness. Using the dataset, they train a reward model to score the generated image and text prompt pair. The reward model consists of a small trainable MLP head over BLIP text-image features. The paper further proposes a baseline method of fine-tuning the generative diffusion model using the score model to increase the generated image alignment with human preferences. The fine-tuning loss is a weighted sum of standard diffusion loss and the one from the scoring model (only at the predicted images at lower timesteps of diffusion).", "questions": "Minor point: some grammatical issue in line 290, 302 and 360", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The annotated dataset of human preference is one of the first datasets of this kind and scale. This will help in further research in both text-to-image model evaluation and improving the generations with higher human preference. \n   \nBoth the reward scoring model and fine-tuning method based on the score model are shown to work on par of better than existing baselines.  \n\nThe paper consists of extensive analysis and details regarding the dataset annotation, scoring model, and its comparison to recent methods. \n", "weaknesses": "1. One limitation of the dataset might be that it becomes less relevant as the generative models improve. Given that the dataset only consists of generated images with their corresponding text prompt and not a plausible ground truth image for the prompt with the best possible human preference score. \n\n2. Only fine-tuning the model on 0-10 timesteps with the scoring model doesn't seem optimal. Specifically, in cases of object omission, the layout has already been decided in the initial stages of diffusion; thus, the reward score guidance at later stages might not be effective. Is there any ablation or analysis regarding what metrics among fidelity and text alignment improve the most? \n\n3. In Eq2, phi is implemented as a ReLU function, as mentioned in line 260. Probably this should be ReLU over the negative of the score function. Because the higher the score, the better. Or is my understanding incorrect? \n\n4. It would be great to expand on the evaluation setup and metrics, which are sometimes not very clear.\n\n    (a) In line 122, are the 100 real user test prompts different than the ImageReward dataset used to train the scoring model? Similarly, 466 and 371 prompts in Table 3. \n\n    (b) How is the \"filter\" evaluation metric calculated in Table 3? Does this calculate the number of times the model didn't select the worst image in top-k?\n    \n    (c) In Table 4, it's unclear how the evaluation numbers are reported. Does it denote the #winrate for each method out of total N samples (N being the sum of the column), or is it a binary comparison of each method vs the baseline? How many generated images per prompt over the 466/77 prompts were used for the evaluation? \n\n    (d) Some of the baseline, e.g., reward weighted fine-tuning method, performs worse than the baseline in Table 4. Is there any analysis regarding that? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions!\n\n### Weakness 1: Usefulness of the ImageReward dataset\n\nThank you for your suggestion. However, there might be some misunderstandings about the function of the dataset and how we improve T2I models upon them:\n1. RMs’ model-agnostic generalization: Regarding the relevance of our dataset after T2I models become improved, we want to clarify that we do not directly use annotated data for fine-tuning a T2I model. Instead, we first train Reward Models (RMs, e.g., our ImageReward), on the annotated data. Once ImageReward learns the general human preference, we then use it to generate rewards for fine-tuning the Latent Diffusion Model (LDM). As a result, ImageReward can help any T2I models to improve human preferences as long as they are imperfect. We also extensively experimented and conducted ablation studies in the paper to show that ImageReward learns general human preferences and has good model-agnostic generalization capabilities (Section 4.1, Table 2-3; Appendix C, Figure 15-17; Section 2.3, Table 1). \n2. Effect of groundtruth image: In response to the mention of ground truth images, during the preparation of the annotated data, we specifically used the clip-retrieval method to collect a set of real images highly corresponding to text prompts in the ImageReward dataset. Each prompt had generated images from the DiffusionDB as well as images retrieved from the LAION-5B dataset using the clip model. However, during the human annotation process, we observed that many of these LAION-5B real images did not consistently align with human preferences, and including this set of real images in the training data did not improve ImageReward's performance. \n\nNevertheless, we believe that future work can explore more diverse sources and qualities of the annotated images to address the issue. We appreciate your feedback and will consider these points for further improvements in our research.\n\n### Weakness 2: ReFL’s effect on layout, fidelity and text alignment\n\nThank you very much for your feedback. We would like to clarify that it is not accurate to say that we only perform fine-tuning on 0-10 timesteps. More precisely, we conduct fine-tuning on the last 25% timesteps of the denoising process. Admittedly, ReFL may have a larger impact on details rather than layouts of generated images, but it does help improve layout. As the multiple denoising steps share the same Unet model, optimizing the Unet parameters in later steps can also impact the denoising in earlier steps, which majorly influence the layout generation. \n\nIn addition to Table 4, we conducted a supplementary human evaluation experiment where five annotators provided scores based on alignment, fidelity, and overall quality dimensions. The results revealed that ReFL outperformed the baseline scores by 0.10 in alignment, 0.29 in fidelity, and 0.14 in overall quality. This indicates that ReFL significantly improves fidelity (i.e., details) but also has a positive impact on alignment (i.e., layout).\n\n|                    | alignment | fidelity | overall quality |\n| ------------------ | --------- | -------- | --------------- |\n| SD v1.4 (baseline) | 5.31      | 5.41     | 5.12            |\n| ReFL (Ours)        | 5.41      | 5.70    | 5.26            |\n| Improvement        | +0.10     | +0.29    | +0.14           |\n\nAdditionally, it's worth noting that in the practical use of LDM, switching random seeds is a dominant method to produce various layouts. Therefore, a probably more practical way to find a better layout is to switch random seeds and select the best one using ImageReward scoring, rather than using ReFL.\n\n### Weakness 3: Typo on Eq2\n\nThanks for pointing it out. On line 260, it is a typo which should be \"objective\" instead of \"loss.\" If it were a loss, we would need to include a negative sign. In our code implementation, it is correct.\n\n### Weakness 4: Clarification on evaluation setup and metrics\n\n(a) Yes, the 100 real user test prompts are different from the ImageReward dataset used to train the scoring model. Specifically, these 100 prompts are sampled from the prompts in the DiffusionDB that are outside the training set. In Table 3, there are 466 prompts specially selected as the test set, and 371 prompts are taken from the test set with corresponding image counts greater than or equal to 8, combined with prompts that were annotated by the researchers in the early stages of the study. \n\n(b) It's not about not selecting the worst image in top-k, but rather selecting the worst image in \"worst-k\" or \"lowest score k\". \n\n(c) In Table 4, \"# Win\" is calculated from all comparisons in the human evaluation, while \"WinRate\" is derived from comparisons against the baseline. All fine-tuned models generate one image per prompt using the same denoising parameters and random seed for evaluation. \n\n(d) \nNote that both RAFT and Reward Weighted do not collect the prompts used by users in real scenarios for fine-tuning, but prompts used in our review are more widely distributed and complex, so the problems with their methods are more clearly exposed.\nRAFT is constrained by the quality of the constructed dataset.  It is important to note that even expert generators have limitations, and when fine-tuning is performed using prompts sampled from real user data, which can be challenging, there may be instances where the expert generator fails to generate high-quality images.\nIn the case of the Reward Weighted method, the coefficient used for the rewards is constrained within the [0, 1] range.  This implies that influence of the non-preferred images is not completely eliminated. Similarly, when utilizing real user prompts, it is likely that there will be non-preferred images (even those relatively the best) in the dataset, which can hinders the effectiveness of the Reward Weighted method.\n\n\nWe will update the above content to paper for clarification in the final version. Thanks for your suggestion!\n"}, {"review_id": "6NWBlFJNtv", "reviewer": "Reviewer_JPbK", "summary": "The paper explores human preferences and introduces an ImageReward mechanism, which can be employed for evaluating text-to-image generation. The study further enhances the performance of existing text-to-image generation models through a Reward Feedback Learning (ReFL) approach. The experiments primarily focus on the alignment between the proposed ImageReward and human preference/judgment, which is annotated by real individuals.", "questions": "1. The number of generative models considered in the experiments seems insufficient. With only six text-to-image generation models used for the evaluation of the proposed ImageReward, the ranking results, as demonstrated in Table 1, may be easily consistent. Moreover, even a slight variation in ranking can lead to significantly different correlation values. To enhance the persuasiveness of the experiments, it would be better to include a more diverse range of generative models, which is more convincing.\n2. In Table 1, the results show that the Spearman correlation between the zero-shot FID score and human judgment is only 0.09. With this relatively low correlation, does it suggest that there is little relevance between the FID score and human preferences?", "limitations": "The authors discuss the limitations and broader impact in the last two sections of the paper and propose potential solutions and outline future directions for further research and development.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem addressed in this paper is vital as it aims to judge the alignment between text-to-image generation models and human preferences during both training and evaluation. The proposed ImageReward model could effectively guide both the training and evaluation processes of text-to-image generation models.", "weaknesses": "There are some concerns about experiments including the considered generative models and the results of the correlations with human preferences/judgments. Please refer to more details in the following.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions!\n\n### Question 1: On sufficiency of ImageReward experiments \n\n> The number of generative models considered in the experiments seems insufficient. With only six text-to-image generation models used for the evaluation of the proposed ImageReward, the ranking results, as demonstrated in Table 1, may be easily consistent. Moreover, even a slight variation in ranking can lead to significantly different correlation values. To enhance the persuasiveness of the experiments, it would be better to include a more diverse range of generative models, which is more convincing.\n\nThanks for your suggestion. There might be some misunderstandings about our evaluation of ImageReward. It is worth noticing that in this work, our evaluation of ImageReward is splitted to two parts:\nEvaluation across images (Table 2 & 3 & 5, Figure 5 & 15 & 16): the major evaluation, where ImageReward is sufficiently evaluated and compared to necessary baselines on judging better generations conditioned on each text prompt.\nEvaluation across models (Table 1, Figure 3): the complementary evaluation, where we endeavor to qualitatively show that ImageReward can also serve as an average metric to compare different T2I models’ quality.\nOur emphasis in this work has been on the evaluation of ImageReward across images, which paves the way for the proposal of ReFL. As a result, we have majorly extensively validated the effectiveness of ImageReward through the first type of evaluation rather than the second.\n\nAdditionally, we also actually identify some practical difficulties when evaluating across a large number of T2I models as in the second type. As we obtain human evaluations by model ranking, when the number of models grows large, it becomes very challenging for humans to consistently make effective and reasonable comparisons of images from different models. In practice, when the number of images to rank goes beyond 8, we find human rankings become inconsistent and inaccurate for valid evaluation.\n\n### Question 2: Relation between FID score and human preferences\n\n> In Table 1, the results show that the Spearman correlation between the zero-shot FID score and human judgment is only 0.09. With this relatively low correlation, does it suggest that there is little relevance between the FID score and human preferences?\n\nYes, we observed that although some models may have higher FID scores compared to others, the images generated by these models better align with human preferences. We would like to conclude that FID scores insufficiently reflect human preferences, which are in fact very crucial for T2I models’ practical applications.\n"}, {"review_id": "j7jIiphVWK", "reviewer": "Reviewer_5G85", "summary": "This study presents ImageReward, a general-purpose text-to-image human preference reward model. They have collected 137k expert preference dataset, which contain a lot of rating and ranking annotation.  Additionally, the authors propose Reward Feedback Learning (ReFL), a direct tuning algorithm designed to optimize diffusion models. The performance of ImageReward surpasses that of existing models and metrics.", "questions": "you may also incorporate evaluation metrics and more complex prompts (e.g. with more spatial grounding\") to collect the human feedback.", "limitations": "it is well discussed in the paper", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is very well presented with clear paper writing and good demonstration.\n2. The idea is very novel, which explores directly using the human preference as the supervision signals to tune the pretrained text-to-image generation models\n3. they also collect a large-scale high-quality human preference dataset, which can inspire many future works. ", "weaknesses": "BLIP, being outdated, falls short in generating accurate and comprehensive image descriptions. An alternative approach is to employ more recent models like MiniGPT-4 or LLaVa, which have the potential to produce superior reward scores. These advanced vision-language models not only comprehend the objects within the image but also grasp the emotional and artistic aspects. It would be beneficial if the authors could include a comparative analysis involving these models to further strengthen their findings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments, we will explain your concerns point by point.\n\n### Weaknesses: More advanced backbone for ImageReward\n\n> BLIP, being outdated, falls short in generating accurate and comprehensive image descriptions. An alternative approach is to employ more recent models like MiniGPT-4 or LLaVa, which have the potential to produce superior reward scores. These advanced vision-language models not only comprehend the objects within the image but also grasp the emotional and artistic aspects. It would be beneficial if the authors could include a comparative analysis involving these models to further strengthen their findings.\n\nThank you for your great suggestion. Indeed, introducing Visual-enhanced Large Language Models (LLMs) to assist with feedback for Text-to-Image (T2I) is a highly valuable avenue for research and exploration. We would like to elaborate a few of our thoughts and discoveries here.\n\nIn general, compared to BLIP, the most important aspect of MiniGPT-4 and LLaVa is the use of LLMs possessing powerful capabilities beyond scoring. We believe LLMs can provide evaluations of T2I results from various dimensions *if properly prompted or fine-tuned*, ranging from accuracy in object attributes, spatial relationships, aesthetics, social ethics, human emotions, and more.\n\nWe first study how directly prompting those visual LLMs would help scoring human preference. Take LLaVa as an example in our experiment, by inputting images and appropriate prompts into LLaVA-Lightning-MPT-7B, we derived model ratings from its responses and evaluated their consistency with human rankings following practices in LLM rating [1]. In our testing, 39.22% of pairs couldn't be tested due to the model not providing scores, 31.88% received identical scores from LLaVa, and the remaining 28.9% exhibited an accuracy of 53.60% in aligning LLaVa's scores with human preferences. Interestingly, this result was lower than any baseline's performance in our paper, indicating a gap in LLaVa's zero-shot scoring.\n\nAdditionally, we also consider how proper fine-tuning with visual LLMs could help. Unfortunately, due to the short response period and potentially complicated tuning strategies required (as is noted in our Section 2.2), we did not finish the fine-tuning experiments on LLaVa. However, we may share some thoughts on potential outcomes if we make it successfully.\n\nIn this work, we primarily compared CLIP and BLIP in the paper since many previous works directly used CLIPScore to assess image-text consistency. It's important to note that we only utilized BLIP's encoder and not its decoder. Specifically, we used ViT-L as the image encoder and a 12-layer BERT as the text encoder, which is comparable in model size to CLIP. The most significant difference between BLIP and CLIP is that BLIP is a single-tower model while CLIP is dual-tower. An insight in our work is that a single-tower model may offer superior joint image-text modeling. We also experimentally demonstrated that BLIP outperforms CLIP on the same dataset scale.\n\nBoth BLIP and CLIP do not employ LLMs, and their resource requirements (including training costs, inference overhead, GPU memory requirements, etc.) are lower. As a result, it is likely that with properly tuned visual LLMs, we can better create better human preference RMs for the T2I process. However, it is also a problem that these visual LLMs are somehow even larger than the T2I models themselves (e.g., LLaVa’s 7B v.s. Stable Diffusion’s 1B), which would make the training inefficient.\n\n\n### Question: \n\n> you may also incorporate evaluation metrics and more complex prompts (e.g. with more spatial grounding\") to collect the human feedback.\n\nThank you for your excellent suggestion! Indeed, exploring more complex prompts is a worthwhile direction for future research. We will consider these improvements in the next version of ImageReward. We will consider more complex prompts such as complex spatial relationships, incorporating textual elements within images, complex object/attribute combinations, intricate grammar structures, and combinations of objects that require imaginative thinking beyond the distribution of the training data. Additionally, prompts can be designed to promote diversity and non-discrimination, among other possibilities. Overall, exploring these avenues holds great promise for future research.\n\nReferences:\n[1] Zheng L, Chiang W L, Sheng Y, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena[J]. arXiv preprint arXiv:2306.05685, 2023.\n"}, {"review_id": "drBz4vSiS8", "reviewer": "Reviewer_6uei", "summary": "The paper presents a human preference model for text-to-image generation and a method to enhance text-to-image models using this preference model. To achieve this, the authors develop a human preference annotation pipeline and create a dataset consisting of generated images and human ratings. The proposed model is trained to predict human preference rankings, and experimental results indicate that it aligns better with human preferences than existing automatic measures. Additionally, the paper introduces a learning method to fine-tune a diffusion model using the human preference model. The experimental result demonstrates that the text-to-image generation model, when adjusted with the proposed method, is preferred by human annotators.", "questions": "- Will the entire human preference dataset be made publicly available? The repository looks to provide only test set.\n- Can the authors clarify the unclear points mentioned in the Weaknesses section?", "limitations": "The limitations section addresses three main issues:\n- Annotation scale, diversity, and quality\n- The heuristic nature of RM training\n- The lack of theoretical background for training the diffusion model with RM feedback\nThe authors provide meaningful suggestions for future research directions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper's clear contribution is the human preference dataset, which features high-quality annotations from a professional data annotation company.\n- The experiment suggests that ImageReward outperforms popular measures like FID and CLIP scores in evaluating text-to-image generation.\n- By training the text-to-image model using the human preference model, the authors achieve improved performance in both automatic and human evaluations.", "weaknesses": "- The paper exceeds the page limit. The authors should carefully follow formatting instructions and revise the manuscript accordingly.\n- The description of ImageReward training lacks detail, which may make reproduction difficult.\n- Information regarding the human evaluation of experiments in Section 4.2 is missing.\n- Certain aspects are unclear. For example, the intention of Figure 7 is unclear for me. What image generation problems are demonstrated in the examples? The numbers in Table 2 are unexplained. What agreement measure is used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions.\n\n### Weakness 1: Formatting issues\n\nWe sincerely apologize for the unintentional formatting issues and would appreciate your leniency towards these easily rectifiable issues. In the final version, we will promptly correct these formatting problems by relocating the \"Limitations\" and \"Broader Impact\" sections from page 10 to the \"Appendix\".\n\n### Weakness 2: Details on ImageReward Training\n\nThere might be some misunderstandings about this part. In fact, we have provided detailed information on the necessary aspects to reproduce the training in the following sections of the paper and supplementary code repos: \n1. In \"Section 2.2 RM Training,\" we provide a comprehensive description of the objective function and how the dataset is utilized. \n2. In the \"Training Setting\" paragraph of Section 4.1, we offer detailed information about the model architecture, hyperparameter settings, and experimental setup.\n3. One-click script to reproduce our experiments: we provide all codes related to reproduce ImageReward training in our code repository (see our abstract), which ensures reproduction with one line of commands. (You can find it at train/src/train.py in the code repository.)\n\nThanks for your pointing out and  for clarification, we will sum up all the reproduction details in a new “Reproduction” section in the final version.\n\n### Weakness 3: Details on human evaluation of ReFL (Section 4.2)\n\nThank you very much for your attention to the details of our experiment! Our human evaluation in Section 4.2 is consistent with Section 2.3 and the form of dataset labeling, which involves humans sorting multiple images under a prompt. We account for the source and number of prompts used in Section 4.2, with different models controlling the same training process as well as parameter settings.\n\n### Weakness 4: Details on others\n\n> Certain aspects are unclear. For example, the intention of Figure 7 is unclear for me. What image generation problems are demonstrated in the examples? The numbers in Table 2 are unexplained. What agreement measure is used?\n\n1. Figure 7 is designed to demonstrate that the LDM generated by ReFL can produce images that are more preferred overall, encompassing aspects such as text-image alignment, fidelity, and aesthetics. For example, in the second row of prompts containing \"long pointy ears,\" only the model fine-tuned with ReFL generates correct ears, while other images either lack ears or have inaccurate representations. This illustrates both alignment and fidelity aspects. \n\n2. \"Agreement\" refers to the probability that two individuals have consistent judgments on which of the two pairs is superior.\n\nThanks for your pointing out and we will update these explanations to the final version.\n\n### Question 1: Dataset Open-sourcing\n\n> Will the entire human preference dataset be made publicly available? The repository looks to provide only test set.\n\nWe have already made the whole 137k human preference dataset public, but due to the double-blind policy we cannot provide the public link here.\n"}]

Author rebuttal:

Meta-review: To improve the image-text alignment, this paper proposes a human preference dataset, provides a reward function aligned with human preferences and tuning method called ReFL.


**strengths**

* Large-scale high-quality human preference dataset for text-to-image generation
* Reward model which can be useful for various downstream tasks
* Tuning method based on learned reward function.

**weaknesses/suggestions**

* Training details for reward learning & investigating efficient labeling 
* Detailed explanation for human evaluation

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version. Also, please take a closer look at ethics reviews and modify the paper by following suggestions.
Decision: Accept (poster)