SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiazheng Xu $^{\diamond \dagger *}$ , Xiao Liu $^{\diamond \ddagger *}$ , Yuchen Wu $^{\diamond}$ , Yuxuan Tong $^{\diamond}$ , Qinkai Li $^{\S \dagger}$ , Ming Ding $^{\ddagger}$ , Jie Tang $^{\diamond}$ , Yuxiao Dong $^{\diamond}$
Tsinghua University ‡Zhipu AI §Beijing U. of Posts and Telecommunications
We present a comprehensive solution to learn and improve text-to-image models from human preference feedback. To begin with, we build ImageReward—the first general-purpose text-to-image human preference reward model—to effectively encode human preferences. Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer. Both automatic and human evaluation support ReFL's advantages over compared methods. All code and datasets are provided at https://github.com/THUDM/ImageReward.
Text-to-image generative models, including auto-regressive [43; 11; 14; 16; 12; 63] and diffusion-based [37; 45; 42; 46] approaches, have experienced rapid advancements in recent years. Given appropriate text descriptions (i.e., prompts), these models can generate high-fidelity and semantically-related images on a wide range of topics, attracting significant public interest in their potential applications and impacts.
Despite the progress, existing self-supervised pre-trained [33] generators are far from perfect. A primary challenge lies in aligning models with human preference, as the pre-training distribution is noisy and differs from the actual user-prompt distributions. The inherent discrepancy leads to several well-documented issues in the generated images [15; 31], including but not limited to:
- Text-image Alignment: failing to accurately depict all the numbers, attributes, properties, and relationships of objects described in text prompts, as shown in Figure 1 (a)(b).
- Body Problem: presenting distorted, incomplete, duplicated, or abnormal body parts (e.g., limbs) of humans or animals, as illustrated in Figure 1 (e)(f).
- Human Aesthetic: deviating from the average or mainstream human preference for aesthetic styles, as demonstrated in Figure 1 (c)(d).
- Toxicity and Biases: featuring content that is harmful, violent, sexual, discriminative, illegal, or causing psychological discomfort, as depicted in Figure 1 (f).
These prevalent challenges, however, are difficult to address solely through improvements in model architectures and pre-training data.
In natural language processing (NLP), researchers have employed reinforcement learning from human feedback (RLHF) [55; 36; 39] to guide large language models [6; 7; 66; 48; 64] towards human preferences and values. The approach relies on learning a reward model (RM) to capture human
*JX & XL contributed equally. Corresponding authors: YD & JT (yuxiaod | jietang@tsinghua.edu.cn).
$^{\dagger}$ Work done when JX interned at Zhipu AI and QL visited Tsinghua University.
Figure 1: (Upper) Top-1 images out of 64 generations selected by different text-image scorers. (Lower) 1-shot generation after ReFL training using ImageReward as feedback. Images get better text coherence and human preference after ImageReward selection or ReFL training. In prompts (from real users, truncated), the bold roughly denotes content, and the italic denotes style or function.
preference from massive expert-annotated model output comparisons. Effective though it is, the annotation process can be costly and challenging [39], as it requires months of effort to establish labeling criteria, recruit and train experts, verify responses, and ultimately produce the RM.
Contributions. Recognizing the importance of addressing these challenges in generative models, we present and release the first general-purpose text-to-image human preference RM—ImageReward—which is trained and evaluated on 137k pairs of expert comparisons in total, based on real-world user prompts and corresponding model outputs. Based on the effort, we further investigate the direct optimization approach ReFL for improving diffusion generative models. Our main contributions are:
- We systematically identify the challenges for text-to-image human preference annotation, and consequently design a pipeline tailored for it, establishing criteria for quantitative assessment and annotator training, optimizing labeling experience, and ensuring quality validation. We build the text-to-image comparison dataset for training the ImageReward model based on the pipeline. The overall architecture is depicted in Figure 2.
- We demonstrate that ImageReward outperforms existing text-image scoring methods, such as CLIP [41] (by  $38.6\%$ ), Aesthetic [50] (by  $39.6\%$ ), and BLIP [26] (by  $31.6\%$ ), in terms of understanding human preference in text-to-image synthesis through extensive analysis and experiments. ImageReward is also proven to significantly mitigate the aforementioned issues, providing valuable insights into how human preference can be integrated into generative models.
- We suggest that ImageReward could serve as a promising automatic text-to-image evaluation metric. Compared to FID [18] and CLIP scores on prompts from real users and MS-COCO 2014, ImageReward aligns consistently to human preference ranking and presents higher distinguishability across models and samples.
Figure 2: An overview of the ImageReward and ReFL. (Upper) ImageReward's annotation and training, consisting of data collection, annotation, and preference learning. (Lower) ReFL leverages ImageReward's feedback to directly optimize diffusion models at a random latter denoising step.
- We propose Reward Feedback Learning (ReFL) for tuning diffusion models regarding human preference scorers. Our unique insight on ImageReward's quality identifiability at latter denoising steps allows the direct feedback learning on diffusion models, which offer no likelihood for their generations. Extensive automatic and human evaluations demonstrate ReFL's advantages over existing approaches including data augmentation [61; 13] and loss reweighing [23].
2 ImageReward: Learning to Score and Evaluate Human Preferences
ImageReward is constructed using a systematic pipeline involving data collection and human annotation from experts. Based on the pipeline, we implement the RM training and derive the ImageReward.
2.1 Annotation Pipeline Design
Prompt Selection and Image Collection. The dataset utilizes a diverse selection of real user prompts from DiffusionDB [58], an open-sourced dataset. To ensure diversity in selected prompts, we employ a graph-based algorithm that leverages language model-based prompt similarity [56; 44; 53]. This selection yields 10,000 candidate prompts, each accompanied by 4 to 9 sampled images from DiffusionDB, resulting in 177,304 candidate pairs for labeling (Cf. Appendix A.1 for details).
Human Annotation Design. Our annotation pipeline involves a prompt annotation stage, which includes categorizing prompts and identifying problematic ones, and a text-image rating stage, where images are rated based on alignment, fidelity, and harmlessness. Subsequently, annotators rank the images in order of preference. To manage potential contradictions in the ranking, we provide trade-offs in our annotation document (completely attached in Appendix B). Our annotation system is composed of three stages: Prompt Annotation, Text-Image Rating, and Image Ranking. Screenshots of our system are provided in Figure 8. Annotators were recruited in collaboration with a professional data annotation company, with a majority having at least college-level education. They are trained using documents that describe the labeling process and criteria. To ensure quality, we employ quality inspectors to double-check each annotation, with invalid annotations reassigned for relabeling. Due to the page limits, please refer to Appendix A.3, A.2, B for comprehensive details and discussion.
Human Annotation Analysis. After 2 months of annotation, we collected valid annotations for 8,878 prompts, resulting in 136,892 compared pairs. A comprehensive analysis of these prompts, annotations, and challenges discovered is discussed in detail in Appendix A.4.
Table 1: Text-to-image model ranking by humans and automatic metrics (ImageReward, CLIP, and FID). *Zero-shot FID (30k) scores of DALL-E 2 is taken from [42]; others are evaluated in  ${256} \times  {256}$  resolution on MS-COCO 2014 validation set following prior practices.
Dataset & Model	Real User Prompts	MS-COCO 2014
Human Eval.	ImageReward	CLIP	ImageReward	Zero-shot FID*
Rank	#Win	Rank	Score	Rank	Score	Rank	Score	Rank	Score
Openjourney	1	507	1	0.2614	2	0.2726	3	-0.0455	5	20.7
Stable Diffusion 2.1-base	2	463	2	0.2458	4	0.2683	2	0.1553	4	18.8
DALL-E 2	3	390	3	0.2114	3	0.2684	1	0.5387	1	10.9*
Stable Diffusion 1.4	4	362	4	0.1344	1	0.2763	4	-0.0857	2	17.9
Versatile Diffusion	5	340	5	-0.2470	5	0.2606	5	-0.5485	3	18.4
CogView 2	6	74	6	-1.2376	6	0.2044	6	-0.8510	6	26.2
Spearman ρ to Human Eval.	-	1.00	0.60	0.77	0.09
2.2 RM Training
Admittedly, human evaluation is after all the touchstone for human preference for synthesized images; but it is limited by labor costs and hard to scale up. We aim to model human preference based on annotations, which can lead to a virtual evaluator free from dependence on humans.
Similar to RM training for language model of previous works [55; 39], we formulate the preference annotations as rankings. We have  $k \in [4, 9]$  images ranked for the same prompt  $T$  (the best to the worst are denoted as  $x_{1}, x_{2}, \ldots, x_{k}$ ) and get at most  $C_k^2$  comparison pairs if no ties between two images. For each comparison, if  $x_{i}$  is better and  $x_{j}$  is worse, the loss function can be formulated as:
$$
\operatorname {l o s s} (\theta) = - \mathbb {E} _ {(T, x _ {i}, x _ {j}) \sim \mathcal {D}} [ \log (\sigma (f _ {\theta} (T, x _ {i}) - f _ {\theta} (T, x _ {j}))) ] \tag {1}
$$
where  $f_{\theta}(T,x)$  is a scalar value of preference model for prompt  $T$  and generated image  $x$ .
Training Techniques. We use BLIP [26] as the backbone of ImageReward, as it outperforms conventional CLIP (Cf. Table 2b) in our preliminary experiments. We extract image and text features, combine them with cross attention, and use an MLP to generate a scalar for preference comparison.
Training ImageReward is of no ease. We observe rapid convergence and consequent overfitting, which harms its performance. To address this, we freeze some backbone transformer layers' parameters, finding that a proper number of fixed layers improves ImageReward's performance (Cf. Section 4.1). ImageReward also exhibits sensitivity to training hyperparameters, such as learning rate and batch size. We perform a careful grid search based on the validation set to determine optimal values.
2.3 As Metric: Re-Evaluating Human Preferences on Text-to-Image Models
Training text-to-image generative models is hard, but evaluating these models reasonably is even harder. In literature [11; 42; 12; 46], it has been a de facto practice to evaluate text-to-image generative models on MS-COCO [28] image-caption dataset against the real images, using fine-tuned or zero-shot FID [18] scores following DALL-E [43] setting. Nevertheless, it remains quite dubious whether the FID really fits the current need [38], especially from the following aspects:
1. Zero-shot Usage: As generative models are now dominantly used by the public in a zero-shot manner without fine-tuning, fine-tuned FID may not honestly reflect models' actual performance in real use. In addition, despite the adoption of zero-shot FID in recent trends, the possible leak of MS-COCO in some models' pre-training data would make it a potentially unfair setting.
2. Human Preference: FID measures the average distance between generated images and reference real images, and thus fails to encompass human preference that is crucial to text-to-image synthesis in evaluation. Moreover, FID's relies on average over the whole dataset to provide an accurate assessment, whereas in many cases we need the metric to serve as a selector over single images.
Seeing these challenges, we propose ImageReward as a promising zero-shot automatic evaluation metric for text-to-image model comparison and individual sample selection.
Better Human Alignment Across Models. We conduct researcher annotation (i.e., by authors) across 6 popular high-resolution (around  $512 \times 512$ ) available text-to-image models: CogView 2 [12],
Versatile Diffusion (VD) [62], Stable Diffusion (SD) 1.4 and 2.1-base [45], DALL-E 2 (via OpenAI API) [42], and Openjourney<sup>1</sup>, to identify the alignment of different metrics to human.
We sample 100 real-user test prompts for the alignment test, with each model generating 10 outputs as candidates. To compare these models, we first pick the best image out of 10 outputs by each model on each prompt. Then, the annotators rank the images from different models for each prompt, following the disciplines for ranking described in Section 2.1. We aggregate all annotators' annotations, and compute the final win count of each model to all others (Cf. Table 1).
For ImageReward and CLIP scores, we report their average for 1,000 text-image pairs per model. We also document all models' zero-shot FID and ImageReward score (30k) on MS-COCO 2014 valid set following prior practices [42; 12], where outputs are unified to  $256 \times 256$  resolution and optimal classifier-free guidance values are selected by grid search (i.e., [1.5, 2.0, 3.0, 4.0, 5.0]). As shown in Table 1, ImageReward aligns well with human ranking, whereas zero-shot FID and CLIP are not.
Better Distinguishability Across Models and Samples. Another highlight is that, compared to CLIP, we observe that ImageReward can better distinguish the quality between individual samples. Figure 3 presents a box plot of ImageReward and CLIP's score distributions on the 1,000 generations per model. The distributions are normalized to 0.0 to 1.0 using minimum and maximum values of ImageReward and CLIP scores per model, and outliers are discarded. As it demonstrates, ImageReward's scores in each model have a much larger interquartile range than that of CLIP, which means ImageReward can well distinguish the quality of images from each other. Besides, in terms of comparison across models, we discover that the medians of the ImageReward scores are also roughly in line with human ranking in Table 1. On the contrary, CLIP's medians fail to present the property.
Figure 3: Normalized distribution of ImageReward and CLIP scores of different generative models (outliers are discarded). ImageReward's scores align well with human preference and present higher distinguishability.
3 ReFL: Reward Feedback Learning Improves Text-to-Image Diffusion
Though ImageReward can pick out highly human-preferred images from many generations of a prompt, the generate-and-then-filter paradigm could be expensive and inefficient in practical applications. Therefore, we seek to improve text-to-image generative models, particularly for the popular latent diffusion models, for allowing high-quality generation in single or very few trials.
Challenge. In NLP, researchers have reported using reinforcement learning algorithms (e.g., PPO [51]) to steer language models to align to human preference [55; 36; 39], which depends on the likelihood of a whole generation to update the model.
However, unlike language models, latent diffusion models (LDMs)'s multistep denoising generation cannot yield likelihoods for their generations, and thus fail to adopt the same RLHF approaches. A potentially similar approach is classifier-guidance [54; 9] technique during LDM inference. Nonetheless, it is for inference only
Prompt: Landscape photography by marc adamus, mountains with some forests, small lake in the center, fog in the background, sunrays, golden hour, high quality.
Figure 4: ImageReward scores of a prompt with different generation seeds along denoising steps. Final image qualities become identifiable after 30 out of 40 steps.
1https://openjourney.art/
and employs a classifier necessarily trained on noisy intermediate latents, which naturally contradicts RMs' annotation where images need to be completely denoised for humans to mark correct preference. Some concurrent works propose some alternative indirect solutions, such as using RMs to filter dataset for fine-tuning [61; 13], or to re-weight losses of training samples according to their qualities [23]. Nevertheless, these data-oriented approaches are virtually indirect. They could rely heavily on proper fine-tuning data distributions and finally only improve the LDMs mildly.
ReFL: Insight and Solution. We endeavor to develop a direct optimization method for improving LDMs according to an RM (e.g., ImageReward). Looking into ImageReward scores along denoising steps (i.e., 40 in our case), we derive an intriguing insight (Cf. Figure 4) that when we directly predict  $x_{t} \rightarrow x_{0}'$  at a step  $t$  (different from the real latent  $x_{0}$  which experiences  $x_{t} \rightarrow x_{t+1} \rightarrow \ldots \rightarrow x_{0}$ ):
- When  $t \leq 15$ : ImageReward scores for all generations are unanimously low.
- When  $15 \leq t \leq 30$ : High-quality generations begin to stand out, but overall we still cannot clearly judge all generations' final qualities based on the current ImageReward scores.
- When  $t \geq 30$ : Generations of different ImageReward scores are generally distinguishable.
Algorithm 1 Reward Feedback Learning (ReFL) for LDMs  
1: Dataset: Prompt set  $\mathcal{V} = \{y_1, y_2, \dots, y_n\}$   
2: Pre-training Dataset: Text-image pairs dataset  $\mathcal{D} = \{(\mathrm{xt}_1, \mathrm{img}_1), \dots (\mathrm{xt}_n, \mathrm{img}_n)\}$   
3: Input: LDM with pre-trained parameters  $w_0$ , reward model  $r$ , reward-to-loss map function  $\phi$ , LDM pre-training loss function  $\psi$ , reward re-weight scale  $\lambda$   
4: Initialization: The number of noise scheduler time steps  $T$ , and time step range for fine-tuning  $[T_1, T_2]$   
5: for  $y_i \in \mathcal{V}$  and  $(\mathrm{xt}_i, \mathrm{img}_i) \in \mathcal{D}$  do  
6:  $\mathcal{L}_{pre} \gets \psi_{w_i}(\mathrm{xt}_i, \mathrm{img}_i)$   
7:  $w_i \gets w_i$  // Update LDM $w_i$  using Pre-training Loss  
8:  $t \gets rand(T_1, T_2)$  // Pick a random time step  $t \in [T_1, T_2]$   
9:  $x_T \sim \mathcal{N}(0, I)$  // Sample noise as latent  
10: for  $j = T, \dots, t + 1$  do  
11: no grad:  $x_{j-1} \gets LDM_{w_i}\{x_j\}$   
12: end for  
13: with grad:  $x_{t-1} \gets LDM_{w_i}\{x_t\}$   
14:  $x_0 \gets x_{t-1}$  // Predict the original latent by noise scheduler  
15:  $z_i \gets x_0$  // From latent to image  
16:  $\mathcal{L}_{reward} \gets \lambda \phi(r(y_i, z_i))$  // ReFL loss  
17:  $w_{i+1} \gets w_i$  // Update LDM $w_i$  using ReFL loss  
18: end for
In light of the observation, we conclude that ImageReward scores for generations  $x_0'$  after 30 steps of denoising, unnecessarily the final step, could serve as reliable feedback for improving LDMs.
We thus propose an algorithm to directly fine-tune LDMs by viewing the scores of an RM as human preference losses to back-propagate gradients (Cf. Algorithm 1) to a randomly-picked latter step  $t$  (in our case  $t \in [30,40]$ ) in the denoising process. The reason for the random selection of  $t$  instead of using the last step is that, if only the gradient of the last denoising step is retained, the training is proved very unstable and the results are bad. In practice, to avoid rapid overfitting and stabilize the fine-tuning, we re-weight ReFL loss and regularize with pre-training loss. The final loss form is written as
$$
\mathcal {L} _ {\text {r e w a r d}} = \lambda \mathbb {E} _ {y _ {i} \sim \mathcal {Y}} (\phi (r (y _ {i}, g _ {\theta} (y _ {i}))) \tag {2}
$$
$$
\mathcal {L} _ {p r e} = \mathbb {E} _ {\left(y _ {i}, x _ {i}\right) \sim \mathcal {D}} \left(\mathbb {E} _ {\mathcal {E} \left(x _ {i}\right), y _ {i}, \epsilon \sim \mathcal {N} (0, 1), t} \left[ \| \epsilon - \epsilon_ {\theta} \left(z _ {t}, t, \tau_ {\theta} \left(y _ {i}\right)\right) \| _ {2} ^ {2} \right]\right) \tag {3}
$$
where  $\theta$  denotes the parameters of the LDM,  $g_{\theta}(y_i)$  denotes the generated image of LDM with parameters  $\theta$  corresponding to prompt  $y_i$ . Meanings of other symbols are detailed in Algorithm 1, while the loss function of  $\mathcal{L}_{pre}$  is taken from [45].
4.1 ImageReward: On Human Preference Prediction
Dataset & Training Setting. Rankings of annotated images are collected to train ImageReward, which contains 8,878 prompts and 136,892 pairs of image comparisons. We divide the dataset according to prompts annotated by different annotators and select 466 prompts from annotators who have a higher agreement with researchers to consist for the model test. Except for prompts for testing, other more than 8k prompts of annotation are collected for training.
Table 2: Data annotation agreement and ablation study on model backbones and dataset sizes. (a) Agreement between different annotators, researchers, and models. Espe- cially, "annotator ensemble" means, for each pair of images, we use the image (b) Ablation study for different backbones (i.e., CLIP and BLIP) considered better by most people as the better one. used in ImageReward.
researcher	annotator	annotator ensemble	CLIP Score	Aesthetic	BLIP Score	Ours
researcher	71.2% ± 11.1%	65.3% ± 8.5%	73.4% ± 6.2%	57.8% ± 3.6%	55.6% ± 3.1%	57.0% ± 3.0%	64.5% ± 2.5%
annotator	65.3% ± 8.5%	65.3% ± 5.6%	53.9% ± 5.8%	54.3% ± 3.2%	55.9% ± 3.1%	57.4% ± 2.7%	65.3% ± 3.7%
annotator ensemble	73.4% ± 6.2%	53.9% ± 5.8%	-	54.4% ± 21.1%	57.5% ± 15.9%	62.0% ± 16.1%	70.5% ± 18.6%
Backbone	Training Set Size	Preference Acc.
CLIP	4k	61.87
8k	62.98
BLIP	1k	63.07
2k	63.18
4k	64.71
8k	65.14
Table 3: Results of ImageReward and comparison methods on human preference prediction. Preference accuracy is from the test set of 466 prompts (6,399 comparisons); Recall and Filter's scores are from another test set of 371 prompts with 8 images each. All scores are averaged per prompt.
Model	Preference Acc.	@1	Recall @2	@4	@1	Filter @2	@4
CLIP Score	54.82	27.22	48.52	78.17	29.65	51.75	76.82
Aesthetic Score	57.35	30.73	53.91	75.74	32.08	54.45	76.55
BLIP Score	57.76	30.73	50.67	77.63	33.42	56.33	80.59
ImageReward (Ours)	65.14	39.62	63.07	90.84	49.06	70.89	88.95
We load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of ImageReward, and initialize MLP head according to  $\mathcal{N}(0,1/(d_{model}+1))$  decaying the learning rate with a cosine schedule. We sweep over several value settings of learning rate and batch size and fix different rates of backbone transformer layers. We find that fixing  $70\%$  of transformer layers with a learning rate of 1e-5 and batch size of 64 can reach up to the best preference accuracy. ImageReward is trained on 4 40GB NVIDIA A100 GPUs, with a per-GPU batch size of 16.
We use the CLIP score, Aesthetic score, and BLIP score as baselines to compare with the ImageReward. CLIP score and BLIP score are calculated directly as cosine similarity between text and image embedding, while the Aesthetic score is given by an aesthetic predictor introduced by LAION[50].
Agreement Analysis. Agreement assesses the likelihood of two individuals sharing consistent preferences for superior images. While most people generally agree on image quality, variations in model-generated images may lead to divergent judgments. Before assessing model performance, it's crucial to measure the likelihood of consensus in selecting superior images. We use other 40 prompts (778 pairs) to calculate preference agreement between researchers, annotators, annotator ensemble, and models. Table 2a shows the result.