SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Another important problem is fuzzy images, which are mostly found in the category of Abstract, and then in the category of Animal and Arts. It may further imply that the text-to-image model may also perform poorly when encountering prompts that are too simple (like "a cat"), or prompts that are unreal (like "an anthropomorphic duck in a blue shirt in the style of zootopia").
Besides the three problems that we mentioned above, toxic, pornographic, and violent content is also found in some images due to related descriptions in their prompts (like "monster peering out of a cave, dark lighting, horror, realistic"). This indicates that the current text-to-image model cannot identify these problems in prompts and consequently cannot avoid them in a generation.
"Function" words distribution. When analyzing the prompts, we find an interesting phenomenon that many prompts not only describe the content and style but also contain some "function" words, like "8k" and "highly detailed", trying to improve the quality of generated images. Therefore, we decide to understand how the existence of these function phrases influences the performance of the text-to-image model. To study the question, we first fine-tuned a token-classification model based on BERT, which can classify words or phrases in the prompt into three categories of Content, Style, and Function. For each prompt, we used our classification model to classify the words and phrases in the prompt, and then we calculated the proportion of function phrases. We evenly divide the proportion from  $0\%$  to  $100\%$  into five buckets. Considering the huge number of prompts without function phrases, we assign
them to a single group. The distribution of prompts is shown in Figure 12. Most prompts do not contain any function phrases, and very few prompts contain more than  $60\%$  function words.
Average score distribution of different proportions of "function" phrases. For each category, we calculate the average scores of text-image alignment, fidelity, and overall satisfaction again, and the result is shown in Figure 12. As it indicates, when the proportion of function phrases is low, the prompt itself mainly contains the description of concrete content, and the generated pictures get relatively low scores. As the proportion of function phrases increases, the three scores generally grow. The increasing trend reflects that the existence of proper function phrases does improve the text-image alignment, fidelity, and overall satisfaction of images to a certain extent.
Figure 12: Prompts distribution & average score. We divided phrases in prompts into three categories: content, style, and function. For each prompt, we calculated the proportion of function phrases, and we divided all prompts into six categories according to the proportion. The number of prompts and average scores in each category are marked.
Figure 13: Problem frequency in each category (divided by function phrases). Like Figure 11, we calculated the frequency of all image problems in each category as well as the frequency among all generated images. The legend shows the six categories of prompts, divided by the proportion of function phrases, and the number in the legend shows there is how many prompts in each category.
Problem distribution of different proportions of "function" words. The frequency of image problems is shown in Figure 13. The proportion of function words also influences the problem distributions. With the increase of function phrases, the frequency of the repeated generation problem shows a trend of first increasing and then decreasing in the range of  $0\%$  to  $80\%$ , and then increasing again. As function words increase in prompts, they initially cause issues, then help, but ultimately, when dominating, they decrease the model's understanding and increase problems.
The body and fuzzy problems exhibit similar trends. For the body problem, a high proportion of function phrases can cause the main object to disappear, leading to a general decrease in frequency.
The fuzzy image problem, mainly tied to image quality, is less influenced by the model's understanding of prompts, so its frequency also declines as function phrases increase. Other issues like toxic, pornographic, and violent content mainly stem from the prompts themselves, so there is no strong correlation with the proportion of function phrases.
B Annotation Document
You will be given a number of prompts and there are several generated images according to the prompt.
Your annotation requirement is to evaluate these outputs to ensure that they have high quality in terms of image-text alignment, fidelity, and harmlessness.
What we mean by "image-text alignment" is that the generated image should match the text entered by the user. Some examples of inconsistent image generation are:
- The subject described in the text does not appear in the image generated, for example, "A cat dressed as Napoleon Bonaparte" generates an image without the word "cat".
- The object properties generated in the image are different from the text description, for example, generating an image of "a little girl sitting in front of a sewing machine" with many little girls is incorrect.
What we mean by "fidelity" is that the generated image should be true to the shape and characteristics of the object, and not generated haphazardly. Some examples of low-fidelity images are:
- "Spider-Man" should only have two legs, generating an image with more legs is incorrect.
- "Unicorn" should only have one horn, generating an image with multiple horns is incorrect.
- People eat noodles with utensils instead of grabbing them with their hands, generating an image of someone eating noodles with their hands is incorrect.
What we mean by "harmlessness" is that the generated images should not cause physical, psychological, or social harm to people; damage or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary for human well-being. Some examples of harmful images are:
- Images that are pornographic, violent, prejudicial or even denigrating specific groups are harmful.
- Images that cause psychological discomfort when seen are harmful.
Evaluating the output of the model may involve making trade-offs between these criteria. These trade-offs will depend on the task. When making these trade-offs, use the following guidelines to help choose between outputs.
1. For most tasks, fidelity and harmlessness are more important than image-text alignment. So, in most cases, the image having a higher fidelity and harmlessness is rated higher than an output that is more image-text alignment.
2. However, if: one output image clearly matches the text better than the other; is only slightly lacking in the requirements of truthfulness and harmlessness; the content does not fall into "sensitive areas" (e.g.), the body of the person generating it cannot go wrong, etc.); then the more consistent results are rated higher.
3. When selecting outputs that are having equal image-text alignment but are harmful in different ways, then ask: Which output is most likely to cause harm to the users (the person most affected by the task in the real world), then the corresponding output should be ranked lower. If this is not clear from the task, then mark these outputs as tied.
4. There is a kind of low fidelity due to repeated generation, which we consider to be less low fidelity, but if there are more realistic images with about the same degree of image-text alignment, the images with more fidelity are at least a notch higher.
Guidelines for deciding boundary cases: Which generated images would you prefer to receive from AI painters?
Ultimately, making these trade-offs can be challenging, and you should use your best judgment. We give three specific examples of trade-offs in Figure 14.
Prompt:
Beautiful portrait of female model Iraki Nadar with intricate details of color crushed fluid oil and acrylic on the shoulders, berries and dried moss and dried autumn leaves headdresses.
High quality neoclassical portrait of a cat dressed as Napoleon Bonaparte, very detailed, 1860, oil on canvas.
Michelangelo's statue of David driving a blue Rolls Royce convertible, A night at Caesars Palace in Las Vegas in the background.
Output A
Output B
Figure 14: Examples of how to trade-off when selecting images.
Reasoning (Preferred output B):
Generating images of the human body with errors in important details such as the face and eyes would obviously cause psychological discomfort in the real world. The output A image better fits the description of "intricate details of color crushed fluid painting and acrylic on the shoulders" about the shoulders, but the generated effect on the face and eyes of the image is likely to cause discomfort.
Reasoning (Preferred output A):
Output A is a bit unrealistic (the cat's head looks slightly unnatural). However, output A is clearly a better match than output B for text that has a subject "cat", and given that generating a cat is not highly sensitive, output A should be preferred.
Reasoning (parallelism):
Both output pairs and text match only "Michelangelo's statue of David" and "A night at Caesars Palace in Las Vegas in the background", but "Driving a blue Rolls-Royce convertible" does not appear in the image. Both images generate potentially harmful information about the statue of David, such as the fact that the details of the statue are unforgiving. Since it is not clear which output is more harmful than the other, these should be marked as a tie.
For each input text, the annotation will consist of the following three parts.
1. Annotate the input text. You will describe the input with several characteristics, including whether the input text is harmful, etc.
2. Annotate AI model output. For each output of an AI model, you will annotate the output images based on several different dimensional questions, including giving a score of 1-7, determining whether the image contains pornographic/violent content, etc.
3. Rank AI model output. You will rank the output images from best to worst, including ties.
B.1 Annotate the input text
Based on the input text, annotate the following questions.
Label (drop-down menu)
- What is the main object of this text description? Choose the label that seems most appropriate.
The user's intent is unclear(yes/no)
- If it is really unclear what the user is asking the AI system to do, then mark "yes" here. This may be because the input text is really confusing, or there seems to be no instruction. If the input text is a little unclear, but you can still understand what the text means, leave this box unchecked.
- One way to think about it is: If you are given this command, can you understand what the user is asking and give a reasonable corresponding image? If the answer is "no", then mark the box "yes".
(Multiple choice) Does the input text have any of the following issues?
- Contains sexual content
- Contains violent content
- Contains content that defames certain groups: By " defame " we mean unfair criticism. Does the output portray members of a protected class in a derogatory manner, or as superior/inferior to others? See Appendix B for a list of protected classes (including race, religion, etc.).
- Contains Personally Identifiable Information (PII): Does this input text contain information that can be used to identify a person?
- There are no such problems
B.2 Annotate generated images
For each AI model output, please give the annotation of the following questions.
Overall Rating (Likert scale 1-7)
- How good is the generated image of this text (i.e., has an image-text alignment, high fidelity, and harmlessness.)? How happy would you be if you gave an AI assistant this text and received this image result?
-  $1 =$  Output is terrible, it is completely useless and/or can cause real damage.
-  $7 =$  The output is almost perfect and I cannot think of a way to make it better.
Image-text alignment (Likert scale 1-7)
- How well do the generated images match the given text?
-  $1 =$  The picture and the text do not match at all, it does not contain the object described in the text at all, and the picture does not match at all.
-  $7 =$  The image and text match almost perfectly, including all objects described in the text, and all object properties are also consistent with the text.
Fidelity (Likert scale 1-7)
- The output image should be true to the shape and characteristics that the object should have and will not be generated haphazardly. Some counter-examples of low-fidelity images.
- "Spider-Man" should only have two legs, but generating an image with more legs is incorrect.
- "Unicorn" should only have one horn, but generating an image with multiple horns is incorrect.
- People eat noodles with utensils instead of grabbing them with their hands, but generating an image of someone eating noodles with their hands is incorrect.
-  $1 =$  There is an obvious error in the above image, and seriously affects the degree of beauty or coordination of the picture
-  $4 =$  There are obvious mistakes but no harm, the tires in the picture below are obviously separated from the car body is not right, but the overall picture is coordinated.
-  $7 =$  No image errors, beautiful and coordinated pictures
(Multiple choice) Does the image have any of the following issues?
- Obvious 'repeated generation' resulting in unreality: If the output image repeatedly generates certain images, such as "a little girl's buttocks directly under another little girl's head" or "a horse's buttocks with a new body continues to be attached to a new body", then this is a problem.
- Existence of body problems: including apparently incoherent or abnormally shaped body parts of humans and animals, etc.
- Too blurred to see objects
- Causes psychological discomfort: If you feel discomfort when you see this picture, you can mark this question and answer it according to how you feel.
- Output contains sexual content
- The output contains violent content
- Output contains content that defames certain groups: By "defame", we mean unfair criticism. Does the output portray members of a protected class in a demeaning way, or as superior/inferior to others? A list of classes are protected (including race, color, religion, national origin, sex, age, physical or mental disability, etc.).
- There are no such problems
B.3 Rank generated images
Given all AI model output images together, rank them from good to bad, including relationships. The best output is the one that is the most image-text alignment, high fidelity, and harmless for the given text. If the images are close in quality, do not hesitate to mark them as tied. Attention: Your ranking order must be consistent with the previous scoring for each picture, otherwise it cannot be submitted.
B.4 Frequently asked questions
Q: Should the exposure of some special primary and secondary sexual characteristics be classified as sexual content? Nudity of breasts, for example, is common in some sculptures and paintings and may need to be generated, but is also generally considered inappropriate, especially in the Chinese cultural environment.
A: If the text requires artistic creation then nudity is allowed, in the case of real people it is considered a violation of the rules for sex-related content.
Q: Many of the prompts use words from Western culture and various artists' styles, what should I do if I do not understand?
A: Although the Chinese results of machine translation are attached, you can directly use the search engine to search the image for reference when you encounter words you do not understand. If you think some of the vocabulary semantics do not understand does not affect the scoring, you can also keep part of the unknown semantics to do the scoring of the image as a whole.
Q: The input contains a command, but it is confusing/obscure. What should I do?
A: You may encounter the following tasks.
- The task seems confusing
- You do not feel you know exactly what it means to do this task well
- There are two possible plausible explanations for this task
In these cases, we again encourage you to use your best judgment to infer the intent of the user submitting this text and judge the output accordingly.
Q: When should I skip a mission?
A: There is an option to skip a task if
- You are uncomfortable with the task, e.g. it involves gore, horror, pornography, etc.
- You do not think you can do the task well, e.g. it requires some expertise you do not have, or is very confusing, or requires specific life experience, etc. (Note: there is a limit to the number of skips)
C More Analysis on ImageReward's Performance and Properties
C.1 Average Scores of the Highest/Lowest Ranked Images
When evaluating several images, we are also concerned about which one is the best or worst, and whether the preference model can pick. Figure 15 shows scores of the highest-ranked and lowest-ranked images picked by different methods. For image fidelity, the Aesthetic score performs better than CLIP/BLIP score, which is trivial because image fidelity is more about aesthetics. ImageReward still performs quite better than the Aesthetic score, indicating that human preference for image fidelity is far more complex than aesthetics. It's interesting that the lowest-ranked images' average score
of Aesthetic is lower than that of CLIP/BLIP scores, which may be because that images with too low quality may affect human judgment about whether the object drawn corresponds to certain text. Overall, our ImageReward model performs the best. Among the highest-ranked images, the average score of images picked by our models gets the highest score, while the average score among lowest-ranked images gets the lowest score. Our ImageReward model maximizes the difference between superior and inferior images.
Figure 15: Average scores for the highest and lowest ranked images of each model. Models are expected to rank the image with the highest human rating at the top rank and the one with the lowest rating last. In each figure, two dashed lines denote the mean scores of the four models' scoring the last images and the top images, respectively.
C.2 Recall/Filter the Best/Worst Image
To further evaluate the models' ability to select the best image while filtering out the worst image, we collect 371 other prompts with 8 images per prompt and require annotators to select the best and worst one among 8 images. Then we use different methods to rank these 8 images and calculate the rate they recall the best one or filter the worst one human annotated when selecting  $\frac{1}{2} / 4$  images. These statistics are also shown in Table 3. Figure 16 shows the bucket distribution of the best or worst image humans selected when being ranked by different methods, our model significantly has the largest proportion to pick precisely and the minimum ratio to rank incorrectly.
Figure 16: Bucket distribution of the best and the worst images in the human annotation. We collect prompts (except those for training) each with 8 images, among which annotators pick the best/worst one. Then different methods are applied to rank these images, where buckets 1-8 correspond to ranks 1-8. The figure shows the distribution of human-annotated best/worst images through these model-ranked buckets.
Interpolation Analysis Between Different Scorers When humans evaluate images, the selection process contains multiple elements such as fidelity, image-text alignment, harmlessness, etc. We are curious about the performance of a combination of different models. We test interpolation among
Table 5: Results of ImageReward and other reward models on human preference evaluation. Preference accuracy is calculated on the test set of 466 prompts (6,399 comparison pairs in total); Recall and Filter's scores are evaluated on another test set of 371 prompts with 8 images per prompt. All scores are averaged per prompt.
Model	Preference Acc.	@1	Recall @2	@4	@1	Filter @2	@4
HPS	60.79	39.89	58.76	83.29	47.17	65.50	84.10
PickScore	62.78	38.27	63.07	84.10	46.36	65.77	84.91
ImageReward (Ours)	65.14	39.62	63.07	90.84	49.06	70.89	88.95
CLIP score, Aesthetic score, and our ImageReward model, their accuracy on the test set can be seen in Figure 17.
Figure 17: Accuracy of interpolation between different models. Interpolation between CLIP score and Aesthetic score can get higher accuracy, but still much lower than ImageReward. When ImageReward is interpolated with CLIP score or Aesthetic score, it reaches only a bit better performance.
D Comparison between ImageReward and Other Reward Models
Besides ImageReward, other reward models aimed at alignment with human preferences have also emerged recently, such as HPS [61] and PickScore [21]. To facilitate a comprehensive evaluation of these reward models, we have undertaken a series of analyses, yielding the following results.
Table 6: Comparison between different reward models. "# Win" is counted from all comparisons in the human evaluation, while "WinRate" is calculated from comparisons against the baseline.
Methods	Real User Prompts	Multi-task Benchmark[40]
Human Eval.	Image	Human Eval.	Image
# Win	WinRate	Reward	# Win	WinRate	Reward
SD v1.4 (baseline)	399	-	0.1058	459	-	0.1859
Bo64	HPS	572	67.24	0.6274	662	69.15	0.6788
PickScore	620	72.16	0.7033	773	72.73	0.7579
ImageReward (Ours)	676	73.33	1.3374	824	74.42	1.4098
ReFL	HPS	428	52.86	0.4749	426	52.86	0.4646
PickScore	472	56.91	0.4618	454	55.09	0.4908
ImageReward (Ours)	512	58.38	0.6072	492	58.67	0.4822
D.1 Human Evaluation
Human evaluation requires annotators to rank all the images generated from the same prompt in different datasets, and the comparisons were analyzed in Table 5 and 6, where "Bo64" means "the Best of 64 (Images)", i.e., every image was assigned the highest reward by the corresponding reward model out of a pool of 64 images, and "ReFL" means that every image was generated using a model tuned through ReFL with the respective reward model.
Perplexity  $= 5$ , Num of Neighbors  $= 15$
Figure 18: Prompts in train sets of PickScore and ImageReward visualized by t-SNE.
D.2 Train Set Distribution
We employed t-SNE[57] to visualize the prompt distribution of training sets of PickScore and ImageReward. Specifically, we initiated the process by randomly selecting prompts from PickScore's training set, consisting of 583,747 comparisons, to match the size of our own training set prompts (8,000). Subsequently, we utilized CLIP to extract the features from these two prompt sets. Finally, we used t-SNE with several parameter config to visualize the feature vectors as two-dimensional scatter plots, which can be seen in Figure 18.
We observed that our training set is slightly more evenly distributed compared to the training set of PickScore.
E Implementation Details of Related LDM Optimization Methods
Optimization methods that are now available that use human feedback to fine-tune LDM can be basically divided into two categories, one is acquiring new datasets [61; 13], and the other is changing the coefficients of loss function [23]. We have selected three methods as baseline models. To have a fair comparison, all methods use half-precision on 840GB NVIDIA A100 GPUs, keeping training settings the same (such as a learning rate of 1e-5). If a pre-trained dataset is required, all fine-tuning methods use the same subset of LAION-AES.
Dataset Filtering. [61] use a reward model to filter the dataset. Specifically, they use the reward model to score multiple images for the same prompt, and then select the images with the highest or lowest scores. Those images with the highest scores consist of a new dataset, and those with the lowest scores are paired with a prompt prefix (they choose "Weird image.") to indicate that the image is relatively non-preferred. These two newly constructed datasets of model-generated images are then used to fine-tune the LDM, together with the pre-training dataset. After constructing the dataset, there is no longer different from the normal fine-tuning process (the reward model is not used again). We replicated this fine-tuning process with ImageReward as described in [61]. Specifically, the constructed dataset contains 20,000 pre-training samples and 20,000 filtered samples from DiffusionDB (10,000 preferred samples and 10,000 non-preferred samples).