SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rater Compensation. Raters were selected based on their performance history, requiring a minimum of 5,000 approved HITs with an approval rate exceeding  $98\%$ . They were required to pass a qualification exam with a perfect score before given access to the task. The hourly compensation was $10, ensuring fair renumeration for their contributions.
Table 4: List of entities and their modifiers in the DVMP dataset. Colors are not restricted to categories.
Category	Entities	Modifiers
General	backpack, crown, suitcase, chair, balloon, bow, car, bowl, bench, clock, camera, umbrella, guitar, shoe, hat, surfboard, skateboard, bicycle	modern, spotted, wooden, metal, curved, spiky, checkered
Fruit	apple, tomato, banana, strawberry	sliced, skewered
Animals	cat, dog, bird, bear, lion, horse, elephant, monkey, frog, turtle, rabbit, mouse, panda, zebra, gorilla, penguin	furry, baby, spotted, sleepy
Color Modifiers
red, orange, yellow, green, blue, purple, pink, brown, gray, black, white, beige, teal
- Please read the following instructions carefully:
- In this task, you will be given a description and two images. Your job is to evaluate the images based on two criteria:
- 1. Concept Separation: How well does the image match the given description?
2. Visual Appeal: Which image looks overall better or more natural?
- Concept Separation: For each image, ask yourself:
- Do you see all objects from the description?
- Are all objects' details correct?
- Are there any details on objects that should not be there?
Choose the image that best matches the description. If they are equally good, choose 'equally good' and if they are equally bad, choose
'equally bad'
- Visual Appeal: After evaluating Concept Separation, decide which image looks better or natural to you.
- Please choose the images based on Concept Separation first, and then consider their Visual Appeal.
Figure 21: The instructions that were given to the raters.
Table 5: The population vote of three raters was normalized to sum to 100 and the standard error mean was added. The table reports the scores for concept separation (how well the image matches the prompt) and visual appeal for different models on each dataset.
Dataset	Model	Concept Separation	Visual Appeal
A&E	SynGen (ours)	38.80 ± 0.48	40.11 ± 0.49
A&E	22.60 ± 0.41	21.47 ± 0.41
Structured Diffusion	09.98 ± 0.29	11.87 ± 0.32
Stable Diffusion	08.85 ± 0.28	09.79 ± 0.29
No majority winner	19.77 ± 0.39	16.76 ± 0.37
DVMP (challenge set)	SynGen (ours)	29.22 ± 0.45	23.55 ± 0.42
A&E	19.83 ± 0.39	19.00 ± 0.39
Structured Diffusion	09.00 ± 0.28	15.56 ± 0.36
Stable Diffusion	09.89 ± 0.29	15.56 ± 0.36
No majority winner	32.06 ± 0.46	26.33 ± 0.44
ABC-6K	SynGen (ours)	33.00 ± 0.47	25.72 ± 0.43
A&E	17.84 ± 0.38	17.28 ± 0.37
Structured Diffusion	13.44 ± 0.34	14.50 ± 0.35
Stable Diffusion	11.72 ± 0.32	14.50 ± 0.35
No majority winner	24.00 ± 0.42	28.00 ± 0.44
Figure 22: Examples given to raters in their instructions. Each example consists of a prompt and two images: A good match (top) and a bad match (bottom) for the concept separation criterion. These examples were accompanied by text explaining why the images are considered a good (or bad) match to the prompt.
Figure 23: A screenshot of the AMT task. The order of images was randomized per HIT. "equally good" and "equally bad" were merged during post-processing into "no winner", to simplify presentation of results.
G.2 Phrases-to-Image Similarity
A common approach to automatically assess text-based image generation is by computing the cosine similarity between an image and prompt, using a vision-language model like CLIP [9]. However, the very challenge we tackle here is rooted in CLIP's failure in establishing correct mapping between syntactic bindings and visual bindings, functioning like a bag-of-words model [10]. As an example, suppose CLIP is prompted with "a blue room with a yellow window". If we present CLIP with an image of a yellow room with a blue window, it may yield a similar score to an image that accurately depicts a blue room with a yellow window.
In an attempt to address this flaw, we segment prompts to phrases containing entity-nouns and their corresponding modifiers (e.g., "a blue room" and "a yellow window"), and compute the similarity between these segmented phrases and the image. We then aggregate the result to a single score by computing the mean. With this approach, we expect CLIP to properly associate the modifiers (e.g., "blue" and "yellow") with the correct entity-noun (i.e., "room" and "window") as there is only one entity-noun in each segment. Unfortunately, this metric achieves relatively low agreement with the majority selection of human evaluation, only  $43.5\%$  of the time, where  $25\%$  is random selection. Despite the low agreement, we note the overall trend of selections of this automatic metric is very similar to the human majority selection. Table 6 shows the results of our automatic evaluation.
"a white bicycle and a spiky bow"
Figure 24: A screenshot of the fine-grained AMT task.
How many objects are described in the text?
How many objects which are described in the text appear in the image?
Looking only at the objects that appear in the text and image, how many attributes are correctly assigned to their object?
How many attributes are mapped to something other than the object that they were assigned to in the text prompt?
□ I'm not confident in my answer
Feel free to leave feedback here (optional)
Table 6: Automatic evaluation of all methods on the three datasets. The table reports scores for concept separation (how well the image matches the prompt) and visual appeal. Values are the fraction of majority vote of three raters, normalized to sum to 100.
Method	DVMP (ours)	ABC-6K	A&E
SynGen (ours)	47.33	41.33	44.63
A&E	27.66	24.33	27.11
Structured Diffusion	12.84	17.84	11.87
Stable Diffusion	12.17	16.50	16.39