SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Color classification. We use the CLIP ViT-L/14 model from OpenAI [31], released under an MIT license, as a zero-shot color classifier. We also test this with the ViT-B/32 model, but find it shows lower human agreement. For each candidate object, the image is cropped down to the bounding box of the object, and the segmentation mask is used to replace all background pixels with gray. We find that this shows higher agreement with human judgment than using the whole image or not replacing the background (Table 4). For the classification, we classify between the 10 candidate colors described above in C.1, with the following class templates:
- "a photo of a [COLOR] [OBJECT]"
- "a photo of a [COLOR]-colored [OBJECT]"
- "a photo of a [COLOR] object"
where [COLOR] is replaced with the color name and [OBJECT] is replaced with the detected object's name. The normalized prompt embeddings for each color are averaged, and the prompt embedding with the highest cosine similarity to the cropped image determines the predicted color.
CLIPScore baseline. We compare our evaluation framework against CLIPScore [15]. This boils down to the cosine similarity between the prompt and image CLIP embeddings, clipped to be at least 0. The original paper multiplies the score by 2.5 to span a wider natural range; since this is has no real impact on comparing CLIPScore values, we simply scale the similarity to be between 0 (orthogonal) and 100 (perfectly aligned).
[15] used the CLIP ViT-B/32 model from [31] available at the time; since newer improved CLIP models have been developed more recently, we test several different potential models and choose the model most aligned with human judgment as a baseline. We compare the ViT-B/32 model against the larger ViT-L/14 model [31], an even larger ViT-H/14 model from OpenCLIP [8], and the EVA-02-CLIP model from [12]. Table 5 displays the results of this comparison. Overall, the OpenCLIP
CLIP
Model	Overall	Single
object	Two
object	Counting	Colors	Position	Attribute
binding	ImageNet-1k
zeroshot
ViT-B/32 [31]	0.773	0.991	0.720	0.604	0.811	0.790	0.723	0.632
ViT-L/14 [31]	0.760	0.985	0.692	0.583	0.796	0.786	0.719	0.753
ViT-H/14 [8]	0.798	0.986	0.828	0.734	0.827	0.682	0.732	0.780
EVA-02-E/14+ [12]	0.600	0.992	0.628	0.612	0.804	0.240	0.322	0.820
GENEVAL	0.834	0.953	0.799	0.823	0.808	0.823	0.799
Interannotator	0.878	0.990	0.872	0.901	0.854	0.797	0.853
Table 5: Human study agreement with CLIPScore using various CLIP models. While the OpenCLIP ViT-H/14 obtains the best overall agreement with human judgment, it is significantly worse at evaluating position, suggesting a qualitative difference in training data compared to [31]. In addition while EVA-02-CLIP has the highest accuracy on ImageNet, this does not appear to transfer to tasks aside from single object recognition.
ViT-H/14 agrees most with human annotators — however, none of the models are consistently better or worse for CLIPScore across all our tasks.
D Human study details
The human study was conducted through Amazon Mechanical Turk. Annotators were asked to answer a series of questions about a displayed image, as enumerated below and shown in Figure 11. There were very limited potential participant risks, if they were to be exposed to an image that was disturbing or not safe for work (NSFW). However, the images we used were sampled from templated prompts which were not in themselves offensive. Furthermore, all images generated by our models or retrieved by CLIP were passed through NSFW filters which would black out any potentially unsafe images.
We annotated 400 images generated from 100 randomly chosen prompts for each of Stable Diffusion v2.1, IF-XL, and CLIP retrieval from LAION-5B, with 5 annotations for each image. Depending on the prompt (which affected the questions asked), each annotation took about 1–2 minutes. Each crowdworker was paid an estimated $15 per hour. In total, across all 6,000 annotations and including Mechanical Turk fees, $3,600 were spent on our human study.
D.1 Full text
The task consists of general instructions and a set of questions specific to each image. For each instance of the task, the “[OBJECT]” in each question is replaced with the names of the objects from the prompt. If the prompt consists of two types of objects (e.g., “a photo of a bird and a skateboard”), questions 2 and 3 refer to each type respectively. If the prompt only has one type of object (e.g., “a photo of four handbags”), question 3 and question 4 are skipped.
Instructions. Thanks for participating in this HIT! You will view an image, potentially generated by an AI image generator. Please answer all of the following questions to the best of your abilities. For each image you may be asked about:
- List of objects: What objects are you able to discern in the image?
- Number of objects: How many objects of a particular type are visible, either fully or partially?
- Color of objects: What color(s) would you consider the object(s) to be?
- Realism: How realistic or structurally correct are the objects?
- Relative position: What is the relative position in the image of one object to another?
- Caption fit: How well does the image correspond to a provided caption?
A few notes:
- Take a look at the examples if any of the questions are unclear to you.
- Complete question 1 before moving on to other questions. If you are unable to recognize a particular object, guess what it is or use a brief description.
- Additional questions may appear based on your previous answers. Make sure to answer all questions shown.
- When asked about color, select at least one color that best matches the true color of the object(s).
- Do not consider color when deciding how realistic an object is; focus on the shape and form compared to real life.
- When asked about position, an object can be both above/below and to the left/right of another object. Mark position from your perspective looking at the 2D image, not from the camera perspective.
- The image may be just a black square, in which case, you should say there are no objects.
- Please answer with care: Some HITs will be checked by hand, and work may be rejected if there are too many errors.
Question 1. Briefly list all the objects visible in the image, separated by commas. If there are no objects in the image, write "none".
Question 2a/3a. How many [OBJECT]s are in the image?
Question 2b/3b. What color of [OBJECT]s are in the image? Choose all that apply.
Question 2c/3c. How realistic are the [OBJECT]s in the image? (1-3) Ignore the color of the object, and focus on any visible defects in the shape or structure.
Question 4a. Is the [OBJECT B] to the left or right of the [OBJECT A]? Provide your answer as a viewer of the image, not from the "camera's perspective".
Question 4b. Is the [OBJECT B] to above or below the [OBJECT A]? Provide your answer as a viewer of the image, not from the "camera's perspective".
Annotation task
Figure 11: Mechanical Turk annotation example.
Question 1: Briefly list all the objects visible in the image, separated by commas. If there are no objects in the image, write "none".
Question 2a: How many teddy bears are in the image?
0 None
Question 2b: What color of teddy bears are in the image? Choose all that apply.
Question 2c: How realistic are the teddy bears in the image? (1-3) Ignore the color of the object, and focus on any visible defects in the shape or structure.
3 Realistic or almost fully realistic
2 Obvious mistakes or deformations
1 Completely unrealistic
Question 5: How well does the image fit the caption below? (1-4) "Elements" refer to individual objects or attributes specified by the text.
a photo of a white teddy bear
The image:
4 Fully captures the meaning of the caption.
3 Mostly captures the meaning, but misses one of many elements in the caption.
2 Misses several or most of the elements in the caption.
1 Is completely unrelated to the caption.