SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Text Prompt: a brown backpack and a blue cow
Figure 4: AMT Interface for the image-text alignment evaluation on attribute binding (color).
Rate the matching degree of objects' color attributes between the Image and Text Prompt:
$\bigcirc$  5 - Perfect: all/both objects match their attributes in the text prompt
$\bigcirc$  4 - Good: basic level of alignment
$\bigcirc$  3 - Not okay: merely aligned with the text prompt
$\bigcirc$  2-Bad: not aligned properly with the text prompt
$\bigcirc$  1 - Poor: almost irrelevant to the text prompt
Text Prompt: an oval sink and a rectangular mirror
Figure 5: AMT Interface for the image-text alignment evaluation on attribute binding (shape).
Rate the matching degree of objects' shape attributes between the Image and Text Prompt:
Text Prompt: The glass jar and fluffy ribbon hold the metallic candy on the wooden table
Figure 6: AMT Interface for the image-text alignment evaluation on attribute binding (texture).
Rate the matching degree of objects' texture attributes between the Image and Text Prompt:
Text Prompt: a vase on the right of a cat
Figure 7: AMT Interface for the image-text alignment evaluation on spatial relationships.
Rate the matching degree of objects' spatial layout between the Image and Text Prompt:
$\bigcirc$  5 - Perfect: correct spatial layout
$\bigcirc$  4 - Good: basically correct spatial layout
$\bigcirc$  3 - Not okay: spatial layout not aligned properly with the text
$\bigcirc$  2-Bad: image not aligned properly with the text
$\bigcirc$  1 - Poor: image almost irrelevant to the text prompt
Text Prompt: A boat is sailing on a lake
Figure 8: AMT Interface for the image-text alignment evaluation on non-spatial relationships.
Rate the matching degree of objects' relationship between the Image and Text Prompt:
$\bigcirc$  5 - Perfect: accurate alignment
$\bigcirc$  4 - Good: basic level of alignment
$\bigcirc$  3 - Not okay: action relationship not correct.
$\bigcirc$  2-Bad: image not aligned properly with the text
$\bigcirc$  1 - Poor: image almost irrelevant to the text
Text Prompt: The crisp apple lay beside the rough stone and the silky fabric
Figure 9: AMT Interface for the image-text alignment evaluation on complex compositions.
Rate the overall alignment of Image and Text Prompt:
$\bigcirc$  5 - Perfect: accurate alignment
$\bigcirc$  4 - Good: basic level of alignment
$\bigcirc$  3 - Not okay: ignored key parts
$\bigcirc$  2-Bad: image not aligned properly with the text
$\bigcirc$  1 - Poor: image almost irrelevant to the text
D.1 Quantitative Results of Seen and Unseen Splits
We provide the seen and unseen splits for the test set, where the unseen set consists of attribute-object pairs that do not appear in the training set. The unseen split tends to include more uncommon attribute-object combinations than seen split. The performance comparison of seen and unseen splits for attribute binding is shown in Table 12. Our observations reveal that our model exhibits slightly lower performance on the unseen set than the seen set.
Table 12: Performances of our model on attribute binding (color, shape, and texture) for seen and unseen sets.
Metric	Color	Shape	Texture
Seen	unseen	Seen	unseen	Seen	unseen
CLIP	0.3422	0.3283	0.2926	0.3068	0.3240	0.3219
B-CLIP	0.7716	0.7612	0.7425	0.6752	0.7569	0.6809
B-VQA	0.7192	0.5426	0.5500	0.3356	0.7647	0.3567
mGPT	0.6626	0.6780	0.6381	0.6307	0.6773	0.6580
mGPT-CoT	0.8082	0.8038	0.7510	0.6888	0.8453	0.7412
D.2 MiniGPT-4 Evaluation without Chain-of-Thought
Table 13 shows the additional results of benchmarking on T2I-CompBench of 6 models with MiniGPT-4 without Chain-of-Thought. Results indicate that MiniGPT-4 evaluation without Chain-of-Thought does not strictly align with human evaluation results.
Table 13: mGPT benchmarking on 6 sub-categories in T2I-CompBench.
Model	Color	Shape	Texture	Spatial	Non-spatial	Complex
mGPT	Human	mGPT	Human	mGPT	Human	mGPT	Human	mGPT	Human	mGPT	Human
Stable v1-4 [1]	0.6238	0.6533	0.6130	0.6160	0.6247	0.7227	0.8524	0.3813	0.8507	0.9653	0.8752	0.8067
Stable v2 [1]	0.6476	0.7747	0.6154	0.6587	0.6339	0.7827	0.8572	0.3467	0.8644	0.9827	0.8775	0.8480
Composable v2 [7]	0.6412	0.6187	0.6153	0.5133	0.6030	0.6333	0.8504	0.3080	0.8806	0.8120	0.8858	0.7520
Structured v2 [8]	0.6511	0.7867	0.6198	0.6413	0.6439	0.7760	0.8591	0.3467	0.8607	0.9773	0.8732	0.8333
Attn-Ext v2 [9]	0.6683	0.8240	0.6175	0.6360	0.6482	0.8400	0.8536	0.4027	0.8684	0.9533	0.8725	0.8573
GORS-unbiased (ours)	0.6668	0.8253	0.6399	0.6573	0.6389	0.8413	0.8675	0.4467	0.8845	0.9534	0.8876	0.8654
GORS (ours)	0.6677	0.8320	0.6356	0.7040	0.6709	0.8573	0.8584	0.4560	0.8863	0.9853	0.8892	0.8680
D.3 Reward models to Select Samples for GORS-unbiased
To avoid the bias from selecting samples by evaluation metrics as reward, we introduce new reward models which are different from our proposed evaluation metrics. Specifically, we adopt Grounded-SAM [53] as the reward model for the attribute binding category. We extract the segmentation masks of attributes and their associated nouns separately with Grounded-SAM, and use the Intersection-over-Union (IoU) between the attribute masks and the noun masks together with the grounding mask confidence to represent the attribute binding performance. We apply GLIP-based [54] selection method for spatial relationships. For non-spatial relationships, we adopt BLIP [13] to generate image captions and CLIP [11, 12] to measure the text-text similarity between the generated captions and the input text prompts. For complex compositions, we integrate the 3 aforementioned reward models as the total reward. Those sample selection models are different from the models used as evaluation metrics. The models trained with the new reward models are denoted as GORS-unbiased.
D.4 Scalability of our proposed approach
To demonstrate the scalability of our proposed approach, we introduce additional 700 prompts of complex compositions to form an extended training set of 1,400 complex prompts. The new prompts are generated with the same methodology as described in the appendix B and they are accessible through this link. We conduct 6 experiments to train the models with different training set sizes, i.e., 25 prompts, 275 prompts, 350 prompts, 700 prompts, 1050 prompts, and 1400 prompts. The results
Table 14: Performances of our model on complex compositons on the 3-in-1 metric
ours (25)	ours (275)	ours (350)	ours (700)	ours (1050)	ours (1400)
0.2596	0.3086	0.3299	0.3328	0.3371	0.3504
in Table 14 show the performance of our model grows with the increase of the training set sizes. The results indicate the potential to achieve better performance by scaling up the training set.
D.5 Qualitative Results of Ablation Study
We show the qualitative results of the variants in ablation study in Figure 10. When only CLIP is fine-tuned with LoRA, the generated images do not bind attributes to correct objects (for example, Figure 10 Row. 3 Col. 3 and Row. 6 Col. 3). Noticeable improvements are observed in the generated images when U-Net is fine-tuned by LoRA, particularly when both CLIP and U-Net are finetuned together. Furthermore, we delve into the effect of the threshold for selecting images aligned with text prompts for fine-tuning. A higher threshold value enables the selection of images that are highly aligned with text prompts for finetuning, ensuring that only well-aligned examples are incorporated into the finetuning process. In contrast, a lower threshold leads to the inclusion of misaligned images during finetuning, which can degrade the compositional ability of the finetuned text-to-image models (for example, Figure 10 last two columns in Row. 2).
Figure 10: Qualitative comparison of ablation study on fine-tuning strategy and threshold.
D.6 Qualitative Results and Comparison with Prior Work
Additional results and comparisons are shown in Figure 11 and Figure 12
Figure 11: Qualitative comparison between our approach and previous methods.
Figure 12: Qualitative comparison between our approach and previous methods.
Figure 13: Failure cases of the evaluation metric BLIP-VQA.
E Limitation and Potential Negative Social Impacts
One limitation of our work is the absence of a unified metric for all forms of compositionality. Future research can explore the potential of multimodal LLM to develop a unified metric. Our proposed evaluation metrics are not perfect. As shown by the failure cases in Fig. 13, BLIP-VQA may fail in challenging cases, for example, the objects' shapes are not fully visible in the image, shape's description is uncommon or the objects are not easy to recognize. The UniDet-based evaluation metric is limited to evaluating 2D spatial relationships and we leave 3D spatial relationships for future study. Researchers need to be aware of the potential negative social impact from the abuse of text-to-image models and the biases of hallucinations from image generators as well as pre-trained multimodal models and multimodal LLMs. Future research should exercise caution when working with generated images and LLM-generated content and devise appropriate prompts to mitigate the impact of hallucinations and bias in those models.