SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Benchmarking on attribute binding (color and shape), with scores unnormalized. Blue represents the proposed metric for the category, and green indicates the human evaluation, applicable to the following Table 3 and Table 4.
Model	Color	Shape
CLIP	B-CLIP	B-VQA-n	B-VQA	mGPT-CoT	Human	CLIP	B-CLIP	B-VQA-n	B-VQA	mGPT-CoT	Human
Stable v1-4 [1]	0.3214	0.7454	0.5875	0.3765	0.7424	0.6533	0.3112	0.7077	0.6771	0.3576	0.7197	0.6160
Stable v2 [1]	0.3335	0.7616	0.7249	0.5065	0.7764	0.7747	0.3203	0.7191	0.7517	0.4221	0.7279	0.6587
Composable v2 [7]	0.3178	0.7352	0.5744	0.4063	0.7524	0.6187	0.3092	0.6985	0.6125	0.3299	0.7124	0.5133
Structured v2 [8]	0.3319	0.7626	0.7184	0.4990	0.7822	0.7867	0.3178	0.7177	0.7500	0.4218	0.7228	0.6413
Attn-Exct v2 [9]	0.3374	0.7810	0.8362	0.6400	0.8194	0.8240	0.3189	0.7209	0.7723	0.4517	0.7299	0.6360
GORS-unbasiéd (ours)	0.3390	0.7667	0.8219	0.6414	0.7987	0.8253	0.3175	0.7149	0.7630	0.4546	0.7263	0.6573
GORS (ours)	0.3395	0.7681	0.8471	0.6603	0.8067	0.8320	0.2973	0.7201	0.7937	0.4785	0.7303	0.7040
where  $(x,y,s)$  is the triplet of the image, text prompt, and reward, and  $z_{t}$  represents the latent features of  $x$  at timestep  $t$ . We adopt LoRA [47] for efficient finetuning.
Evaluated models. We evaluate the performance of 6 text-to-image models on T2I-CompBench. Stable Diffusion v1-4 and Stable Diffusion v2 [1] are text-to-image models trained on large amount of image-text pairs. Composable Diffusion [7] is designed for conjunction and negation of concepts for pretrained diffusion models. Structured Diffusion [8] and Attend-and-Excite [9] are designed for attribute binding for pretrained diffusion models. We re-implement those approaches on Stable Diffusion v2 to enable fair comparisons. GORS is our proposed approach which finetunes Stable Diffusion v2 with selected samples and their rewards. Since calculating the rewards for GORS with the automatic evaluation metrics can lead to biased results, we also provide alternative reward models (Appendix D.3) which are different from the evaluation metrics, which is denoted as GORS-unbiased.
Implementration details. Please find the implementation details in the appendix.
6.2 Evaluation Metrics
We generate 10 images for each text prompt in T2I-CompBench for automatic evaluation.
Previous metrics. CLIPScore [11, 12] (denoted as CLIP) calculates the cosine similarity between text features and generated-image features extracted by CLIP. BLIP-CLIP [9] (denoted as B-CLIP) applies BLIP [13] to generate captions for the generated images, and then calculates the CLIP text-text cosine similarity between the generated captions and text prompts. BLIP-VQA-naive (denoted as B-VQA-n) applies BLIP VQA to ask a single question (e.g., a green bench and a red car?) with the whole prompt.
Our proposed metrics. Disentangled BLIP-VQA (denoted as  $B$ -VQA) is our proposed evaluation metric for attribute binding. UniDet is our proposed UniDet-based spatial relationship evaluation metric. 3-in-1 computes the average score of CLIPScore, Disentangled BLIP-VQA, and UniDet, as the evaluation metric for complex compositions. MiniGPT4-Chain-of-Thought (denoted as mGPT-CoT) serves as a potential unified metric for all types of compositional prompts based on multimodal LLM.
Human evaluation. For human evaluation of each sub-category, we randomly select 25 prompts and generate 2 images per prompt, resulting in 300 images generated with 150 prompts per model in total. The testing set includes 300 prompts for each sub-category, resulting in 1800 prompts in total. The prompt sampling rate for human evaluation is  $8.33\%$ . We utilize Amazon Mechanical Turk and ask three workers to score each generated-image-text pair independently based on the image-text alignment. The worker can choose a score from  $\{1,2,3,4,5\}$  and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers.
Table 3: Benchmarking on attribute binding (texture) and spatial relationship.
Model	Texture	Spatial
CLIP	B-CLIP	B-VQA-n	B-VQA	mGPT-CoT	Human	CLIP	B-CLIP	UniDet	mGPT-CoT	Human
Stable v1-4 [1]	0.3081	0.7111	0.6173	0.4156	0.7836	0.7227	0.3142	0.7667	0.1246	0.8338	0.3813
Stable v2 [1]	0.3185	0.7240	0.7054	0.4922	0.7851	0.7827	0.3206	0.7723	0.1342	0.8367	0.3467
Composable v2 [7]	0.3092	0.6995	0.5604	0.3645	0.7588	0.6333	0.3001	0.7409	0.0800	0.8222	0.3080
Structured v2 [8]	0.3167	0.7234	0.7007	0.4900	0.7806	0.7760	0.3201	0.7726	0.1386	0.8361	0.3467
Attn-Exct v2 [9]	0.3171	0.7206	0.7830	0.5963	0.8062	0.8400	0.3213	0.7742	0.1455	0.8407	0.4027
GORS-unbiased (ours)	0.3216	0.7291	0.7778	0.6025	0.7985	0.8413	0.3237	0.7882	0.1725	0.8241	0.4467
GORS (ours)	0.3233	0.7315	0.7991	0.6287	0.8106	0.8573	0.3242	0.7854	0.1815	0.8362	0.4560
Table 4: Benchmarking on the non-spatial relationship and complex compositions.
Model	Non-spatial	Complex
CLIP	B-CLIP	mGPT-CoT	Human	CLIP	B-CLIP	3-in-1	mGPT-CoT	Human
Stable v1-4 [1]	0.3079	0.7565	0.8170	0.9653	0.2876	0.6816	0.3080	0.8075	0.8067
Stable v2 [1]	0.3127	0.7609	0.8235	0.9827	0.3096	0.6893	0.3386	0.8094	0.8480
Composable v2 [7]	0.2980	0.7038	0.7936	0.8120	0.3014	0.6638	0.2898	0.8083	0.7520
Structured v2 [8]	0.3111	0.7614	0.8221	0.9773	0.3084	0.6902	0.3355	0.8076	0.8333
Attn-Exct v2 [9]	0.3109	0.7607	0.8214	0.9533	0.2913	0.6875	0.3401	0.8078	0.8573
GORS-unbiased (ours)	0.3158	0.7641	0.8353	0.9534	0.3137	0.6888	0.3470	0.8122	0.8654
GORS (ours)	0.3193	0.7619	0.8172	0.9853	0.2973	0.6841	0.3328	0.8095	0.8680
6.3 Quantitative and Qualitative Evaluation
The quantitative evaluation results are reported on attribute binding (color), attribute binding (shape) (Table 2), attribute binding (texture), spatial relationship (Table 3), non-spatial relationship, and comprehensive compositions (Table 4), respectively. Qualitative results are shown in Figure 3.
Comparisons across evaluation metrics. Previous evaluation metrics, CLIP and BLIP-CLIP, predict similar scores for different models and cannot reflect the differences between models. Our proposed metrics, BLIP-VQA for attribute binding, UniDet for spatial relationship, CLIP for non-spatial relationship, and 3-in-1 for complex compositions, highly align with the human evaluation scores.
Comparisons across text-to-image models. (1) Stable Diffusion v2 consistently outperforms Stable Diffusion v1-4 in all types of compositional prompts and evaluation metrics. (2) Although Structured Diffusion built upon Stable Diffusion v1-4 shows great performance improvement in attribute binding as reported in Feng et al. [8], Structured Diffusion built upon Stable Diffusion v2 only brings slight performance gain upon Stable Diffusion v2. It indicates that boosting the performance upon a better baseline of Stable Diffusion v2 is more challenging. (3) Composable Diffusion built upon Stable Diffusion v2 does not work well. A similar phenomenon was also observed in previous work [9] that Composable Diffusion often generates images containing a mixture of the subjects. In addition, Composable Diffusion was designed for concept conjunctions and negations so it is reasonable that it does not perform well in other compositional scenarios. (4) Attend-and-Excite built upon Stable Diffusion v2 improves the performance in attribute binding. (5) Previous methods Composable Diffusion [7], Structure Diffusion [8] and Attend-and-Excite [9] are designed for concept conjunction or attribute binding, so they do not result in significant improvements in object relationships. (6) Our proposed approach, GORS, outperforms previous approaches across all types of compositional prompts, as demonstrated by the automatic evaluation, human evaluation, and qualitative results. The evaluation results of GORS-unbiased and GORS significantly exceed the baseline Stable v2. Besides, GORS-unbiased achieves on-par performance with GORS, indicating that our proposed approach is insensitive to the reward model used for selecting samples, and that the proposed approach works well as long as high-quality samples are selected.
Comparisons across compositionality categories. According to the human evaluation results, spatial relationship is the most challenging sub-category for text-to-image models, and attribute binding (shape) is also challenging. Non-spatial relationship is the easiest sub-category.
Figure 3: Qualitative comparison between our approach and previous methods.
6.4 Human Correlation of the Evaluation Metrics
We calculate Kendall's tau  $(\tau)$  and Spearman's rho  $(\rho)$  to evaluate the ranking correlation between automatic evaluation and human evaluation. The scores predicted by each evaluation metric are normalized to 0-1 for better comparison.
The human correlation results are illustrated in Table 5. The results verify the effectiveness of our proposed evaluation metrics, BLIP-VQA for attribute binding, UniDet-based metric for spatial relationships, CLIPScore for non-spatial relationships, and 3-in-1 for complex compositions. MiniGPT4-CoT does not perform well in terms of correlation with human perception, but we believe that the multimodal LLM-based evaluation metrics have the potential to become a unified evaluation metric in the future. We leave explorations on boosting the performance of multimodal LLM for compositional prompts to future work.
6.5 Ablation study
We conduct ablation studies with the attribute binding (color) sub-category on our proposed GORS approach and MiniGPT4-CoT evaluation metric.
Table 5: The correlation between automatic evaluation metrics and human evaluation. Our proposed metrics demonstrate a significant improvement over existing metrics in terms of Kendall's  $\tau$  and Spearmanr's  $\rho$ .
Metric	Attribute-color	Attribute-shape	Attribute-texture	Spatial rel	Non-spatial rel	Complex
τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)
CLIP	0.1938	0.2773	0.0555	0.0821	0.2890	0.4008	0.2741	0.3548	0.2470	0.3161	0.0650	0.0847
B-CLIP	0.2674	0.3788	0.1692	0.2413	0.2999	0.4187	0.1983	0.2544	0.2342	0.2964	0.1963	0.2755
B-VQA-n	0.4602	0.6179	0.2280	0.3180	0.4227	0.5830	-	-	-	-	-	-
B-VQA	0.6297	0.7958	0.2707	0.3795	0.5177	0.6995	-	-	-	-	-	-
UniDet	-	-	-	-	-	-	0.4756	0.5136	-	-	-	-
3-in-1	-	-	-	-	-	-	-	-	-	-	0.2831	0.3853
mGPT	0.1197	0.1616	0.1282	0.1775	0.1061	0.1460	0.0208	0.0229	0.1181	0.1418	0.0066	0.0084
mGPT-CoT	0.3156	0.4151	0.1300	0.1805	0.3453	0.4664	0.1096	0.1239	0.1944	0.2137	0.1251	0.1463
mGPT-CLIP	0.2301	0.3174	0.0695	0.0963	0.2004	0.2784	0.1478	0.1950	0.1507	0.1942	0.1457	0.2014
Table 6: Ablation studies on fine-tuning strategy and threshold.
Metric	FT U-Net only	FT CLIP only	Half threshold	0 threshold	GORS (ours)
B-VQA	0.6216	0.5507	0.6157	0.6130	0.6570
mGPT-CoT	0.7840	0.7663	0.7886	0.7879	0.7899
Finetuning strategy. Our approach finetunes both the CLIP text encoder and the U-Net of Stable Diffusion with LoRA [47]. We investigate the effects of finetuning CLIP only and U-Net only with LoRA. As shown in Table 6, our model which finetunes both CLIP and U-Net performs better.
Threshold of selecting samples for finetuning. Our approach fine-tunes Stable Diffusion v2 with the selected samples that align well with the compositional prompts. We manually set a threshold for the alignment score to select samples with higher alignment scores than the threshold for fine-tuning. We experiment with setting the threshold to half of its original value, and setting the threshold to 0 (i.e., use all generated images for finetuning with rewards, without selection). Results in Table 6 demonstrate that half threshold and zero threshold will lead to worse performance.
MiniGPT4 without Chain-of-Thought. We compare the evaluation metric of MiniGPT-4 with and without Chain-of-Thought, and with MiniGPT-4 for captioning and CLIP for text-text similarity, denoted as mGPT-CLIP. As shown in Table 5, Chain-of-Thought improves the human correlation of MiniGPT4-based evaluation by a large margin.
7 Conclusion and Discussions
We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 prompts from 3 categories and 6 sub-categories. We propose new evaluation metrics and an improved baseline for the benchmark, and validate the effectiveness of the metrics and method by extensive evaluation. One limitation is that we do not have a unified evaluation metric for all categories of the benchmark. Please refer to appendix for failure cases of our proposed metrics. Additionally, our dataset primarily focuses on 2D spatial relationships. We suggest that assessing 3D spatial relationships could integrate depth maps into our existing UniDet-based evaluation metric, which we leave for future work. When studying generative models, researchers need to be aware of the potential negative social impact, for example, it might be abused to generate fake news. We also need to be aware of the bias from the image generators and the evaluation metrics based on pretrained multimodal models. More discussions on the limitation and potential negative social impact are provided in the appendix.
This work is supported in part by HKU Startup Fund, HKU Seed Fund for PI Research - Basic Research, HKU Seed Fund for PI Research - Translational and Applied Research, and HKU FinTech Academy R&D Funding Scheme.
[51] https://github.com/huggingface/diffusers/blob/main/examples/text_to_image.
[52] Ilya Loshchilov and Frank Hutter. "Decoupled weight decay regularization". In: arXiv preprint arXiv:1711.05101 (2017).
[53] Shilong Liu et al. "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection". In: arXiv preprint arXiv:2303.05499 (2023).
[54] Liunian Harold Li et al. "Grounded language-image pre-training". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 10965-10975.
Evaluation details. We employ pre-trained models for our proposed evaluation metrics. For BLIP-VQA, we utilize the BLIP w/ ViT-B and CapFilt-L [13] pretrained on image-text pairs and fine-tuned on VQA. We employ the UniDet [46] model Unified_learned_COIM_RS200_6x+2x trained on 4 large-scale detection datasets (COCO [38], Objects365 [48], OpenImages [49], and Mapillary [50]). For CLIPScore, we use the "ViT-B/32" pretrained CLIP model [12, 11]. Finally, for MiniGPT4-CoT, we utilize the Vicuna 13B of MiniGPT4 [15] variant with a temperature setting of 0.7 and a beam size of 1.
Training details. We implement our proposed FT-SSWL upon the codebase of diffusers [51] (Apache License), and finetune the self-attention layers of the CLIP text encoder and the attention layers of U-net using LoRA [47]. The model is trained by AdamW optimizer [52] with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 1e - 8$ , and weight decay of 0.01. The batch size is 5. The model is trained on 832GB NVIDIA v100 GPUs, for 50000-100000 steps.
B T2I-CompBench Dataset Construction
This section provides the details of the prompts that ChatGPT uses for generating the text prompts in T2I-CompBench. The text prompts in T2I-CompBench is available at this link.
Color. The prompt for ChatGPT is: Please generate prompts in the format of "a {adj} {noun} and a {adj} {noun}" by using the color adj., such as "a green bench and a red car".
Shape. (1) For fixed sentence template, the prompt for ChatGPT is: Please generate prompts in the format of "a {adj} {noun} and a {adj} {noun}" by using the shape adj.: long, tall, short, big, small, cubic, cylindrical, pyramidal, round, circular, oval, oblong, spherical, triangular, square, rectangular, conical, pentagonal, teardrop, crescent, and diamond. (2) For natural prompts, the prompt for ChatGPT is: Please generate objects with shape adj. in a natural format by using the shape adj.: long, tall, short, big, small, cubic, cylindrical, pyramidal, round, circular, oval, oblong, spherical, triangular, square, rectangular, conical, pentagonal, teardrop, crescent, and diamond.
Texture. (1) We generate 200 natural text prompts by ChatGPT with the following prompt: Please generate objects with texture adj. in a natural format by using the texture adj.: rubber, plastic, metallic, wooden, fabric, fluffy, leather, glass. (2) Besides the ChatGPT-generated text prompts, we also provide the predefined texture attributes and objects that can be described by each texture, as shown in Table 7. We generate 800 text prompts by randomly selecting from the possible combinations of two objects each associated with a textural attribute, e.g., "A rubber ball and a plastic bottle".
Table 7: Textural attributes and associated objects to construct the attribute-texture prompts.
Textures	Objects
Rubber	band, ball, tire, gloves, sole shoes, eraser, boots, mat
Plastic	Bottle, bag, toy, cutlery, chair, phone case, container, cup, plate
Metallic	car, jewelry, watch, keychain, desk lamp, door knob, spoon, fork, knife, key, ring, necklace, bracelet, earring
Wooden	chair, table, picture frame, toy, jewelry box, door, floor, chopsticks, pencils, spoon, knife
Fabric	bag, pillow, curtain, shirt, pants, dress, blanket, towel, rug, hat, scarf, sweater, jacket
Fluffy	pillow, blanket, teddy bear, rug, sweater, clouds, towel, scarf, hat
Leather	jacket, shoes, belt, bag, wallet, gloves, chair, sofa, hat, watch
Glass	bottle, vase, window, cup, mirror, jar, table, bowl, plate
Non-spatial relation. The prompt for ChatGPT is: Please generate natural prompts that contain subjects and objects by using relationship words such as wear, watch, speak, hold, have, run, look at, talk to, jump, play, walk with, stand on, and sit on.
Complex. (1) For 2 objects with mixed attributes, the prompt for ChatGPT is: Please generate natural compositional phrases, containing 2 objects with each object one adj. from {color, shape, texture} descriptions and spatial (left/right/top/bottom/next to/near/on side of) or non-spatial relationships. (2) For 2 objects with multiple attributes, the prompt for ChatGPT is: Please generate natural compositional phrases, containing 2 objects with several adj. from {color, shape, texture} descriptions and spatial (left/right/top/bottom/next to/near/on side of) or non-spatial relationships. (3) For multiple
objects with mixed attributes, the prompt for ChatGPT is: Please generate natural compositional phrases, containing multiple objects (number>2) with each one adj. from {color, shape, texture} descriptions and spatial (left/right/top/bottom/next to/near/on side of) non-spatial relationships. (4) For multiple objects with multiple attributes, the prompt for ChatGPT is: Please generate natural compositional phrases, containing multiple objects (number>2) with several adj. from {color, shape, texture} descriptions and spatial (left/right/top/bottom/next to/near/on side of) or non-spatial relationships.
C.1 Prompts for MiniGPT4-CoT and MiniGPT4 Evaluation
MiniGPT4-Chain-of-Thought. In this part, we detail the prompts used for the MiniGPT4-CoT evaluation metric. For each sub-category, we ask two questions in sequence: "describe the image" and "predict the image-text alignment score". Specifically, Table 8 shows the MiniGPT4-CoT prompts for evaluating attribute binding (color, shape, texture). Table 9, Table 10, and Table 11 demonstrate the prompt templates used for spatial relationships, non-spatial relationships, and complex compositions, respectively.
Table 8: Prompts details for mGPT4-CoT evaluation on attribute binding.
Describe	You are my assistant to identify any objects and their color (shape, texture) in the image. Briefly describe what it is in the image within 50 words.
Predict	According to the image and your previous answer, evaluate if there is {adj.+noun} in the image. Give a score from 0 to 100, according the criteria:
100: there is {noun}, and {noun} is {adj}.
75: there is {noun}, {noun} is mostly {adj}.
20: there is {noun}, but it is not {adj}.
10: no {noun} in the image.
Provide your analysis and explanation in JSON format with the following keys: score (e.g., 85), explanation (within 20 words).
Table 9: Prompts details for mGPT4-CoT evaluation on spatial relationship.
Describe	You are my assistant to identify objects and their spatial layout in the image.
Briefly describe the image within 50 words.
Predict	According to the image and your previous answer, evaluate if the text "#xxx" is correctly portrayed in the image.
Give a score from 0 to 100, according the criteria:
100: correct spatial layout in the image for all objects mentioned in the text.
80: basically, spatial layout of objects matches the text.
60: spatial layout not aligned properly with the text.
40: image not aligned properly with the text.
20: image almost irrelevant to the text.
Provide your analysis and explanation in JSON format with the following keys: score (e.g., 85), explanation (within 20 words).
Table 10: Prompts details for mGPT4-CoT evaluation on non-spatial relationship.
Describe	You are my assistant to identify the actions, events, objects and their relationships in the image.
Briefly describe the image within 50 words.
Predict	According to the image and your previous answer, evaluate if the text "#xxx" is correctly portrayed in the image.
Give a score from 0 to 100, according the criteria:
100: the image accurately portrayed the actions, events and relationships between objects described in the text.
80: the image portrayed most of the actions, events and relationships but with minor discrepancies.
60: the image depicted some elements, but action relationships between objects are not correct.
40: the image failed to convey the full scope of the text.
20: the image did not depict any actions or events that match the text.
Provide your analysis and explanation in JSON format with the following keys: score (e.g., 85), explanation (within 20 words).
MiniGPT-4 without Chain-of-Thought To guide miniGPT4 in addressing specific compositional problems, we utilize predefined prompts that prompt miniGPT4 to provide a score ranging from 0
Table 11: Prompts details for mGPT4-CoT evaluation on complex compositions.
Describe	You are my assistant to evaluate the correspondence of the image to a given text prompt.
Briefly describe the image within 50 words, focus on the objects in the image and their attributes (such as color, shape, texture), spatial layout and action relationships.
Predict	According to the image and your previous answer, evaluate how well the image aligns with the text prompt: {xxx}.
Give a score from 0 to 100, according the criteria:
100: the image perfectly matches the content of the text prompt, with no discrepancies.
80: the image portrayed most of the actions, events and relationships but with minor discrepancies.
60: the image depicted some elements in the text prompt, but ignored some key parts or details.
40: the image did not depict any actions or events that match the text.
20: the image failed to convey the full scope in the text prompt.
Provide your analysis and explanation in JSON format with the following keys: score (e.g., 85), explanation (within 20 words).
to 100. For attribute binding, we focus on the presence of specific objects and their corresponding attributes. We utilize a prompt template such as "Is there {object} in the image? Give a score from 0 to 100. If {object} is not present or if {object} is not {color/shape/texture description}, give a lower score." We leverage this question for each noun phrase in the text and compute the average score. For the spatial relationships, non-spatial relationships, and complex compositions, we employ a more general prompt template such as "Rate the overall alignment between the image and the text prompt {prompt}. Give a score from 0 to 100."
C.2 Human Evaluation
We conducted human evaluations on Amazon Mechanical Turk (AMT). Specifically, we ask the annotators to rate the alignment between a generated image and the text prompt used to generate the image. Figure 4, 5, 6, 7, 8, 9 show the interfaces for human evaluation over the 6 sub-categories. We randomly sample 25 prompts from each sub-category and each model, and then randomly select 2 images per prompt. In total, we gather 1,800 text-image pairs for human evaluation experiments. Each image-text pair is rated by 3 human annotators with a score from 1 to 5 according to the image-text alignment. The estimated hourly wage paid to each participant is 9 USD. We spend 270 USD in total on participant compensation.