SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kaiyi Huang $^{1}$  Kaiyue Sun $^{1}$  Enze Xie $^{2}$  Zhenguo Li $^{2}$  Xihui Liu $^{1*}$
<sup>1</sup> The University of Hong Kong <sup>2</sup> Huawei Noah's Ark Lab
{huangky, kaiyue}@connect.hku.hk
{xie.enze, li.zhenguo}@huawei.com xihuiliu@eee.hku.hk
Attribute-color
A red book and a yellow vase
Attribute binding
Attribute-shape
An oval coffee table and a rectangular rug
Attribute-texture
A metallic spoon and a glass vase
Spatial relationship
A book on the left of a bird
Figure 1: Failure cases of Stable Diffusion v2 [1]. Our compositional text-to-image generation benchmark consists of three categories: attribute binding (including color, shape, and texture), object relationship (including spatial relationship and non-spatial relationship), and complex compositions.
Non-spatial relationship
A woman is holding a yoga mat
Complex compositions
The sharp blue scissors cut through
the thick white paper
Despite the stunning ability to generate high-quality images by recent text-to-image models, current approaches often struggle to effectively compose objects with different attributes and relationships into a complex and coherent scene. We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). We further propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation and explore the potential and limitations of multimodal LLMs for evaluation. We introduce a new approach, Generative mOdel finetuning with Reward-driven Sample selection (GORS), to boost the compositional text-to-image generation abilities of pretrained text-to-image models. Extensive experiments and evaluations are conducted to benchmark previous methods on T2I-CompBench, and to validate the effectiveness of our proposed evaluation metrics and GORS approach. Project page is available at https://karine-h.github.io/T2I-CompBench/.
Recent progress in text-to-image generation [2, 1, 3, 4, 5, 6] has showcased remarkable capabilities in creating diverse and high-fidelity images based on natural language prompts. However, we observe that even state-of-the-art text-to-image models often fail to compose multiple objects with different attributes and relationships into a complex and coherent scene, as shown in the failure cases of Stable Diffusion [1] in Figure 1. For example, given the text prompt "a blue bench on the left of a green car", the model might bind attributes to the wrong objects or generate the spatial layout incorrectly.
Previous works have explored compositional text-to-image generation from different perspectives, such as concept conjunction [7], attribute binding (focusing on color) [8, 9], and spatial relationship [10]. Most of those works focus on a sub-problem and propose their own benchmarks for evaluating their methods. However, there is no consensus on the problem definition and standard benchmark of compositional text-to-image generation. To this end, we propose a comprehensive benchmark for open-world compositional text-to-image generation, T2I-CompBench. T2I-CompBench consists of three categories and six sub-categories of compositional text prompts: (1) Attribute binding. Each text prompt in this category contains at least two objects and two attributes, and the model should bind the attributes with the correct objects to generate the complex scene. This category is divided into three sub-categories (color, shape, and texture) based on the attribute type. (2) Object relationships. The text prompts in this category each contain at least two objects with specified relationships between the objects. Based on the type of the relationships, this category consists of two sub-categories, spatial relationship and non-spatial relationship. (3) Complex compositions, where the text prompts contain more than two objects or more than two sub-categories mentioned above. For example, a text prompt that describes three objects with their attributes and relationships.
Another challenge is the assessment of compositional text-to-image models. Most previous works evaluate the models by image-text similarity or text-text similarity (between the caption predicted from the generated images and the original text prompts) with CLIPScore [11, 12] or BLIP [13, 14]. However, both metrics do not perform well for compositionality evaluation due to the ambiguity of image captioning and the difficulty of compositional vision-language understanding. To address this challenge, we propose several evaluation metrics for different categories of compositional prompts. We propose disentangled BLIP-VQA for attribute binding evaluation to overcome the ambiguous attribute correspondences, UniDet-based metric for spatial relationship evaluation, and 3-in-1 metric for complex prompts. We further investigate the potential and limitations of multimodal large language models such as MiniGPT-4 [15] with Chain-of-Thought [16] for compositionality evaluation.
Finally, we propose a new approach, Generative mOdel finetuning with Reward-driven Sample selection (GORS), for compositional text-to-image generation. We finetune the state-of-the-art Stable Diffusion v2 [1] model with generated images that highly align with the compositional prompts, where the fine-tuning loss is weighted by the reward which is defined as the alignment score between compositional prompts and generated images. This approach is simple but effective in boosting the model's compositional abilities and can serve as a new baseline for future explorations.
In summary, our contributions are three-folded. (1) We propose a comprehensive benchmark for compositional text-to-image generation which consists of 6,000 prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). (2) We propose evaluation metrics that are specifically designed for compositional text-to-image evaluation. Experiments validate that the proposed evaluation metrics are highly correlated with human perception. (3) We benchmark several previous text-to-image models on our proposed benchmark and evaluation metrics, and propose a simple and effective approach, GORS, for boosting the compositionality of text-to-image models.
Text-to-image generation. Early works [18, 19, 20, 21, 22, 23] explore different network architectures and loss functions based on generative adversarial networks (GAN) [24]. DALL-E [25] achieves impressive results with a transformer and discrete variational autoencoder (VAE) trained on web-scale data. Recently, diffusion models have achieved remarkable success for text-to-image
Table 1: Comparison of compositional text-to-image benchmarks.
Benchmark	Prompts number and coverage	Vocabulary diversity
CC-500 [8]	500 attr bind (color)	196 nouns, 12 colors
ABC-6K [8]	6,000 attr bind (color)	3,690 nouns, 33 colors
Attn-Exct [9]	210 attr bind (color)	24 nouns, 11 colors
HRS-comp [17]	1,000 attr bind (color, size), 2,000 rel (spatial, action)	620 nouns, 5 colors, 11 spatial, 569 actions
T2I-CompBench	3,000 attr bind (color, shape, texture)	2,316 nouns, 33 colors, 32 shapes, 23 textures
2,000 rel (spatial, non-spatial), 1,000 complex	7 spatial rel, 875 non-spatial rel
generation [26, 27, 1, 28, 29]. Current state-of-the-art models such as Stable Diffusion [1] still struggle to compose multiple objects with attributes and relationships in a complex scene. Some recent works attempt to align text-to-image models with human feedback [30, 31]. Concurrent work RAFT [32] proposes reward-ranked fine-tuning to align text-to-image models with certain metrics. Our proposed GORS approach is a simpler finetuning approach that does not require multiple iterations of sample generation and selection.
Compositional text-to-image generation. Researchers have delved into various aspects of compositionality in text-to-image generation to achieve visually coherent and semantically consistent results [7, 8, 33, 10]. Previous work focused on concept conjunction and negation [7], attribute binding with colors [8, 9, 34], and spatial relationships between objects [35, 10]. However, those work each target at a sub-problem, and evaluations are conducted in constrained scenarios. Our work is the first to introduce a comprehensive benchmark for compositional text-to-image generation.
Benchmarks for text-to-image generation. Early works evaluate text-to-image on CUB birds [36], Oxford flowers [37], and COCO [38] which are easy with limited diversity. As the text-to-image models become stronger, more challenging benchmarks have been introduced. DrawBench [3] consists of 200 prompts to evaluate counting, compositions, conflicting, and writing skills. DALL-EVAL [39] proposes PaintSkills to evaluate visual reasoning skills, image-text alignment, image quality, and social bias by 7,330 prompts. HE-T2I [40] proposes 900 prompts to evaluate counting, shapes, and faces for text-to-image. Several compositional text-to-image benchmarks have also been proposed. Park et al. [34] proposes a benchmark on CUB Birds [36] and Oxford Flowers [37] to evaluate the models' ability to generate images with object-color and object-shape compositions. ABC-6K and CC500 [8] benchmarks are proposed to evaluate attribute binding for text-to-image models, but they only focused on color attributes. Concurrent work HRS-Bench [17] is a general-purpose benchmark that evaluates 13 skills with 45,000 prompts. Compositionality is only one of the 13 evaluated skills which is not extensively studied. We propose the first comprehensive benchmark for open-world compositional text-to-image generation, shown in Table 1.
Evaluation metrics for text-to-image generation. Existing metrics for text-to-image generation can be categorized into fidelity assessment, alignment assessment, and LLM-based metrics. Traditional metrics such as Inception Score (IS) [41] and Frechet Inception Distance (FID) [42] are commonly used to evaluate the fidelity of synthesized images. To assess the image-text alignment, text-image matching by CLIP [11] and BLIP2 [14] and text-text similarity by BLIP [13] captioning and CLIP text similarity are commonly used. Some concurrent works leverage the strong reasoning abilities of large language models (LLMs) for evaluation [43, 44]. However, there was no comprehensive study on how well those evaluation metrics work for compositional text-to-image generation. We propose evaluation metrics specifically designed for our benchmark and validate that our proposed metrics align better with human perceptions.
3 T2I-CompBench
Compositionality of text-to-image models refers to the ability of models to compose different concepts into a complex and coherent scene according to text prompts. It includes composing attributes with objects, composing different objects with specified interactions and spatial relationships, and complex compositions. To provide a clear definition of the problem and to build our benchmark, we introduce three categories and six sub-categories of compositionality, attribute binding (including three sub-categories: color, shape, and texture), object relationships (including two sub-categories: spatial relationship and non-spatial relationship), and complex compositions. We generate 1,000 text prompts (700 for training and 300 for testing) for each sub-category, resulting in 6,000 compositional text
prompts in total. We take the balance between seen v.s. unseen compositions in the test set, prompts with fixed sentence template v.s. natural prompts, and simple v.s. complex prompts into consideration when constructing the benchmark. The text prompts are generated with either predefined rules or ChatGPT [45], so it is easy to scale up. Comparisons between our benchmark and previous benchmarks are shown in Table. 1.
3.1 Attribute Binding
A critical challenge for compositional text-to-image generation is attribute binding, where attributes must be associated with corresponding objects in the generated images. We find that models tend to confuse the association between attributes and objects when there are more than one attribute and more than one object in the text prompt. For example, with the text prompt "A room with blue curtains and a yellow chair", the text-to-image model might generate a room with yellow curtains and a blue chair. We introduce three sub-categories, color, shape, and texture, according to the attribute type, and construct 1000 text prompts for each sub-category. For each sub-category, there are 800 prompts with the fixed sentence template "a {adj} {noun} and a {adj} {noun}" (e.g., "a red flower and a yellow vase") and 200 natural prompts without predefined sentence template (e.g., "a room with blue curtains and a yellow chair"). The 300-prompt test set of each sub-category consists of 200 prompts with seen adj-noun compositions (adj-noun compositions appeared in the training set) and 100 prompts with unseen adj-noun compositions (adj-noun compositions not in the training set).
Color. Color is the most commonly-used attribute for describing objects in images, and current text-to-image models often confuse the colors of different objects. The 1,000 text prompts related to color binding are constructed with 480 prompts from CC500 [8], 200 prompts from COCO [38], and 320 prompts generated by ChatGPT.
Shape. We define a set of shapes that are commonly used for describing objects in images: long, tall, short, big, small, cubic, cylindrical, pyramidal, round, circular, oval, oblong, spherical, triangular, square, rectangular, conical, pentagonal, teardrop, crescent, and diamond. We provide those shape attributes to ChatGPT and ask ChatGPT to generate prompts by composing those attributes with arbitrary objects, for example, "a rectangular clock and a long bench".
Texture. Textures are also commonly used to describe the appearance of objects. They can capture the visual properties of objects, such as smoothness, roughness, and granularity. We often use materials to describe the texture, such as wooden, plastic, and rubber. We define several texture attributes and the objects that can be described by each attribute. We generate 800 text prompts by randomly selecting from the possible combinations of two objects each associated with a textural attribute, e.g., "A rubber ball and a plastic bottle". We also generate 200 natural text prompts by ChatGPT.
3.2 Object Relationship
When composing objects in a complex scene, the relationship between objects is a critical factor. We introduce 1,000 text prompts for spatial relationships and non-spatial relationships, respectively.
Spatial relationships. We use "on the side of", "next to", "near", "on the left of", "on the right of", "on the bottom of", and "on the top of" to define spatial relationships. The two nouns are randomly selected from persons (e.g., man, woman, girl, boy, person, etc.), animals (e.g., cat, dog, horse, rabbit, frog, turtle, giraffe, etc.), and objects (e.g., table, chair, car, bowl, bag, cup, computer, etc.). For spatial relationships including left, right, bottom, and top, we construct contrastive prompts by swapping the two nouns, for example, "a girl on the left of a horse" and "a horse on the left of a girl".
Non-spatial relationship. Non-spatial relationships usually describe the interactions between two objects. We prompt ChatGPT to generate text prompts with non-spatial relationships (e.g., "watch", "speak to", "wear", "hold", "have", "look at", "talk to", "play with", "walk with", "stand on", "sit on", etc.) and arbitrary nouns.
Figure 2: Illustration of our proposed evaluation metrics: (a) Disentangled BLIP-VQA for attribute binding evaluation, (b) UniDet for spatial relationship evaluation, and (c) MiniGPT4-CoT as a potential unified metric.
3.3 Complex Compositions
To test text-to-image generation approaches with more natural and challenging compositional prompts in the open world, we introduce 1,000 text prompts with complex compositions of concepts beyond the pre-defined patterns. Regarding the number of objects, we create text prompts with more than two objects, for example, "a room with a blue chair, a black table, and yellow curtains". In terms of the attributes associated with objects, we can use multiple attributes to describe an object (denoted as multiple attributes, e.g., "a big, green apple and a tall, wooden table"), or leverage different types of attributes in a text prompt (denoted as mixed attributes, e.g., the prompt "a tall tree and a red car" includes both shape and color attributes). We generate 250 text prompts with ChatGPT for each of the four scenarios: two objects with multiple attributes, two objects with mixed attributes, more than two objects with multiple attributes, and more than two objects with mixed attributes. Relationship words can be adopted in each scenario to describe the relationships among two or more objects. For each scenario, we split 175 prompts for the training set and 75 prompts for the test set.
4 Evaluation Metrics
Evaluating compositional text-to-image generation is challenging as it requires comprehensive and fine-grained cross-modal understanding. Existing evaluation metrics leverage vision-language models trained on large-scale data for evaluation. CLIPScore [11, 12] calculates the cosine similarity between text features and generated-image features extracted by CLIP. Text-text similarity by BLIP-CLIP [9] applies BLIP [13] to generate captions for the generated images, and then calculates the CLIP text-text cosine similarity between the generated captions and text prompts. Those evaluation metrics can measure the coarse text-image similarity, but fails to capture fine-grained text-image correspondences in attribute binding and spatial relationships. To address those limitations, we propose new evaluation metrics for compositional text-to-image generation, shown in Fig. 2. Concretely, we propose disentangled BLIP-VQA for attribute binding evaluation, UniDet-based metric for spatial relationship evaluation, and 3-in-1 metric for complex prompts. We further investigate the potential and limitations of multimodal large language models such as MiniGPT-4 [15] with Chain-of-Thought [16] for compositionality evaluation.
4.1 Disentangled BLIP-VQA for Attribute Binding Evaluation
We observe that the major limitation of the BLIP-CLIP evaluation is that the BLIP captioning models do not always describe the detailed attributes of each object. For example, the BLIP captioning model might describe an image as "A room with a table, a chair, and curtains", while the text prompt for generating this image is "A room with yellow curtains and a blue chair". So explicitly comparing the text-text similarity might cause ambiguity and confusion.
Therefore, we leverage the visual question answering (VQA) ability of BLIP [13] for evaluating attribute binding. For instance, given the image generated with the text prompt "a green bench
and a red car", we ask two questions separately: "a green bench?", and "a red car?". By explicitly disentangling the complex text prompt into two independent questions where each question contains only one object-attribute pair, we avoid confusion of BLIP-VQA. The BLIP-VQA model takes the generated image and several questions as input and we take the probability of answering "yes" as the score for a question. We compute the overall score by multiplying the probability of answering "yes" for each question. The proposed disentangled BLIP-VQA is applied to evaluate the attribute binding for color, shape, and texture.
4.2 UniDet-based Spatial Relationship Evaluation
Most vision-language models perform poorly in reasoning spatial relationships such as "left" and "right". Therefore, we introduce a detection-based spatial relationship evaluation metric. We first use UniDet [46] to detect objects in the generated image. Then we determine the spatial relationship between two objects by comparing the locations of the centers of the two bounding boxes. Denote the center of the two objects as  $(x_{1},y_{1})$  and  $(x_{2},y_{2})$ , respectively. The first object is on the left of the second object if  $x_{1} < x_{2}$ ,  $|x_{1} - x_{2}| > |y_{1} - y_{2}|$ , and the intersection-over-union (IoU) between the two bounding boxes is below the threshold of 0.1. Other spatial relationships "right", "top", and "bottom" are evaluated similarly. We evaluate "next to", "near", and "on the side of" by comparing the distances between the centers of two objects with a threshold.
4.3 3-in-1 Metric for Complex Compositions Evaluation
Since different evaluation metrics are designed for evaluating different types of compositionality, there is no single metric that works well for all categories. We empirically find that the Disentangled BLIP-VQA works best for attribute binding evaluation, UniDet-baused metric works best for spatial relationship evaluation, and CLIPScore works best for non-spatial relationship evaluation. Thus, we design a 3-in-1 evaluation metric which computes the average score of CLIPScore, Disentangled BLIP-VQA, and UniDet, as the evaluation metric for complex compositions.
4.4 Evaluation with Multimodal Large Language Models
By aligning a pretrained visual encoder with a frozen large language model, multimodal large language models such as MiniGPT-4 [15] have demonstrated great abilities in vision-language cross-modal understanding. We leverage MiniGPT-4 with Chain-of-Thought as an evaluation metric by feeding the generated images to the model and asking two questions: "describe the image" and "predict the image-text alignment score". More information on the prompt design is provided in the appendix. We believe more advanced multimodal LLMs have the potential to be a unified evaluation metric in the future, but the current models exhibit limitations such as inaccurate understanding of images and hallucination issues.
We introduce a simple but effective approach, Generative mOdel finetuning with Reward-driven Sample selection (GORS), to improve the compositional ability of pretrained text-to-image models. Our approach finetunes a pretrained text-to-image model such as Stable Diffusion [1] with generated images that highly align with the compositional prompts, where the fine-tuning loss is weighted by the reward which is defined as the alignment score between compositional prompts and generated images.
Specifically, given the text-to-image model  $p_{\theta}$  and a set of text prompts  $y_{1},y_{2},\dots ,y_{n}$ , we first generate  $k$  images for each text prompt, resulting in  $kn$  generated images  $x_{1},x_{2},\dots ,x_{kn}$ . Text-image alignment scores  $s_1,s_2,\dots ,s_{kn}$  are predicted as rewards. We select the generated images whose rewards are higher than a threshold to fine-tune the text-to-image model. The selected set of samples are denoted as  $\mathcal{D}_s$ . During fine-tuning, we weight the loss with the reward of each sample. Generated images that align with the compositional prompt better are assigned higher loss weights, and vice versa. The loss function for fine-tuning is
$$
\mathcal {L} (\theta) = \mathbb {E} _ {(x, y, s) \in \mathcal {D} _ {s}} \left[ s \cdot \| \epsilon - \epsilon_ {\theta} (z _ {t}, t, y) \| _ {2} ^ {2} \right], \tag {1}
$$