Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?
Authors: Xingyu Fu
Muyu He
Yujie Lu
William Yang Wang
Dan Roth
Venue: COLM (2024)
Keywords: text to image generation, alignment with reality, commonsense reasoning
TL;DR: This paper finds that text-to-image generation models do not align with the reality, in real world commonsense.

Abstract: We present a novel task and benchmark for evaluating the ability of text-to-image(T2I) generation models to produce images that align with commonsense in real life, which we call Commonsense-T2I. Given two adversarial text prompts containing an identical set of action words with minor differences, such as *a lightbulb without electricity* vs. *a lightbulb with electricity*, we evaluate whether T2I models can conduct visual-commonsense reasoning, e.g. produce images that fit *The lightbulb is unlit* vs. *The lightbulb is lit* correspondingly.
Commonsense-T2I presents an adversarial challenge, providing pairwise text prompts along with expected outputs.
The dataset is carefully hand-curated by experts and annotated with fine-grained labels, such as commonsense type and likelihood of the expected outputs, to assist analyzing model behavior. We benchmark a variety of state-of-the-art (sota) T2I models and surprisingly find that, there is still a large gap between image synthesis and real life photos--even the DALL-E 3 model could only achieve 48.92% on Commonsense-T2I, and the stable diffusion XL model only achieves 24.92% accuracy. Our experiments show that GPT-enriched prompts cannot solve this challenge, and we include a detailed analysis about possible reasons for such deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation benchmark for T2I commonsense checking, fostering advancements in real life image generation.

Published as a conference paper at COLM 2024
Commonsense-T2l Challenge: Can Text-to-lmage Genera- tion Models Understand Commonsense?
Xingyu Fu F Muyu Hel Yujie Lušt, William Yang Wang Dan Roth
University of Pensylvania University of California, Santa Barbara
(xingyuf2, muyuhe, Kanroth)@sean upenn.edu. (yujielu, wangwi vangwilliamyang Sechsaannneuuue....oooooo @ucsb. edu https://zeyofu. gi weriii///ooonn DC/ooonneeeeeT22
DALL-E 3
Stable Diffusion XL
Openjourney v4
Playground v2.5
prompt: lightbulb without electricity
Figure 1: An example prompt in Commonsense-T2I and failure cases from DALL-E3 (Betker et al., 2023), Stable Diffusion XL (Rombach et al., 2022), Openjourney v4, and Playground v2.5 (Li et al., 2024). The expected output for the prompt is "The lightbulb is unlit".
Abstract
We present a novel task and benchmark for evaluating the ability of text-to- image([2I) generation models to produce images that align with common- sense in real life, which we call Commonsense-122 Given two adversarial text prompts containing an identical set of action words with minor dif- ferences, such as "a lightbulb wi without electricity" vs. "a lightbulb with electricity" we evaluate whether T2I models can conduct visual- commonsense reasoning, e.g. produce images that fit "The lightbulb is unlit" vs. "The lightbulb is lit" correspondingly. Commonsense-T2r presents adversarial challenge, providing pairwise text prompts along with expected outputs. The dataset carefully hand-curated by experts and annotated with fine-grained labels, such commonsense type and likelihood of the expected outputs, to assist analyzing model behavior. We benchmark variety of state-of-the-art (sota) T2I models and surpris- ingly find that, there is still large gap between image synthesis and real life photos-even the DALL-E model could only achieve 48.92% on Commonsense-T2 T21,, and the stable diffusion XL model only achieves 24.92% accuracy. Our experiments show that GPT-enriched prompts cannot solve this challenge, and we include detailed analysis about possible reasons for such deficiency. We aim for Commonsense-T21 to serve as a high-quality eval-
These authors contributed equally this work.

---

Published as a conference paper at COLM 2024
uation benchmark for T2I commonsense checking, fostering advancements in real life image generation.
1 Introduction
Recent advances in generative modeling have allowed text-to-image (T2I) synthesis to achieve drastic performance improvements (Ramesh et al., 2021; 2022; Rombach et al., 2022; Betker et al., 2023; Li et al., 2024). While it seems that we can realize complete transition between a text prompt and an image, an image is worth a thousand words, and it is inevitable to lose information between the transition, due to the simplicity nature of language. Therefore, it is not possible to provide a text prompt to T2I model that covers every single detail about an image- the T2T model must try to understand the text prompt first and self-imagine a real life scenario that contains every missing piece of detail, before generating the required images. As shown in Figure 1, the visual differences of "lightbulb" between "A lightbulb without electricity" and "A lightbulb with electricity" are blatantly obvious - but will the same still be true for machines? At least current T2I models Betker et al. (2023); Rombach et al. (2022); Li et al. (2024) fail to reason the commonsense for this example, namely "lightbulb is unlit wi thout electricity".
We observe that many samples in existing T2I generation evaluations are straightforward composition of objects and their attributes, e.g. "A red car and a white sheep" (Saharia et al., 2022; Park et al., 2021; Cho et al., 2023; Feng et al., 2022). They focus on understanding of object-related and attribute-related tokens the prompts, such as size, color, object, and shape, and fail to cover complicated commonsense reasoning required scenarios. There- fore, it remains unknown whether current generative Al systems can reach human-level intelligence and generate images that align with commonsense in reality, using existing evaluations.
In this paper, we specifically focus the following question: can text-to-image models generate images that align with commonsense in reality? Motivated by this, we propose novel evaluation task, called Commonsense-T21, for measuring commonsense reasoning capabilities in generative models. As illustrated in Figure 2, Commonsense-T21 presents a high-quality expert-curated test set, with each data sample containing two adversarial text prompts, their corresponding expected output descriptions, likelihood score for each expected output, and commonsense category. We design the prompts in a pairwise format that both prompts contain an identical set of action words with minor differences, ordered in such a way that the images must show noticeable differences to align with commonsense in real life. To perform well on Commonsense-T22, T21 models must not only encode the su- perficial meaning of each token in the prompts, but also be able to synthesize commonsense reasoning across the two modalities.
One natural question is how to conduct commonsense reasoning assessment on T2I models. Most metrics evaluate the models on image-text alignment (Radford et al., 2021; Hessel et al., 2021), focusing on divers subjects including fidelity (Heusel et al., 2017; Jayasumana et al., 2023), faithfulness (Hu al., 2023), and compositionality (Lu et al., 2023c; Chen, 2023). None of them examines commonsense reasoning in generative models. While some recent methods evaluate models by human feedback (Lu et al., 2024), they are effort-demmaninng do not focus on commonsense, and only include relative comparison without gold answers. In this paper, we present an automatic evaluation pipeline using our collected expected output descriptions and multimodal large language models (LLMs) (OpenAT, 2023; Team et al., 2023), that is tested to align well with human perceptions under our evaluation metric.
The primary purpose of Commonsense- T2I is to support commonsense probing tasks for T2T models. We experiment with a variety of generative models including Stable Diffusion models, Playground v2.5, Openjourney v4, and DALL-E 3(Rombach et al., 2022; Betker et al., 2023; Li et al., 2024). Surprisingly, the state of the art (sota) DALL-E model 3 only achieves 48.92% accuracy, and all other models hover around 15-30% accuracy. Our findings indicate that the commonsense reasoning capabilities have not emerged in existing T2I models. Additional experiments($3.3) show that GPT-revised enriched prompts cannot
2

---

Published as a conference paper at COLM 2024
P1: balloon filled with helium in bedroom.
P2: A balloon filled with the bedroom.
D1: The balloon floating.
D1: The balloon stays on the floor.
DALL-E:
Playground 2.5
Physical Laws
Stable Diffusion XL
Figure 2: Illustration of one data example from Commonsense-T21, where PL, P2 are pairwise prompts and D1, D2 are expected output description($2), along with selected generated images from DALL-E 3, Playground v2.5, and Stable Diffusion XL. More examples in $3.3
solve the Commonsense-T21 challenge, and we include detailed analysis on possible reasons for such deficiency across all the T2I models.
In summary, our contributions are threefold. (1) We propose a high-quality expert-annotated benchmark for evaluating commonsense reasoning required in text-to-image generation. (2) We propose an automatic evaluation pipeline using multimodal LLMs for the task, and show that it highly correlated with human evaluation.
(3) We benchmark wide range of T2I models on Commonsense-722 and show that there is still a huge gap between all current models and human level intelligence, along with detailed analyses. We hope that Commonsense-T22 will stimulate the community to help T2I models catch up with human-level commonsense reasoning capabilities, fostering further progress in the field.
2 The T2 Benchmark
Our goal is to faithfully evaluate whether T2T models understand commonsense. We introduce a novel benchmark, Tommonsense--2I, designed to enable both quantitative and qualitative evaluation of the real-life commonsense reasoning capabilities of T21 models. We unfold this section by illustrating the overall design of Commonsense-T2 ($2.1) and discussing its unique features. Then we provide an in-depth explanation of the data curation process (52.2). We further present the evaluation metric designed for the t5s.3)..3).
2.1 Overview of Commonsense-T2r
Commonsense-T21 comprises of 150 manually curated examples, where each example has pair of adversarial prompts: P1 and Pz, their corresponding expected output descriptions: D, and D2, likelihood scores for each output to happen, and commonsense category. Complete data example is in A.1. A data sample satisfies the Commonsense-T22 criteria if and only if:

---

Published as a conference paper at COLM 2024
Pj and P2 have the same set of action words but different subjects or adjectives.
P1 and P2 have the same subjects but different action words.
Dj and D2 are completely contrastive and cannot co-exist in one image.
will lead D1 and P2 will lead to D2 in daily life under common sense.
Category
Examples
Percent
A glass of water fell on the floor: spilled water.
Physical Laws
32.7
A hot cup of water in winter: steam rises.
A man at a wedding: in suit, looking cheerful.
Human Practices
30.0
A person eating burger: eating with hands.
Oak trees during winter: no leaves on branches.
Biological Laws
11.3
A wheat field in spring: green field.
A small bag of party balloons for sale: balloons are flat.
Daily Items
14.0
A phone with a drained battery: dark screen.
A peacock attracting mate: spreading feathers.
Animal Behaviors
12.0
A penguin sliding on ice: sliding on its belly.
Table Commonsense knowledge categories and percentage of data samples.
2.2 Dataset Collection Process
The Commonsense-T2 T2I dataset is entirely hand-curated by experts. We first decide the cate- gories of commonsense knowledge required for text-to-image generation, and then use GPT- 4 (Brown et al., 2020) to help generate multiple examples requiring cisual-commonsense knowledge as inspirations, and manually curate each test sample of pairwise prompts and expected outputs.
Commonsense Category. Commonsense knowledge is huge portion of human experience, encompassing knowledge about the spatial, physical, social, temporal, and psychological aspects of typical everyday life (Liu & Singh, 2004), and widely studied in Al research (Bosselut et al., 2019; Zellers et al., 2019). To build Commonsense-T2,, we manually select five categories of knowledge that naturally require visual commonsense understanding, as illustrated in Table 1.
Inspiration Generation. We prompt the GPT-4-turbo model iteratively to generate mas- sive pool of specific examples as inspirations for each commonsense category. Specifically, we first use GPT-4 to generate natural sentence examples given a commonsense category, e.g. "butter melts when heating" given category physical laws. Additionally, we require GPT to only provide examples that visually salient and easy to visualize. For instance, "a bowl of milk put outside the fridge for a few days" is bad case because entails non-visual commonsense knowledge that milk would turn prompt GPT-4 to locate the subject and expected output, and discover specific real scenarios as PI, e.g. the subject is "butter", scenario "in heated pan", and expected output is "butter melts in a heated pan"" Finally, we ask GPT-4 to generate one counter-example for each real scenario with slightly different subject different action for the same subject, as the temporary candidate for Pz. However, the quality of generated examples are often not guaranteed, and may not test the desired commonsense knowledge.
For each example
Manual Curation. Prepared with the auto-generated examples as inspirations, we manually create the pairwise prompts and expected output descriptions, and verify the commonsense type, add the likelihood score. For the pairwise prompts, we rewrite to have natural sentences that do not reveal the expected outputs. For instance, we turn "an untied air balloon" into "A balloon filled with air" as in Figure 2. Also, we revise to keep minimum difference between P1 and P2, while leading to contrasting outputs per criteria. For the expected output descriptions, changed the GPT output "A mirror reflecting nothing in the room" to "A barely visible room" for "A mirror in room wi ththuut light". For the likelihood score, rate to answer "How
rewrite to be faithful to the prompts. For example,
4

---

Published as a conference paper at COLM 2024
many out of ten generated images do we believe should show the expected outputs?", and discard examples with likelihood score lower than 7.
Data quality control: To guarantee the quality of Commonsense-T21, manually go through all collected data again and filter out data that are ambiguous.
P1: Bear eating
bwer calching
X
bear whole salmon fish.
11.01.22
mage description?
Incorrect
Person person salmon.
D2 person is eating from plate.
No
110
12:
Humen
GFT-VV
0:01 1:01.D21 No
person eating filet.
02,022 Yes
02,0111
fo f2,02,22: Yos
Figure 3: The evaluation pipeline for Commonsense-T22 T2I. More details are in Section 2.3.
Evaluation Metrics
Commonsense-T21 is designed pairwise challenge set and we evaluate performance of T2I models according to the following criteria: only when both of the pairwise prompts Pt and P2 have the generated images I and l2 match the expected output descriptions D1 and D2 at the same time, we count the sample as correct. Specifically,
f1, 1, image 4; fits the description Dj
fit(I,D;)
0, otherwise
is indicator function evaluating whether the generated image I fits the description Dj where i, E [1,2] Then, the score for data sample n containing pairwise prompts P/t and Pa is calculated as
1, if fil()DD)) + ( - - fi((T-D))) =
scoren =
0, otherwise
where it only correct when the generated images for the pairwise prompts are both correct at the same time. For instance, if the T2I model generates images for Pi correctly but images for P2 incorrectly, namely (i(IDD)) = and (2) = O, then we consider the sample as incorrect, because the model fails to conduct the required commonsense reasoning. In our experiments, we generate multiple times for each data sample and take the average score for fair comparison. The final accuracy is then calculated
1
Accuracy N
Σ score
where N the total number of data samples. Details can be found in Figure 3.
3 Experiment
In this section, we first describe the baseline T2I models and experimental setup (53.1). Then we present a comprehensive evaluation of both human and existing multimodal models ($3.2). We demonstrate that while T2I models can generate high-quality images, Commonsense-T21 challenging for existing models. Finally, we provide detailed analyses using multimodal large language models (LLMs) auto evaluators, whether GPT- enriched prompts can solve the problem, possible reasons for dificiency in current models, and error analysis across different T2I models(53.3).

---

Published as a conference paper at COLM 2024
3.1 Experimental Setup
Baseline T2I Models:
We evaluate Commonsense-T2I variery of T2I models, including
three Stable Diffusion (Rombach et al., 2022) models: (1) Stable Diffusion v2.1¹ (SD-21), (2) Stable Diffusion XL (SD-XL), and (3) Stable Diffusion Medium (Esser et al., 2024) (SD-3). For the models SD-XL, and SD-3, we include a new setting that uses pairwise prompts as negative prompts in input: (4) SD-XI. w/ negative prompt and (5) SD-3 w/ negative prompt. Developed upon the Stable Diffusion XL model, (6) Playground v2.5 (Li et al., 2024)* is included as it provides high-quality images that are preferred by human over Stable Diffusion XL and DALL-E 3 (Betker et al., 2023) as in the paper. We also include (7) Openjourney v4* from PromptHero which is finetuned upon Stable Diffusion v1.5 using Midjourney images, and (8) the Latent Consistency Models (LCMs) (Luo al., 2023) which is distilled from Dreamshaper v7 fine-tune of Stable Diffusion v1.5. Flux models developed by Black Forest Labs" are recently released and have shown great performances. We include (9) Flux Dev and (10) Flux Schenel for comparison. As for API-based models, we evaluate on the DALL-E model (Betker et al., 2023). The DALL-E 3 model by default enriches and revises the given text prompt using GPT model (Brown et al., 2020) before generation, adding more details to the prompt since more detailed prompts generally result in higher quality images. Therefore, we include two variants of the model: (11) DALL-E 3, which is the original model, and (12) DALL-E 3 w/o revision, which turns off the GPT- revision function-we follow the OpenAl instruction 10 and add the following to our prompts: NEED test how the tool works with extremely simple prompts. NOT add any detail, just use it AS-IS: {prompt].
Evaluation Protocol: We assign two experts (coauthors) for each data sample
in Commonsense-T21 and present their average scores as human performance. As stated in Section 2.3, evaluators are expected to separately determine whether the content in image I fits the description DV, and also whether image I fits the description D for data sample n. In order to conduct fair comparison and avoid randomness, we generate the images for each data sample four times, evaluate separately, and take the average score for each data sample to calculate final accuracy. Namely, score, is averaged over four times of generations.
Notice that due to the demanding nature of the task, we have conducted manual evaluations only on the following models: SD-21, SD-XL, DALL-E 3, and DALL-E 3 w /o revision.
Considering the effort-demanding nature of human evaluation, experiment with multimodal LLMs to conduct automatic evaluation on Commonsense- Two models are tested in our paper: GPT-4V(ssion) (OpenAI, 2023), which is known to be one of the most powerful multimodal models to date, and we evaluate using three gpt-4-turbo, and gpt-4o; and GeminiPro (Team et al., 2023), which is one of the most widely used multimodal models, and we use the Gemini 1.0 Pro Vision version it. Specifically, for each image and description D, we get fit D) with the following prompt: Can you tell me if the image generally fits If it generally fits the descriptions, then return 1, otherwise, return 0. Give me number 1 or 0 only. Notice that in several cases where the multimodal LLM fails to tell fit(IDD) correctly and believes the image fits
Automatic Evaluation:
checkpoint models: spt-4-vision-previee
the descriptions "(description)""
"https://hugginngaaac Intpp://haggingffcccc/////b/bbb/ http://hhh.......oaaao.oaaa///ad
anttps://hugginggface
Pnttps://huggingfaac
#nttps://huggingfaa
hhttps://ppromptherro.oo
onttps://www.nlddoouree com/
comm/home home
/nttps://huggingf
9nttps://huggingffa ace. co/Lyykon/dreammhaperr
9https://blaccforestll
1Ptttps:/pplaffrmm www.aaarooooooo.....mmmmmom

---

Published as a conference paper at COLM 2024
CLIP
Gemini Pro
GPT- preview
Turbo (OpenAl, 2023)
GPT-4o (OpenAl, 2023)
Model/Evaluator
Human
(Radford et al, 2021) (Team et al,, 2023) (OpenAl, 2023)
Open-scuree T21 models
SD-21 (Rombach et al., 2022)
24.67
21.67
15.83
21.17
21.00
18.83
SD-XL (Rombach 2022)
26.00
29.67
23.50
26.67
25.17
24.92
SD-XL w/ negative prompt
44.83
23.50
30.67
31.67
Openjoumey
26.83
26.67
20.17
22.33
Playground v2.5 (Li et al., 2024)
24.83
29.00
18.50
20.33
25.83
LCMs (Luo et al,, 2023)
23.33
23.50
17.17
20.67
23.83
SD-3 (Rombached 2022)
26.17
22.00
21.67
SD-3 w/ negative prompt
47.17
30.00
35,83
37.67
Flux Dev
24.50
19,00
Flux Schenel
27.50
29.67
26.17
28.00
pmiprittrry T21 modils
DALL-E3 3 (Betker et al., 2023)
40.17
45.50
43.83
45.83
48.83
48.92
DALL-E3 w/o revision
34.83
36.50
33.50
35.00
37.50
34.00
Table 2: Main results on the Commonsense-T2) challenge set. The columns row shows the T2T models that we evaluate on, and the first row shows the evaluator choices. The best performance model under each evaluator is in-bold. Notice that some Gemini 1.0 Pro Vision scores are left blank because the model was deprecated since July 2024.
CLIP
GeminiPro
GPT-40 Human
0.8
40
0.6
30
0.4
02
DALL-E w/o revision
0.2
0.4
0.5
0.0
SD-21 SD-XL
DALL-E
CLIP SimilarttyPP,,
Figure 4: Comparison between using multi- modal LLMs vs. humans as evaluators. We can see that the evaluators CLIP and Gem- iniPro are both not ideal. GPT-4o, how- ever, consistently provides similar evalua- tion scores to humans and can be good candidate for automatic evaluation.
Figure 5: Illustration of the CLIP embedding similarity of prompts P1, P2 against human evaluated performance scores. suggests that T2I models perform badly when their text en- coders fail differentiate between P1 and Pz, and perform well when P1 and P2 are correctly embedded to be far.
both expected output descriptions Dj and D2, we randomly select one description such that fit(I D1) + fit(I D2) =
We also include CLIP (Radford et al., 2021) (ViT-L/14) as an additional evaluator, considering that it is one of the most widely used multimodal encoders. Specifically, we give CLIP the generated image and two expected output descriptions D1 and D2, and let fit(I_D,) = 1, fit = 0 if the cosine similarity between and D1 is larger than that between and D2, or (1) O, (2) if vice versa.
3.2 Main Results
Human Evalution Results: As illustrated Table 2, the mean accuracy of open-source stable diffusion models hover below 25%, with the XL model achieving 6.09% improve- ment over the v2.1 model. We surprisingly find that even the sota DALL-E 3 model only achieves 48.92%, meaning that it fails the commonsense challenge for more than half
7

---

Published as a conference paper at COLM 2024
of the cases in Commonsense-T21. After turning off the GPT revision function of D 3, DALL-E 3 w/o revision achieves 15.92% worse than the DALL-E 3 performance, suggesting the possible improvements brought by enriched text prompts.
Automatic Evaluation Results: Can Multimodal LLMs replace human evaluators? As shown in Table 2, with the most advanced models, GPT-4o and Gemini Pro, automatic evaluation performances similar to that of humans, with the maximum difference being 5.09% and 5.25% respectively. Their performance trends are consistent with that of humans, as shown in Figure 4. Notably, GPT-4V consistently rates lower than humans, while Gemini Pro always rates higher than humans except on DALL-E 3 images. We believe that multimodal LLMs, especially GPT-4V, can represent the T2I model performances generally. However, CLIP evaluation scores are relatively divergent from that of human, and cannot represent the trends of '2T model performences well.
achieve
3.3
Analysis
Are T2I models limited by text embedding? Since all the Stable Diffusion (based) T2I models score under 35% accuracy on Commonsense-T21, we investigate the possible reason behind this phenomena: these models might be biased by the text embedding of the prompts. The motivation is follows: if the embeddings of P1 and P2, which are inputs to the T2I models, are very similar, then they could lead the T21 models to generate similar images for P1 and P3, while the expected outputs should different. We deploy the CLIP (Radford et al., 2021) (ViT/L14 encoder, which is the default text encoder for Stable Diffusion (based) models, to encode the pairwise prompts P1 and P2 in Commonsense- We compare the similarity between CLIP embedding of P1 and P2 against performance score as in Figure 5. Notice that we adopt min-max normalization project the embedding similarity values into [0,1].
Can GPT augmented prompts solve the Commonsense-T21 problems? To answer this question, we analyze the error cases of DALL-E 3, which automatically uses GPT-auggmetted revised-prompts, and check whether the revised prompts include correct expected outputs. As in Figure 6, we show the difference between DALL-E 3 outputs and DALL-E 3 w/o revision outputs, along with the GPT-revised prompt used by DALL-E We can see that in the first two failed cases, the GIT revised prompts can provide the exact correct expected output information; while in the last two cases, they provide partially correct information. In short, GPT augmented prompts can help to some extent, with DALL-E 3 achieving 14.92% improvement over DALL-E3 3 w /o revision. However, they cannot solve the Commonsense- T2I challenge- they either fail to provide comprehensive correct details, or the T2T part fails to visualize the correct details.
Do different T2I models make same errors? The Stable Diffusion based models: SD-21, SD-XL, Playground v2.5, and Openjourney v4 fail on most samples in Commonsense-T21, even for the easy ones, such as "prompt: peacock sleeping, expected: The peacock has its feathers closed" from "Animal Behaviors". Additional failed examples are shown in Figure Meanwhile, DALL-E 3, as illustrated in Figure 6, often fails on more complicated cases, e.g. it mostly succeeds on "Animal Behaviors" and "Biological Laws" samples and fails on uncommon situations with commonly seen objects, such as unlit lightbulbs, di dintingisheed candles, fully melted ice cream...,ffc.
4 Related Work
T2I Models and Benchmarks Text-to-image synthesis models are typically trained to generate images conditioned on text. Early studies widely use GANs (Reed et al., 2016b;a). Starting from DALL-E (Ramesh et al., 2021), image generation models started to show impressive results. Recently, diffusion models have achieved remarkable success on text- guided image generation (Nichol et al., 2021; Saharia et al., 2022; Ramesh et al., 2022; Rombach et al., 2022; Li et al., 2024; Ku et al., 2024). Another series of work (Lu et al., 2023b; Zhu et al., 2023) enables the collaboration between the LLMs and ext-tto-mmage models. Multiple benchmarks evaluating different aspects of T2I models have been introduced.
8

---

Published as a conference paper at COLM 2024
prompt: A lightbulb without electricity expected: Lightbulb is unlit
DALL-E 3
DALL-E w/o revision
revised prompt: Visualize clear glass lightbub. The filament inside untouched and frail, but there'sss no glow emanating from it.
The metal base screwed tightly, waiting for the moment to be set in power source. However, currendyy it's devoid electricity. Around it, the surroundings dark and still, amplifying its ghostly inert state. lightbu stands testament technology yet symbolizing e paradoxical illustration of darkness in this instance. Please show prominently, laking center slage image.
prompt: pen placed in cup of water expected: The submerged part of the pen appears bent
DALL-E 3
DALL-E w/o revision
revised prompt:
An image depicting realistic scenario where ballpoint pen immersed in clear glass cup fillad with water. The flat and clean surface, and light should
should be resting reflecting off water and the glass, giving them glossy look, The inside water is distorted duu to the refraction of light. Shadlws from pen the cup
on surface.
prompt: hourglass just finishing counting expected: Sand is mostly in the bottom part
3
DALL-E w/o revision
revised prompt: An ultra realistic image a vintage hourg ass right the end of is countdown, the last from the top chamber the bottom one. The hourglass glass has an ornate design and is crafted from dark polished wood and glistening, clear gless. The fine grains sand within are golden in color. stands richly varnished waoden table and behind it is nearly fading background.
sand dropping
prompt: sundae untouched for several hours expected: The
DALL-E 3
has melted into liquid form
DALL-E 3 w/o revision
Generate detailed ed and hyper realistic image of been untouched for several portray glear signs of melting and decompostion of its ingredients) which typically includes whipped cream, ice sauces like chocolate and caramel, nuts, cherries, and a glass serving dish. There should be visible signs of the condensation on the outer surface
revised prompt:
sundae
should
of various flavors,
surface of the dish. background can include some splattors and drops of malted ice cream and ingradients.
Figure 6: Error cases of DALL-E 3, Prompts and expected outputs are in the green box. DALL-E images are generated with the revised prompts returned by DALL-E 3, and DALL-E 3 w/o revision images are generated with the original prompt. The highlighted sentences are (partially) correct expected output descriptions in revised prompts.
Most of them focus on straighforward text prompts, e.g. "a read book and yellow vase.", ", and evaluate direct visual attributes such as counting, color, and shape (Saharia et al., 2022; Park et al., 2021). Others focus on object detection and relations (Cho et al., 2023; Bakr et al., 2023), compositionality. presence of multiple objects and their attributes (Bakr et al., 2023; Huang et al., 2023), and fairness (Cho et al., 2023; Bakr et al., 2023). But none of them evaluates multimodal commonsense understanding of generative models.
T2I Evaluation Most metrics evaluate the models on fidelity (Salimans et al., 2016; Heusel et al., 2017; Jayasumana et al., 2023), image-text alignment (Radford et al., 2021; Hessel et al., 2021; Li et al., 2023), Recent metrics try to use large language models (LLMs) (Lu et al., 2023c; Zhang et al., 2023; Chen, 2023), or VQA (Hu et al., 2023), human (Ku et al., 2024; Lu et al., 2024) for evaluation. However, there is no comprehensive study on how well those evaluation metrics work for commonsense T2I generation. We propose evaluation metrics

---

Published as a conference paper at COLM 2024
specifically designed for our task and validate that our proposed metrics align well with human perceptions.
Multimodal Large Language Models With the recent development of multimodal Large Language Models (LLMs) (Alayrac et al., 2022; Li et al., 2023; Liu et al., 2023a; OpenAI, 2023; Liu et al., 2023b; Team al., 2023; Zhang al., 2023), more reasoning-related research questions about multimodality have been studied (Thrush et al., 2022; Marino et al., 2019; Fu et al., 2022; Lu et al., 2023a; Fu et al., 2023a; Gupta Kembhavi, 2023; Surís et al., 2023; Fu et al., 2024; 2023b; Wang et al., 2024; Hu et al., 2024). One paper (Zhang et al., 2022) studies visual commonsense knowledge in pretrained multimodal models. Concurrent work (Meng et al., 2024) specifically evaluates on the physics phenomenon in real life in text-to-image generation. general commonsense in reality for T2I models.
there are no comprehensive studies on alignment with
Conclusion
We introduce Commonsense-T21, a novel task for evaluating commonsense reasoning abilities of T2I models. We provide high-quality expert-annotated test set for the task including pairwise prompts and expected outputs. Our experiments show that current T2I models score between 15-50% on our dataset, fostering future research in this direction.
Limitations We would like to emphasize that the size of Commonsense-T21 is limited by the need to manually revise all the samples (each including five entries) by experts. We propose Commonsense-T21 as high-quality expert-curated test set and believe it can serve as good evaluation benchmark for the goal of this paper. Nevertheless, using the inspiration generation method in $2.2, one can easily generate huge amount of weak-supervision data.
Acknowledgements
We thank Wenhu Chen for the insightful discussions. This work was funded in part by ONR Contract N00014-23-1-2417, and supported by NSF grant IIS-2212433.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain BarT, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23777 2022.
Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to- image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20041-20053, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. httpss//cdn. openai. com/papers/dalle-3 pdf, 2(3):8, 2023.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. Xiv preprint ar XrXio:1966.05111 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvínd Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yixiong Chen. X-iqe: explainable image quality evaluation for text-to- text-to-image generation with visual large language models. Xro preprint ar arXio:2305.11 10843, 2023.
10

---

Published as a conference paper at COLM 2024
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3043-3054, 2023.
Probing the reasoning skills and social
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint ar Xrii:2212.000:: 2022.
Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, and Dan Roth. There'sa sa time and place for reasoning beyond the image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 1138-1149, 2022.
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, and Bing Xiang- Generate then select: Open-ended visual question answering guided by world knowledge. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 2333 2346. Association for Computational Linguistics, July 2023a.
Xingyu Fu, Ben Zhou, Sihao Chen, Mark Yatskar, and Dan Roth. Interpretable by design visual question answering. arXiv preprint ar Xrii:2200..14883 2023b.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. Xiv preprint ar Xxio::2444.12390 2024.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14953-14962, June 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint ar XrXi::2104.08718, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable image faithfulness evaluation with question answering. ar Xiv preprint ar xrXi::2303.11897, 2023.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. ar arviv preprint ar wxx:::2406.094000 2024.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint ar Xiv:2307 .06350, 2023.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. Xiv preprint ar KY0:2401.09600 2023.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at COLM 2024
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Play- ground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image genera- fion. Xiv preprint ar arXiv:2402.17245 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arxiv preprint arXiv:2301.12597, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Hugo Liu and Push Singh. Conceptnet-a practical commonsense reasonnni tool-kit. BT technology journal, 22(4):211-226, 2004.
Yujie Lu, Xiujun Li, William Yang Wang, and Yejin Choi. Vim: Probing multimodal large
language models for visual embedded instruction following, 2023a.
Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting, 2023b.
Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Limscore: Unveiling the power of large language models in text-to-image synthesis evaluation. arXiv preprint ar Xiv:23305.11f16 2023c.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision arena: Benchmarking multimodal iims in the wild, February 2024. URL
https://huggingf ace. co/ httpaccs/iioiiiiiooo//././oo///e
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency
models: Synthesizing high-resolution images with few-step inference. arXio preprint #xXiv:2310.03378, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, Yu Qiao, et al. Phybench: A physical commonsense benchmark for evaluating text-to-image models. arxiv preprint ar #xXi:2246.11802, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint ar arXio:2112.1072 2021.
OpenAL. Gpt-4 technical report, 2023.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 77448-8733. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 6821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. Xiv preprint arXiv:2204.06125, 1(2):3, 2022.
12

---

Published as a conference paper at COLM 2024
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016a.
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw. Advances in neural information processing systems, 29, 2016b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEÊE/CVE conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35:36479-3666 36494,, 2022.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances neural information processing systems, 29, 2016.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. Xrv preprint ar arxi::2303.081 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. Xiv preprint arXio:2312.11805, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistä compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248 2022.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. ar Xiv preprint ar Xiv::44406 Stt:22406.09411,,, 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel-Eskin. common- sense in pretrained unimodal and multimodal models. Xiv preprint arXiv:2205.01857 2022.
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-Av(ision) as generalist evaluator for vision-language tasks, 2023.
Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Collaborative generative ai: Integrating gpt-k for efficient editing in text-to-image generation, 2023.
A Appendix
A.1 Complete Data Example
A complete data sample in Commonsense-T2r looks as the following:
13

---

Published as a conference paper at COLM 2024
Pt: birthday cake after making wish
P2: A birthday cake before making a wish
D1: The candles are extinguished
D2: The candles on the cake are lit category: Human Practices 1ikelihood:9
A.2 Dataset Inspiration Generation Prompts
We illustrate the prompts used to generate the data inspirations from GPT-4-turbo following: Figure and 8.
want you to generate some prompts for image generation Al. These prompts require the Al to have common sense to infer visual information and depict correctly. For example:
lightbulb without electricity cannot give light requires infer that the lightbulb dark category: physical laws prompt: A lightbulb without electricity.
A crosswalk with red pedestrian signal means stop requires Ai to infer that no one is crossing the streets because of the red light category: cultural practices prompt: A crowded crosswalk with red pedestrian signal.
A guitar with broken strings doesn't have all six complete strings requires the to infer that at least one of the six strings on the guitar is broken and curled up category: visual common sense prompt: guitar with broken strings.
Bats sleep during the day requires the to infer that bats are not flying around in broad daylight category: animal behavior prompts: bats during the day. Now generate five more examples the above format, without introduction filler words.
Figure 7: Prompts we use to generate data inspirations. (1/2)
For each of the examples above, want you put them in this format:
(!"prompt!": "<object> + <visible scenario> "prompt2": '<a contrasting prompt about the same subject and common sense>", "category": "<category (number)'s name>', "description"". "A+ <visible state> + <object>", "description2": "A <visible state> <object>"]
In this format, dlescription1 is about prompt1, and description2 is about prompt2.
For example:
C'prompt1": "A lightbulb without electricity', "prompt2": lightbulb with electricity", "description"::
"The lightbulb is dark.", 'description2": "The lightbulb is lit.")
want the language of the prompt to be concise and natural.
want descriptiont and description2 to be an easily scenari..
Figure 8: Prompts we use to generate data inspirations. (2/2)
A.3 Error Cases Examples
In Figure 9, we illustrate some error cases by SD-XL and Playground v2.5.
14

---

Published as a conference paper at COLM 2024
prompt: A person eating salmon expected: A person is eating from a plate; A person is eating a salmon fillet
Playground 2.5
Stable Diffusion XL
prompt: A piece of butter in a heated pan expected: The piece of butter is melting
Playground 2.5
Stable Diffusion XL.
prompt: A flag on a pole on a windless day expected: The flag hanging down from the pole
Playground 2.5
Stable Diffusion XL
prompt: A cup of coffee with no milk expected: The cup contains black coffee
Playground 2.5
Stable Diffusion XL
Figure 9: Error cases of SD-XL andPlayground v2.5. The prompt and expected output description are provided in green box for each example.
15


Reviews:
[{"review_id": "3ZVhRFNGML", "reviewer": "Reviewer_FMFR", "summary": "The paper present a new benchmark called *Commonsense-T2I* challenge. The main contribution is that the author collected a set of text prompts involving commonsense and some of them are long-tail scenarios -- things won't happen often and they won't usually appear in the training corpus. The authors find that most common T2I models fail this challenge significantly. The best among all is DALLE3 and revision of prompts usually won't help -- even though the revision contains the correct commonsense reasonings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions.\n\n**Expand T2I models**\n\nAs for the closed-sourced SD-3 model, due to limited computation resources, we manually tested 10% examples from our benchmark, and found that the accuracy is around 28%. \n\nAs for latent consistency models, we used https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7 model and the GPT-4O evaluation result is: 22.3%. We will include more details in our final version.\n\n**Potential Improvements**\n\nThe purpose of our benchmark is to raise the novel problem of commonsense in T2I and to provide an evaluation benchmark. We agree that people can follow our way to collect data and fine-tune T2I models, and such fine-tuning will be helpful to the models. However, we doubt that the core problem lies in the limited expression of CLIP text embeddings, as shown in Sec 3.3 Analysis, and we need a better text embedding for T2I models to really solve this problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Clear motivation**. The motivation of benchmarking T2I model's commonsense is welcome. The benchmark is also very needed given the conceptions that the common mass belief T2I is a solved problem but in fact it's not. With the introduction of commonsense T2I, people would see a new direction of improving T2I models. \n\n- **Complete Ablation Study**. Though ablation study for datasets/benchmark papers are not fully required but it's nice to see the authors provide the ablation of different evaluator (automatic ones such as GPT-4 and Gemini, together with human raters). It's nice to see their consistencies and would confirm the benchmark is well designed.", "reasons_to_reject": "1. I would recommend the authors expand the number of T2I models used for evaluation. Probably including recent latent consistency models and SD-3 to see if they have different biases against SD-21 and SD-XL.\n\n2. I would love to hear more about the authors comments on potential ways to improve commonsense T2I. As the authors can collect these amount of benchmark data, the data can also be used to fine-tuning T2I models. For example, using fully-automated methods such as DreamSync and its data-efficient followup work SELMA, improving commonsense T2I seems possible and these methods are very simple baselines. I would love to see the authors reasonings on why these methods can or cannot improve commonsense T2I. If they can't, it could further strengthen the necessity of having this benchmark.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odYPy1O8MI", "reviewer": "Reviewer_XsVU", "summary": "This paper establishes a novel benchmark, named Commonsense-T2I, for evaluating the ability of text-to-image (T2I) models to understand commonsense in the text prompt. A dataset is created with the help of GPT4 and then manually filtered and annotated. Automatic and manual evaluation metrics are explored to analyze the performance of state-of-the-art (SOTA) T2I models. The paper shows that although SOTA diffusion-based T2I models can generate high-quality images, they fail to properly capture and generate the underlying commonsense expressed by the text input, which indicates a potential direction to improve text-to-image generation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions.\n\n**Commonsense Categories**\n\nThe categories are summarized from observations during data curation preparation by the authors. Before we manually collect the benchmark, we observe daily life examples from the real-world that require visual commonsense but are extremely difficult for T2I models. We summarize the representative and most common cases to decide our commonsense knowledge categories. We will release all data, code and data curation pipeline for further scaling needs. \n\nThe ratio in Table1 is the percentage of numbers of examples in our benchmark of each category.\n\n**Naming of D1 and D2**\n\nWe apologize for the confusion. We intend to mean the descriptions of the output of tested T2I models. We will clarify the naming in our final version.\n\nTypo: Thanks for pointing it out. We will fix it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Understanding the commonsense in text input and reflecting it in the generated image is an important capacity of text-to-image generation models. This paper proposes an interesting benchmark to analyze existing T2I models and provides insights for further improving text-to-image generation.\n\n- The dataset collection process is clever and reasonable. The powerful GPT4 is used to propose inspirations for commonsense knowledge, followed by manual curation. \n\n- The author first conducts manual evaluation and further explores using CLIP or multimodal LLMs for automatic evaluation. The conclusion suggests that multimodal LLMs can make the benchmark's evaluation process much more scalable.\n\n- The paper provides detailed error cases of DALL-E 3, indicating that the SOTA T2I models have limitations in 1) understanding the commonsense expressed by the text and 2) correctly reflecting the straightforward \"translated/extended commonsense\" into pixels. The conclusions of the paper can provide helpful inspiration for future work in the T2I generation.", "reasons_to_reject": "- It is unclear how the categories of commonsense knowledge are determined. If the five categories come from a referred paper, please clarify the source. Also, there is no explanation of how the ratios of the five categories are determined.\n\n- The naming and explanation of D1 and D2 are vague and hard to understand -- the name \"expected output descriptions\" is kind of ambiguous and could mean that the descriptions are the output of some model. There is a typo at the top of Figure 2: The \"D1\" there should be \"D2\".", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Own3UwtUUK", "reviewer": "Reviewer_4uPG", "summary": "The paper aims to evaluate the capability of text-to-image (T2I) generation models on understanding commonsense. The paper first collects a new manual-curated dataset as a new benchmark for evaluation. The paper also proposes a manual and an automatic pipeline to evaluate T2I model performance on the new dataset. The paper compares a few T2I models including DALL-E, Openjourney v4 and more.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions.\n\n**likelihood score and ambiguity removal**\n\nThe likelihood score was only necessary and used during our curation process, for filtering out ambiguous examples. Specifically, it represents “At least how many out of ten generated images should show the expected outputs” for the given prompt. For example, human experts believe at least 5 out of 10 generated images for prompt (“people holding an umbrella in a rainstorm”) should fit the expected output (“umbrella is inside out”), then the likelihood score is 5. Since each data sample has two prompts, we assign the lower likelihood score of the prompts to be the score for the sample. According to our criteria that samples with likelihood lower than 7 are ambiguous and should be discarded, this example will then be removed.\n\nWe only keep the likelihood scores in the benchmark for completion purposes.\n\n**Data Quality Control**\n\nThe human experts (authors) went through three iterations of data curation together for better data quality, focusing on multiple aspects including but not limited to: prompt quality (to focus on visual commonsense cases), ambiguous case removal (rating likelihood scores), data category balancing (to find more examples of certain categories), evaluation descriptions quality (to be succinct), etc.\n\n\n**Evaluation detail**\n\nWe take the average score of two experts for each image, and take the average score for the four images of each data sample to calculate final accuracy. The average agreement between human experts over four manually evaluated models is 92%.\n\n**Enlarge this benchmark**\n\nIt is a great question. During our curation process, we find that it is very difficult to automatically generate high quality data samples – the generated prompts in most cases are not common sense required, but only complex descriptive sentences. We leave this question to future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is overall well-written with clear motivation.\n\nThe paper collects a useful new benchmark for the community to evaluate the capability of T2I models.\n\nThe paper proposes an automatic pipeline to evaluate T2I capability on understanding commonsense, which is correlated with the manual evaluation process.\n\nThe paper concludes with a clear direction for future works to improve T2I models.", "reasons_to_reject": "Some details in the methodology are missing.\n\n1. Section 2.2 Page 4, it is unclear how the likelihood score was assigned based on what criteria.\n\n2. Section 2.2 Page 4, how was the data quality control implemented? How to decide data is ambiguous?", "questions_to_authors": "Please see \"Reasons to Reject\"\n\n1. How were likelihood scores used eventually? I.e. why they were necessary?\n\n2. Page 6 Evaluation Protocol, how was the disagreement between two experts resolved?\n\n3. It will be interesting to know how to enrich/enlarge this benchmark in the future and maintain good quality.", "extra_scores": {}}, {"review_id": "67OMYv5Mha", "reviewer": "Reviewer_673E", "summary": "This paper proposes a benchmark for evaluating text-to-image models at commonsense implicature. For instance, given a prompt like \"a lightbulb without electricity\", an illustration should show a dark lightbulb, not one that's lit up.\n\nThe paper considers 150 manually selected prompts, grouped in pairs P1 and P2 that cannot cooccur together. Then, models can be scored by how well an image generation for P1 matches P1 and all the things it entails (versus P2). The paper considers humans as well as CLIP and image caption evaluators.\n\nOverall, it seems that models struggle at this task, even when provided ultra-descriptive captions.\n\n--\noverall: seems like all reviews are positive, so should be pretty clearcut to accept this paper!", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions.\n\n**Digital Art concerns**\n\nIt is a great point and worth further exploration. For generality, we used the same prompt for every model. However, certain T2I models like Dall E 3 tend to generate digital art images. Specifically for the melting-ice example, we tried the Dall E 3 model again and changed the prompt into “Give me a realistic image, a real world photo of ” + original prompt. While the generated images look more real, they still make the same mistake as in Figure 5. We leave this question to future research.\n\n**Commonsense concern**\n\nIt is a great point that these examples might be summarized as entailment problems. However, considering that all reasoning questions can be summarized as entailment questions, we argue that commonsense might be a better title here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This seems like an interesting benchmark that could be a useful diagnostic for the community. Many text-to-image model generations are a bit weird and different from how humans perceive the world and create art, and quantifying this gap seems useful.\n* The experiments seem well done at least to this reviewer. In particular, it seems interesting that ultra-descriptive captions for DALL-E 3 help but do not fully solve the problem.", "reasons_to_reject": "* I am still a fan of this paper, but I'm a bit iffy on some of these prompts in the digital art case. There are ostensibly multiple goals with digital art and representing the physical world accurately is just one of them (along with emphasizing key features). For instance, the melting ice cream sundae example in Figure 5 indeed shows something _in the process of being melted_, emphasizing the second part of the prompt. If there was a way to restrict these models to only generating naturalistic real-world images, maybe this could be avoided, but I think the lack of specification might be a challenge here. I'd guess that more realistic text-to-image models (trained more naturalistic images versus digital art) should do better on this task, which might not be intended.\n* To this reviewer, it's not 100% clear whether commonsense is a great label for this task as opposed to just simply entailment (like SNLI-style).", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Commonsense-T2I, a novel commonsense benchmark for image generation tools. The benchmark contains uncommon prompts such as "A lightbulb without electricity". The authors show that several SOTA models fail to capture key requirements from the prompts (e.g., generate a lit lightbulb on this prompt). The reviewers all praise the work, the idea and its implementation. Relatively minor concerns seem to be addressed in the author-response.
Decision: Accept