Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?

Authors: xingyu  fu
muyu  he
yujie  lu
william yang wang
dan  roth
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-18 03:54:34

Abstract: We present a novel task and benchmark for evaluating the ability of text-to-image(T2I) generation models to produce images that align with commonsense in real life, which we call Commonsense-T2I. Given two adversarial text prompts containing an identical set of action words with minor differences, such as *a lightbulb without electricity* vs. *a lightbulb with electricity*, we evaluate whether T2I models can conduct visual-commonsense reasoning, e.g. produce images that fit *The lightbulb is unlit* vs. *The lightbulb is lit* correspondingly.
Commonsense-T2I presents an adversarial challenge, providing pairwise text prompts along with expected outputs.
The dataset is carefully hand-curated by experts and annotated with fine-grained labels, such as commonsense type and likelihood of the expected outputs, to assist analyzing model behavior. We benchmark a variety of state-of-the-art (sota) T2I models and surprisingly find that, there is still a large gap between image synthesis and real life photos--even the DALL-E 3 model could only achieve 48.92% on Commonsense-T2I, and the stable diffusion XL model only achieves 24.92% accuracy. Our experiments show that GPT-enriched prompts cannot solve this challenge, and we include a detailed analysis about possible reasons for such deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation benchmark for T2I commonsense checking, fostering advancements in real life image generation.

Main paper: 
 introduction: 
 Recent advances in generative modeling have allowed text-to-image (T2I) synthesis to achieve drastic performance improvements (Ramesh et al., 2021; 2022; Rombach et al., 2022; Betker et al., 2023; Li et al., 2024) . While it seems that we can realize a complete transition between a text prompt and an image, an image is worth a thousand words, and it is inevitable to lose information between the transition, due to the simplicity nature of language. Therefore, it is not possible to provide a text prompt to a T2I model that covers every single detail about an image-the T2I model must try to understand the text prompt first and self-imagine a real life scenario that contains every missing piece of detail, before generating the required images. As shown in Figure 1 , the visual differences of "lightbulb" between "A lightbulb without electricity" and "A lightbulb with electricity" are blatantly obvious -but will the same still be true for machines? At least current T2I models Betker et al. (2023) ; Rombach et al. (2022) ; Li et al. (2024) fail to reason the commonsense for this example, namely "lightbulb is unlit without electricity".We observe that many samples in existing T2I generation evaluations are straightforward composition of objects and their attributes, e.g. "A red car and a white sheep" (Saharia et al., 2022; Park et al., 2021; Cho et al., 2023; Feng et al., 2022) . They focus on understanding of object-related and attribute-related tokens in the prompts, such as size, color, object, and shape, and fail to cover complicated commonsense reasoning required scenarios. Therefore, it remains unknown whether current generative AI systems can reach human-level intelligence and generate images that align with commonsense in reality, using existing evaluations.In this paper, we specifically focus on the following question: can text-to-image models generate images that align with commonsense in reality? Motivated by this, we propose a novel evaluation task, called Commonsense-T2I, for measuring commonsense reasoning capabilities in generative models. As illustrated in Figure 2 , Commonsense-T2I presents a high-quality expert-curated test set, with each data sample containing two adversarial text prompts, their corresponding expected output descriptions, likelihood score for each expected output, and commonsense category. We design the prompts in a pairwise format that both prompts contain an identical set of action words with minor differences, ordered in such a way that the images must show noticeable differences to align with commonsense in real life. To perform well on Commonsense-T2I, T2I models must not only encode the superficial meaning of each token in the prompts, but also be able to synthesize commonsense reasoning across the two modalities.One natural question is how to conduct commonsense reasoning assessment on T2I models. Most metrics evaluate the models on image-text alignment (Radford et al., 2021; Hessel et al., 2021) , focusing on divers subjects including fidelity (Heusel et al., 2017; Jayasumana et al., 2023) , faithfulness (Hu et al., 2023) , and compositionality (Lu et al., 2023c; Chen, 2023) . None of them examines commonsense reasoning in generative models. While some recent methods evaluate models by human feedback (Lu et al., 2024) , they are effort-demanding, do not focus on commonsense, and only include relative comparison without gold answers. In this paper, we present an automatic evaluation pipeline using our collected expected output descriptions and multimodal large language models (LLMs) (OpenAI, 2023; Team et al., 2023) , that is tested to align well with human perceptions under our evaluation metric.The primary purpose of Commonsense-T2I is to support commonsense probing tasks for T2I models. We experiment with a variety of generative models including Stable Diffusion models, Playground v2.5, Openjourney v4, and DALL-E 3 (Rombach et al., 2022; Betker et al., 2023; Li et al., 2024) . Surprisingly, the state of the art (sota) DALL-E model 3 only achieves 48.92% accuracy, and all other models hover around 15-30% accuracy. Our findings indicate that the commonsense reasoning capabilities have not emerged in existing T2I models. Additional experiments( §3.3) show that GPT-revised enriched prompts cannot (2) We propose an automatic evaluation pipeline using multimodal LLMs for the task, and show that it is highly correlated with human evaluation. (3) We benchmark a wide range of T2I models on Commonsense-T2I and show that there is still a huge gap between all current models and human level intelligence, along with detailed analyses. We hope that Commonsense-T2I will stimulate the community to help T2I models catch up with human-level commonsense reasoning capabilities, fostering further progress in the field. 
the commonsense-t2i benchmark: 
 Our goal is to faithfully evaluate whether T2I models understand commonsense. We introduce a novel benchmark, Commonsense-T2I, designed to enable both quantitative and qualitative evaluation of the real-life commonsense reasoning capabilities of T2I models. We unfold this section by illustrating the overall design of Commonsense-T2I ( §2.1) and discussing its unique features. Then we provide an in-depth explanation of the data curation process ( §2.2). We further present the evaluation metric designed for the task( §2.3).Commonsense-T2I comprises of 150 manually curated examples, where each example has a pair of adversarial prompts: P 1 and P 2 , their corresponding expected output descriptions: D 1 and D 2 , likelihood scores for each output to happen, and commonsense category. Complete data example is in A.1. A data sample satisfies the Commonsense-T2I criteria if and only if:• P 1 and P 2 have the same set of action words but different subjects or adjectives.• P 1 and P 2 have the same subjects but different action words.• D 1 and D 2 are completely contrastive and cannot co-exist in one image.• P 1 will lead to D 1 and P 2 will lead to D 2 in daily life under common sense.Physical Laws A glass of water fell on the floor: spilled water. 32.7 A hot cup of water in winter: steam rises.A man at a wedding: in suit, looking cheerful. 30.0 A person eating a burger: eating with hands.Oak trees during winter: no leaves on branches. 11.3 A wheat field in spring: green field.A small bag of party balloons for sale: balloons are flat. 14.0 A phone with a drained battery: dark screen.A peacock attracting a mate: spreading feathers. 12.0 A penguin sliding on ice: sliding on its belly.Table 1 : Commonsense knowledge categories and percentage of data samples.The Commonsense-T2I dataset is entirely hand-curated by experts. We first decide on the categories of commonsense knowledge required for text-to-image generation, and then use GPT-4 (Brown et al., 2020) to help generate multiple examples requiring visual-commonsense knowledge as inspirations, and manually curate each test sample of pairwise prompts and expected outputs.Commonsense Category. Commonsense knowledge is a huge portion of human experience, encompassing knowledge about the spatial, physical, social, temporal, and psychological aspects of typical everyday life (Liu & Singh, 2004) , and widely studied in AI research (Bosselut et al., 2019; Zellers et al., 2019) . To build Commonsense-T2I, we manually select five categories of knowledge that naturally require visual commonsense understanding, as illustrated in Table 1 .Inspiration Generation. We prompt the GPT-4-turbo model iteratively to generate a massive pool of specific examples as inspirations for each commonsense category. Specifically, we first use GPT-4 to generate natural sentence examples given a commonsense category, e.g. "butter melts when heating" given category physical laws. Additionally, we require GPT to only provide examples that are visually salient and easy to visualize. For instance, "a bowl of milk put outside the fridge for a few days" is a bad case because it entails a non-visual commonsense knowledge that milk would turn sour. For each example, we prompt GPT-4 to locate the subject and expected output, and discover specific real scenarios as P 1 , e.g. the subject is "butter", scenario is "in a heated pan", and expected output is "butter melts in a heated pan" . Finally, we ask GPT-4 to generate one counter-example for each real scenario with a slightly different subject or different action for the same subject, as the temporary candidate for P 2 . However, the quality of generated examples are often not guaranteed, and may not test the desired commonsense knowledge.Manual Curation. Prepared with the auto-generated examples as inspirations, we manually create the pairwise prompts and expected output descriptions, and verify the commonsense type, add the likelihood score. For the pairwise prompts, we rewrite to have natural sentences that do not reveal the expected outputs. For instance, we turn "an untied air balloon" into "A balloon filled with air" as in Figure 2 . Also, we revise to keep minimum difference between P 1 and P 2 , while leading to contrasting outputs per criteria. For the expected output descriptions, we rewrite to be faithful to the prompts. For example, we changed the GPT output "A mirror reflecting nothing in the room" to "A barely visible room" for "A mirror in a room without light". For the likelihood score, we rate to answer "How many out of ten generated images do we believe should show the expected outputs?", and discard examples with likelihood score lower than 7.To guarantee the quality of Commonsense-T2I, we manually go through all collected data again and filter out data that are ambiguous. P1: Bear eating salmon.A bear is in a river catching a fish.A bear is eating a whole raw salmon fish.A person is eating from a plate.A person is eating a salmon fillet. Commonsense-T2I is designed as a pairwise challenge set and we evaluate performance of T2I models according to the following criteria: only when both of the pairwise prompts P 1 and P 2 have the generated images I 1 and I 2 match the expected output descriptions D 1 and D 2 at the same time, we count the sample as correct. Specifically,f it(I i D j ) = 1, if image I i fits the description D j 0, otherwiseis an indicator function evaluating whether the generated image I i fits the description D j where i, j ∈ [1, 2]. Then, the score for data sample n containing pairwise prompts P n 1 and P n 2 is calculated asscore n = 1, if f it(I n 1 D n 1 ) + f it(I n 2 D n 2 ) -f it(I n 1 D n 2 ) -f it(I n 2 D n 1 ) = 2 0, otherwisewhere it is only correct when the generated images for the pairwise prompts are both correct at the same time. For instance, if the T2I model generates images for P 1 correctly but images for P 2 incorrectly, namely f it(I 1 D 1 ) = 1 and f it(I 2 D 2 ) = 0, then we consider the sample as incorrect, because the model fails to conduct the required commonsense reasoning. In our experiments, we generate multiple times for each data sample and take the average score for fair comparison. The final accuracy is then calculated asAccuracy = 1 N N ∑ n=1 score nwhere N is the total number of data samples. Details can be found in Figure 3 . 
experiment: 
 In this section, we first describe the baseline T2I models and experimental setup ( §3.1).Then we present a comprehensive evaluation of both human and existing multimodal models ( §3.2). We demonstrate that while T2I models can generate high-quality images, Commonsense-T2I is challenging for existing models. Finally, we provide detailed analyses on using multimodal large language models (LLMs) as auto evaluators, whether GPTenriched prompts can solve the problem, possible reasons for dificiency in current models, and error analysis across different T2I models( §3.3).Baseline T2I Models: We evaluate Commonsense-T2I on a variery of T2I models, including three Stable Diffusion (Rombach et al., 2022) models: (1) Stable Diffusion v2.1 1 (SD-21), (2) Stable Diffusion XL 2 (SD-XL), and (3) Stable Diffusion 3 Medium (Esser et al., 2024) (SD-3).For the models SD-XL and SD-3, we include a new setting that uses pairwise prompts as negative prompts in input: (4) SD-XL w/ negative prompt and (5) SD-3 w/ negative prompt. Developed upon the Stable Diffusion XL model, (6) Playground v2.5 (Li et al., 2024) 3 is included as it provides high-quality images that are preferred by human over Stable Diffusion XL and DALL-E 3 (Betker et al., 2023) as in the paper. We also include (7) Openjourney v4 4 from PromptHero 5 , which is finetuned upon Stable Diffusion v1.5 using Midjourney 6 images, and (8) the Latent Consistency Models (LCMs) (Luo et al., 2023) 7 , which is distilled from Dreamshaper v7 8 fine-tune of Stable Diffusion v1.5. Flux models developed by Black Forest Labs 9 are recently released and have shown great performances. We include (9) Flux Dev and (10) Flux Schenel for comparison. As for API-based models, we evaluate on the DALL-E 3 model (Betker et al., 2023) . The DALL-E 3 model by default enriches and revises the given text prompt using GPT model (Brown et al., 2020) before generation, adding more details to the prompt since more detailed prompts generally result in higher quality images. Therefore, we include two variants of the model: (11) DALL-E 3, which is the original model, and (12) DALL-E 3 w/o revision, which turns off the GPTrevision function-we follow the OpenAI instruction 10 and add the following to our prompts: I NEED to test how the tool works with extremely simple prompts. DO NOT add any detail, just use it AS-IS: {prompt}.We assign two experts (coauthors) for each data sample in Commonsense-T2I and present their average scores as human performance. As stated in Section 2.3, evaluators are expected to separately determine whether the content in image I n 1 fits the description D n 1 , and also whether image I n 2 fits the description D n 2 , for data sample n. In order to conduct fair comparison and avoid randomness, we generate the images for each data sample four times, evaluate separately, and take the average score for each data sample to calculate final accuracy. Namely, score n is averaged over four times of generations.Notice that due to the demanding nature of the task, we have conducted manual evaluations only on the following models: SD-21, SD-XL, DALL-E 3, and DALL-E 3 w/o revision.Automatic Evaluation: Considering the effort-demanding nature of human evaluation, we experiment with multimodal LLMs to conduct automatic evaluation on Commonsense-T2I. Two models are tested in our paper: GPT-4V(ision) (OpenAI, 2023), which is known to be one of the most powerful multimodal models to date, and we evaluate using three checkpoint models: gpt-4-vision-preview, gpt-4-turbo, and gpt-4o; and GeminiPro (Team et al., 2023) , which is one of the most widely used multimodal models, and we use the Gemini 1.0 Pro Vision version of it. Specifically, for each image I and description D, we get f it(I D) with the following prompt: Can you tell me if the image generally fits the descriptions "{description}"? If it generally fits the descriptions, then return 1, otherwise, return 0. Give me number 1 or 0 only. Notice that in several cases where the multimodal LLM fails to tell f it(I D) correctly and believes the image fits Published as a conference paper at COLM 2024 Model/Evaluator CLIP (Radford et al., 2021) Gemini Pro (Team et al., 2023) GPT-4 preview Figure 5 : Illustration of the CLIP embedding similarity of prompts P 1 , P 2 against human evaluated performance scores. It suggests that T2I models perform badly when their text encoders fail to differentiate between P 1 and P 2 , and perform well when P 1 and P 2 are correctly embedded to be far.both expected output descriptions D 1 and D 2 , we randomly select one description such thatf it(I D 1 ) + f it(I D 2 ) = 1.We also include CLIP (Radford et al., 2021) (ViT-L/14) as an additional evaluator, considering that it is one of the most widely used multimodal encoders. Specifically, we give CLIP the generated image I and two expected output descriptions D 1 and D 2 , and let f it(I D 1 ) = 1, f it(I D 2 ) = 0 if the cosine similarity between I and D 1 is larger than that between I and D 2 , or f it(I D 1 ) = 0, f it(I D 2 ) = 1 if vice versa.Human Evalution Results: As illustrated in Table 2 , the mean accuracy of open-source stable diffusion models hover below 25%, with the XL model achieving a 6.09% improvement over the v2.1 model. We surprisingly find that even the sota DALL-E 3 model only achieves 48.92%, meaning that it fails on the commonsense challenge for more than half of the cases in Commonsense-T2I. After turning off the GPT revision function of DALL-E 3, DALL-E 3 w/o revision achieves 15.92% worse than the DALL-E 3 performance, suggesting the possible improvements brought by enriched text prompts.Automatic Evaluation Results: Can Multimodal LLMs replace human evaluators? As shown in Table 2 , with the most advanced models, GPT-4o and Gemini Pro, we can achieve automatic evaluation performances similar to that of humans, with the maximum difference being 5.09% and 5.25% respectively. Their performance trends are consistent with that of humans, as shown in Figure 4 . Notably, GPT-4V consistently rates lower than humans, while Gemini Pro always rates higher than humans except on DALL-E 3 images. We believe that multimodal LLMs, especially GPT-4V, can represent the T2I model performances generally. However, CLIP evaluation scores are relatively divergent from that of human, and cannot represent the trends of T2I model performences well.Are T2I models limited by text embedding? Since all the Stable Diffusion (based) T2I models score under 35% accuracy on Commonsense-T2I, we investigate the possible reason behind this phenomena: these models might be biased by the text embedding of the prompts.The motivation is follows: if the embeddings of P 1 and P 2 , which are inputs to the T2I models, are very similar, then they could lead the T2I models to generate similar images for P 1 and P 2 , while the expected outputs should different. We deploy the CLIP (Radford et al., 2021) (ViT/L14) encoder, which is the default text encoder for Stable Diffusion (based) models, to encode the pairwise prompts P 1 and P 2 in Commonsense-T2I. We compare the similarity between CLIP embedding of P 1 and P 2 against performance score as in Figure 5 . Notice that we adopt min-max normalization to project the embedding similarity values into [0,1].problems? To answer this question, we analyze the error cases of DALL-E 3, which automatically uses GPT-augmented revised-prompts, and check whether the revised prompts include correct expected outputs.As in Figure 6 , we show the difference between DALL-E 3 outputs and DALL-E 3 w/o revision outputs, along with the GPT-revised prompt used by DALL-E 3. We can see that in the first two failed cases, the GPT revised prompts can provide the exact correct expected output information; while in the last two cases, they provide partially correct information. In short, GPT augmented prompts can help to some extent, with DALL-E 3 achieving 14.92% improvement over DALL-E 3 w/o revision. However, they cannot solve the Commonsense-T2I challenge -they either fail to provide comprehensive correct details, or the T2I part fails to visualize the correct details.The Stable Diffusion based models: SD-21, SD-XL, Playground v2.5, and Openjourney v4 fail on most samples in Commonsense-T2I, even for the easy ones, such as "prompt: A peacock sleeping, expected: The peacock has its feathers closed" from "Animal Behaviors". Additional failed examples are shown in Figure 9 . Meanwhile, DALL-E 3, as illustrated in Figure 6 , often fails on more complicated cases, e.g. it mostly succeeds on "Animal Behaviors" and "Biological Laws" samples and fails on uncommon situations with commonly seen objects, such as unlit lightbulbs, distinguished candles, fully melted ice cream,...,etc. 
related work: 
 T2I Models and Benchmarks Text-to-image synthesis models are typically trained to generate images conditioned on text. Early studies widely use GANs (Reed et al., 2016b; a) . Starting from DALL•E (Ramesh et al., 2021) , image generation models started to show impressive results. Recently, diffusion models have achieved remarkable success on textguided image generation (Nichol et al., 2021; Saharia et al., 2022; Ramesh et al., 2022; Rombach et al., 2022; Li et al., 2024; Ku et al., 2024) . Another series of work (Lu et al., 2023b; Zhu et al., 2023) enables the collaboration between the LLMs and text-to-image models. Multiple benchmarks evaluating different aspects of T2I models have been introduced. Most of them focus on straighforward text prompts, e.g. "a read book and a yellow vase.", and evaluate direct visual attributes such as counting, color, and shape (Saharia et al., 2022; Park et al., 2021) . Others focus on object detection and relations (Cho et al., 2023; Bakr et al., 2023) , compositionality: presence of multiple objects and their attributes (Bakr et al., 2023; Huang et al., 2023) , and fairness (Cho et al., 2023; Bakr et al., 2023) . But none of them evaluates multimodal commonsense understanding of generative models.specifically designed for our task and validate that our proposed metrics align well with human perceptions.With the recent development of multimodal Large Language Models (LLMs) (Alayrac et al., 2022; Li et al., 2023; Liu et al., 2023a; OpenAI, 2023; Liu et al., 2023b; Team et al., 2023; Zhang et al., 2023) , more reasoning-related research questions about multimodality have been studied (Thrush et al., 2022; Marino et al., 2019; Fu et al., 2022; Lu et al., 2023a; Fu et al., 2023a; Gupta & Kembhavi, 2023; Surís et al., 2023; Fu et al., 2024; 2023b; Wang et al., 2024; Hu et al., 2024) . One paper (Zhang et al., 2022) studies visual commonsense knowledge in pretrained multimodal models. Concurrent work (Meng et al., 2024) specifically evaluates on the physics phenomenon in real life in text-to-image generation. However, there are no comprehensive studies on alignment with general commonsense in reality for T2I models. 
conclusion: 
 We introduce Commonsense-T2I, a novel task for evaluating commonsense reasoning abilities of T2I models. We provide a high-quality expert-annotated test set for the task including pairwise prompts and expected outputs. Our experiments show that current T2I models score between 15-50% on our dataset, fostering future research in this direction.Limitations We would like to emphasize that the size of Commonsense-T2I is limited by the need to manually revise all the samples (each including five entries) by experts. We propose Commonsense-T2I as a high-quality expert-curated test set and believe it can serve as a good evaluation benchmark for the goal of this paper. Nevertheless, using the inspiration generation method in §2.2, one can easily generate huge amount of weak-supervision data. 
 
 Appendix: T2I EvaluationMost metrics evaluate the models on fidelity(Salimans et al., 2016;Heusel et al., 2017;Jayasumana et al., 2023), image-text alignment(Radford et al., 2021;Hessel et al., 2021;Li et al., 2023), Recent metrics try to use large language models (LLMs)(Lu et al., 2023c;Zhang et al., 2023;Chen, 2023), or VQA(Hu et al., 2023), or human(Ku et al., 2024;Lu et al., 2024) for evaluation. However, there is no comprehensive study on how well those evaluation metrics work for commonsense T2I generation. We propose evaluation metrics 


Decision: This paper presents Commonsense-T2I, a novel commonsense benchmark for image generation tools. The benchmark contains uncommon prompts such as "A lightbulb without electricity". The authors show that several SOTA models fail to capture key requirements from the prompts (e.g., generate a lit lightbulb on this prompt). The reviewers all praise the work, the idea and its implementation. Relatively minor concerns seem to be addressed in the author-response.

Reviews: review: paper_summary: paper_summary: The paper present a new benchmark called *Commonsense-T2I* challenge. The main contribution is that the author collected a set of text prompts involving commonsense and some of them are long-tail scenarios -- things won't happen often and they won't usually appear in the training corpus. The authors find that most common T2I models fail this challenge significantly. The best among all is DALLE3 and revision of prompts usually won't help -- even though the revision contains the correct commonsense reasonings.
strength_weakness: strength_weakness: 1. I would recommend the authors expand the number of T2I models used for evaluation. Probably including recent latent consistency models and SD-3 to see if they have different biases against SD-21 and SD-XL.

2. I would love to hear more about the authors comments on potential ways to improve commonsense T2I. As the authors can collect these amount of benchmark data, the data can also be used to fine-tuning T2I models. For example, using fully-automated methods such as DreamSync and its data-efficient followup work SELMA, improving commonsense T2I seems possible and these methods are very simple baselines. I would love to see the authors reasonings on why these methods can or cannot improve commonsense T2I. If they can't, it could further strengthen the necessity of having this benchmark.
paper_summary: paper_summary: This paper establishes a novel benchmark, named Commonsense-T2I, for evaluating the ability of text-to-image (T2I) models to understand commonsense in the text prompt. A dataset is created with the help of GPT4 and then manually filtered and annotated. Automatic and manual evaluation metrics are explored to analyze the performance of state-of-the-art (SOTA) T2I models. The paper shows that although SOTA diffusion-based T2I models can generate high-quality images, they fail to properly capture and generate the underlying commonsense expressed by the text input, which indicates a potential direction to improve text-to-image generation.
strength_weakness: strength_weakness: - It is unclear how the categories of commonsense knowledge are determined. If the five categories come from a referred paper, please clarify the source. Also, there is no explanation of how the ratios of the five categories are determined.

- The naming and explanation of D1 and D2 are vague and hard to understand -- the name "expected output descriptions" is kind of ambiguous and could mean that the descriptions are the output of some model. There is a typo at the top of Figure 2: The "D1" there should be "D2".
paper_summary: paper_summary: The paper aims to evaluate the capability of text-to-image (T2I) generation models on understanding commonsense. The paper first collects a new manual-curated dataset as a new benchmark for evaluation. The paper also proposes a manual and an automatic pipeline to evaluate T2I model performance on the new dataset. The paper compares a few T2I models including DALL-E, Openjourney v4 and more.
strength_weakness: strength_weakness: Some details in the methodology are missing.

1. Section 2.2 Page 4, it is unclear how the likelihood score was assigned based on what criteria.

2. Section 2.2 Page 4, how was the data quality control implemented? How to decide data is ambiguous?
questions: questions: Please see "Reasons to Reject"

1. How were likelihood scores used eventually? I.e. why they were necessary?

2. Page 6 Evaluation Protocol, how was the disagreement between two experts resolved?

3. It will be interesting to know how to enrich/enlarge this benchmark in the future and maintain good quality.
paper_summary: paper_summary: This paper proposes a benchmark for evaluating text-to-image models at commonsense implicature. For instance, given a prompt like "a lightbulb without electricity", an illustration should show a dark lightbulb, not one that's lit up.

The paper considers 150 manually selected prompts, grouped in pairs P1 and P2 that cannot cooccur together. Then, models can be scored by how well an image generation for P1 matches P1 and all the things it entails (versus P2). The paper considers humans as well as CLIP and image caption evaluators.

Overall, it seems that models struggle at this task, even when provided ultra-descriptive captions.

--
overall: seems like all reviews are positive, so should be pretty clearcut to accept this paper!
strength_weakness: strength_weakness: * I am still a fan of this paper, but I'm a bit iffy on some of these prompts in the digital art case. There are ostensibly multiple goals with digital art and representing the physical world accurately is just one of them (along with emphasizing key features). For instance, the melting ice cream sundae example in Figure 5 indeed shows something _in the process of being melted_, emphasizing the second part of the prompt. If there was a way to restrict these models to only generating naturalistic real-world images, maybe this could be avoided, but I think the lack of specification might be a challenge here. I'd guess that more realistic text-to-image models (trained more naturalistic images versus digital art) should do better on this task, which might not be intended.
* To this reviewer, it's not 100% clear whether commonsense is a great label for this task as opposed to just simply entailment (like SNLI-style).