Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
Authors: Jaemin Cho
Yushi Hu
Jason Michael Baldridge
Roopal Garg
Peter Anderson
Ranjay Krishna
Mohit Bansal
Jordi Pont-Tuset
Su Wang
Venue: ICLR 2024 poster (2024)
Keywords: text-to-image generation
text-to-image evaluation
Davidsonian semantics
large language models
scene graphs
visual question answering
question generation
benchmark
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Evaluating text-to-image models is notoriously difficult. A strong recent approach for assessing text-image faithfulness is based on QG/A (question generation and answering), which uses pre-trained foundational models to automatically generate a set of questions and answers from the prompt, and output images are scored based on whether these answers extracted with a visual question answering model are consistent with the prompt-based answers. This kind of evaluation is naturally dependent on the quality of the underlying QG and VQA models. We identify and address several reliability challenges in existing QG/A work: (a) QG questions should respect the prompt (avoiding hallucinations, duplications, and omissions) and (b) VQA answers should be consistent (not asserting that there is no motorcycle in an image while also claiming the motorcycle is blue). We address these issues with Davidsonian Scene Graph (DSG), an empirically grounded evaluation framework inspired by formal semantics, which is adaptable to any QG/A frameworks. DSG produces atomic and unique questions organized in dependency graphs, which (i) ensure appropriate semantic coverage and (ii) sidestep inconsistent answers. With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above. Finally, we present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts, covering a wide range of fine-grained semantic categories with a balanced distribution. We release the DSG-1k prompts and the corresponding DSG questions.

Published as a conference paper at ICLR 2024
DAVIDSONIAN SCENE GRAPH: IMPROVING RELIABILITY IN FINE-GRAINED EVALUA- TION FOR TEXT-TO-IMAGE GENERATION
Jaemin Cho Yushi Hu
Roopal Garg
Peter Anderson 3 Ranjay Krishna
Jason Baldridge
Mohit Bansal'
Jordi Pont-Tuset
Su Wang²
'University of North Carolina at Chapel Hill https://google http://ggoole.....u.....
University of Washington GGogle Research
ABSTRACT
Evaluating text-to-image models notoriously difficult. A strong recent approach for assessing text-image faithfulness is based on QG/A (question generation and answering), which uses pre-trainn pre-trained foundational models to automatically gener- ate set of questions and answers from the prompt, and output images are scored based on whether these answers extracted with a visual question answering model are consistent with the prompt-based answers. This kind of evaluation is naturally dependent on the quality of the underlying QG and VQA models. We identify and address several reliability challenges in existing QG/A work: (a) QG questions should respect the prompt (avoiding hallucinations, duplications, and omissions) and (b) VQA answers should be consistent (not asserting that there is motor- cycle in an image while also claiming the motorcycle is blue). We address these issues with Davidsonian Scene Graph (DSG), empirically grounded evalua- tion framework inspired by formal semantics, which is adaptable to any QG/A frameworks. DSG produces atomic and unique questions organized in depen- dency graphs, which (i) ensure appropriate semantic coverage and (ii) sidestep inconsistent answers. With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above. Finally, we present DSG-IK Ik, an open-sourced evaluation benchmark that includes 1,060 prompts, covering a wide range of fine- tee-graandd semantic categories with a balanced distribution. We release the DSG-1K 1k prompts and the corresponding DSG questions.
INTRODUCTION
Text-to-Image (T2I) generation models are the "talk of the town" (Saharia et al., 2022; Yu et al., 2022: Chang et al., 2023; Rombach et al., 2022; Ramesh al., 2022). Until recently, the standard practice to assess T2I alignment was to compute similarity scores between the prompt and the generated image using captioning models (Hong et al., 2018), multimodal encoders (Xu et al., 2018; Hessel et al., 2022), or object detectors (Hinz et al., 2022; Cho et al., 2023a). To provide a more fine- grained and interpretable (hence informative) evaluation of T2I alignment, recent line of work (Hu et al., 2023; Yarom et al., 2023; Cho et al., 2023b) proposes to use Question Generation (QG) module create
of validation questions and expected answers from text description. For example, from text "a blue motorcycle parked by paint chipped doors." " it generates the question "is there a motorcycle?" and the expected answer "Yes". Then Visual Question Answering (VQA) module answers the questions based on the generated image, and score is computed by comparing the responses to the expected answers. We refer to these as QG/A frameworks (see Fig. 1). QG/A is well motivated by successful applications in other areas such as summarization, where QA is used to validate the quality of automatically generated summaries (Deutsch et al., 2021; Min et al., 2023).
Compared to the popular single-score similarity-based metrics (e.g., CLIPScore, Hessel et al. 2022; CLIP R- R-Preiision, Park et al. 2021), QG/A approaches are more calibrated and interpretable, and
Work done Student Rescarcher Google Research.

---

Published as a conference paper at ICLR 2024
Question Generation (QG)
Question Answering (QA)
Is there motoreycle?
Are there some doors?
"A blue motoreyele parked paint
Is the motoreycle bluc?
Score 5/6
chipped doors."
Are the doors painted?
Is the paint chipped!?
Is the motoreycle parked
near door?
Figure 1: QG/A workflow: Generate questions from text prompt (QG), answer the questions via VQA models (QA), and average the answer scores to obtain a single summary score.
they correlate well with human judgments. For calibration, e.g., a CLIPScore of 0.3 is empirically on the low side for realistic images but on the high end for pixel art. For interpretability, the question- answer pairs provides a more granular view of the model behavior. E.g., it is possible to assess that a generated image depicts the entities and their spatial relations correctly while having the colors wrong. For human correlation, Hu et al. (2023) for example achieves 47.2 Kendall's T in a sizable correlation analysis with human 1-5 Likert judgments, with CLIPScore T = 23.1.
Nevertheless, we identify several nontrivial reliability issues in existing QG/A methods. Extracted questions often involve multiple details that introduce ambiguity interpretation (e.g., the answer to "is there a blue motorcycle" covers both the color and presence of the object). Duplicated and hallucinated questions are common (c.g.. "is there a motorcycle?" & "what type of vehicle is this?" are duplicates). Finally, the VQA invalid query failure is prevalent (e.g.. "is there motorcycle?" (root), model: "no"; "is the motorcycle blue?" (dependent), model: "yes").
Focusing first the QG step, we argue that good framework should have the following properties:
A. Atomic Questions. Each question should cover the smallest possible semantic unit, so that (i) its answer is unequivocally interpretable and (ii) it facilitates the task of the VQA model person answering it. E.g. "is there a blue motorcycle?" inquires about more than one detail and thus the answer is unnecessarily harder assess is there motorcycle is the color incorrect?
B. Full Semantic Coverage and No Hallucinations. All contents of the prompt, and only the contents of the prompt, should be represented by the generated questions.
C. Unique Questions. Having "is there a motorcycle?" and "what type of vehicle is this?" over- represents one part of the prompt and thereby skews the final metric.
D. Valid Question Dependency. Depending on the answers, some questions become invalid and thus should not be asked to the VQA module. For instance, if the answer to "is there a motor- cycle?" is no, dependent questions like "is the motorcycle blue?" should be skipped VQA models may often say "motorcycle doesn't 't exist but it's blue" (more examples in Appendix C).
To capture these desired properties, we introduce Davidsonian Scene Graph (DSG), structured representation of text descriptions that naturally fulfills the above requirements. linguistic formalisms (Davidson, 1965; 1967a;b) that represent sentence's 's semantics by decom- posing it recursively down to a set of atomic propositions (A) that collectively represent the entirety of the semantics of the sentence, no more and no less (B). We implement the QG step as Directed Acyelic Graph (DAG) where the nodes represent the unique questions (C) and the directed edges represent semantic dependencies between them (D). In the VQA validation step, these dependencies ensure that if the answer to a question is negative, then all the dependent questions in the sub-graph are skipped. DSG is implemented as a modular pipeline based on state-of-the-ar Large Language Models - LLMs (e.g., PaLM 2 (Anil et al., 2023), GPT-3.5/4 (OpenAI, 2022; 2023)) for the first QG stage. The second QA stage is handled by sfate-of-the-ar VQA modules (Li al., 2022: Chen et al., 2023; Dai et al., 2023), where the dependent questions are skipped by VQA models accord- ing to the structure in the dependency graph (Figure 2). We comparatively experiment with various LLMs and VQA modules and present our pipeline with a combination of PaLM 340B (Anil al., 2023) and PaLI 17B (Chen et al., 2023). To further facilitate research in T2I alignment evaluation, collect DSG-1k, fine-grained (Table 1) with a balanced distribution of semantic categories and styles (Fig. 4). The prompts are sourced from a wide range of existing public datasets.
inspired by
benchmark with diverse set of 1,060 prompts
Through comprehensive experimentation (Sec. 4), we found that (a) DSG addresses the aforemen- tioned reliability issues well; (b) latest VQA models are capable support the QG/A approach in
2

---

Published as a conference paper at ICLR 2024
the motorcycle packed?
moturcycle blue?
motorcycle parked?
mosurccycle blue
Is the motorcycle next to the doors?
Is the mutuncycle yll mext doces?
there motorcyele?
Is there a motorvycle?
Is the door paint chipped?
Are the doors painted?
Are there some doors?
the door paintt chipped??
Are the doees painted??
Are there doors?
Score:
677 0.86
(b) Score: 3/7 0.43
Figure 2: DSG T2I alignment evaluation. The dependencies among questions avoid invalid queries (e.g., there isn't 't a motorcycle, so don't ask about its color, the right example).
some semantic categories but fail in others. Finally we evaluate SoTA T21 models and thereby demonstrating DSG's s strength in fine-grained diagnosis which helps advancing model development.
2 RELATED WORK
Single-summary scoring frameworks. The most common approaches for Text-to-Image (T2I) generation include text-image embedding similarity scores from multimodal encoders (e.g., CLIP- Score (Hessel al., 2022) and R-precisoon (Xu et al., 2018; Park et al., 2021)) and text similarity based on image captioning (Hong et al., 2018). These metrics are often uncalibrated (e.g.. 0.2 CLIP- Score might be high for pixel art but low for realistic photos). addition, summarizing text and im- age in single embeddings obscures the semantic granularity involved in T2I alignment. SOA (Hinz et al., 2022) and DALL-Eval (Cho et al., 2023a) introduced object detection in validating text-image alignment a finer-grained scale (and naturally, calibration by object alignment accuracy). These methods are however limited validation scope, e.g., relations among objects and global stylistic features often lie beyond the purview of object detection capabilities.
QG/A frameworks. Question Answering (QA)-based evaluation emerged early in text summa- rization to gauge the amount of key information contained in summary: manually in Narayan et al. (2018), automatically in Durmus et al. (2020); Eyal et al. (2019), and more comprehensively in Deutsch al. (2021). In the multimodal domain, with advancing capabilities of large pre-traue trained foundation models, a line of work has emerged around the idea of validating alignment by applying Visual Question Answering (VQA) on questions generated from the prompt. This approach (col- lectively referred to as QG/A) allows for more fine-grained evaluation (e.g., How many objects are matching? Are relations and attributes correctly bound to specified objects?). TIFA (Hu al., 2023) generates questions with GPT-3 (Brown et al., 2020) in various semantic categories (e.g., color, shape, counting) and validates with VQA modules such as mPLUG (Li et al., 2022). Yarom et al. (2023) propose similar approach with VQ² A (Changpinyo et al., 2022) the question generator and improved VQA model-human correlation through data synthesis and high quality negative sam- pling. VPEval (Cho al., 2023b) further refines the process by introducing object detection and Optical Character Recognition (OCR) modules, and orchestrates the querying the form of con- trollable visual programming with ChatGPT (OpenAI, 2022). Our work follows the QG/A method- ology, but it takes inspiration from formal semantics (Davidson, 1965) to address several nontrivial reliability issues observed in prior work (e.g.. duplicated and non-atomic question).
DAVIDSONIAN SCENE GRAPH
Briefly sketched, DSG is set of T21 alignment validation questions structured in (directed) scene graph, produced from the prompt as the ground truth. The questions are posed a pre-trained VQA module in the context of the image evaluated. The final result consists of per-question answers and an aggregated VQA accuracy score. Figure 2 illustrates the process.
Dependency-aware atomic decomposition of scene description. Like other QG/A- T2I eval- uation frameworks, given a text prompt and generated image é, we generate a set of validation questions q and their expected answers a from t. We then obtain answers from VQA model ā to in the context of i. By comparing à and a, we can compute an alignment score between and the closer ā and are, the more aligned the image is to the prompt.
3

---

Published as a conference paper at ICLR 2024
Global
Entities 40.9%
Attributes 23.5%
Relations 24.3%
11.3%
Whole
Part
State
Color
Typo MORSIT
Caunt
SINp
Spatial
Scale
Global
Semantic category breakdown for TIFA-160 (manual) annotation. Entity: whole (entire entity, e.g., chair), part (part of entity, e.g., back of chair). Attribute: color (e.g., red book), type (e.g., aviator goggles), material (e.g., wooden chair), count (e.g., 3 geese), texture (e.g., rough surface), text rendering (e.g., letters "Macaroni"),, shape (e.g.. triangle block), size (e.g., large fence). Relation: spatial (e.g.. next B); action (A kicks B). Global (e.g., bright lighting). Appendix B presents details on per-category question counts.
Figure 3:
Tuples -y Questions
1. Is there motoreycle?
2. Are some doors?
DSG DAG
"A blue motoreycle parked by paint chipped doors."
3. Is the motorcycle blue?
Are
painted?
Arsibak coler motoncycle blue?
5. Is the door paint chipped??
metooeyccle
6. motorcycle parked?
Prompt Tuplcs
7. the motorcycle next doors?
Emityy - whole (motoreycle)
Easty whole
whale
Enirry whole (dooss
there motoncycle?
Attributc- color (metoncyclee, bluc)
Tuples - Dependencies
Artribute- state (doers, painted)
Independent
Attribute- state (door paint, chipped)
2. Independent
Eeialin spaiid motorcycle next the doors?
Are the doors painted?
Attribute state (motorcycle, parked)
3. Needs heing
Relation spatial (motorcycle, doors, next
4. Nceds 2 being true
Needs being
Artihr Saae the door chipged?
6. Nceds being truc
7. Needs and being true
Figure 4: Automatic generation of DSG. We first generate semantic tuples from text prompt. Then we generate questions and dependencies from the semantic tuples, in the form of DAG.
Inspired by Davidsonian Semantics (Davidson, 1965; 1967a;b), we represent t's semantics as Directed Acyelic Graph (DAG), where the nodes are atomic propositions and edges are entailment dependencies among them. An atomic proposition is statement which (i) has a truth value that can be verified against and (ii) cannot further decomposed into constituent propositions with truth values. A child proposition and its parent are in entailment dependency if the truth of the parent necessary for the child to be considered. For the prompt "a blue motorcycle", the proposition "there is motorcycle" is parent of "the motorcycle is blue": this dependent's 's truth (i.e. whether the motorcycle is blue) can only be evaluated if the parent is true (i.e. there is a motorcycle at all).
We identify four types of atomic propositions: entities, attributes, relationships, and globals. Each type further subsumes detailed semantic categories (Fig. 3). We represent each of these with m- ples: entities and globals are 1-tuples (the entity or global feature itself), attributes are 2-tuples (the attribute and the entity it refers to), and relationships are 3-tuples (relationship, subject entity, and object entity). Each tuple can then be translated into an atomic question that be fed the VQA module (Fig. 2). We call the resulting DAG Davidsonian Scene Graph (DSG).
Automatic generation of DSGs. We propose a three-step pipeline for automated generation of DSGs (Fig. 4): Prompt + Tuples: A list of atomic semantic tuples is generated from the input prompt, each of it with numerical identifier. Tuples Questions: Each semantic tuple is translated into a natural language question with an expected binary yes/no answer. Tuples + Dependencies: The dependency between the tuples is established by generating pairs of parent-children identifiers.
Each step is implemented by an LLM given task-specific in-context examples: we prompt an LLM with a preamble (with input and output sampled from manual annotations with fixed seeds) to elicit tuple annotations of the same format for new inputs. The details on the preamble engineering is in Appendix A. This three-step pipeline proved more effective than a single-step pipeline where an LLM the whole DSG once, since it allows to include more task-specific in-context examples for each stage (more than 20 vs. fewer than 10).
For generation pipeline such that the question sets resulted fulfill the four reliability properties (i.e.. atomicity, full semantic coverage, uniqueness. and valid dependency, see Sec. 1) as faithfully possible, we conduct two layers of quality assurance. First, ensure that all three human
4

---

leftmost moai at Ahu Tongariki on Published as a conference paper at ICLR 2024
Table 1: DSG-1k overview. To comprehensively evaluate T2I models, DSG-Ik provides 1,060
prompts covering diverse skills and writing styles sampled from different datasets.
Feature
Source
Sample
Example
TIFA
Assorted categories
(Hu et al.. 2023)
160
"A Christmas tree with lights and teddy bear""
"There is cat in the shelf. Under the shelf are two small barbels, the shelf are also DVD players
Stanford paragraphs (Krause et al,, 2017)
100
Paragraph-type captions
Beside the shelf big bottle of white
wooden case."
picture can food on the surface. At the right corner of the image those
on plate, which
Localized Narratives
(Pont-Tuset 2020)
100
looking like fingers of a person."
CountRench
view of
Counting
(Phiss et al,, 2023)
100
Easter Island"
"person at table. person has face. person wear shirt. person wear shirt. chair to table. shirt on person. person wear glasses. person hold phone"
VRD
Relations
(Lu ct al., 2016)
"a painting of huangshan. a matte puinting by mare si- monetti, deviantart, fantasy art, apocalypse landscape, matte painting. apocalypse art"
DiffusionDB (Wang et al., 2023)
Written T2I real users
"furry caterpillar, pupa, screaming evil demon, fangs, red hands. borror. 3 dimensional. delicate. sharp. lifelike, photorealistic, deformed, wet, shiny, slimy"
Midjourncy-prompge (Ture Nemade, 2022)
100
PoseScript
"subject squatting, torso leaning holding up subject. right arm is straight forward, head is leun- ing left looking forward"
arm
Human poses
(Delmas 2022)
100
Commonsense- defying
Whoops
100 "A man riding jet ski through the desert"
(Bitton-Guese et al., 2023)
Draw Ceet-Cretvvve (Liu 2023)
"a painting of a landscupe. with handwritten note that says "thi painting was not painted by me""
Text rendering
100
experts reach consensus that each question / question set is valid for the properties for in-context examples. Second, we administer both automatic and human evaluation on the generated DSG outputs (Sec. Tab. 2).
DSG-1k dataset. We construct the DSG- Ik prompts by combining publicly available prompts that cover different challenges (c.g., counting correctly, correct color/shapp/text/ett.. rendering, seman- tie categories, and writing styles. The core of DSG- Ik is TIFA160, from TIFA v1.0 (Hu et al., 2023), which come with Likert-scale T2I alignment ratings by human annotators. While TIFA160 prompts come from different datasets, its size (160 prompts) limits the diversity of features, so we sample 900 more prompts from 9 different datasets, covering different writing styles (e.g., short cap- tions, paragraph-level captions, synthetic relation tuples, alt text, prompts written by real users), and skills (e.g. text rendering, counting, relations), enabling evaluation of more diverse aspects of T2I generation capabilities with source-specific scores. DSG-1k 1k has 1,060 prompts in total (Table 1). For the annotation details for DSG-Ik, see Appendix B.
EXPERIMENTS AND DISCUSSION
This section covers the validation experiments we conduct in question generation and answer and T2I evaluation results representative SoTA T2I models. In regards to QA, we focus on pushing the limit on the direct VQA query strategy. A summary on the mixed technique involving non-VQA modules (e.g., dedicated OCR module for text rendering) is in Appendix
4.1 QUESTION GENERATION
In the following, evaluate the DSG questions in terms of how accurate they are (precision/recall), how simple/straightforwaadd they are (atomicity), whether they are overlapping each other (unique- ness), and how accurate their inter-dependencces are.
Do generated DSG tuples match human annotations? Precision & Recall. Given a prompt, set of annaa-en semantic tuples T {(1,..., 4 and model-generated questions
5

---

Published as a conference paper at ICLR 2024
Data/ Evaluator
QG Mothod
Precision (%)
Recall
Table 2: Evaluation of generated ques- tions on TIFA160 prompts with man- ual human evaluation (on 30 prompts)
TIFA160 (30 samples) manual TIFA160 (160; full) / GPT-3.5
DSG (ours)
92.2
100.0
98.3
96.0
Data/ Evaluntor
QG Method
Atomicity (%6
Uniqueness (5%)
and GPT-3.5 based automated evaluation (on 160 prompts). Top: DSG achieves high precision/recall. Bottom: DSG and
TIFA
78,5
65.4
TIFAT6D sumples) / mamal
VO'A DSG (ours)
3.9
96.5
97.5
TIFA
73.4
higher atomicity and uniqueness than the bascline question generation methods.
TIFAT60 (160:: fully
VQ"A
79.8
DSG (surs)
Q ........ . 912 }: let Pe.g be q matches t (e.g., q = "is there a motorcycle?" ?" matches t (entity- motorcycle)). From the tuples and questions, we calculate precision = = /tt./// I and recall- = Σ /t.' The evaluation is done both manually and automatically. For the former, we sample 30 prompts from TIFA 160 and ask domain experts provide match judgments (1(,g) For the automatic evaluation, we apply an LLM (GPT-3.5 (OpenAI, 2022)) to generate question-tuple matching by showing in-context examples (see Appendix A for details).
We report results in Table 2 (top). Overall, the DSG questions are close to perfect in matching the source semantic tuples, in both manual (precision 92.2% / recall 100%) and automatic evalua- tions (precision 98.3% / recall 96.0%). On examining manual annotations, we observe three types of mismatches: Granularity of relation decomposition, e.g., in "A table topped with bags of lug- gage and purses.". the human annotation holds the relationship between the purses and table as implicit, whereas DSG includes the question explicitly. Granularity in referential scope, e.g., in "[...] two wheels [...] one behind the other", the human annotation considers the wheels single en- tity, whereas DSG generates questions both about front and back wheels. Subjective details, c.g.,, in "A group of giraffes are gathered logether in their enclosure." ", the possessive relationship the enclo- sure is the giraffes' isn't 't included in the human annotation whereas DSG produces "is the enclosure theirs (the giraffes)?". The matter with granularity is relatively minor but subjectivity deserves a dedicated solution as valid QG/A method should distinguish between the details that can/cannot be visually verified and exclude the latter.
Are the questions simple and not overlapping each other? Atomicity & Uniqueness. A question atomic if it only asks about one fact (an entity/atribute/rrlattin)) e.g., "is this a motorcycle?"" is atomic whereas "is this a blue motorcycle?" is not (asks about the entity "motorcycle" and its color attribute "blue"). We ask two experts to make judgment on the same 30 prompt samples above, and calculate the percentage of atomic questions for TIFA, VQ2A, and DSG. We observe that LLMs do not yield reliable answers to this concept yet. We define uniqueness as the proportion of unique questions in question set. For example, in Q = ('is there a motorcycle?", "what type of vehicle is this?"), the questions are duplicates, thus uniqueness is 1/2 = 0.5.
We report results in Table 2 (bottom) both for manual evaluation 30 samples (3 expert annotators) and automatic one on all of TIFA160. Overall, DSG achieves strong performance in keeping ques- tions atomic (easier to obtain unambiguous answers) and unique (thereby avoiding skewing metric calculation). For atomicity, we observe a large number of non-atomic questions VQ² A, leading to very low atomicity of (3.9)), e.g. "A human is right to what?" points all entities the right of the person. This frequently causes the VQA model to give a valid answer which happens to not match the ground truth. For uniqueness, VQ A and DSG fare relatively better. Figure 9 in appendix exemplifies disagreement between auto and human evals.
Are the question dependencies valid? DSG produces parent-child annotations where the child question is only valid (as a VQA query) if the answer to the parent question is positive (e.g., "is the motorcycle blue?" is only valid if the answer to "is there a motorcycle?" is positive). We formulate the evaluation as binary judgments for all parent-child pairs, and conduct it manually on 30 samples (3 expert annotators) and automatically on the full TIFA 160. In the automatic evaluation, we check whether the antity/attibute/ellation mentioned in the parent is also included in the child. On the 30 samples from TIFA160, the valid ratio is 100%. For the full TIFA160, barring a few failure cases with parsing errors, the ratio remains close to perfect, at 99%. Given the qualitative benefit of dependency annotations in avoiding incurring invalid VQA answers, this result strongly supports the reliability of DSG in this regard.

---

Published as a conference paper at ICLR 2024
QG methods
Per-item evaluation VQA models combined with different QG methods. We compare Spearman's 'ss and Kendall's т correlation between VQA score and human 1-5 Likert TIFA (FFA160 prompts. The correlation sec- tion in Appendix E further contextualizes the results with sungle-summary metric (e.g. CLIPScore) results.
Table 3:
VQA models
TIFA
VQ
DSG
mPLUG-large
0.352/0.259
0.212/0. 164
0.463/0.380
Instruct-BLIt
0.460/0.360
0.426/0.352
0,442/0.364
on
PaLI
0.431/0.323
0.207/0.157
0.571/0.458
DSG-Ik Question Types
VQA-Hummn Answer match accuracy on three T2I gener- ation models on DSG-IK Ik prompts, averaged over questions. We DSG-PaLM2 questions.
Table 4:
VQA models
Entity
Attribute
Relation
Global
Overall
mPLUG-large
81.4
61.3
74.7
45.6
72.1
Instruct-BLAn
79.6
59.5
72.0
48.3
70.4
PaLI
83.0
62.3
77.2
47.2
73.8
4.2 QUESTION ANSWERING
With questions properly generated, good VQA module is expected to answer the questions in high alignment to human answers. Following previous work (Hu et al., 2023; Yarom et al., 2023; Cho et al., 2023b), we first investigate the correlation between VQA accuracy and Likert human ratings (1-5) of text-image pairs. In addition, examine per-question answers from VQA modules vs. human raters. We collect per-item (text-image pair) for TIFA 160 prompts, and per-question human judgments for the full DSG-Ik 1k prompts. A per-item judgment is 1-5 Likert text-image consistency scale, and a per-question judgment is a binary (DSG and VQ² A questions) or multi-choice (TIFA questions) answers. For per-item judgment, following Hu et al. (2023),
the five text-to-imaae generation models: three versions of Stable Diffusion (SD) v1.1/v1... v1.1/v1.5/v2.1 (Rombach et al., 2022), minDALL-E (Kim al., 2021), and VQ-diffusion (Gu al., 2022) on TIFA160 prompts. For per- question judgment, we collect three answers per questions on three text-to-image generation models: Stable Diffusion v2.1, Imagen* (Saharia et al., 2022), and MUSE* (Chang al., 2023) (see Sec. 4.3 for details of Imagen* and MUSE*). See Appendix H for the annotation UI. Leveraging the rich annotations collected, we conduct both per-item (VQA-human Likert score correlation) and per- question evaluation (VQA-huma) matching accuracy). For specific VQA modules, we experiment with three state-of-the-arr models Pal.I (Chen et al., 2023), mPL.U-large (Ye et al., 2023), and Instruct-BLID (Dai et al., 2023). In addition, we present supporting evidence for the value of DSG dependency annotations (as presented here) in an ablation study in Appendix D.I.
Per-item eval. In Table 3, we report the correlation between the average VQA scores and human (1-5 Likert) ratings on per-item text-image alignment, measured with Spearman's p and Kendall's т. We find that DSG+PaLI combination achieves the best correlation (p 0.563, 0.458) which is also strong on the absolute scale (conventionally interpreted as "moderate to strong" correlation). As the DSG questions
DSG questions achieve the highest correlations among all QG methods, for the per-question evaluation of VQA described below.
Per-question eval. Table 4 summarizes the proportion of matching questions between VQA mod- els and human raters. Overall, PaLI performs the best with 73.8% matching ratio, slightly higher than mPLUG-large and Instruct-BLIP. Breaking down by broad semantic categories (i.e., entity/ portiyy/atin htt//ttriin wwwiiiiueeeeaattte to entity questions than attribtee/rreaaions/glaall questions.
global), the all three VQA models output more accurate answers
Table further looks at the detailed semantic categories. For entities, the matching ratio is high (and at similar level) for both whole (e.g., cat) and part (e.g., neck of a car). Similar observation is made for relations between spatial and action. The metric distributes unevenly for attributes: visually explicit categories such as color, texture, etc. are relatively much easier to handle than abstract / higher-order ones like count and text rendering, For text rendering in particular, the model- hallen matching ratios are (well) below 40% (the "random" ratio for the multichoice questions which are primarily binary). The observation, data scaling for recent large VQA models does not translate well into helping them learn "deep semantics" which are visually more implicit (e.g., PaI.I does not handle text rendering ing better than
believe, supports the view that training
7

---

Published as a conference paper at ICLR 2024
Table 5: VQA-Human answer match accuracy on images from T2I generation models on DSG- 1k prompts, by fine-grained semantic categories. We use DSG-PaLLM2 questions and images generated by T2I models: SDv2.1, Imagen*. R and MUSE" ent: entity, rel: relation, att: attribute.
DSG-IK Question typs
VQA models
(ent) whole
(rel) spatial
(att) state
(ont) part
(global)
(att) color
(rel) action
(art) type
mPLUG-lare -lage
80.9
75.4
60.5
83,8
45.6
81.4
70.2
52.6
Instruct-BLiP
78.5
72.3
60.7
84.3
48.3
78.8
70.1
53.7
PaLl
82.6
78.3
62.8
84,6
47.2
83.6
70.3
55,5
(att) count
(art) text rendering
(att) material
(att) shape
(at1) style
(att) texture
(att) size
mPLUG-large
67.2
24.9
56.9
$4.6
23,6
69.4
75.2
Instruct- DI.IP
21.1
67.6
PaLl
67.4
13.7
59.1
53.9
21.1
70.4
72.4
Subjectivity
Proegt coferpure tolicoo Maaki fight
Proneet: actueco bast laplle
hocept eneple upil rolie vili Lott?
Frorge mocilei
Unicc plow
dael an Shen Coup
lace best VOA enneer
Questoo dey Rocero
Garmaa mower anawer
araver
VID ennmer
Domain knowleedge
Pronge Colori) ve:
Premet cyberuur detaiic ned lights. Quest
Trompt lover
Prement vor
lowehe windoes
dhodk giol deranii (ilass ciniulieu
VQA
Figure 5: Questions involving subjectivity or domain knowledge. For subjectivity, in addition the frequent VQA-human disagreement, consensus among human raters is hard to achieve. e.g.. how detailed should the image be to count as "highly detailed"? For domain knowledge, depending on their knowledge, average human raters may find it difficult to give reliable judgments for this type of questions. egg., what counts as "breath of the wild" or "Studio Ghibli" styles?
mPLUG-large despite the size advantage and superior performance in visual explicit categories such as color or shape.). Table 5 also reveals the difficulties of VQA models in dealing with questions in categories like material, shape, style, etc. Manually examining model predictions, believe the issue primarily lies with a) subjectivity and b) domain knowledge. For subjectivity, e.g. human raters often disagree with VQA on if an objective "roundish" (shape), or whether painting is "highly detailed" (style); and they also disagree among themselves. For domain knowledge, it is generally impractical to assume the average human rater possesses the knowledge required to make an objective judgment. Fig. 5 exemplifies.
"made of alloy" (material), whether it is
4.3 TEXT-TO-IMAGE EVALUATION
We compare three recent T2I generation models: SD v2.1 (Rombach et al., 2022), Imagen* (Saharia al., 2022), and MUSE (Chang al., 2023), by measuring the average VQA scores calculated by our DSG. In addition to the global average scores, we calculate the scores on subsets of prompts (Table 1) of DSG-Ik understand how prompt styles affect the performance of the models. In particular, through this experiment, we intend to understand the extent to which the QG/A approach reliable in automating T2I alignment evaluation. For the evaluation of T2I generation models, we use the best QG+QA combination from the evaluations described above: DSG questions + PaLI.
Tab. 6 summarizes our results. Overall, PaLI predictions match human ratings in ranking model performance as Imagen* MUSE* SD v2.1. The score scaling however shows PaLI is overly optimistic (~ 80% accuracy) per human ratings (~ 60% accuracy). Observing the results per data sources, PaL.I ratings track humans' closely in the categories [TIFA160, Paragraph, Relation,
"Imagen* and MUSE model variants based the Imagen (Saharia et al., 2022) and Muse (Chang al., 2023) models, but parameterized differently and trained internal data sources.
8

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of three T2I models: SD v2.1 vs. Imagen vs. MUSE# in VQA scores by different DSG-IK prompt sources. QG: DSG-PaLMM;; QA: both PaLI and human raters.
DSG-IK
Prompt
T21 models
TIFA-160
Paragraph
Relation
Counting
Real users
Poses
Commonsensse defying
Text
Overall
Answerer:
SID v
88.
85.1
34.4
70.4
89.7
89.6
84.4
83.7
80.5
Imagen" MUSE"
95,1
92.1
40.6
72.1
91.1
89.3
82.8
56.0
83.9
91.8
39.5
72.5
89.7
84,9
Answerer: Human
74.9
48.0
46.9
Imagen" MUSE
88.0
86.2
32.8
47.0
49.4
73.1
79.6
56.4
66.2
86.3
85.5
31.2
45.2
44.2
69.4
73.4
52.8
63.1
Pronge phowod 1m01 am real
Proept signtet edegg smiting and Questher Dbes sign
Pronpe: bowdor the menragg the tow cerou Guentor Doen the Plumm arewer PC
Prompnt Appooti poovrtationn karg kest Dan paic Mockess
LAAA CORGIS
Gueston Does Be cran VGA WOAeERD
ASR
DOGS
Hu orawer NG
On Ouwertoe Covn bent
DFFISSION
Haman enower
Figure 6: Judgment matching between VQA (PaLI here) and human raters in text (text rendering) questions. Example I: Match, only account for ~30% of the cases. Example 2: Mismatch, prag- matics humans interpret the question "does the sign ONLY say 'no dogs ??". Example & 4: Mismatch, VQA almost invariably answer YES as long as some text is present.
Commonsenss-ddefyinge (the difference is explainable with PaLI's good yet imperfect VQA accu- racy (Tab, 4, at 73.8%)), while being considerably higher in categories [Counting Real users, Text]. For Counting, even the latest VQA models struggle to count reliably to 5: found PaLI is incorrect in ~ 40% of the cases (corroborated by recent research e.g. Paiss et al. (2023)). For Text, both T2I and VQA models fail the majority of the time (Fig. 6 exemplifies). In general, VQA per- formance is much lower in more abstract semantically higher-order categories like these (Tab. 16 presents further evidence with accuracy by fine fine-graing grained categories). For Real users, our findings coincide with Sec, 4.2: Subjectivity. Many prompts involve details such as "is the cyberpunk city highly detailed?". "is the art elegant?" which lead both VQA-human disagreement as well as disagreement among human raters. Domain knowledge. Some semantic elements are not reliably judgeable even by human raters, e.g., "is this image 8k", etc. Fig. exemplifies. Based on the results, believe for model-level eval semantically more concrete categories, current VQA models are capable to support QG/A approach, yet b) at the level particular semantic details (in particular the "difficult categories" shown above), the VQA predictions are yet sufficiently reliable.
CONCLUSION
The QG/A framework (Hu al., 2023; Yarom al., 2023; Cho et al., 2023b) leverages the power of pretrained LLMs and VQAs to have enabled much more fine-grained (thus informative) diagnostics T2I evaluation. However, they have various reliability issues, such as duplicated, invalid, and ambiguous questions. In this work, inspired by formal semantics (Davidson, 1965; 1967a;b), we propose Davidsonian Scene Graph (DSG), QG/A approach to present a solution. Comprehensive experimentation and human evaluation demonstrate that DSG better addresses the reliability issues than previous work. addition, its fine- lieeggrined question typing allows for even more in-depth diagnostics. Finally, we collect a benchmark dataset DSG-Ik to facilitate research in this area. Our work casts light on future directions in the QG/A framework: In both QG and QA, we observe that (a) subjectivity and domain knowledge may lead to model-huma disagreement, and (b) some semantic categories remain beyond the capabilities existing SoTA VQA models (e.g., text rendering).
well Human-human

---

Published as a conference paper at ICLR 2024
STATEMENTS
6.1 ETHICS STATEMENT
Our work involves human evaluation where human raters hired from Google's crowdsoure- ing platform, and asked to look at images and and text prompts then answer questions regarding text-image grounding. This work was carried out by participants who are paid contractors. Those contractors received a standard contracted wage, which complies with living wage laws in their country of employment. Due to global privacy concern,, we cannot include more details about our participants, e.g., estimated hourly wage or total amount spent on compensation.
6.2 REPRODUCIBILITY STATEMENT
All the experiments involving publicly available packages and tools are reproducible (the main text and appendix provide necessary details). This includes
GPT-3.5 (OpenAI, 2022) based precision, recall, and uniqueness evaluation described in Tab. 2;
VQA query results in Tab. 4, and Tab. with mPLUG-large (Ye et al., 2023) and Instruct- BLIP (Dai et al., 2023);
Stable Diffusion (Rombach al., 2022) produced images used in all parts of Sec. 4.3.
The other LLMs and VQAs used, specifically PaLM2 (Anil et al., 2023) and PaI.I (Chen et al., 2023) yet released. Further, the human evaluation results may not be exactly reproduced due to the use different human raters, however the general trend should retain to the extent that the conclusions drawn this work should not be impacted.
ACKNOWLEDGMENTS
We would like to thank Yonatan Bitton, Dennis Shtatnov, Shuai Tang, Dilip Krisnan for their gen- crous help through the course of the project. We give thanks to Muhammad Adil, Aditya Saraswat, Sudhindra Kopalle, Yiwen Luo and all the anonymous human annotators for assisting nate and complete the human evaluation tasks. are grateful to Arjun Akula and Radu Soricut for reviewing the paper and providing feedback. Thanks to Jason Baldridge for help with resource coordination.
to coordi-
REFERENCES
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellsterm, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hermandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan. Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li. YaGuang Li, Jian Li, Hyeontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu. Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif, Bryan Richter. Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue. Pengcheng Yin, Jiahui Yu, Qiao Zhang Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
10

---

Published as a conference paper at ICLR 2024
Nitzan Bitton-Guette Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and lannanug-- bench- mark of synthetic and compositional images. In ICCV, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Husnn Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In ICML, 2023.
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. In NAACL, 2022.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver. Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Sori- cut. Pali: A jointly-scaled multilingual language-image image model. In ICLR, 2023.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. ICCV, 2023a.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. In NeurlPS, 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, volume 0w./2305.06500 2023. URL https://api. semanticscholes wat. wtrg//CorusID:: 25861 12666.
Donald Davidson. Theories of meaning and learnable languages. In In Yehoshua Bar-Hillel (ed.), ), Proceedings of the 1964 International Congress for Logic, Methodology, and Philosophy of Sci- ence. Amsterdam: Nolth-Holland, pp. 383-394, 1965.
Donald Davidson. The logical form of action sentences. In N. Rescher (ed.) The Logic of Decision and Action, Pittsburgh: University of Pittsburgh, 1967a.
Donald Davidson. Truth and meaning. In Inquiries into Truth and Interpretation; Soames, chapter of PATC, 1967b.
Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francese Moreno- -oggue, and Grégory Ro- gez. PoseScript: 3D Human Poses from Natural Language. In ECCV, 2022.
Daniel Deutsch, Tania Bedrax-Weiss and Dan Roth. Towards Question-Answering as Automatic Metric for Evaluating the Content Quality of a Summary. Transactions of the Association for Computational Linguistics, 9:774-789, 08 2021. ISSN 2307-387X. doi: 16.1/6//tacl 00397.
URL httpps://of org/ 10. 1162/tacl 00397.
Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 5055-5070 Online, July 2020. As- sociation for Computational Linguistics. doi: 10. 10.18653/v 1/2020 acl- m5in.454. URL https: //aclanthology wrgpy/org//22// acl-main.455 454.
11

---

Published as a conference paper at ICLR 2024
Matan Eyal, Tal Baumel, and Michael Elhadad. Question answering an automatic evalua- tion metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 3938-3948, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/w1/N19-13955 URL http:s//aclannholoo..rrrro wwwpp//..og//11111330
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. CVPR, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2022.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text- to-image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3): 1552-1565, mar 2022. doi: 20.1009/paii..220.3022 20.22000022220000 URL https:://tii.oorrtte/oo 2Ftpami 2020. 3021209.
1109%
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurIPS, 2020. URL hntt:://arxia org/abs/2006 11239.
Seunghoon Hong, Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring Semantic Layout for Hierarchical Text-to-lmage Synthesis. In CVPR, 2018.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Accurate and interpretable texi-to-image faithfulness evaluation with question an- swering, In ICCV, 2023.
Smith. Tifa:
Jaided AI. Easyoer:
Ready-to-use ocr with 80+ supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc., 2023.
Saehoon Kim, Sanghun Cho, Chiheon Kim, Doyup Lee, and Woonhyuk Baek. mindall-e on con- 2021.
ceptual captions. https://githht: wwtp//aaaobrraanniiinaaaaaaa
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. hierarchical approach for generating descriptive image paragraphs. In 2017.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. In EMNLP, 2022.
Rosanne Liu, Daniel H Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, R. Mical, Mohammad Norouzi, and Noah Constant. Character-aware models im- prove visual text rendering. In ACL, 2023. URL /atps///tlaaaaoooo/ooooo www.//23....2./. oor....................o...g 900.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision long form text generation. In EMNLP, 2023.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection, 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive summariza- tion with reinforcement learning. In Proceedings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1747-1759, New Orleans, Louisiana, June 2018. Association for URL https: / ///////aaaaanlgggaaa/a/ae org/
Computational Linguistics. doi: (0.18653/v1/N18-1
N18-1158.
12

---

Published as a conference paper at ICLR 2024
OpenAl. Chatgpt, 2022. URL http:tss//ppppa..cc////////haaa
OpenAl. Gpt-4 technical report, 2023.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In ICCV, 2023.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume Cur- neurips...... cc/ papperrffiles// Para pdf.
ran, 2021. URL
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connect- ing vision and language with localized narratives. In ECCV, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. ArXiv, 2204.06125, 2022. URL http://
arxiv. org/ htt.:/ggg/bb///////// 2204.. 06125.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding. In NeruPPS, 2022.
Jascha Sohk-DDcks, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICMI,, 2015. ISBN 97815108 0881510110000 0810587.
Iulia Ture and Gaurav Nemade. Midjourney user prompts & generated images (250k), 2022. URL https: kaggle. cwtccs/ 2349267.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondh: A large-scale prompt gallery dataset for text-to-image gener- ative models. In ACL, 2023. URL htttps///rxiv. wrt..bg///////o/e abs. 14896.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang. and Xiaodong Text to Image Generation with Attentional Generative Adversarial Networks. CVPR, 2018. ISBN 978 178366664,,,
He. AttnGAN:
doi: (t/.////...01....000
Michal Yarom, Yonatan Bitton, Soravit Changpinyo. Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. In htpp://aartivorr/aa//23..3 10400, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content- Rich Text-to- Image Generation. Transactions on Machine Learning Research, 2022. URL http: //arxiv. htt.v/org/gbbb///////o 10789.
Appendix
In this appendix, we include LLM preambles (Appendix A), DSG-Ik dataset details (Appendix B), invalid VQA query examples (Appendix C), DSG abalation studies (Appendix D), additional QG/QA/T2I evaluation results (Appendix E), experiments mixed QA evaluation models (Ap- pendix F) Pseudocode of T21 evaluation with DSG (Appendix G), and human evaluation setup (Ap- pendix H).
13

---

Published as a conference paper at ICLR 2024
A LLM PREAMBLES
Fig. 7 presents the preambles for the three-stage automatic DSG generation pipeline, with PaLM2 (Anil et al., 2023).
Fig. presents the preambles for question evaluation in precision, recall, and uniqueness, with GPT-3.5 (OpenAI, 2022).
Tuple generation
given input prompts, describe each scene with skill-specific tuples. Do not generate tuples again. Do generate tuples that are not explicitly described in the prompts.
Task:
output format: id tuple $
Question generation
Task:
given input promots and skill-specific tuples, re-write tuple each in natural language question.
output format: id question
$(HUMAN ANNOTATION EXAMPLES}
Dependency generation
Task: given input prompts and tuples, describe the parent tuples of each tuple. output format: id dependencies (comma separated)
//HUMAN.ANOTAATOON.EAMPPP
Figure 7: The preambles for the three-step automatic DSG generation pipeline.
Precision
Task: model generated questions to check semantics of images generated from prompts. Given prompts (ground-truth), tuples that decompose the prompts (ground-truth), and questions generated by the model, check the ids the questions that are not entailed with the tuples. entailed questions: id of questions ids of tuples entailed wi th each question wrong questions: ids of questions that are not entailed with GT tuples.
Recall
Task: model generated questions to check semantics of images generated from prompts. Given prompts (ground-truth), tuples that decompose the prompts (ground-truth), and questions generated by the model, check the ids of tuples are covered by the questions. covered tuples: id of tuples ids of questions covering each tuple missed tuples: ids of missed tuples
Uniqueness
Task: model generated questions to check semantics of images generated from prompts. Given prompts (ground-truth), and questions generated by the model, check the ids of duplicated questions. output format: ids of duplicated questions
Figure 8: The preambles for the automatic evaluation of questions in precision/reaalluuiqqeeess.
B DSG-IK DATASET DETAILS
Images. We generate images from the prompts DSG-Ik using three recent SoTA text-to-image models (SoTA at the time of experimentation): Imagen* (Saharia et al., 2022). MUSE* (Chang et al., 2023), and SD v2.1 (Rombach et al., 2022). Imagen and SD are denoisng diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020). MUSE* is encoder-decoder model with parallel image decoding steps. In total, we provide 3x 1,,0003,,800 model -ener-egee images. For fair comparison with uniform model evaluation and human annotation, published by TIFA for TIFA160. This includes SD v1.1, v1.5 (Rombach al., 2022), miLL (Kim et al., 2021), and VQ-diffusion (Gu al., 2022).
also make of the generated images
14

---

Published as a conference paper at ICLR 2024
Questions, Answers and Alignment Scores. On TIFA160 prompts, set of three human experts manually and independently generate the semantic tuples and their dependencies (scene graph). Then, with the LLM- based DSG generation pipeline (Sec. 3), generate DSG (semantic tuples, questions, and dependencies) on the entire DSG-1k prompts. We show the generated questions together with the generated images to human annotators (i) annotate their validity (e.g., is not valid to ask whether the motorbike is red if there is no motorbike in the image) and (ii) answer the valid questions with Yes No. We also answer the same questions with automatic VQA models, specifically mPLUG-large (Ye et al., 2023), Instruct-BLII BLIP (Dai et al., 2023), and PaLI (Chen et al., 2023). Following TIFA, ask human annotators to collect 1-to-5 Likert-scale scores about T2I alignment on TIFA 160 for seven models, for a total of 7 x 160 = 1,120 text-image pairs. In experiments, we use human Likert annotations to measure the correlation between human and QG/A judgments.
Additional Statistics. The #questions tables below complement Fig. on the statistics for DSG-1k, In particular, the counts demonstrate the by-category analyses are valid in size statistically.
DSG-Ik
entity
attribute
relation
global
# questions
3,398
2,251
2,082
527
Table 7: DSG-IK statistics: The number of questions per broad semantic category.
ent-whole
ent-part
att-state
att-color
att-type
att-material
att-count
questions
2,751
930
427
98
att-size
utt-texture
att-text rendering
att-shape
rel-spatial
rel-scale
global
questions
1,808
274
Table 8: DSG-Ik Ik statistics: The number of questions per detailed semantic category. ent: entity, rel: relation, att: attribute.
Questions generated by TIFA
Text: A chair in the corner on a boat..
Text:
an astriich standing couch, an ostrich?
q1
what type of vehicte is this?
Is there couch?
the chaii in the carner?
Is ostrich standing?
bost?
GPT-3.5
Questions and 4 are diplicates because they both ask about the and location of the chair in the corner. duplicates:
Questions and 4 are duplicates because they both ask about the ostrich's paxition. duplicatess qa,ef
GPT-3.5 eval:
ql.04 uniqueness nyunique n_total
uniqueness
3/4
8.8
Human eval:
Huren eval:
dupl irates: Kone
Scplecat::: (g1,e4), (a?, 43) uniqueness
uniqueness n.anique nutocal 4/4 1.8
nunnique
Figure 9: Automatic uniqueness evaluation: despite the general high agreement between GPT-3.5 5 and human,
GPT-3.5 is sometimes distracted by text overlap to err. Left: duplidates q2 & 3 differ in wording. Right: q3 & 4 similar wording but are unique questions.
C INVALID VQA QUERY EXAMPLES
Fig. 10 provide examples of invalid VQA queries described in Sec. 1. In the bottom right image, VQA model answered "No" to the question "is there car?", but the VQA model answered "Yes" to the question "is the car playing soccer?". Such examples are more common when T2I model miss entities (more frequent in weaker models we evaluated, such minDALL-E and VQ-Diffusion).
15

---

Published as a conference paper at ICLR 2024
Prompt: man wet suit is surfing.
Prompt: cube made of denim. A cube with the texture of denim.
Root: there man? VQA NO
Root: there cubee? VQA answer: NO
Dependent: Is the man surfing? VQA YES
Dependent: the cub denim? VQA answer: YES
Prompt: photo of bike and stop sign: stop sign is below bike.
Prompt: pair of glasses under computer monitor.
Root: there bike? VQA answer: NC
Root: there corr corpputer monitor?
VQA answer: NO
700
Dependent: stop sign below the bike? VQA answer: YES
Dependent: of asses under the computer monitor?
VQA answer: YES
Prompt: photo of suitcase and bench; bench is left suitcase
Prompt: A car playing soccer, digital art.
Root: there car? VQA answer: NO
Root: there suitcase?
VQA answer: NC
Dependent: car playing soccer?
Dependent: the bench left to the suitcase?
VQA answer: YES
VQA answer: YES
Figure 10: Examples of invalid VQA queries where a dependent question gets positive answer when its root question gets negative. For example, in the bottom right image, VQA model answered "No" to the question "is there a car?", but the VQA model answered "Yes" to the question "is the car playing soccer?". Such examples in weaker models we evaluated, such as minDALL-E and VQ-Diffusion). Fundamentally however this is issue of posing invalid queries VQA that should not be posed in the first place.
common when T21 model miss entities (more frequent
D DSG ABLATION
D.1 WAYS TO UTILIZE DEPENDENCY
We ablate on three ways in which DSG questions are administrated to VQA modules based on question dependencies.
1. DSG. This the method we present in the main text - the answer to a parent question is NO, all the answers its child questions counted as NO well (e.g., "is there bike?" NO + "is the bike blue?" :- := NO).
2. DSG (drop). the answer to a parent question NO, drop its child questions by not counting their answers towards the calculaton of the VQA accuracy for the text-image pair item.
3. DSG (w/o dep). Disregarding the dependency annotation and factoring in the answers to all questions equally in VQA accuracy calculation.
In gist, we found
16

---

Published as a conference paper at ICLR 2024
A. In relation the correlation between VQA accuracy (auto) and 1-5 Likert judgments (hu- man), the different ways to handle dependency does not lead to significant effect. Tab. 9. B. In the per-question cies (with either DSG DSG (drop) improves performance substantially. Tab. 10.
matching accuracy, on the other hand, utilizing dependen-
Based on the observation, we argue that A as an aggregated reflection on the general correlational trend does not tell the whole story B the other hand indicates that human raters do implicitly take the question dependency into consideration, which highlights the value of incorporating de- pendencies in improving the reliability and interpretability of automatic evaluation results the fine-grained per-item/question level.
VQA models
DSG
DSG (drop)
DSG (w/o dep)
laLuG-large
0.463/0.380
0.462/0.370
0,464/0.378
Instruct-BLIP
0.442/0.363
0.437/0.355
0,436/0.355
PaLI
0.571/0.458
0.561/0.451
0.570/0.457
Table 9: DSG ablation wrt. the correlation (Spearman's p Kendall's r) between VQA accuracy and human 1-5 Likert score.
VQA models
DSG
DSG (drop)
DSG (w/o dep)
mPLUU-large
72.1
69.9
64.7
Instruct-BIIP
70.4
68.4
64.0
PaLI
73.8
71.4
65.2
Table 10: DSG ablation wrt. the per-question VQA-human matching accuracy.
D.2 BY-CATEGORY ABLATION FOR TIFA, VQ"A, AND DSG
TIFA160 comes with data splits:
(a) COCO. General purpose;
(b) PaintSkill. Focuses on spatial relations.
(C) PartiPrompts. Focuses on attributional failure cases observed (per Yu et al. (2022));
(d) DrawBench. Focuses on compositionality.
Leveraging the full human annotations on TIFA160, we ablate across TIFA, VQ2A, and DSG on the data splits in the correlation between VQA results and human Likert judgments, as a further breakdown and extension for Tab. 3. Tab. 11 summarizes the results.
COCO
PaintSkill
PartiPrompts
DrawBench
Overall
TIFA
0.30/0.22
0.56/0.40
0.49/0.37
0.59/0.46
0.43/0.32
VQ DSG
0.17/0.11
0.25/0.17
0.28/0.21
0.45/0.37
0.21/0.16
0.53/0.42
0.64/0.53
0.61/0.49
0.56/0.44
0.57/0.46
Table 11: Ablation by data split for TIFA160 across QG/A methods the correlation between VQA score and human 1-5 Likert score (Spearman's p Kendall's T). The VQA model used: PaLI.
E ADDITIONAL QG/QA/T2I EVALUATION RESULTS
Single-summary metric context. To contextualize the correlation results between VQA scores and human 1-5 Likerts (Tab. 3) in wider context with single-summary metrics (e.g., CLIPScore), present the correlation results between the single-summary scores investigated in TIFA (Hu et al., 2023) with the human scores we collected. Following Hu al. (2023), report 5 single-summary
Note: The results reported in the Table 2 in Hu et al. (2023) are not directly applicable as the human collected with different human
17

---

Published as a conference paper at ICLR 2024
metrics: BLEU-4, ROUGE-L. METEOR, SPICE, (with BLIP-2 Captioning) and CLIPScore (with CLIP ViT-B/32), in Tab. 12. While some single-summary correlations are higher than TIFA / V in some configurations, DSG, the other hand, results in much higher correlations with all VQA module (see Tab. 3 for details).
Metric BLEU-4
ROUGE-L
METEOR
SPICE
CLIPScore
P/T 0.261/0.183
0.342/0.239
0.379/0.269
0.329/0.236
0.276/0.191
Table 12: Spearman (p) / Kendall (r) correlation between single-summary metrics and human 1-5 Likerts on TIFA160 prompts (across the same 5 models per Hu et al. (2023)).
QG evaluation. Fig. we show some example mismatch case human evaluation and GPT-3.5 based automatic evaluation uniqueness, described in Sec. 4.1.
QA evaluation. Table 13, we also summarize the VQA-human match accuracy by DSG-1K 1k data sources. The trend matches the observation in the by-category results in Table 4 and 5.
DSG-Ik Prompt Sources
VQA models
TIFA-160
Paragraph
Relation
Counting
Real users
Poscs
Commonsense-defying
Text
mPLUG-large
86.4
84.7
80.4
65.7
46.1
69.0
80.4
60.3
Instruct-BLIIP
86.3
83.6
71.1
60.2
47.3
69.9
81.0
$7.0
PaLI
87.9
84.9
66.1
47.6
82.4
Table 13: VQA- answer match accuracy on 3 T2I generation models on DSG-Ik prompts, by data sources/segments. We use DSG-PaL.M2 questions.
minDALL-E
VQ-Diffusion
SD vl.1
SD v1.5
SD v2.1
Imagen*
3.839
3.608
3.731
3,990
4.146
4.399
Table 14: Avg. Likert (1-5) on T2I generation models on TIFA 160 prompts.
T21 evaluation. Table 14 shows the average Likert (1-5) scores of T2I generation models on TIFA160 prompts. Table 15 and Table 16 present VQA-human correlation and accuracy results by fine une-grained semantic categories. In Table 15, we calculate for categories with 2 30 samples and keep results with p < 4.
F EXPERIMENTS ON MIXED QA EVALUATION MODELS
In the QG/A framework. in relation question answering specifically, an alternative strategy has been explored in concurrent work. One prominent representative is VPEval (Cho et al., 2023b), where non- non-VQA VQA modules are applied in corresponding semantic categories specifically a) object detection (OD) for counting, and b) optical character recognition (OCR) for text rendering.
Concretely, for evaluation queries measuring counting skill (e.g., "Are there five Halloween ban- ners?"), an OD module detect bounding boxes with the target object name mentioned in the ques- tion. The queries are answered as 'yes' if the number of detected bounding boxes is equal to the number mentioned in the question. For evaluation queries measuring text rendering skill (e.g., "Do the letters say 'Contour'?" ?"), an OCR module on detects text from images. The queries are answered 'yes' if the target text captured.
Overall, with the assistance of domain sppeifc models, we observe difference performance in the VQA-human alignment slightly lower performance with OD (Tab. 17), and noticeable (yet not categorically substantial) bump with OCR (Tab. 18). The results are consistent with the findings in Cho et al. (2023b).
18

---

Published as a conference paper at ICLR 2024
Table 15: VQA-Human correlation. VQA: proportion of correct answers; Human: 1-5 Likert. Split by fine-grained semantic categories, measured with Spearman's 's and Kendall's 7. ent: entity, rel: relation, aff: attribute. (Data: TIFA160; Categories: DSG- Ik question types).
VQA models
DSG-IK Question types
(out) whole
(rel) spatial
(att) state
(ent) part
(global)
(att) color
(rel) action
(alt) type
PaLI
0,499/0,397
0,518/0.423
0,401/0,330
0,243/0.213 0,4100.332
0.291/0.267
O. 01389/0.311
0.359/0,310
(att) count
(art) text rendering (att) material
(att) shape
(att) style
(art) texture
(ai1)
P
09346/0.319
Table 16: T2I model VQA accuracy (SD v2.1 Imagen* MUSE*) (averaged over text-image items). By fine-grained semantic categories. Questions generated with DSG-PaL.M2, answered with both Pal. and human. ent: entity, rel: relation, atf: attribute.
DSG- Question types
121 models
(ent) whole
(rel) spatial
(att) state
(cat)
(global)
(att) color
(rel) action
(att) type
Answerer: PuL.
SD v2.1
$1.9
63.8
85.9
84.4
90.7
76.8
73.9
78.2
Imagen" MUSE
71.2
87.6
87.6
90.7
87.4
78.3
81.9
83.8
70.1
88.0
88.2
93.1
87.1
75.0
80.2
Answerer: Human
SD v2.1
68.2
46.3
52.6
76.1
48.9
67.3
46.9
44.7
Imagen MUSE*
71.4
55.8
58.3
81,3
46.2
78.7
54.6
41.6
(att) count
(all) text rendering
(alt) material
(att) shupe
(alt) stylle
(att) texture
(att) size
PaLl
SD v2.1 Imagen" MUSE
73.5
70.6
73.2
82.0
79.6
71.8
80.0
80.1
70.6
81.4
86.0
77.6
94.9
85.7
77.3
72.5
78.4
88.0
83.7
89.7
94.3
Answerer: Human
v2.1
44.9
5.1
40.7
45.3
14.6
47.4
53.2
Imagen
49.7
51.2
55.6
17.6
72.4
63.1
51.5
46.1
56.9
62.2
OD for counting. Across TIFA, VQ"A, and PaLI, we compare direct VQA query with the al- ternative query strategy with OWWL2-base-patee (Minderer et al., 2023) the DSG-Ik split dedicated to counting CountBench (Paiss et al., 2023). The result is summarized in Tab. 17.
VQA models
direct VQA query (ours)
+OD for counting
mPL.UG largee
65.7
64.1 (-1.6)
Instruct-BLIP
58.9 (-1.3)
Pal.1
66.1
64.9
Table 17: VQA-human matching accuracy of VQA models with direct VQA query vs. +OD (w/ VPEval), Data split: CountBench.
OCR for text rendering. Similar to the experiment with OD, on the DSG-IK Ik split on text rendering - DrawText (Liu al., 2023), again compare direct VQA query with the alternative query strategy with EasyOCR (Jaided AI, 2023) integrated per VPEval. The result summarized in Tab. 18.
VQA models
disect VQA query (ours)
+OCR for text readering
UPLUG-large
60.3
64.5 (+4.2)
Instruct-BLiP
57.0
63.5 (+6.5)
PaLl
60.3
66.8 (+6.5)
Table 18; VQA-human matching accuracy of VQA models with direct VQA query vs. +OCR (w/ VPÉval). Data Split: DrawText.
19

---

Published as a conference paper at ICLR 2024
G PSEUDOCODE OF T21 EVALUATION WITH DSG.
Algorithm shows Python pseudocode demonstrating the T2I evaluation pipeline with DSG.
Algorithm Python pseudocode of T2I evaluation with DSG
PROMPT_TUPEE """Task: given input prompts, describe each scene with skill-specific tuples
PROMPT_ DEPENNEEPY """Task: given input prompts and tuples, describe the parent tuples of each tuple
PROMPT_QUESTICN """Task: given input prompts and skill-specific tuples, re-write tuple each in natural language question ...
mun
def (texx,, LLM):
generate DSG (tuples, dependency, and questions) from text""*
# 1) generate atomic semantic tuples
# output: dictionary of (tuple id: semantic tuple}
id2tuples LLM(text, PROMPT_TUPLE)
# generate dependency graph from the tuples
# output: dictionary of (tuple id: ids of parent tuples)
idddependency LLM(text, id2tuples, PROMPT_DEPENDENCY)
# 3) generate questions from the tuples
# output: dictionary of (tuple id: ids of generated questions)
id2questions LLM(text, id2tuples, PROMPT_QUESTION return id2tuples, id2dependency, id2questions
def evaluate__nne iiaaae_dag(ett
pnperated_imagee VQA, LLM): """evaluate generated image with DSG # generate DSG from text
id2tuples, id2dependency, id2questions generate_dsg(text, LLM)
# 2) answer questions with the generated image id2scores O
for id, question in id2questions. items itmeiiiooonniii(())) ()
answer VQA(generated__iegg question) id2scores[id] (anawer 'yes')
# 3) zero-out scores from invalid questions for id, parent_ids in id2dependency items():
# zero-out scores if parent questions answered 'no' uny_parrent_anwerred..o
False
for parent_id in parent_ide
if id2scores(parent_idd == 0:
any_paarent_answered.mm True break wwy_parret__nnserred.og
id2scores[id] =
# 4) calculate the final score by averaging
average_score sum(id2scores. values()) / (en(id2scores) return average_score
H HUMAN EVALUATION SETUP
Fig. 11 and Fig. exemplify the data collection UI used in our human answer/rating collection.
20

---

Published as a conference paper at ICLR 2024
PROMPT
stained window depicting calm tyrannosaurus
Is this stained glass window?
YES INVALID QUESTION
what window this?
O O
is there Syrannosuurus rex?
O YES O INVALID QUESTION
what animal the picturo?
O TYRANNOSAURUS REX O TRICERATOPS
O STEGOSAURUS O VELOCIRAPTOR
O INVALID QUESTION
the stained glass window depicting tyranossuurus rex?
O YES O INVALID QUESTION
is the tyrannosaurus rEx calm?
O YES O NO O INVALID QUESTION
Is the tyrannosaurus rex calm angry?
CALM O ANGRY INVALID QUESTION
SAD HAPPY
Figure 11: for data collection per-question human judgment annotation.
PROMPT
white cat with black ears and markings
Consistency: How consistent the image with the text description?
Consistent
O 5 Consistent O Unsure
Figure 12: UI for data collection in per-item human judgment annotation (1-5 Likert consistency rating).
21


Reviews:
[{"review_id": "rXP9hxnXeK", "reviewer": "Reviewer_tDEJ", "summary": "The paper addresses the task of evaluating text-to-image models, specifically focusing on the question generation and answering method. This method automatically generate questions and answers from the prompt, and the faithfulness of the image is assessed based on the consistency of the answers from both prompt and visual question answering models. The authors identify and tackle key reliability challenges in this approach, including the quality of generated questions and the consistency of visual question answering.\nTo overcome these challenges, the authors introduce the Davidsonian Scene Graph (DSG), which produces atomic and unique questions organized in dependency graphs, to ensure questions cover semantic of the prompt and that answers are consistent.\nThe paper provides experimental results and human evaluations, demonstrating that DSG addresses the reliability challenges mentioned earlier. Additionally, the authors introduce DSG-1k, an open-sourced evaluation benchmark with 1,060 prompts covering a wide range of fine-grained semantic categories.", "questions": "1.\tIn 4.1, to validate the question dependencies, authors evaluate manually on 30 samples and automatically on the full TIFA160. However, the consistency of manual and automatic evaluation is not presented.\n\n2.\tThe comparison of runtime among different methods should be added. \n\n3.\tIn table 3, for Instruct-BLIP, the Spearman’s ρ of DSG is lower than that of TIFA, authors should explain this phenomenon briefly.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper introduces Davidsonian Scene Graph (DSG) to improve the faithfulness of the text-to-image evaluation. Compared to previous QG/A methods, this framework generates  atomic questions with full semantic coverage and valid question dependency. The authors implement QG step as a Directed Acyclic Graph (DAG) where the nodes represent the unique questions and they explicitly model semantic dependencies with directed edges. Additionally, they collect a fine-grained human-annotated benchmark called DSG-1k including 1,060 diverse prompts with a balanced distribution to facilitate research in this area.", "weaknesses": "1.\tThis paper improves the existing QG/A methods with Davidsonian Scene graph, which is generated based on LLMs. The approach of the work could be enriched with more details and techniques. \n\n2.\tAblation on separate steps of DSG should be presented. For example, without establishing dependencies, measure the changes of consistency between VQA score and the human 1-5 Likert Scores.\n\n3.\tApart from TIFA and VQ2A, more methods could be compared, including CLIPScore and caption based approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "goqirvrQUN", "reviewer": "Reviewer_A5bo", "summary": "This paper focuses on text-to-image generation evaluation by asking text-related questions and checking whether a VQA model can answer it given the generated image. It suggests previous QG/A frameworks usually ask ambiguous, duplicated, and invalid questions. The paper parses the input text input atomic entity/attribute/relation tuples, translates each tuple into questions, and obtains their dependencies through an LLM. The experimental results show the generated questions are unique, have valid entailment, and query atomic semantics. The paper lastly constructs a dataset with the proposed methods.", "questions": "Can a better VQA model lead to better alignment in Tables 3, 4, and 6?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has strong motivation. It first analyses the drawbacks of previous QG/A methods, then proposes some principles for the generated questions, and lastly introduces a three-step prompting method to resolve it.\n\nThe experimental results are strong enough to support the effectiveness of the proposed method in the proposed uniqueness, valid dependency, and human alignment.\n\nThe paper is well-written, and the related work is sufficient.", "weaknesses": "The VQA model is not good enough and hinders the final alignment to humans in T2I evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WjY1aSyAxu", "reviewer": "Reviewer_p5Xv", "summary": "This paper studies the problem of fine-grained evaluation of text-to-image (T2I) alignment. Following a line of recent works, this paper formulates T2I alignment evaluation as visual question answering (VQA), which involves question generation from the text prompt using LLMs and question answering using VQA models (referred to as the QG/A framework).\n\nThe main contributions of this paper are:\n* It identifies the issues of existing QG/A methods regarding the question generation step and proposes four properties that the generated questions should satisfy.\n* It proposes a QG method that constructs the questions of a given text prompt as a Davidsonian Scene Graph (DSG). DSG considers the dependency between questions and is designed to satisfy the four properties.\n* It collects a test set of 1060 prompts, which covers different challenges, semantic categories and writing styles, together with Likert-scale T2I alignment rating by humans.\n* The experiments demonstrate that DSG, when combined with different VQA models, achieves a higher correlation with human evaluation compared with existing QG/A frameworks.\n* It further reveals two challenges of QG/A frameworks for T2I evaluation: (1) Some question categories (e.g., shape, style and text rendering) are beyond the capability of current SOTA VQA models to evaluate. (2) For questions that involve “subjectivity” and “domain knowledge”, agreement is hard to achieve even between humans.", "questions": "* It is stated on page 4 that DSG only involve binary yes/no questions. Why are there multi-choice questions in the human annotation UI in Figure 10?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This work reveals the issues of existing QG/A methods in terms of the QG step, which is well-motivated and the proposed four desired properties of QG are reasonable.\n* The experiments demonstrate that the proposed DSG achieves solid improvement over existing QG/A frameworks in terms of correlation with human evaluation, which advances the reliability of automatic T2I alignment evaluation.\n* This work is transparent about its limitations, shedding light on directions for future studies to work on.\n* The paper is well-written and easy to follow.", "weaknesses": "### The design of DSG and DSG-1k\n* It is not well-explained which part of DSG is designed to address the **Atomic Question** and **Full Semantic Coverage** properties. Why the QG methods of TIFA and VQ^A cannot achieve these two properties?\n* Some categories are under-represented in the DSG-1k dataset. According to Table 9, six categories have less than 30 examples and VQA-Human correlation is not computed for these categories.\n\n### Evaluation\n* There is no ablation study showing which design choice in DSG contributes to each of the four properties.\n* It is unclear whether DSG correlates better with humans, compared with TIFA and VQ^A, in each fine-grained category.\n* The relationship between the claimed desired question properties and the final VQA-Human correlation is not well demonstrated. In other words, the relationship between Precision/Recall/Atomicity/Uniqueness in Table 2 and Spearman/Kendall correlation in Table 3 is unclear.\n* There is no comparison with TIFA and VQ^2A in terms of precision and recall in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)