Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)
Authors: Michael Saxon
Fatima Jahara
Mahsa Khoshnoodi
Yujie Lu
Aditya Sharma
William Yang Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: text-to-image
t2i
metric
meta-metric
analysis
Primary area: evaluation
TL;DR: We propose a meta-metric for objectively evaluating text-to-image faithfulness metrics, by assessing their ability to sort and discriminate images of increasing error counts in a "Semantic Error Graph", and many existing metrics are surprisingly bad.

Abstract: With advances in the quality of text-to-image (T2I) models has come interest in benchmarking their prompt faithfulness---the semantic coherence of generated images to the prompts they were conditioned on. A variety of T2I faithfulness metrics have been proposed, leveraging advances in cross-modal embeddings and vision-language models (VLMs). However, these metrics are not rigorously compared and benchmarked, instead presented with correlation to human Likert scores over a set of easy-to-discriminate images against seemingly weak baselines. 

We introduce T2IScoreScore, a curated set of semantic error graphs containing a prompt and a set of increasingly erroneous images. These allow us to rigorously judge whether a given prompt faithfulness metric can correctly order images with respect to their objective error count and significantly discriminate  between different error nodes, using meta-metric scores derived from established statistical tests. Surprisingly, we find that the state-of-the-art VLM-based metrics (e.g., TIFA, DSG, LLMScore, VIEScore) we tested fail to significantly outperform simple (and supposedly worse) feature-based metrics like CLIPScore, particularly on a hard subset of naturally-occurring T2I model errors. TS2 will enable the development of better T2I prompt faithfulness metrics through more rigorous comparison of their conformity to expected orderings and separations under objective criteria.

Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T215core5core (TS2)
Michael Saxon
Fatima Jahara 802
Mahsa Khoshnoodi
Yujie Lu
Aditya Sharma
William Yang Wang @
University of California, Santa Barbara
Rutgers University Equal contribution
Fatima Al-Fihri Predoctoral Fellowship
Contact: saxon@ucsb. edu
T2IScoreScore. giw.hu..biitttiioo........
Dataset
Meta-metrics
Node pai-wise soparation scores
Prompt A
shirt fruit
Imagesiise scores
Falthfunnes Metrics
Separation:
Node pair Sep.
Semantic Error Graph
Img
K-8 = 0.5
0.1a
BIT
0.75
0, 1b
0.5
5-U
18
0.56
1w,
GfT
CLiPScore
1:2
3-1
16
16
0.50
0.79
Ordering: Sevemen Den
Node seq Ord.
0.18 Za 0.86
2-01
28
0b 2a 0.57
2m
Metric
1b, 2b 0.68
Pg
SEG walk--wriee ardering
Figure 1: Overview of T2IScoreScore. T2I evaluation metrics scored based their ability to correctly organize images in a semantic error graph (SEG) relative to their generating prompt, checking ordering (Spearman's p) and separation of nodes (Kolmogorov-Smiroov statistic).
Abstract
With advances in the quality of text-oo-mmae (T2I) models has come interest benchmarking their prompt faithfulness-the semantic coherence of generated images to the prompts they were conditioned on. A variety T2I faithfulness metries have been proposed, leveraging advances in cross-modal embeddings and lision-language models (VLMs). However, these metrics are not rigorously compared and benchmarked, instead presented with correlation to human Likert scores over a set of masy-to-diseriminate images against seemingly weak baselines.
We introduce T2IScoreScore, curated set of semantic error graphs containing a prompt and set of increasingly erroneous images. These allow us to rigorously judge whether given prompt faithfulness metric can correctly order images with respect to their objective error count and significantly discriminate between different error nodes, using meta-metric scores derived from established statistical tests. Surprisingly, we find that the state-of-the thee-arrt VLM-based metrics (e.g.. TIFA, DSG, LLMScore, VIEScore) tested fail significantly outperform simple (and supposedly worse) ciature-based metrics like CLIPScore, particularly on hard subset of naturally-occurrinn T2I model errors. TS2 will enable the development of better T2I prompt faithfulness metrics through more rigorous comparison of their conformity to expected orderings and separations under objective criteria.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Text-to-image (T2I) models improving breakneck pace in terms of quality, fidelity, and coherence of generated images to their conditioning prompts [1-4]. Despite this, persistent challenges in achieving poage-prompp faithfulness [5. 6] remain-partieularyy particularly in freely available models that don't I sit behind proprietary APls. Indeed, many techniques to improve T21 models have been proposed of late, aiming reduce hallucination [7, 8), duplication [9], composition errors [8, 10], and missing objects (11, 12]. However, there is no consensus on how to best compare these many models and methods, so it is hard objectively track T2I progress [13, 14].
Recent work has proposed a litany of automated image-prompt coherence metrics which rate the faithfulness of generated images: the degree to which a they satisfy the implicit requirements set forth in the generating prompt [15-18]. These proposed metrics vary considerably in design; as rating how well an image matches to its prompt is nontrivial multimodal challenge [14, 19, 20].
This variety itself presents mela-evaluation problem: there is no consensus on how these faithful- ness metries ought to be compared, and consequently each new metric is validated on its own ad-hoc test set against prior baselines. Typically these self-evaluations consist of a set of prompt-image pairs with accompanying human annotations (usually simple Likert scores [16, 19]), and metrics are judged on their correlation to these human judgements [20].
Such self-evaluation is not ideal; authors may unwittingly tilt the scales by using evaluation examples which cater to the particular strengths of their proposed method, and variance of metric performance between different evaluation sets (containing different images and prompt semantics [21, 22]) is high [23]. Additionally. relying on correlation to human judgements of small sets of images across different prompts is highly subjective [24, 25] and prone to including judgements of quality and style that are orthogonal to prompt coherence. We need consistent and objective meta-evaluation.
To this end we propose TZIScoreScore (TS2), a benchmark and set of meta-metrics for evaluating T2I faithfulness metrics. While it contains a similar number of images to previously proposed coherence metric evaluation sets, it contains fewer prompts. This high image-to-promp, ratio allows us to organize the images along semantic error graphs, SEGs (fig. 1), where each edge corresponds to specific error with respect to the prompt that a child image set possesses but its parent images do not. These semantic error graphs permit objective scoring of a metric by answering:
1. Can a metric correctly order increasingly wrong images against their generating prompt?
2. Can a metrie reliably separate sets of images that differ by a specifie semantic error?
3. Does the metric confidently separate the image sets within its dynamic range?
We adapt existing statistical tests [26, 27] to the SEG setting answer these questions for broad set of T2I faithfulness metrics. We find some surprising results: despite their inferior performance in correlating human preferences against complicated vision-language model (VLM)-base metrics [6, 16-18], simple bebbeedding-oorelattii methods like CLIPScore [15] are actually quite performant on our meta-metrics, and Pareto-optimal with respect compute cost ($5). In summary, we:
Formalize the task of objectively assessing T2I prompt coherence metrics by their ability to correctly order and separate image populations within semantic error graphs (SEGs). ($2)
Present T2IScoreScore (TS2), our evaluation for this task: a carefully-curated benchmark dataset of SEGs each containing between 4 and 76 images, permitting 93,000 total pairwise image comparisons and meta-metrics for ordering and separation in SEGs. ($2, $3)
Evaluate a broad and representative set of T2I faithfulness benchmarks using TS2, demonstrate that it identifies novel failure cases, and motivate future work on improved metrics. (54 $5, $6)
1.1 Related Work
Most evaluations in the T2I space test the quality of generating models based on a fixed faithfulness metric's 's scores over a fixed benchmark set of prompts. Often these reference prompt sets are designed for testing a single specific capability. DrawBench [4], 121-CompBench [28], and ABC-6k [8] focus on attributes like compositionality, cardinality, and spatial relations, in strictly text-guided image generation, while ImagenHub [14] tests them in broader set of settings like image editing and subject-driven synthesis. Other evaluation dimensions such as multilinguality [29, 30] and stereotype
bias [31] have also been explored. These prompts are usually sourced from some combination of

---

# Images
Ing. per Prompt
# Img Comparisons
Ad-hoc
Dataset
Total
T2n-Gen.
Avg
Per equiv. pref
Total T21 Errors
Benchmarks for captioning models.
Flickr8k [32]
8k
0
0.2
0.2
8k
0
Flickr30k [33]
31k
0
0.2
0.2
31k
0
MSCOCO Captions [34]
330k
0
0.22
0.22
330k
0
Benchmarks for image retrieval/matching models. (Could be used for T2I metric evaluation)
SecTRUD [38]
31k
0
1
31k
0
Pick-a-Pic [37]
500k
500k
2)
7M
0
Benchmarks for T2I faithfulnnss metrics.
TIFA v1.0 [16]
800
800
5
4k
0
DSG-IK l [16]
1k
1k
1
1
1k
0
T2IScoreScore
2.8k
2690
17
3,4
93k
3.1k
Table 1: Comparison of benchmark datasets that can be used to evaluate T2I faithfulness. Per equiv pref means the average number of images for each prompt that are assigned the same preference or correctness score. Bold numbers are best overall, italic are best of the T2I metric benchmarks.
existing natural image captioning resources [32-34] and sets of in-the- wild conditioning prompts produced by real users [35, 36). These benchmarks assess image quality either by direct human analysis or automated metrics [14] including those we analyze in this work. Often the goal of these benchmarks is to analyze how well a model generates images that comport with human preferences [14, 37], and directly elicit opinions from users through web interface for this purpose.
To meta-evaluate faithfulness metrics benchmark containing a fixed set of images and prompts is required. However, all existing benchmarks for this purpose suffer from two key limitations:
1. Evaluating on noisy human preference rather than explicitly labeled objective differences Low mmage-to-promp ratio limiting evaluation of discriminatory power over similar images
Captioning benchmarks [32-34] are poor candidates for faithfulness metric evaluation as single images are paired with multiple prompt candidates rather than vice versa. Image matching and entailment benchmarks such as SecTRUE [38] and Pick-a-Pic [37] are also limited by low ratio.
The few extant deliberately-designei
faithfulness evaluation sets are limited by both factors. TIFA v1.0 [16] and DSG- 1k [6] were proposed ad-hoc to demonstrate the utility of their accompanying metrics by relating the scores assigned by the metric to human preferences. These done over small sets of images (800 & 1000 respectively) with slightly more images- per- per-prompt (5 & 1).
The two limitations of these prior evals are linked. A reliance on human preference correlations is natural consequence of having few and poorly organized images to compare to each prompt. A lack of meta-metrics designed for evaluating structured aspects other than human preference correlation means limited utility in collecting larger, structured sets of images for each prompt. By providing both meta-metrics and structured eval set T2IScoreScore overcomes these limitations (table 1).
2 T2IScoreScore meta-metrics
We introduce three measures of ordering and separation by a given metric within semantic error graphs (SEGs). We define SEG S as prompt P and a directed acyclic graph of nodes n; containing one or more images I, sharing the same errors wrt. the prompt. We label each node by its error count and type (eg, [0, la, 1b] has 1 node with 0 errors, and 2 nodes with I error each of different type).
good prompt coherence metric will correctly rank images along each walk of increasing error counts within SEG, and separate the scores assigned to images in successive nodes. Our metrics assess this by evaluating each walk separately. For ease notation, we refer to each SEG as a set of walks W E S over nodes of increasing error count (eg, (0, 1a, 2a). (0, 1a, 2b), etc). where each walk the in-order set of all (image, prompt, num. error) triples (T, P, N) E W. For example, the first walk in fig. is [(0-0. j.jp...p................... P. 0), (0- 1. jpg P. 0), (1- jpg. P. (). j.g. P.2). ...].
We introduce measures of metric for SEG S: rankm, (S), (S). sep (S) & deltam (S), assessed over every walk W E S in all SEGs in the TS2 dataset to score metric. They 'r defined as:
Although this benchmark has 14 pairs of images per prompt, each pair is separately annotated. With no way compare between pairs, the effective number of images per prompt with many repeated prompts.
3

---

2.1 Ordering score over walks: rankm
We use Spearman's ' rank correlation coefficient p [26] between image-level error count and metric- assigned score over every walk on a SEG to assess how a metric's faithfulness score aligns to our objective structure error counts. Spearman's p the PCC the rank order of variables X, Y:
p(x, Y) cov(H(X),
R(X)
E 1(s <x) E 2,eX
(I)
R((
Thus, our case the SEG-level rank order score rankm (5) for scoring model m is defined as:
1
rankmm (S) =
ra((m((, P,N) E WT. S
(2)
One limitation of Spearman"s p for characterizing scores is that it is undefined if one R(U) exclusively contains identical elements, as ((U) = O. For tractability in these scenarios we define p(s, R(U)) := O. If a metric assigns identical scores to all examples across different error levels, it presents discernible relationship between error severity and score for that image set.
2.2 Statistical separation of error populations score: sepm
We assess the two-sample Kolmogorov Simooo statistic [27] pairwise between the populations of metric mn's scores assigned to each sample between two error nodes n, and ny as populations. The Kolmogoroy-Sminov statistic is non-parametrie measure of the separation between two distributions [39, 40], defined as the maximum vertical difference between their empirical cumulative distribution functions (s):
DRs(X,Y) sup - Fy (x))
arE R.
Where Fx (ar) is proportion of samples in population X for which the metric-assigned score m(i) S 2, (see fig. for a visual depiction). We compute Dxs for every pair of adjacent error nodes in each tree walk W and report the average over all of these the SEG separation score sep... (S):
sep sep...S) (S)
Σ Dks((m((.,D((PP)) E m.), (IP,) E eme)a)
(4)
S
2.3 Separation of nodes within dynamic range: deltam
While sep, nonparametrically estimates whether pairs nodes are drawn from different distributions (and thereby distinguished), it provides no information about the distance by which they are separated within the metric's s dynamic range. This measure gives an alternative look at separation between nodes: the more separation metric provides between nodes, the less severe slight variations in assigned score will be to ignoring errors in generated images. We assess it as:
avg({m(P,D)((p)) e N.)) (e((Hm(P,D))()))) E N+11) (5) deltam(S) = S (MS) N:EW
Σ
Where mi(s) is the standard deviation of scores from metric m on all images in all SEGs in TS2. Our score is the average distance between the mean metric score of all adjacent nodes in all SEGs, rescaled by the standard deviation score to normalize against the metric's dynamic range.
3 The T2IScoreScore Dataset
We now turn to describing the TS2 dataset collection process. We use three different semantic error graph collection procedures produce a diverse set of SEGs. Each contains prompt and between 4 and 76 images assigned to error nodes. Each node usually contains more than one image, though for simplicity in presentation only show image assigned each node in this section.
We use individual nodes n, E S containing pairs of prompt, image (P,2) to make this equation easier read.

---

Synthetic Error, Image
Synth. Error, Natural Image (Nat)
Real Error, Synth. Image (Reaf)
(Synth)
Starting Christmas with lights and bodaly bear
Image
Free Online Stock
Starting gry olephaan and pisi Ramingo
Image Repositories
Error Graph Design (by accurnlsation)
Image Synthesis Attempts (T21
lights
No errors
lights or beer
No bear
Human curation and sorting relation graph
Graph-informed prompt edits
Brown conut
Orange conut.
Orange
Errors human-counted
with lggtt
and lights:
mill--lghttee and--Goa
EC: 0
EC:
EC: 1
Buth hassy
graph from error taxonomy
Image Synthesls (Stable diffusion, etc)
"Prompt" chosen to structure relations Into en error graph. error counts assigned.
Flamingo
No ertons
Prompt happy woman eating an orange donut
No emars
No elephant
Libopgy
Binan
EC:0
EC:
EC: 2
Final Error Counts, graph structure human-verifled.
Final Error Counts human-verified.
EC: 1
EC:
Figure 2: The three semantic error graph production procedures. Synth. (images generated from multiple prompts written to populate a SEG), Nat. (natural images populate a SEG), and Real (real errors from image generation attempts from one prompt populate SEG).
3.1 Dataset Collection Procedure
Figure 2 depicts the three procedures by which we produce and populate SEGs with images: synthetic images from synthetic graph (Syuth), graph from natural images (Nar), and graph from real errors of synthetic images (Real), differentiated by prompt source, image source, and the order of production.
Synth. Synthetic SEGs are produced "graph first." From an initial prompt we list all entities and properties it contains, then ablate them to produce an error graph. We then manually write prompts describing each node, generate their images, and manually check image-node faithfulness. For example, in the left panel of fig. 2, the initial prompt "a Christmas tree with lights and teddy bear" converted to error prompts such "a Christmas tree with lights."
Nat. The natural error trees exclusively contain real images sourced from the free stock image repository Pexels. We generate SEGs in "image, graph, prompt" order. We source sets of natural images that share objects and models. We organize them by relation graphs describing how the images differ by objects, actions, attributes, and composition. We then select a head node this relation graph and write "prompt" describing this head node (eg., fig. 2 center panel). We produced SEGs of natural images to assess whether distributional differences between synthetic images and real images might lead to measurable impacts performance for the faithfulness metrics that typically use base models pretrained exclusively on natural images [41].
The real error, synthetic image SEGs are produced following "prompt, image, graph" order. From a seed prompt (both manually written and sourced from COCO or PartiPrompts) we simply generate a large set of images using T2I model, then annotate the errors in each generated image. These labeled images are then organized into final error graph for the SEG. This procedure is documented in the right panel of fig. 2.
Real.
3.2 Dataset structure, size, and validity
Each of the 165 SEGs in TS2 is manually checked by three human annotators. The head node of each SEG contains at least one image that has been assessed by the anmotators to contain no errors of verbal information (eg, entity in the image isn't performing the described action), compositionality (eg, object described as "on top" but is beneath object), missing objects, or incorrect object attributes.
Each edge on the SEG represents enror of one of the aforementioned types. Each node is labeled with the number of edges along its shortest path back to node O, representing its error count (fig. 1, fig. 8). Each node contains at least one image which is erroncous according to the described errors.
5

---

Distribution of image sources
Distribution of prompt source
Distribution of error
Missing objects
41.3%
46.0%
SD 2.1
18.0%
47.0%
41.0%
Manual
9.0%
Comp.
7.7%
14.0%
0%
Vers
15.0%
SD 1.5
SDXL
Natural
DALL-E2
PartiPrompl
Wrong attribure
(a) Image Source
(b) Prompt Source
(c) Error node (SEG edge) type
Figure 3: Overview of the distribution sample types in TS2: (a) Where source images came from: 5% of images in the benchmark are real photographs from Pexels, while the remainder were generated by Stable Diffusion (SD) or DALL-E variants. (b) Source the eliciting prompt; either existing resources us (Manual). (c) Distribution of enror types edges in all SEGs.
The synthetic images are generated with several T2I models moodels-inlludngg including DALL-E [2], Stable Diffusion 1.5 [3], 2.0, 2. and SDXL [42]. We use MS-COCO [43], PartiPrompt [35], and manually written prompts in head nodes for the Synth and Real subsets. Image sources, prompt sources, and error type statustics are documented in fig. 3.
The total number images and prompts is roughly in line with previous benchmarks that have been used to verify methods such as TIFA [16] and DSG [6] in their own papers. and permits significantly more image comparisons-por perpoonmppppppppppppp than any prior benchmark (Table 1). which we submit the primary type of comparison required to verify that prompt- faithflless assessments work.
4 Experiments
Using TS2 evaluated three classes of T21 evaluation metrics: embedding--oe cocerelattoooo (comparing embeddings of prompt and image), QG/A (using VQA to check requirement questions generated from the prompt are satisfied) and caption-based (comparing captions extracted from the generated images to the prompt). We evaluate these metrics with multiple backend VLMs ($A.2).
For each metric, score every image against its SEG's prompt. We report the results of our Ordering and Separation metrics across all SEGs, as well as for our the Synth, Nat, and Real SEG subsets.
4.1 Embeddingg Metrics
CLIPScore [15] is a popular prompt faithfulness metric based on simple text-image similarity. CLIPScore is computed as the cosine similarity of the Lg-normalized CLIP-assessed [41] image and text embeddings. Equations and details in $A.1.
ALIGNScore (not to be confused with the text-only AlignScore [44]) is variant embedding- similarity score produced using the ALIGN [45] embedding model rather than CLIP embed the prompt and image. Other than model it is equivalent to CLIPScore.
4.2 Question Generation Answering (QG/A) Metrics
Question Generation & Answering Metrics use an LM MMG to produce a set of requirement question/answert pairs (q, a) Q from prompt p, and then use vision-language model My L to check each requirements against the image, reporting satisfaction rate as the image's 's faithfulness score. QG/A metrics vary by how questions generated and relate each other. Equations in $A.1.
TIFA [16] prompts an (GPT-3) to generate a set of multiple choice and yes-no questions and their expected answers relative to the prompt. Then vision language model Mv produces "free-form" answers to each question, which are converted into multiple choice answers a' using an SBERT model. The TIFA score for given image is then the rate of correct answers.

---

LLaVA 1.5 (alt)
LLvV 1.5 (alt)
LLMSeore EC
LLMSSure Over
ALIGNScore
InstructBL.I
InstructBL.IP
CLiPSoure
LLaVA
LLaVA 1.5
VIEScore
BPLDG
GPT4-V
#PLUG
6PT4.V
BLIPT
Fuyu
BLIPI
Fuyu
Emb-hassd
TIFA
DSG
Caption-based
Avg
71.4
73.9
71.0
74.5
74.4
76.5
73.8
38.7
77.9
70.4
76.2
75.0
79.0
76.6
29.5
79.6
48.8
57.7
37.8
Synth
75.0
77.6
72.6
79.2
79.2
80.2
78.8
44.5
83.6
74.6
80.1
81.6
85.1
81.6
35.4
82.6
50.2
61.6
42.5
58.0
70.2
66.9
62.8
64.0
65.1
23.5
71.6
20.5
73.7
36.2
44.4
22.4
Real
69.3
62.6
68.2
66.7
64.5
71.6
64.0
29.7
69.5
58.4
70.0
54.2
62.0
61.2
14.2
73.0
54.4
54.1
33.2
Avg
90.7
92.8
80.6
82.5
81.9
85.0
81.8
67.2
83.2
78.4
83.1
80.3
84.2
80.8
63.6
84.2
73.6
73.5
51.8
seg
Synth
90.5
94.1
80.6
85.5
85.2
86.7
84,1
67.3
86.2
80.9
85.7
83.9
87.8
84.9
65.8
86.6
71.1
72.8
53.7
91.5
92.6
84.2
75.1
75.6
77.9
73.3
68.6
81.7
80.5
76.7
Real
90.3
887.9
77.4
76.8
74.4
80.5
76.4
59.3
73.7
75.1
74.5
69.4
71.9
70.8
50.3
76.6
77.3
73.6
50.7
deltam
Avg.
89.7
95.6
92.4
97.5
97.1
99.8
94.0
92.3
104.7
102.0
110.6
103.7
37.7
110.2
66.9
56.3
Synth
89.9
95.6
92.5
97.6
97.3
99.9
93.9
44.6
92.5
94.4
104.7
102.1
110.7
103.6
37.8
110.1
67.2
56.7
46.7
92.5
98.6
94.4
98.7
98.8
46.1
105.5
103.0
111.6
104.6
111.5
56.4
Real
95.1
100.8
96.0
100.5
100.4
102.6
96.9
46.9
95.9
97.3
106.8
104.2
113.0
105.9
39.4
113.0
65.0
56.0
46.3
FLOP/run
688M
224T
224T
224T
224T
224T
224T
1.66P
140T
140T
140T 140T 140T
140T
860T
7.011
7.011 2.6T
Table 2: Spearman ordering score rankm, and Kolmogorov- Sminmov separation score sepm average dynamic range delta deltam. (all reported as % for readability) and estimated FLOPs to score an image for each model. Best bold, within 2% of best underlined, top four colored by type (emb-based, TIFA. DSG, caption-based). See $A.3 for information
how we estimate compute costs.
DSG (Davidsonian scene graph) [6] shares the QA structure of TIFA, but generates set of require- ment questions which are non-overlapping. have exclusively yes/no answers, and sit on directed acyclic graph such that a question only satisfied if it and all its parent questions are answered yes.
Backend VLMs. All QG/A metrics rely on the use of a generative vision-language model (VLM) either for performing question answering (Mrr, in $4.2) or captioning (Mc in $4.3). Thanks to the simple decomposable framework of the QG/A methodologies, we were able to efficiently test the performance of both TIFA and DSG using several VLMs as visual question question-answertie answering backends MVLMVy.. We used mPLUG, LLaVA, BLIP, InstructBLIP, Fuyu, and GPT-4V as VLM backends for the QG/A metries, Details and reference for each VLM is provided appendix $A.2.
4.3 Caption-comparison Metries
LLMSccre [17] captures the fine-grained similarity between the image and text with rationales by leveraging the visual details understanding capability from vision experts and the reasoning capability of I.LMs. The visual information is parsed in hierarchical scene descriptions with global and local captions. Then the text-only LLM (we use GPT3) will compare the multi-granulaan descriptions with the input text prompt to give a score according to the evaluation guideline prompt.
visual
VIEScore [18] rates aspects of semantic consistency (SC) and perceptual quality (PQ) ultimately providing a rating score on a scale of 0 and 10. We use O-shot VL. as the backbone MLLM to evaluate how successfully the image follows the text-to- rext-to-image prompt.
5 Results
Table 2 shows the results for the Ordering feature rankm and Separation features sep,.. and deltam, for each metric assessed, on average for all SEGs (Avg), and the three SEG subsets, as well as an approximate FLOP cost per run for each metric ($A.3).
We found that the Synth set consisting of hand-designed (and probably more obvious) errors the casiest subset for all metrics to correctly order. The average rankm score for Synth across all metrics was 70%, for Nat 55% and for Real 56%. However, different subsets were hard for different classes of metrics. For the QG/A metrics, Real was hardest, while Nat was harder for the other classes.
As the embedding correlatiooor metrics came first, TIFA [16], DSG [6], LLMScore [17], and VIEScore [18] all compare themselves against CLIPScore baseline [15]. Despite the superiority these metrics supposedly held on their respective ad-hoc evaluations, the computationally cheaper CLIPScore and ALJGNScore are Pareto-optimal in most cases (Figure 4), sharing the optimality frontier with DSG or TIFA with GPT-4V, methods that are Re 6 orders of magnitude more computationally expensive.
7

---

(All)
(Nat)
(Real)
1.0
1.0
1.0
DSG- TeV
Ordering Score (rank.
ALIGNScore
Ordering Score (rank.
DSG GFG-GTTAA T4V
0.5
CLIPS
Score
Fuyu
0.4
Ordering
0.4
ViEScore
TIFA
Fuyi
0.0
0.00
0.0
107
10
10
16
10
10
10
10
10'
10
10
10
Estimated per image cost (FLOP))
Estimated per image cost (FLOPs)
Estimated per imageaaee cost (FLOPs)
(a) Ordering (rank..) ) vs estimated cost/image (FLOPs), all metrics
(All)
(Nat)
(Real)
1.50
1.50
1,50
Separation Score (delta.m)
1.25
DSG GP TeV
Separation Score (delta_m mm)m
1:25
DSG- T4V
Separation (de]tnum)
1.25
DSG PPSC-PPTTT TeV
1.000
ALIGINSoore
1.000
ALIGNScore
1.00
CraPS
0.75
0.75
0.75
Viore
VIEScore
TiFA-Fuyu
ViEScore
Flyu
0.50
0.25
0.000
0.00
0.00
107
10
10
10
102
10
10
10
10
10
10
10
Estimated per image cost (FLOPs)
Estimated image cost (FLOPs)
Estimated per image cost (FLOPs)
(b) Separation (deltam) estimated cost/image (FLOPs)
(All)
(Nat)
(Real)
DSG-GPT4V
Ordening Score (rank.nn)
0.8
Ordering Score (cmnk.m)
0.8
Ordening (rank.n)
0.8
CLIPScore
0.6
0.6
0.6
0.4
0.4
0.4
TIFA- Fuyu
TIFA-
0.2
0.2
0.00
0.25
0.00
0.25
0.00
Separation Score (sepum)
Separation Score (depum)
Separation Score (nepum)
Ordering (rank...) ) Separation (sep...)
Figure 4: Plots of ordering and separation scores against estimated per-image metric evaluation costs in FLOPs and each other. For all analyses, the Pareto optimal metrics DSG and TIFA with GPT-4 and the vastly less expensive embeddiig--corrllttion correlation ALIGNScore and CLIPScore.
Discussion
The headline takeaway from our findings is that, contrary to claims of superiority leveled their own papers, for all the QG/A and caption-comparison metrics [6. 16-18), the cheap metrics such as CLIPScore are sufficient or even preferable at capturing objective semantic errors relative to fixed prompts. We view this capacity accurately discriminate similar images relative to a prompt as the core feature a good prompt faithfulness metric must possess.
c effectively evaluating T2I metrics as relative score regressors functions that are predicting a specific score for image. However, there are additional desirable elements to in TS2 meta-evas evaataatvlaaiin.... Though the QG/A
Human aesthetic preferences are are-by by and caption-comparison metrics fail meaningfully outperform the cheap embedding-corred metrics, they may have advantages that are not captured by TS2, such as in modeling human aesthetic preferences. Paired with standalone benchmark human aesthetic preferences over error-free images, metric's s error assessment and aesthetic fidelity could be measured independently.

---

when provided by small pool of annotators, correlation to As the first objective evaluation of faithfulness metrics based on structural semantic errors, TS2 en- ables more fine-gres ainiigr measurement of metric desiderata, leading researchers build better metrics, and empowering developers to make trade off-informed metric choices.
6.1 Pareto frontiers with compute cost
Why do we care about compute cost? When evaluating T2I models at release, compute costs are not very important-aa an evaluation only has to run on a small set of images from benchmark prompts.
However, during training or in online monitoring, compute cost for faithfulness metrics becomes quite important. Faithfulness metrics could be used as reward signals while training a T2I model (or prompt generator), or called repeatedly during validation passes. Faithfulness metrics could be deployed in applications to guide an online prompt refinement system, to trigger second call in user-facing applications, or to analyze prompt corpora to surface challenging examples for further training or analysis. all of these settings, performant, low-cost model such as CLIPScore is valuable. TS2 demonstrates that the performance premium for the expensive metrics is quite small.
6.2 Considering error graphs enables objective evaluation
Previous evaluations of coherence metrics have evaluated metrics as human preference score regres- sors over single images, or over image pairs. The challenge with such an evaluation is that human preferences are not objective- such scores is an unclear signal.
However, by instead evaluating walks over error counts in SEGs, the TS2 captures a more objective notion of correctness, by ignoring the subjective relationships between pairs of unconnected nodes.
For example, consider the SEG presented Figure 1, where two different single-error nodes are shown. Given the prompt "a bot in green shirt poses with some fruit," one of these nodes contains images without fruit, and the other contains boys wearing a blue shirt, rather than green. Which of these types of images are actually worse with respect to the prompt? This is subjective decision some annotators may find the missing fruits more important than the incorrectly colored shirt. TS2 ignores this distinction, the difference between those nodes is subjective, the difference between both of these nodes and their shared child node one where fruits are missing and the shirt is incorrectly colored-is objective.
nodes of equivalent eπror counts are connected in any SEG. While
6.3 Human baselines and metric ignorance of ranking task
It is also important to note that metrics under test are not aware of the implicit ranking task in TS2, as they are evaluated as score regressors. Objective human annotation was only possible because the annotators were aware that relative ranking the goal.
human performance were judged on the task of simple Likert scoring of image-prompt accuracy without instructions, humans may not significantly outperform the metrics. However, if the human annotators were instructed to count the number of errors, we suspect they would perform quite well, without the other images for comparison over which the ranking task is performed.
Though we provide no human baseline, we do not think this is significant weakness human per- formance on the inherently synthetic task of image quality scoring is not as important as performance on ranking along objective errors.
6.4 Systematic advantages for some metrics
One disadvantage of using Spearman's 's p that it "expects" ties be the same in both distributions.
For example, if a set of images has error count (0, 1, 1. 2), the ordering (1, 0.5, 0.5, 0) will have a perfect p = 1, while the ordering (1, 0.51, 0.49, 0) will be penalized, despite it also presenting a correct ordering. This means that our Ordering score rankm systematically punishes the embedding- based metrics relative the VLM--based ones, as the embedding-coree correlatioo metrics CLIPScore and ALIGNScore can take continuous values, whie TIFA, DSG, L.MSScore, and VIEScore have a discrete range. In light of this systematic disadvantage for cecheedding-oorrllatti metrics, even more striking that CLIP/ALIGNScore still performant and on the optimality frontier.

---

6.5 Near-perfect performance on rankm and smpm possible
Although the scores of many models on our metric are high, this meta-evaluation is far from "solved." In principle it should be possible to get much closer to 100 average for both meta-metries than find. We view use of TS2 as a necessary secondary evaluation for any new proposed T21 faithfulness metrie; if it has high correlation to subjective human judgements but does not perform well T2IScoreScore, skepticism might be warranted.
6.6 Impact of future VLM advances
Ultimately, all image coherence evaluation metrics stand to improve from further advances in general VLM quality. As a considerably more performant model than LLaVA, mPLUG, etc, was unsurprising that GPT4-V worked much better as backbone for TIFA and DSG than the aforementioned. However, there do appear to be diminishing returns, as the order of magnitude going from mPLUG- GPT4-V-based evaluation yielded sub-1% improvement in rankm performance on the most difficult and construct-valle Real set. Better constraint-generatin processes may be required to push VLM-based evaluation metrics further.
7 Conclusion
We introduced T2IScoreScore, first-of-its kind objective evaluation for text-to-image faithfulness metrics that utilizes a high image-to-prompl ratio to organize its reference images along semantic error graphs, through which faithfulness metric be assessed by novel graph-based meta-metrics. Our study reveals a surprising finding: more expensive and recent "state the art" VLM-metrics actually only have modest gains in performance at best. Indeed, these cheap metrics such as CLIPScore and ALIGNScore are actually Pareto optimal along with the vastly more expensive and slightly more performant GPT-4V-based QG/A metrics, even when strictly comparing ordering and separation capabilities (leaving compute cost aside).
metrics simpler and cheaper embedding-
This underscores the necessity for capable of capturing the subtle semantic nuances between prompts and generated images. The establishment of T2IScoreScore structured way to rigorously test and improve T21 prompt faithfulness metrics, ensuring they can more accurately reflect the semantic coherence between prompts and generated images, thereby facilitating the development of more reliable and effective T2I models.
nuanced approach benchmarking and developing metrics
benchmarking tool is significant step forward, offering
Limitations, ethical considerations, and impact
Limitations to our work are discussed throughout. For example, in $6.4 we discuss how our meta- metries limited by intrinsic biases of rank-correlation metrics among ties (many of which occur when multiple images occupy one node on a SEG). Additionally, compared to other evaluation sets, total number of prompts is modest (this is required to achieve high image-to to-pooppt ratio, however, which a core strength of our work). Finally, due to its secretive nature, we are only able to produce rough estimates of the compute cost of GPT-3 and GPT-4 based metrics. We estimate them to the best of our ability using third-party information ($A.3).
This research will steer the development of more effective faithfulness metrics, which in turn will guide T2I model development. T2I models are inherently dual-use: they can be used to produce misinformation and other harmful content in addition to useful and entertaining imagery. Any work that contributes to improving their overall performance necessarily drives a small amount of both positive and deleterious impact in this way.
Acknowledgements
Thank you to the Fatima Al-Fihri Predoctoral Fellowship program for compute support. This work was supported in part by the National Science Foundation Graduate Research Fellowship Grant No. 1650114, CAREER Award Grant No. 2048122, and the Neal Fenzi Resonant Founder Fellowship.
10

---

Contribution Statement
MS checked SEGs, designed the benchmark and meta- metries, implemented the SEG tree iteration process and evaluation code for the ordering and separation scores, collated the QG/A answers into ID-Ievel scores, and assessed the final scores.
FJ produced and annotated the Synth SEGs, produced and annotated a subset of the Real SEGs, and checked all others. FJ collected answers for Fuyu for the QG/A metrics and cleaned and organized the final dataset release.
MK produced the Nat SEGs and produced, annotated, and checked the other SEGs. MK generated the TIFA and DSG questions for all prompts, implemented and evaluated CLIPScore and ALIGNScore, collected answers for the QG/A metries from BLIP, InstructBLIP, GPT-4V and refactored code.
YL evaluated LLMScore for the examples and conceived of measuring faithfulness errors in T21 faithfulness metrics. AS collected answers for the QG/A metrics from LLaVA and VIEScore.
References
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. 6
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[4] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffussion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479-36494, 2022. 2
[5] Vitali Petsiuk, Alexander E Siemenn, Saisamrit Surbehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hicke, Bryan A Plummer, Ori Kerret, et al. Human evaluation of text-to-image models on multi-task benchmark. arXiv preprint arXiv:2211. 12/112 2022.
[6] Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi bust-, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-lo-image generation, 2024. 2, 3, 6, 7, 8, 16, 18, 22
[7] Shengqiong Wu, Hao Fei, Hanwang Zhang, and Tat-Seng Chua. Imagine that! abstract-to- intricate text-to-image synthesis with scene graph hallucination diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[8] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:22/2.0503: 2022. 2
[9] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounde diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv: 13655, 2023. 2
[10] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423 433. Springer, 2022. 2
[11] Jezia Zakraoui, Moutaz Saleh, Somaya Al-Maadeed, and Jihad Mohammed Jaam. Improving text-to-image generation with object layout guidance. Multimedia Tools and Applications, 80
(18):27423-27443 2021. 2
[12] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances Neural Information Processing Systems,
36, 2024. 2
11

---

[13] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 2
[14] Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models, 2023. 2,3
[15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 14248,52, 2021. 2, 6,
[16] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. TIFA: Accurate and Interpretable Text-to-Image Image Faithfulness Evaluation with Question Answering. URL http: http:://.a.x....aaa/ss//////
11897. 2, 3, 6. 7. 8, 16, 22
[17] Yujie Lu, Xianjun Yang. Xiujun Li, Xin Eric Wang, and William Yang Wang. LImscore: Unveiling the power of large language models in text-to-image synthesis evaluation. arXiv preprint arXiv:2305. 11116,, 2023. 7
[18] Max Ku, Dongfu Jiang. Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867 2023. 2,
7, 8, 22
[19] Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015.
[20] Kimin Lee. Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuging Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023. 2
[21] Michael Saxon, Xinyi Wang, Wenda Xu, and William Yang Wang. Peco: Examining single sentence label leakage in natural language inference datasets through progressive evaluation of cluster outliers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 0053-3066, 2023. 2
[22] Joseph P McKenna, Samridhi Choudhary, Michael Saxon, Grant P Strimel, and Athanasios Mouchtaris. Semantic complexity in end-to-end spoken language understanding, arXiv preprint arXiv:2008.022 02858, 2020. 2
[23] Wanrong Zhu, Xin Eric Wang, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang. Towards understanding sample variance in visually grounded language generation:
Evaluations and observations. arXiv preprint arXiv:2010. 2020. 2
[24] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 1125-1134, 2017. 2
[25] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 21ti::2008.01007. 2021. 2
[26] Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 00029556. URL http://www. 2,
jstor wtwprrorg//ssso httgg//taabee1441111111
[27] Andrey Nikolaevich Kolmogorov. Sulla determinazione empirica di una legge didistribuzione.
Giorn Dell' Dnst Ital Degli Art, 4:89-91, 1933. 2,4
[28] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 12i-comphench: A compre- hensive benchmark for world compositional text-to-image generation, 2023. 2
[29] Michael Saxon and William Yang Wang. Multilingual conceptual coverage in text-to-image models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 4831 -8448 Toronto, Canada, July 2023. Association for Computational Lin- guistics. doi: 10. 16533v1/2023...... aal.:oog.cco-eneee 266. URI. htttss:/aalaanthooooo org./ggg..org/233
acl- long. 266. 2
12

---

[30] Michael Saxon, Yiran Luo, Sharon Levy, Chitta Baral, Yezhou Yang, and William Yang Wang. Lost in translation? translation errors and challenges for fair assessment of rext-to-image models on multilingual concepts. arXiv preprint arXiv:24033 / 1092, 2024. 2
[31] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily Accessible Text- to-Image Generation Amplifies Demographic Stereotypes at Large Scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAceT '23, pages 1493-1504. Association for Computing Machinery. ISBN 9798400701924. doi: 10.1 3593013.35940999 URL https: //dl. acm wtt//oo/l/..o/oo// 111111191///1..../ 3594999 3594095. 2
[32] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. In Michael Wooldridge and Qiang Yang, editors, IJCAI 2015 Proceedings of the 24th International Joint Conference on Artificial Intelligence, UCAI International Joint Conference on Artificial Intelligence, pages 41 88-4192. International Joint Conferences on Artificial Intelligence, 2015. 24th International Joint Conference on Artificial Intelligence, IJCA 2015; Conference date: 25-07-2015 Through 31-07-2015. 3
[33] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. 3
[34] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
3
[35] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5,
2022. 3, 6
[36] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models, 2023. 3
[37] Yuval Kirstain, Adam Polyak. Uriel Singer, Shahbuland Matiana, Joc Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024. 3
[38] Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation, 2023. 3
[39] John W Pratt, Jean D Gibbons, John W Pratt, and Jean D Gibbons, Kolmogorov-smimov two-sample tests. Concepts of nonparametric theory, pages 318-344, 1981, 4
[40] Vance W Berger and Yan Zhou. Kolmogorov-smirnov test: Overview. Wiley statsref: Statistics reference online, 2014.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8766 PMLR, 2021. 5,
[42] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. 6
[43] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[44] Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. Alignscore: Evaluating factual consis- tency with a unified alignment function. arXiv preprint 2.Xi::2305.16733 2023.
[45] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and langone representation learning with noisy text supervision. International conference on machine learning, pages 4906
PMLR, 2021. 6
13

---

[46] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient -laguale learning by cross-modal skip-connections. arXiv preprint arXiv:2205. /2005, 2022. 15
[47] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi. Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023. 15
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 15
[49] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 15
[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
15
[51] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 15
[52] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, 2022. 15
[53] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 15
[54] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşurlar. Introducing our multimodal models (fuyu-8b), 2023. URL https:
/ www./adept.ai/blog/ffaeeeoe fatg//uuuu88hh0 8b. 15
[55] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:23033.8772 2023. 15
[56] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20Xi::2011083666 2020. 15
[57] Avijit Thawani. Jay Pujara, and Filip Ilievski. Numeracy enhances the literacy of language models. In Marie-Franen Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6960-6967, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18655.0 1..18833/v/209 http:/222..eemmmmeeeee mimllp-min.557. URL https://aclantthlll. orgp://orr//22/22 emnlp- main. 557. 22
[58] Dominic Petrak, Nafise Sadat Moosavi, and Iryna Gurevych. Improving the numerical reasoning
skills of pretrained language models. arXiv preprint arXiv:2205.061:
2022. 22
[59] Karthik Valmeckam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change).
arXiv preprint arXiv:2206. 10498, 2022. 22
[60] Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Lims can't 't plan, but can help planning in Ilm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024. 22
14

---

A Supplementary Details
A.1 Equations for evaluated metrics
Embedding correlation metrics CLIPScore and ALIGNScore are computed using positive cosine similarity between text and image features, from feature extractor model M as:
clip-s(p, i) max(cos(MM(())MM
(6)
VQA metrics VILM-VQA metrics like TIFA and DSG are assessed by a multiple choice question assessment model (Mg and vision-language model My over questions Q generated by an LLM based on the prompt
s(pa-s(p,i)
1 [Q]
Σ I(Mg(Mv((())== a.) (g.a)EQ
A.2 VLM details
mPLUG is class of vision- lanonnla models that use skip connections between visual encoder embedding layers between cross-modal attention blocks in the transformer stack [46]. We use the mPLUG-OWL [47] 7b checkpoint which uses LLaMA 7b [48] the pretrained text encoder.
LLaVA is fine-tune of Vicuna [49] (decoder-only transformer model) that uses learned MLP "vision-language connector" layer to map a single input image's ' CLIP encodings into a shared embedding space [50, 51J. We use LLaVA 1.5 13b. Because LLaVA was instruction fine-tuned for chat applications, we experiment with variant system prompt that requests concise answers from the system. We mark this alternate option LLaVa 1.5 (alt) plots and figures.
BLIP is jointly-traine self-attention VIT trained with cross attention to multiple transformer encoder and decoder pipelines with different tasks [52]. We use the BLIP encoder/causal LM decoder combination as transformer encoder-decoder model to produce VQA answers from the blip-vqa- base checkpoint.
InstructBLIPP extends BLIP by including an instruction fine-tuned "Q-Former" that selects salient cntructioo-relateer visual features from frozen VIT for input to frozen LLM that answers the query conditioned on the selected features [53]. We use instructblip f
Fuyu is a decoder-only VLM that splits an input image into sequence of patches that are separately projected directly into the transformer embedding space, which jointly learns ViT and LM behaviors [54]. We use Fuyu-8b.
GPT4-V is the largest state-of-the-art VLM provided by OpenAl [55]. It is expensive to run!
A.3 Compute cost estimates
Estimating FLOPs per inference pass for each model. We use OpenAl's estimate [56] of z 2N OPs per forward pass for a large transformer model, where N is the total number of parameters.
Obtaining parameter count estimates for closed models. TIFA, DSG, and LLMScore use some combination of GPT3 and GPT4 whose exact parameter counts and FLOP/iferrece costs haven't been publicly disclosed. We use estimates from SemiAnalyss to get an approximate FLOP cost. While these numbers are likely imperfect, their orders of magnitude are accurate we can get.
Obtaining metric FLOP cost per single image eval. Given FLOP/forward pass estimates for each model, our estimates of the FLOP cost to evaluate an image is function of the number of model calls and estimated tokens per call. The -oreedddinncoorlatiionn metrics require 2 calls to an embedding model, matmul and sum operations to get the cosine similarity. TIFA requires on average questions, with GPT-3 guestion-generating calls of average 40 tokens, and VQA model calls of average length 20 tokens. For DSG these numbers are 5, 40, 15. The costs of calling the ceafform-to-multipe choice model are negligible. We estimate that LLMScore and VIEScore both require approximately
15

---

50 tokens of LLM or VLM compute to score an image. LLMScore's use of mPLUG to caption is negligible alongside the cost of running GPT-3,
Total compute cost of our study. In total, we estimate our study took 9.89 10's FLOPs, mostly through OpenAl's service, but also on lab-owned NVIDIA Titan-X and A-100 GPUs.
A.4 Semantic Error Graph Structure
For more information the structure of the semantic error graphs (SEGs), provide examples here. SEG 85 one example with a more interesting topology than the example Figure Figure has a structure including two-error edges, single-parent nodes, single-child nodes, multi-parent nodes, and multi-child nodes in the same graph, corresponding to prompt "guy with umbrella hat sitting at table with another person with hat under red umbrella."
SEG guy
sitting table person hat red umbrella.
guy, other person.
person, no umbrella
No
other
person
3b
No other person, no hat
3c
No table,
3a
na other person.
other
No other person, table, umbrella,
umbrella
guy, lable,
other person, no hat
Example of a SEG (85) with more complex structure. Some nodes have multiple child nodes, and some edges correspond to more than one error (dark red).
Figure 5:
A.5 Scoring within SEGs
Figure exemplifies why we choose to only score rank order along walks the graph, rather than between all pairs of nodes. priori there's no the beach-less images should be worse than the umbrellas, yet metrics consistently rate the beach error more severe.
B
Supplementary Results
B.1 Comparing DSG question evaluation using DSG and TIFA score accumulation methods
The first few steps of TIFA [16] and Davidsonian Scene Graph (DSG) [6] scoring methods are nearly identical: an LLM generates set of requirements as questions, and VQA system answers them. However, the two methods differ chiefly in how the answers are combined into single image-level score. TIFA simply scores images by the correct answer rate, while DSG uses the graph structure of the requirements to build in some robustness: if an upstream requirement is not met (e.g., is there a boy? no), then downstreum requirements are all also assessed as not being met, regardless of answer. In the example provided, if the question "is the boy's 's shirl green?" were answered yes, the DSG accumulation technique would still score this requirement as being not met, due to the upstream requirement, while the TIFA accumulation method would score it as being met.
16

---

SEG 109: Mesoamerican pyramid surrounded by jungle. Detailed charcoal sketch.
0
No jungle
No Mesoamerican pyramid
JUNGLE
DUNGE
Figure 6: Example of a SEG (109) with simpler structure. We show multiple images for each of the three nodes in this SEG.
Avg Scores:
Avg Scores:
Node
Node
CLIPScore:
CLIPScore: .99
Scores:
TIFA-L:
78
TIFA-L:
.44
CLIPScore:
37
TIFA-mP:
.78
TIFA-L:
DSG-L: 78
DSG-L:
22
TIFA-mP:
24
DSG-mP:
.89
LLM-EC: 0.33
LLM-EC: 1.6
DSG-mP:
30
U
LLM-EC:
5.2
No beach
Figure 7: Examples from SEG 71 (The beach is crowded with red and white umbrellas). Even though both nodes and have the same error count (1) they systematically differ across metries: all metrics punish the images where the umbrellas are just in water (no beach, 1b) more than they penalize an empty beach with no umbrellas (la).
CLIPScore CDF
(a)
0.6
0.5
Node
0
2b
CLiPScoret
LLMBoore
TIFA-LLaVA node-wise CDF
(b)
0.8
0.6
1
0.4
CLIPScore1 0.65
0.67
II1 AALAA
LLMScore
Figure& 8: Examples of scores assigned by three metries to examples from an easy (a) and hard (b) semantic error graph (left). Computation of the separation score sep,,, (S) for two metrics is depicted at the right. Color coding of each cell corresponds to the metric's score for the image being better (blue) worse (red); more correlated measures (presenting higher rank order score rank.. (S)) will show the same progression from red to blue (a), while harder-to-rank examples will not (b).
17

---

LLaVA 1.5 (alt)
LLaVA 1.5 (alt)
InstructBLIP
InstructBLII
LLaVA 1.5
LLaVA 1.5
mPLUG
mPLUG
BLIP1
B1 BLIPI
Fuya
Fuyu
DSG w/ TIFA accumulation
DSG w/ DSG accumulation
Avg
70.4
76.2
75
79.0
76.6
29.5
68.8
80.0
75.6
80.2
76,9
35.8
Ord.
Synth
74.6
80.1
81.6
85.1
81.6
35.4
73.5
83.8
82.1
86.1
81.7
45.5
Nat
65.3
65.9
68.8
70.7
71.6
20.5
61.9
74.9
68.9
70.2
71
21.5
Real
58.4
70.0
54.2
62
61.2
14.2
56.4
69.6
55.9
65.8
62.8
10
Avg
78.4
83.1
80.3
84.2
80.8
63.6
75.5
82.5
80.5
84.3
80.6
66
Sep.
Synth
80.9
85.7
83.9
87.8
84.9
65.8
77.1
85.5
83.8
88.8
84.1
68.7
Nat
71.2
80.9
76.7
81.8
73.3
68.6
70.6
75.1
77.2
81.5
75.1
71
Real
75.1
74.5
69.4
71.9
70.8
50.3
73.1
76.8
70.6
68.9
71.4
50.8
Table 3: Comparing how using DSG vs TIFA-style accumulation for scoring each image by DSG questions impacts performance along both our metrics. The right half of this table is identical to the DSG section in Table 2, and bold, italic, and highlighting follows the same rules, except cells the TIFA half are marked as if they were replacing the right half cells in the DSG section in Table 2.
As a supplementary experiment, compare how accumulating the DSG questions using the DSG technique compares to accumulating them with the TIFA technique in Table Interstingly, the impact of this change differs between strong and weak VLMs, between ordering and and separation scores, and between the easier and harder subsets. For example, switching from DSG to TIFAstyle acculumation consistently improves ordering performance for mPLUG, while worsens performance for LLaVA, InstructBLIP, and BLIPI. For Fuyu, the weakest model, DSGstyle accumulation significantly improves performance over TIFA. This strengthens the claim from [6] that using the scene graph to check requirements adds robustness; it makes a lot of sense that this robustness benefits the lowest-pertee pewvestperforming VQA systems the most.
For separation scores, TIFA accumulation improves performance of more models. In particular, TIFA accumulation pushes InstructBLIP into the top for separation the Synth subset, while DSG metric using DSG accumulation breaks into the top 3 (red highlighted cell).
B.2 Modelwise Spearman Ordering Score Histograms
Here we provide full histograms for our Spearman Ordering and Kolmogoroy-Smiroy Separation scores, across every SEG, for all metries we assessed.
413
62
0.0
122
642
43
12
15
18

---

40.5
-0.5
+2.8
033
-24
-21
86
B.3 Modelwise K-S Separation Score Histograms
65
19

---

20

---

Here we provide line plots for set of metrics and SEGs. Note that for no rmal Liied_ rank, higher worse (more errors). sigh-correlatin assessed when the metric lines (higher better) go down as the metric lines go up.
Seree ID
lave, kifa
<lpeore pomm
Imager
Bov Lll
mptag IEA
Image 13
Scatts for 50
livs 42.
istmehip du
bagee
Soores for ID 106
Bon-el. dag
Byy, dkg
insruchip
cljooxe
nomallssd
Isnget
Sooms for 1D 148
tnsmethip..t.
cipence men
21

---

C
Supplementary Analysis
Another interesting weakness of the QG/A metries is that many unlucky situations where the VLM backend presents mix of true and false positives that cause incorrect rankings or poor separation (DSG fails to order samples while CLIPScore succeeds in fig. 9) to occur. However, these VLM failures cases are interpretable and can be targeted; T2IScoreScon work in making LMs more robust to these sorts errors for VQA to mitigate this issue. In addition to these interpretability advantages, the more sophisticated VLM- metrics still do present better subjective human preference correlation than CLIPScore [6, 16-18). By focusing exclusively on objective similar-imag ordering and separation, TS2 is effectively orthogonal to these preference evals.
will hopefully drive future
Given the documented biases LLMs have in directly outputting numbers [37, 58], it isn't surprise that the technique which directly prompts VLMs to output a numerical preference value (VIEScore) is at present the least robust.
In general it seems that the most successful methods that leverage VLMs (TIFA and DSG) still ultimately produce scores using a deterministic algorithm. They use VLMs in a perceptual manner to separately check each requirement, but the final score is the accuracy estimate from each separate VQA question. This comports with the theories of L.LM function that treat it as a "system 1" [59]: effectively TIFA and DSG are examples of VLM-modulo frameworks outperforming pure LLMs on the task of prompt coherence scoring [60].
Prompt: A school bus and red and car.
SEG 143
Node
Q1: there a yellow school bus?
Yellow school
VQA: Yes there yellow school bus the image. (Correct, shoutd be yes) there
Correct
bus
Blue car
stop sign?
Red stop sign
VQA: No (Incomett, should be yes)
(3/3)
0.33
0.67
Q3: a blue car?
Targef Score
VQA: No (Incorrect, should be yes)
Q1: there yellow
STOP
school bus X
school bus inc the image. (Incorrcci, should be no)
VQA: Yes there
Correct
Blue car x
Q2: there
sign?
Red
VQA: No (Incorect, should be yes)
0.67 (2/3) 0.33
0.56
Q3: there blue
Targot Score 0.67
VQA: No (incorect, should be yes)
Figure 9: Example of two images nodes 0 and from hard SEG that are correctly separated (and ranked) by CLIPScore but are not separated by DSG-LLaVA. VLM hallucinations are key hinderance to QG/A performance on TS2.
22

---

Are the same SEGs hard for the same models? fig. 10 and fig, 11 present correlation plots between SEG-wise rank,, and sep,. scores respectively between each pair of metrics. For both we show (a) the correlations over all SEGs, and (b) the correlations between only SEGs in the Real subset. These plots show that broadly, similar methods have similar "blind spot" SEGs, while different ones can vary wildly in terms of which examples they succeed and fail at ordering and separating. Note that all TIFA or DSG QG/A metrics have appreciable correlation to each other, provided they use a strong enough VLM. The metrics employing weak VLMs such Fuyu do not perform well. Similarly, the two LLMScore metrics are highly correlated to each other; the pure VLM numerical rating methods not producing random noise. These correlations are stronger in the full set of SEGs (including natural images and the easy, pre-designed Synth SEGs) than they are in the hardest Real set of SEGs.
Ausinge Comisioo Sooin
Syntali Ererr Consiction Score
32
ALION
30
CJrSoor
CLI"Soore
0664 Raye
39
35
34
24
35
060 Ray
30
34
100
37
30
050-inttue
TEA
130
DSC-LINA
35
90
50
DECALLAA
50
20
TOV
15
59
TIA-LLAA
120
TFA-LLAA
LLMBowe
LLVBooo
DS2 kr
CSC-Mhii
TIF MFNg
82 24
88
8
100
vi2tooe
24
00G-LET
TRA
Die Lenh pM
10G Low
TPA LuNk
LMMii Oue
0S0 MPMs
TRAMPO
000-ELET
TEG CLPSON
pio Looh
TIR Lowk
LiVisn Ove
(a) All SEGs
(b) Synth
Natural Image Comsistio Scores
Rasl Onar Cammittrr Score
A K.vw
A1 11
P1
100
20
20
46
020 950-0UPM
57
100
72
4D
CD
T-A P1
25
TIA 9.
22
100
TIFA
96
35
TIFA
160 Martar
34
35
15
36 130
SW
De0k Intt P
26
1:00
31
50
37
24
33
100
27
51
45
31
27
59
29
39
S7
120
000 LLMMInn
76
-33
TIAALLIA
120
120
LLMSOW RC
2G
30
100 74
LLOBove Owe
8
-13
D50 Strke
Curfor
000 aocert
The DS0 LL on
TTALLONN un
Dox Loik
TOKLLAA
LEMSire Der
200-079
THAR P-
CUPtoR
500 AmmeEPP
19% P
LLAN
LEVSOO Dar
200-011
(c) Nat
(d) Real SEGs only
Figure 10: Correlation between the Spearman correlation score for each prompt tree for each metric.
for all SEGs (a), for the synthetic error SEGs (b), for the natural image/synthetic error SEGs (c) and for the real error subset (d).
23

---

Averagg Commatten Soon
Synthlic
Scones
ALKGNSoore
050 P1
10
10C
20
D10 0.P1
100
74
30
30
T-A R1
83
192
16
55
TIA
F1
103
26
53
CJPScoe
CLIPSore
100% instarrtt
TPA
50
100
26
05
120
TERA-LAVA
TERA-LOW
LEMA R0
26
100
LLM Ft:
LLVBore Ove
20
23
LLMbore Over
8
13
MPAk
-10
AL ALO0SOO
CLE
TIM
DiAA AAALLAANN (al)
000 Lonk
DSG-0Pha
TIA 0P
VISSON
DiG sandti
DSGLLAN (a
DOG-Mh
VISSoN
T
(a) All SEGs
(b) Synth
Natural Comsistion Scorea
Real Oiaa Scare
ALICN
TPA 9UM
CJrSoome
CLI CLIrS
25
100
12
30C-Pax
100
-5
20C-Tox
10
100
42
40
DSG-LatMreg
TEU
SV
EOC-LINA
11
DSCALINA
TUV
TIA-LLAA
TFA-LLA
LLMiooe
DS2 krrku
CSC-Mhig
TIR MFAuug
TRAMPP
2
ViSSooe
100% pal
Dia Look
1PO Lun
Unbiice Doe
DSG 90%
oSa LLWW
IRA LuWk
LLUSON
Libie Dee
CSG
(c)
(d) Real SEGs only
Figure 11: Correlation between the K S Separation score for each prompt tree for each metric, for all
SEGs (a), for the synthetic error SEGs (b), for the natural image/synthetic error SEGs (c) and for real error subset (d).
24

---

1.00
1.00
0.75
0.50
DSG
TIFA-LLVA
0.25
0.25
0.00
-0.25
-0.0
-0.50
Synthetic Image
Synthotic Error, Synthetic Image (Synth)
-0.75
Synthotic Error, Synthetir Image (Synth)
Synthetic Natural Image
0.75
Natural Error, Synthatic image (Real)
-1.00
Natural Error, Synthetic Image (Real)
-1.00
0050 -025
0.50
1.00
TIFA BLIP1
TIFA LIFA-LLaVA (alt)
Top-1
(b) Top-2
1.00
0.75
0.50
AUGGScoor
BLPScore
0.25
0.2
0.00
0.0
-0.25
0.2
-0.50
Synthetic Error, Synthetic Image (Synth)
Synthetic Error, Synthetic Image (Synth)
-0.4
Synthetic Error, Natural image (Nat)
-0.75
Synthetic Error, Natural image (Nat)
Natural Error, Synthetic Image (Real)
Natural Error, Synthetic Image (Real)
-0.6
-1.00
-0.75
-0.00 -0.25 0.00
0.25 0.50
-100
0.75
-0.0 -0.25 0.00 0.25
0.50 0.75
1.00
BUPScore
CLiPScore
(c) Bottom-1
(d) Bottom-2
Figure 12: Scatter plots comparing the two most correlated metrics (a, b) by Spearman correlation
Ordering score across the Synth, Nat, and Real populations, and the two least-correlated (c, d). Note that the two highest-correlate metrics are both QG/A metrics using the same underlying VLM (DSG and TIFA using BLIP1, (a); TIFA using LL with different system prompts, (b)).
Figure 12 and Figure 13 show scatter plots for the Ordering (Spearman) and Separation (KS statistic) scores for every SEG between the most highly-correlated (a, b) and low-correlation (c, d) pairs of metrics under evaluation, respectively.
25

---

0.8
0.8
TIFA-UFI
0.6
TIFA-BU11
0.6
0.2
0.2
Synthetic Error, Synthetic Image (Synth)
Synthetic Error, Synthetic Image (Synth)
Synthetic Error, Natural image Nat)
Synthetic Error, Natural Image (Nat) Natural Error, Synthetic Image (Real)
Natural Error, Synthetic image (Real)
0.2
1.0
0.2
1.0
DSG-aLIP1
TIFA- int.-usstuuuLIPP
(a) Top-1
(b) Top-2
1.0
1.0
0.8
0.8-
TFA-LLaVA
LLMScore EC
0.4
0.4
0.2
0.2
Synthetic Error, Synthetic (Synth) Synthetic Error, Natural Image (Nat)
Synthetic Error, Synthetic (Synth) Synthetic Error, Natural Image (Nat)
0.0
Natural Synthetic image (Real)
0.0
Natural Error, Synthetic Image (Real)
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
DSG-Fuyu
(c) Bottom-1
(d) Bottom-2
Figure 13: Scatter plots comparing the most correlated metries (a. b) by Kolmogorov-Sming
Separation score across the Synth, Nat, and Real populations, and the two least-correlater (c. d). Note that the two highest-correlated metrics are both QG/A metrics using the same related underlying VLMs (DSG and TIFA using BLIP1, (a); TIFA using BLIP1 and InstructBLIP, (b)).
Both of these sets of figures confirm that similar underlying VLMs by-and-large "think" similarly in different sets of questions (TIFA and DSG). This suggests that
terms of scoring models, development of overall better VLMs will generalize to many different types of VLM evaluations.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our focus is rigorous and objective evaluation of T2I faithfulness metrics, and we indeed support the suprising finding teased abstract.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See p. 10.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results provided.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detail given throughout. Project page link (including code) on page
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Project page link (including code and data) on page 1.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See sections 3, 4, 5, Appendix, code.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Variation of seed or temperature are not controllably supported by all meth- ods (eg, API-gated GPT.) Additionally, multiple runs of some methods such as question generation are not coherently expressible in statistical significance. Finally, compute costs of rerunning the models over all comparisons multiple times where possible would be prohibitive.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
27

---

Justification: See appendix section A.3.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /etpss:/neurips: http://p..../oooeeeesssssueeeeeo
Answer: [Yes]
Justification: Ethical considerations in Limitations & Impact section.
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: Impacts discussed in Limitations & Impact section.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Manually produced benchmarks such as ours based on images we synthesized sourced from creative commons free stock images and manually checked do require safeguards.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Pexels (Stock image source) acknowledged as required by CC license. All other images in data are created by us. Prompt sources (PartiPrommpt and MSCOCO) referenced.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See section 2, 3, project page
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Annotation exclusively by authors.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
28


Reviews:
[{"review_id": "70U4jfdjL1", "reviewer": "Reviewer_vRNq", "summary": "The paper introduces T2IScoreScore (TS2), which aims to evaluate how good newly developed text-to-image (T2I) evaluation metrics/methods are. The authors formalize the task of evaluating t2i metrics as their abilities to *order* images correctly within SEGs.", "questions": "Instead of pairwise ranking/comparisons, which might not always be robust, have the authors design multi-images ranking instead of only two images for pairwise comparisons? Something like bradley-terry style ranking could make the evaluation more robust.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors identify a very important task -- to evaluate T2I metrics. The introduction of T2IScoreScore and the use of semantic error graphs (SEGs) to evaluate T2I faithfulness metrics are novel and innovative.\n\n2. Experiments are good. The methodology is rigorous, and the experiments are well-designed to test the core claims of the paper.", "weaknesses": "1. Limited Scope: The evaluation is primarily focused on a specific subset of T2I models (many variants of SD, and Dalle-2) and metrics. Expanding the scope to include a broader range of models and datasets would strengthen the generalizability of the findings. Potentially should consider other synthetic images from models such as OpenMUSE or aMUSEd (https://huggingface.co/blog/amused) with totally different generation architectures than diffusion, etc. Alternatively, text-2-image is an old task, even GAN and VAE can probably have image distribution other hand SD and Dalle-2 which heavily depends on CLIP. \n\n2. Intrinsic Bias: The reliance on rank-correlation metrics, which have intrinsic biases, might affect the evaluation results. A more thorough discussion of these biases and potential alternatives could enhance the robustness of the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are heartened by your recognition of our **rigorous methodology and well-designed experiments** approaching the important task of T2I metric assessment. We will briefly address your weaknesses and questions:\n\n### Weaknesses\n\n- **Scope of image-generating models**\n\nIndeed there are many other generative image systems out there beyond the set we used to produce SEG-populating images. While it would be interesting to include images from OpenMUSE or aMUSEd in this approach, we believe showing a metric’s poor performance to rank on *any* set of T2I models is sufficient to demonstrate a need for improvement. Though we do not use a comprehensive set of T2I models to produce the test SEGs, we never claim our test is comprehensive, **only that it is sufficient in showing the surprising lack of differentiation between tested methods**.\n\nHowever, in the future images from additional SOTA models should be added to TS2 as advanced metrics are introduced. Our findings stand well with the current images but in future work we will definitely add more.\n\n- **Bias in the rank-order metrics**\n\nAs we note in sec 6.1, Spearman’s rho (rank order corr) is indeed biased in favor of discrete scoring methods over continuous methods such as CLIPScore, because of the way they handle ties. Luckily, the continuous methods are the weak baseline that the QG/A and Caption-based methods (which are discrete) are intended to beat. The ranking score does indeed penalize these metrics---**yet they still compete with or outperform the rewarded metrics**. This makes the finding even more striking; despite being at a disadvantage, CLIPScore and ALIGNScore still win, to our surprise.\n\nIn the future, as we maintain the TS2 leaderboard and other metric papers use it, it will become important to add metrics that do not penalize continuous metrics, as perhaps more advanced ones will be introduced. For the results in this paper though, **the bias actually strengthens the conclusion**.\n\nWe will clarify this point in the camera ready.\n\n### Questions\n\n- **Pairwise vs multi-image ranking**\n\nYou ask about multi-image ranking instead of pairwise ranking. To clarify, *we do not do two image pairwise ranking*, **all of our evaluation metrics are multi-image-based**. Indeed, this is our work’s primary advantage over all prior T2I metric evaluations.\n\nOur metrics are either **full-walk** scoring ($rank_m$) or **node-pair** scores ($sep_m$ and $delta_m$). For the full-walk scores, ranking over each descending sequence of nodes (each of which contains multiple images)\n\nWe only refer to pairwise comparisons in Table 1 as a way to distinguish our benchmark against prior work. Our metrics are exclusively multi-image (and our *per-equiv pref* score in Table 1 represents that on average, each node contains 3.4 images of equivalent correctness)\n\n\n### Future updates to TS2\n\nWe are excited to incorporate these points into the camera ready, and your suggestions for additional ranking metrics and more images are great ideas for future contributions as we expand TS2 as a living resource. However, we believe the resource as it currently stands represents a significant and timely contribution that advances our understanding of **current** text-to-image faithfulness metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WGL161HhD0", "reviewer": "Reviewer_cvSS", "summary": "The paper proposes an evaluation framework for holistically assessing text-to-image (T2I) evaluation methods. Since most of them are primarily established through simplistic correlational evidence and only compared to the CLIPScore baseline, this approach presents a more detailed way of assessment and also benchmarks existing promising evaluations. While there isn't a single clear winner, results suggest that CLIPScore is still a very competitive candidate and is especially successful when considering the much lower compute costs.", "questions": "None", "limitations": "Sufficiently addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- important contribution to investigate and improve automatic T2I evaluation strategies\n- interesting dataset construction which appears to build a more challenging test bed compared to previous methods, allowing more detailed insights and distinctions\n- thoughtful discussion of the results & exploration of limitations\n- very informative figure 2", "weaknesses": "- I like the general setup but I'm unsure about the accuracy of the \"number of errors\" counting system. I'll give two examples from within the paper. Take the last example in Figure 3 with the prompt \"A gray elephant and a pink flamingo\". An image with two flamingos is categorized as containing one error because there is no elephant. However, if there additionally was an elephant, it would still have an error since there are two instead of one flamingo. So one could argue that there are in fact two errors: a missing elephant and an additional flamingo. Or you say it's one error because there is one animal that is a flamingo but should be an elephant. So this is inherently ambiguous. However, in any ranking solution, this can actually matter quite a lot, so I'm worried that this introduces noise into the analysis process that is hard to reason over. I'm wondering to what extent this is taken into account by the design and how sensitive the results are to this. (Second example to illustrate from Figure 1 SEG: it's noted that when the shirt is not green, it's counted as one error. What if the shirt was additionally also suddenly a hoody. Is that then two errors or still only one? When the boy is gone overall, it's two errors because the shirt isn't green and there is no boy -- but what if there was now a grey shirt in the picture?) \n- Given that the evaluation framework provides many different evaluations for varying setups and (as discussed in the paper) those might come with their own biases, what is the recommendation to those who are thinking about using this framework for when they can call their metric successful? Defining an overall evaluation aggregate might also help with the adoption of the framework.\n- (Minor: It's hard to see which numbers are italicized in the results table.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We appreciate your recognition of our *important contribution* which you find *interesting*, that enables *detailed insights* and has a *thoughtful discussion.* We hope you will find our response to your questions about the counting system satisfactory. \n\n### Weaknesses\n\n- **Ambiguity introduced in the error counting approach**\n\nFor count errors, we considered “a X” to remain correct if there is more than one “X”---a picture of multiple flamingoes does contain “a flamingo.” However, when specific numbers were provided in the prompt, like “one flamingo,” containing more than one counts as an error. \n\nYou are right to point out that there is not necessarily a single objective answer for how to handle multiple attributes that could be incorrect about an object simultaneously, such as your example where “a boy in a green shirt” has a grey hoodie. In this case, under our annotation scheme, we still counted it as one error—”no green shirt,” but the case could be made for it to be two. \n\nWhile it is important to clearly document these annotation nuances for reproducibility, **these issues are inconsequential to our results, because our meta-metrics only evaluate rankings along descending walks in the error graph.** An image of the boy in a grey shirt and the image of a boy in a grey hoodie are not child or parent nodes of each other, instead being related only to images that may also have no boy at all. So regardless of whether a grey shirt or grey hoodie count as the same number of errors, a *grey shirt with no fruit* has more errors than *a green shirt with no fruit*, and has fewer errors than *no boy and no fruit* at all. The relative difference of errors between nodes that aren’t connected on the error graph do not actually matter, as our metrics are only assessed over walks.\n\nWe will ensure this deeper documentation of the error counting process, as well as this explanation for the role the error counts play in evaluation (relative to directed connected nodes, absolute values unimportant) are both provided in the camera ready.\n\n- **What are our recommendations to future system builders**\n\nWe have a few thoughts on this front. We think the right way to use TS2 in metric evaluation is to treat it as *an orthogonal evaluation axis to human preference.* While human preference correlations are great for capturing total output image quality, metrics that are well-correlated with TS2 will be ideal for measuring structural correctness. Authors might want to consider releasing dual metric evaluations, one which performs well on TS2, and another that performs well on image quality and aesthetic human preference correlations, to give more fine-grained feedback to models.\n\nThe pareto optimality evaluation plays a particularly important role when introducing metrics that may be employed for automated feedback, such as in a reinforcement learning setup or as a post-filter to select the best candidate image from a set of outputs.\n\nAs for metrics that may perform better on our evaluation, we think authors should consider better ways to gain faithful outputs from VQA modules, perhaps using techniques such as chain of thought or self-consistency prompting.\n\nThank you for the stimulating question, we will include this discussion in our camera ready.\n\n- **Difficult to read italics in results table**\n\nThank you for pointing this out. Rather than italicizing we will underline the runner-up scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cfdq8eqVJN", "reviewer": "Reviewer_h3XZ", "summary": "This paper presents a rigorous evaluation for text-to-image alignment metrics. This is primarily done by introducing a dataset with several images for each prompt, allowing the construction of semantic graphs that can be used to measure the accuracy of the alignment metrics. From the analysis on the benchmark, a major conclusion is that CLIPScore provides an excellent tradeoff (or is at least on the pareto-optimal frontier) between speed and alignment. VQA-based metrics (e.g TIFA, DSG) while improving over CLIPScore in many cases, come with much higher costs (in some cases orders of magnitudes higher), highlighting important considerations for text-image alignment methods.", "questions": "Overall, I really like the paper and would like to see this accepted. I have a couple of questions for my curiosity. \nWhile 2.8k prompts is still a lot more than TIFA, DSG, is this size still not too small/limited diversity to be able to comprehensively make conclusions about alignment methods?\n\nIs there any idea of what 'human-performance' would look like on these benchmarks, and if existing models are very far off from that?", "limitations": "No major concern.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The dataset collected in the paper is quite valuable, and would be useful for evaluating text-image alignment metrics in the future. The methodology in the paper also seems quite sound to me. I also think the analysis in the paper is very sound, highlighting the cost of running the evaluation metric is an important aspect which is often missed in these methods. The paper is also written very clearly, and is easy to read.", "weaknesses": "In terms of models/methods evaluated, I see 2 notable omissions: human-preference models such as ImageReward would also be a good addition since they might also capture some notion of text-image alignment while also cheap to use. Another good addition would be VQAScore (much more recent), but seems to show extremely strong results on several image-text matching benchmarks, while not being nearly as expensive as the Question-Generation methods (e.g. TIFA, DSG). \n\nMinor: While I totally agree with the issue of methods evaluating on their own proposed test set, the column \"ad-hoc\" in Tab. 1 makes little sense without an explanation (line 85 seems too limited). Either a more complete explanation should be given, or it would be better to replace this column with something more objective/concrete.  \n\n\n[a] Lin et al. \"Evaluating Text-to-Visual Generation with Image-to-Text Generation\", 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We appreciate your praise of our work’s *rigorous evaluation*, *valuable dataset contribution*, *sound methodology and analysis*, and *clear writing*. **We are overjoyed to hear that you really like [our paper] and hope to see it accepted.** These are all very heartening! We hope you will find our responses to your questions satisfactory.\n\n### Weaknesses\n\n- **Omitted metrics**\n\nImageReward did compare itself against CLIPScore and BLIPScore in its introductory paper, so it is reasonable to consider evaluating it in our study. We decided not to and to instead focus on the correlation metric vs. VQA metric angle, and because it is simultaneously attempting to measure the orthogonal characteristics of image aesthetic quality and image-prompt faithfulness as mediated and combined by human preference annotators. You raise a good point that it still can be evaluated using TS2, and that the findings from this experiment may have interesting implications for the trade-off between capturing aesthetic preferences and prioritizing structural correctness. We will discuss this possibility (although we do not have results for it) in the future work section, and investigate including ImageReward in our final leaderboard that will be linked in the camera ready.\n\nVQAScore is very interesting, and we did see the paper late into the writing process, and before its source code was fully ready for our use. While we are unable to evaluate it in the paper, we do look forward to adding it to our leaderboard. Though we do not have results for it, we will include a reference to it in the camera ready.\n\n- **Explanation of Ad-hoc evaluations**\n\nYes, we should clarify, the reason we define “ad-hoc” evaluation benchmarks as test sets that were released alongside proposed methods, and call them out in the discussion of related work, is because of the potential concern that these benchmarks—showing the superiority of the metrics they are introduced alongside—may contain some bias (probably unintentional) in favor of the proposed metric they are motivation, and that these benchmarks are not the primary contributions of their introductory works, and thus have less documented design considerations and production methods. By centering the design considerations of our evaluation in this paper, and *not* introducing a new metric, we are able to focus on the rigor of our evaluation, identify weak points in prior evaluations, and as neutral arbiters demonstrate our surprising finding.\n\n### Questions\n\n- **Regarding size of TS2**\n\nWe do believe the size of the dataset is sufficient to defend our findings. In particular, the high **total number of comparisons** that TS2 enables is its primary strength. By organizing the images along semantic error graphs containing multiple walks, the images are able to be reused much more efficiently than in other datasets, to analyze how well models can order them along different sets of accumulated semantic errors. For example, in figure one the image “1-2.jpg” is simultaneously used to check if a metric attends over a boy missing from the picture, as well as if fruit is missing, by comparing it to both “2-0.jpg” and “2-1.jpg”. This effect, applied over all the SEGs, greatly amplifies the utility of each image.\n\n- **Notions of human performance**\n\nThough our dataset was produced through human annotation with a high annotator agreement, it is important to note that **the human annotators were aware of the implicit ranking task, while the metrics under test are not.** We do not think this is a significant weakness of the work, as human performance on the inherently synthetic task of image quality scoring is not as important as performance on ranking along objective errors.\n\nThus, if human performance were judged on the task of simple Likert scoring of image-prompt accuracy without instructions,  humans may not significantly outperform the metrics. However, if the human annotators were instructed to count the number of errors, we suspect they would perform quite well, even without the other images for comparison over which the ranking task is performed.\n\nThank you for this stimulating idea, we look forward to including this discussion in our camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CqvbbcoUN3", "reviewer": "Reviewer_Xy9h", "summary": "The authors propose T2IScoreScore (TS2), a benchmark and set of meta-metrics for evaluating text-to-image (T2I) faithfulness metrics. Compared to existing relevant benchmarks, TS2 has higher image-to-prompt ratios, which allows users to organize semantic error graphs (SEGs), where each edge corresponds to a specific error with respect to the prompt that a child image set possesses but its parent images do not. Based on SEG, the authors evaluate T2I metrics, including embedding-based (CLIPScore/ALIGNScore), QG/A-based (TIFA/DSG), and caption-based (LLMScore/VIEScore) metrics, by how the metrics properly order and separate images. Different metrics show different advantages, and the authors highlight that simple embedding-based metrics outperform other more computational metrics in separation criteria.", "questions": "See weaknesses", "limitations": "There is no significant negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduction of meta-metric benchmarks of recent T2I metrics, including collection of large image-text pairs and semantic error graphs.\n\n2. Comprehensive experiments, including using different VLM backbones for TIFA/DSG.", "weaknesses": "**1. Simply treating QG/A metrics as score regressors.**\n\nOne of the major motivations behind using QG/A metrics, even though they are computationally expensive, is that they divide multiple aspects of prompts and provide comprehensive skill-specific performances of the T2I model. While the QG/A metrics can be used as a score regressor, comparing them with embedding-based metrics ignores their biggest advantages. This background and limitation needs to be clarified in introduction section, otherwise this could misleading readers who are new to T2I metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are heartened to hear that you view our meta-metrics as a novel strength, and that you appreciate our comprehensive experiments that really attempted to make the strongest possible case for the QG/A metrics by analyzing multiple backends. We hope we can satisfactorily answer your question.\n\n- **On the weakness of evaluating T2I faithfulness metrics as score regressors**\n\nYou are right to point out that one of the motivations of the QG/A metrics, which we do not inspect in detail here, is that they can provide fine-grained, (ideally) explainable assessments of T2I outputs over defined axes, as opposed to the other metrics providing simple numerical outputs. It is important to note this as a strength.\n\nHowever, we are not the only ones who give the QG/A metrics a score regressor treatment—**in their own works introducing the QG/A metrics, the authors justify their contributions with the same score regressor methodology, against the very same correlation-based metrics we analyze**. In their works, using their ad-hoc evaluation sets and simple metrics, they claim significant gains in this score regression performance over the correlation metrics.\n\nSimply by building a more carefully considered test set and metrics that are tailored to evaluate **objective, relative errors alone** without contamination by other forms of human preference such as aesthetics, we find surprising contrary results—that the regressor improvements claimed in the works introducing TIFA and DSG are illusory when considering structural correctness of the images alone.\n\nThat being said, you are right to point out that contextualizing the exact implications of our findings is crucial for inexperienced readers, and we will do so. In our camera ready, we will state the following points regarding the purpose of our evaluation:\n1. TS2 is intended to provide a meta-assessment of metric quality **that isolates structural correctness in relative comparison of related images** \n2. TS2's evaluation **is orthogonal to human aesthetic preferences,** and that a comprehensive meta-evaluation should also take those preferences into account\n3. That our findings, *while important, surprising, and soundly demonstrated,* do not tell the full story of T2I metric quality; **some metric strengths such as explainability and contextualization cannot be captured numerically**\n\nWe appreciate you bringing this point to our attention and look forward to using it to strengthen our camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers’ thoughtful and detailed analyses of our work. 

We are excited that multiple reviewers identified each of our work’s key strengths, including:
1. That our meta-evaluation setting is a **timely and important task** (vRNq) that has not been approached before and is “often missed” (h3ZX) and constitutes an **important contribution** (cvSS) to the T2I evaluation field more broadly
2. That our approach is **novel and innovative** (vRNq), using an interesting dataset (cvSS) that is *quite valuable for future work* (h3XZ)
3. That our methodology is rigorous (vRNq) and “quite sound” (h3ZX)
4. That our “experiments are well-designed to test the core claims” (vRNq) and are comprehensive for including a wide set of VLMs in evaluating VLM-based metrics (Xy9h)
5. That our findings are **detailed** (cvSS), surprising and interesting, and our analysis is “very sound” (h3ZX)
6. That our discussion is thoughtful (cvSS) and our paper is “clearly written and easy to read” (h3ZX).

In particular we are pleased to hear that rev. h3ZX liked our paper overall and hopes to see it accepted!

Furthermore, **we are thankful for your thought-provoking and diverse questions and suggestions** implied in your weaknesses. We believe we have solid answers for most of your suggestions, and that the necessary changes to address them will greatly strengthen our paper. We hope that you find our answers useful or convincing.

Thank you!

Meta-review: **Summary:** This paper addresses the issue of how to evaluate evaluation metrics for text-to-image models. They introduce a dataset of error graphs, which are prompts alongside images of with increasing number of semantic error graphs, such as incorrect color or count of objects. Evaluation metrics are scored by how accurately they rank images. Images that are more consistent with prompt, e.g. higher in the graph, should be ranked higher as well. They find that simpler, embedding space metrics such as CLIPScore tend to outperform more complex, GQ/A based metrics.

**Aggregate Strengths/Weaknesses:** The reviewers find that this is a structured approach to evaluating T2I metrics. They view the datasets as a valuable contribution and the experiments as in-depth and rigorous. The weaknesses are that there is not a single cohesive takeaway when comparing metrics using TS2. It’s not clear how researchers should decide on a single metric and TS2 also does not manage to incorporate other aspects of each metric, such as the fine-grained analysis GQ/A metrics provide. As a smaller point, the reviewers suggest including more models (even GANs/VAE) and more metrics (VQAScore, ImageReward). Time-permitting, it would be nice to see VQAScore in the final version. I'm especially curious about the results given its simplicity (no question generation, the difficulty of GQ/A approaches) while still leveraging a VQA model.

**Recommendation:** Every reviewer recommended this paper for acceptance. This work addresses an important problem of better understanding and selecting our evaluation metrics given the rise of T2I models. The dataset is well-structured and makes comparing two different metrics straightforward. The paper was well-received by all reviewers. I therefore recommend this paper for acceptance.
Decision: Accept (spotlight)