Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)

Authors: michael  saxon
fatima  jahara
mahsa  khoshnoodi
yujie  lu
aditya  sharma
william yang wang
fatima  al-fihri
predoctoral  fellowship
meta-metrics  dataset
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:51:04

Abstract: With advances in the quality of text-to-image (T2I) models has come interest in benchmarking their prompt faithfulness---the semantic coherence of generated images to the prompts they were conditioned on. A variety of T2I faithfulness metrics have been proposed, leveraging advances in cross-modal embeddings and vision-language models (VLMs). However, these metrics are not rigorously compared and benchmarked, instead presented with correlation to human Likert scores over a set of easy-to-discriminate images against seemingly weak baselines. 

We introduce T2IScoreScore, a curated set of semantic error graphs containing a prompt and a set of increasingly erroneous images. These allow us to rigorously judge whether a given prompt faithfulness metric can correctly order images with respect to their objective error count and significantly discriminate  between different error nodes, using meta-metric scores derived from established statistical tests. Surprisingly, we find that the state-of-the-art VLM-based metrics (e.g., TIFA, DSG, LLMScore, VIEScore) we tested fail to significantly outperform simple (and supposedly worse) feature-based metrics like CLIPScore, particularly on a hard subset of naturally-occurring T2I model errors. TS2 will enable the development of better T2I prompt faithfulness metrics through more rigorous comparison of their conformity to expected orderings and separations under objective criteria.

Main paper: 
 introduction: 
 Text-to-image (T2I) models are improving at a breakneck pace in terms of quality, fidelity, and coherence of generated images to their conditioning prompts [1] [2] [3] [4] . Despite this, persistent challenges in achieving image-prompt faithfulness [5, 6] remain-particularly in freely available models that don't sit behind proprietary APIs. Indeed, many techniques to improve T2I models have been proposed of late, aiming to reduce hallucination [7, 8] , duplication [9] , composition errors [8, 10] , and missing objects [11, 12] . However, there is no consensus on how to best compare these many models and methods, so it is hard to objectively track T2I progress [13, 14] .Recent work has proposed a litany of automated image-prompt coherence metrics which rate the faithfulness of generated images: the degree to which a they satisfy the implicit requirements set forth in the generating prompt [15] [16] [17] [18] . These proposed metrics vary considerably in design; as rating how well an image matches to its prompt is a nontrivial multimodal challenge [14, 19, 20] .This variety itself presents a meta-evaluation problem: there is no consensus on how these faithfulness metrics ought to be compared, and consequently each new metric is validated on its own ad-hoc test set against prior baselines. Typically these self-evaluations consist of a set of prompt-image pairs with accompanying human annotations (usually simple Likert scores [16, 19] ), and metrics are judged on their correlation to these human judgements [20] .Such self-evaluation is not ideal; authors may unwittingly tilt the scales by using evaluation examples which cater to the particular strengths of their proposed method, and variance of metric performance between different evaluation sets (containing different images and prompt semantics [21, 22] ) is high [23] . Additionally, relying on correlation to human judgements of small sets of images across different prompts is highly subjective [24, 25] and prone to including judgements of quality and style that are orthogonal to prompt coherence. We need a consistent and objective meta-evaluation.To this end we propose T2IScoreScore (TS2), a benchmark and set of meta-metrics for evaluating T2I faithfulness metrics. While it contains a similar number of images to previously proposed coherence metric evaluation sets, it contains fewer prompts. This high image-to-prompt ratio allows us to organize the images along semantic error graphs, or SEGs (fig. 1 ), where each edge corresponds to a specific error with respect to the prompt that a child image set possesses but its parent images do not. These semantic error graphs permit objective scoring of a metric by answering:1. Can a metric correctly order increasingly wrong images against their generating prompt? 2. Can a metric reliably separate sets of images that differ by a specific semantic error? 3. Does the metric confidently separate the image sets within its dynamic range?We adapt existing statistical tests [26, 27] to the SEG setting to answer these questions for a broad set of T2I faithfulness metrics. We find some surprising results: despite their inferior performance in correlating to human preferences against complicated vision-language model (VLM)-based metrics [6, [16] [17] [18] , simple embedding-correlation methods like CLIPScore [15] are actually quite performant on our meta-metrics, and Pareto-optimal with respect to compute cost ( §5). In summary, we:• Formalize the task of objectively assessing T2I prompt coherence metrics by their ability to correctly order and separate image populations within semantic error graphs (SEGs). ( §2) • Present T2IScoreScore (TS2), our evaluation for this task: a carefully-curated benchmark dataset of SEGs each containing between 4 and 76 images, permitting 93,000 total pairwise image comparisons and meta-metrics for ordering and separation in SEGs. ( §2, §3) • Evaluate a broad and representative set of T2I faithfulness benchmarks using TS2, demonstrate that it identifies novel failure cases, and motivate future work on improved metrics. ( §4 §5, §6)Most evaluations in the T2I space test the quality of generating models based on a fixed faithfulness metric's scores over a fixed benchmark set of prompts. Often these reference prompt sets are designed for testing a single specific capability. DrawBench [4] , T2I-CompBench [28] , and ABC-6k [8] focus on attributes like compositionality, cardinality, and spatial relations, in strictly text-guided image generation, while ImagenHub [14] tests them in a broader set of settings like image editing and subject-driven synthesis. Other evaluation dimensions such as multilinguality [29, 30] and stereotype bias [31] have also been explored. These prompts are usually sourced from some combination of existing natural image captioning resources [32] [33] [34] and sets of in-the-wild conditioning prompts produced by real users [35, 36] . These benchmarks assess image quality either by direct human analysis or automated metrics [14] including those we analyze in this work. Often the goal of these benchmarks is to analyze how well a model generates images that comport with human preferences [14, 37] , and directly elicit opinions from users through a web interface for this purpose.To meta-evaluate faithfulness metrics a benchmark containing a fixed set of images and prompts is required. However, all existing benchmarks for this purpose suffer from two key limitations:1. Evaluating on noisy human preference rather than explicitly labeled objective differences 2. Low image-to-prompt ratio limiting evaluation of discriminatory power over similar imagesCaptioning benchmarks [32] [33] [34] are poor candidates for faithfulness metric evaluation as single images are paired with multiple prompt candidates rather than vice versa. Image matching and entailment benchmarks such as SeeTRUE [38] and Pick-a-Pic [37] are also limited by a low ratio.The few extant deliberately-designed faithfulness evaluation sets are limited by both factors. TIFA v1.0 [16] and DSG-1k [6] were proposed ad-hoc to demonstrate the utility of their accompanying metrics by relating the scores assigned by the metric to human preferences. These are done over small sets of images (800 & 1000 respectively) with slightly more images-per-prompt (5 & 1) .The two limitations of these prior evals are linked. A reliance on human preference correlations is a natural consequence of having few and poorly organized images to compare to each prompt. A lack of meta-metrics designed for evaluating structured aspects other than human preference correlation means limited utility in collecting larger, structured sets of images for each prompt. By providing both meta-metrics and a structured eval set T2IScoreScore overcomes these limitations (table 1 ).2 T2IScoreScore meta-metricsWe introduce three measures of ordering and separation by a given metric within semantic error graphs (SEGs). We define SEG S as prompt P and a directed acyclic graph of nodes n i containing one or more images I j sharing the same errors wrt. the prompt. We label each node by its error count and type (eg, [0, 1a, 1b] has 1 node with 0 errors, and 2 nodes with 1 error each of different type).We introduce measures of metric m for SEG S: rank m (S), sep m (S) & delta m (S), assessed over every walk W ∈ S in all SEGs in the TS2 dataset to score a metric. They're defined as:2.1 Ordering score over walks: rank mWe use Spearman's rank correlation coefficient ρ [26] between image-level error count and metricassigned score over every walk on a SEG to assess how a metric's faithfulness score aligns to our objective structure error counts. Spearman's ρ is the PCC of the rank order of variables X, Y :EQUATIONThus, in our case the SEG-level rank order score rank m (S) for scoring model m is defined as:EQUATIONOne limitation of Spearman's ρ for characterizing scores is that it is undefined if one set R(U ) exclusively contains identical elements, as σ R(U ) = 0. For tractability in these scenarios we define ρ(•, R(U )) := 0. If a metric assigns identical scores to all examples across different error levels, it presents no discernible relationship between error severity and score for that image set. 
statistical separation of error populations score: sep m: 
 We assess the two-sample Kolmogorov-Smirnov statistic [27] pairwise between the populations of metric m's scores assigned to each sample between two error nodes n i and n j as populations.The Kolmogorov-Smirnov statistic is a non-parametric measure of the separation between two distributions [39, 40] , defined as the maximum vertical difference between their empirical cumulative distribution functions F X (s):EQUATIONWhere F X (x) is proportion of samples in population X for which the metric-assigned score m(i) ≤ x, (see fig. 8 for a visual depiction). We compute D KS for every pair of adjacent error nodes in each tree walk W2 , and report the average over all of these as the SEG separation score sep m (S): While sep m nonparametrically estimates whether pairs of nodes are drawn from different distributions (and thereby distinguished), it provides no information about the distance by which they are separated within the metric's dynamic range. This measure gives an alternative look at separation between nodes: the more separation a metric provides between nodes, the less severe slight variations in assigned score will be to ignoring errors in generated images. We assess it as:deltam(S) = 1 |S|σ m(∀S) N i ∈Wavg({m(P, I)|(P, I) ∈ Ni}) -avg({m(P, I)|(P, I) ∈ Ni+1}) (5) Where σ m(∀S) is the standard deviation of scores from metric m on all images in all SEGs in TS2.Our score is the average distance between the mean metric score of all adjacent nodes in all SEGs, rescaled by the standard deviation of the score to normalize against the metric's dynamic range.3 The T2IScoreScore DatasetWe now turn to describing the TS2 dataset collection process. We use three different semantic error graph collection procedures to produce a diverse set of SEGs. Each contains one prompt and between 4 and 76 images assigned to error nodes. Each node usually contains more than one image, though for simplicity in presentation we only show one image assigned to each node in this section. Synth. Synthetic SEGs are produced "graph first." From an initial prompt we list all entities and properties it contains, then ablate them to produce an error graph. We then manually write prompts describing each node, generate their images, and manually check image-node faithfulness. For example, in the left panel of fig. 2 , the initial prompt "a Christmas tree with lights and a teddy bear" is converted to error prompts such as "a Christmas tree with lights." 
dataset collection procedure: 
 Nat. The natural error trees exclusively contain real images sourced from the free stock image repository Pexels. We generate SEGs in "image, graph, prompt" order. We source sets of natural images that share objects and models. We organize them by relation graphs describing how the images differ by objects, actions, attributes, and composition. We then select a head node in this relation graph and write a "prompt" describing this head node (eg., fig. 2 center panel). We produced SEGs of natural images to assess whether distributional differences between synthetic images and real images might lead to measurable impacts on performance for the faithfulness metrics that typically use base models pretrained exclusively on natural images [41] .Real. The real error, synthetic image SEGs are produced following a "prompt, image, graph" order. From a seed prompt (both manually written and sourced from COCO or PartiPrompts) we simply generate a large set of images using a T2I model, then annotate the errors in each generated image. These error-labeled images are then organized into a final error graph for the SEG. This procedure is documented in the right panel of fig. 2 .Each of the 165 SEGs in TS2 is manually checked by three human annotators. The head node of each SEG contains at least one image that has been assessed by the annotators to contain no errors of verbal information (eg, entity in the image isn't performing the described action), compositionality (eg, object described as "on top" but is beneath object), missing objects, or incorrect object attributes.Each edge on the SEG represents an error of one of the aforementioned types. Each node is labeled with the number of edges along its shortest path back to node 0, representing its error count (fig. 1 , fig. 8 ). Each node contains at least one image which is erroneous according to the described errors. The synthetic images are generated with several T2I models-including DALL-E 2 [2] , Stable Diffusion 1.5 [3] , 2.0, 2.1, and SDXL [42] . We use MS-COCO [43] , PartiPrompt [35] , and manually written prompts in head nodes for the Synth and Real subsets. Image sources, prompt sources, and error type statustics are documented in fig. 3 .The total number of images and prompts is roughly in line with previous benchmarks that have been used to verify methods such as TIFA [16] and DSG [6] in their own papers, and permits significantly more image comparisons-per-prompt than any prior benchmark (Table 1 ), which we submit is the primary type of comparison required to verify that prompt-faithfulness assessments work. 
experiments: 
 Using TS2 we evaluated three classes of T2I evaluation metrics: embedding-correlation (comparing embeddings of prompt and image), QG/A (using VQA to check if requirement questions generated from the prompt are satisfied) and caption-based (comparing captions extracted from the generated images to the prompt). We evaluate these metrics with multiple backend VLMs ( §A.2).For each metric, we score every image against its SEG's prompt. We report the results of our Ordering and Separation metrics across all SEGs, as well as for our the Synth, Nat, and Real SEG subsets.CLIPScore [15] is a popular prompt faithfulness metric based on simple text-image similarity. CLIPScore is computed as the cosine similarity of the L 2 -normalized CLIP-assessed [41] image and text embeddings. Equations and details in §A.1.ALIGNScore (not to be confused with the text-only AlignScore [44] ) is a variant embedding-based similarity score we produced using the ALIGN [45] embedding model rather than CLIP to embed the prompt and image. Other than model it is equivalent to CLIPScore.Question Generation & Answering Metrics use an LM M QG to produce a set of requirement question/answer pairs (q, a) ∈ Q from prompt p, and then use a vision-language model M V L to check each requirements against the image, reporting satisfaction rate as the image's faithfulness score. QG/A metrics vary by how questions are generated and relate to each other. Equations in §A.1.TIFA [16] prompts an LM (GPT-3) to generate a set of multiple choice and yes-no questions and their expected answers relative to the prompt. Then a vision language model M V L produces "free-form" answers to each question, which are converted into multiple choice answers a ′ using an SBERT model. The TIFA score for a given image is then the rate of correct answers. Backend VLMs. All QG/A metrics rely on the use of a generative vision-language model (VLM) either for performing question answering (M V L in §4.2) or captioning (M C in §4.3). Thanks to the simple decomposable framework of the QG/A methodologies, we were able to efficiently test the performance of both TIFA and DSG using several VLMs as visual question-answering backends MVLM V L . We used mPLUG, LLaVA, BLIP, InstructBLIP, Fuyu, and GPT-4V as VLM backends for the QG/A metrics. Details and reference for each VLM is provided in appendix §A.2.LLMScore [17] captures the fine-grained similarity between the image and text with rationales by leveraging the visual details understanding capability from vision experts and the reasoning capability of LLMs. The visual information is parsed in hierarchical scene descriptions with global and local captions. Then the text-only LLM (we use GPT3) will compare the multi-granularity visual descriptions with the input text prompt to give a score according to the evaluation guideline prompt.VIEScore [18] rates aspects of semantic consistency (SC) and perceptual quality (PQ) ultimately providing a rating score on a scale of 0 and 10. We use 0-shot LLaVA-1.5 as the backbone MLLM to evaluate how successfully the image follows the text-to-image prompt. 
results: 
 Table 2 shows the results for the Ordering feature rank m and Separation features sep m and delta m for each metric we assessed, on average for all SEGs (Avg), and the three SEG subsets, as well as an approximate FLOP cost per run for each metric ( §A.3).We found that the Synth set consisting of hand-designed (and probably more obvious) errors was the easiest subset for all metrics to correctly order. The average rank m score for Synth across all metrics was 70%, for Nat 55% and for Real 56%. However, different subsets were hard for different classes of metrics. For the QG/A metrics, Real was hardest, while Nat was harder for the other classes.As the embedding-correlation metrics came first, TIFA [16] , DSG [6] , LLMScore [17] , and VIEScore [18] all compare themselves against a CLIPScore baseline [15] . Despite the superiority these metrics supposedly held on their respective ad-hoc evaluations, the computationally cheaper CLIPScore and ALIGNScore are Pareto-optimal in most cases (Figure 4 ), sharing the optimality frontier with DSG or TIFA with GPT-4V, methods that are ≈ 6 orders of magnitude more computationally expensive. Figure 4 : Plots of ordering and separation scores against estimated per-image metric evaluation costs in FLOPs and each other. For all analyses, the Pareto optimal metrics are DSG and TIFA with GPT-4, and the vastly less expensive embedding-correlation ALIGNScore and CLIPScore. 
discussion: 
 The headline takeaway from our findings is that, contrary to claims of superiority leveled in their own papers, for all the QG/A and caption-comparison metrics [6, [16] [17] [18] , the cheap embedding-correlation metrics such as CLIPScore are sufficient or even preferable at capturing objective semantic errors relative to fixed prompts. We view this capacity to accurately discriminate similar images relative to a prompt as the core feature a good prompt faithfulness metric must possess.T2IScoreScoreis effectively evaluating T2I metrics as relative score regressors-functions that are predicting a specific score for an image. However, there are additional desirable elements to a Human aesthetic preferences are-by design-ignored in TS2 meta-evaluation. Though the QG/A and caption-comparison metrics fail to meaningfully outperform the cheap embedding-correlation metrics, they may have advantages that are not captured by TS2, such as in modeling human aesthetic preferences. Paired with a standalone benchmark of human aesthetic preferences over error-free images, a metric's error assessment and aesthetic fidelity could be measured independently.As the first objective evaluation of faithfulness metrics based on structural semantic errors, TS2 enables more fine-grained measurement of metric desiderata, leading researchers to build better metrics, and empowering developers to make trade off-informed metric choices.Why do we care about compute cost? When evaluating T2I models at release, compute costs are not very important-an evaluation only has to be run on a small set of images from benchmark prompts. However, during training or in online monitoring, compute cost for faithfulness metrics becomes quite important. Faithfulness metrics could be used as reward signals while training a T2I model (or prompt generator), or called repeatedly during validation passes. Faithfulness metrics could be deployed in applications to guide an online prompt refinement system, to trigger a second call in user-facing applications, or to analyze prompt corpora to surface challenging examples for further training or analysis. In all of these settings, a performant, low-cost model such as CLIPScore is valuable. TS2 demonstrates that the performance premium for the expensive metrics is quite small.Previous evaluations of coherence metrics have evaluated metrics as human preference score regressors over single images, or over image pairs. The challenge with such an evaluation is that human preferences are not objective-especially when provided by a small pool of annotators, correlation to such scores is an unclear signal.However, by instead evaluating walks over error counts in SEGs, the TS2 captures a more objective notion of correctness, by ignoring the subjective relationships between pairs of unconnected nodes.For example, consider the SEG presented in Figure 1 , where two different single-error nodes are shown. Given the prompt "a bot in a green shirt poses with some fruit," one of these nodes contains images without fruit, and the other contains boys wearing a blue shirt, rather than green. Which of these types of images are actually worse with respect to the prompt? This is a subjective decisionsome annotators may find the missing fruits more important than the incorrectly colored shirt. TS2 ignores this distinction, as no nodes of equivalent error counts are connected in any SEG. While the difference between those nodes is subjective, the difference between both of these nodes and their shared child node-one where fruits are missing and the shirt is incorrectly colored-is objective.It is also important to note that metrics under test are not aware of the implicit ranking task in TS2, as they are evaluated as score regressors. Objective human annotation was only possible because the annotators were aware that relative ranking was the goal.If human performance were judged on the task of simple Likert scoring of image-prompt accuracy without instructions, humans may not significantly outperform the metrics. However, if the human annotators were instructed to count the number of errors, we suspect they would perform quite well, even without the other images for comparison over which the ranking task is performed.Though we provide no human baseline, we do not think this is a significant weakness-human performance on the inherently synthetic task of image quality scoring is not as important as performance on ranking along objective errors.One disadvantage of using Spearman's ρ is that it "expects" ties to be the same in both distributions. For example, if a set of images has error count (0, 1, 1, 2), the ordering (1, 0.5, 0.5, 0) will have a perfect ρ = 1, while the ordering (1, 0.51, 0.49, 0) will be penalized, despite it also presenting a correct ordering. This means that our Ordering score rank m systematically punishes the embedding-based metrics relative to the VLM-based ones, as the embedding-correlation metrics CLIPScore and ALIGNScore can take continuous values, whie TIFA, DSG, LLMScore, and VIEScore have a discrete range. In light of this systematic disadvantage for embedding-correlation metrics, it is even more striking that CLIP/ALIGNScore still are so performant and on the optimality frontier.Although the scores of many models on our metric are high, this meta-evaluation is far from "solved." In principle it should be possible to get much closer to 100 on average for both meta-metrics than we find. We view use of TS2 as a necessary secondary evaluation for any new proposed T2I faithfulness metric; if it has high correlation to subjective human judgements but does not perform well on T2IScoreScore, skepticism might be warranted.Ultimately, all image coherence evaluation metrics stand to improve from further advances in general VLM quality. As a considerably more performant model than LLaVA, mPLUG, etc, it was unsurprising that GPT4-V worked much better as a backbone for TIFA and DSG than the aforementioned. However, there do appear to be diminishing returns, as the order of magnitude going from mPLUG-to GPT4-V-based evaluation yielded a sub-1% improvement in rank m performance on the most difficult and construct-valid Real set. Better constraint-generating processes may be required to push VLM-based evaluation metrics further. 
conclusion: 
 We introduced T2IScoreScore, a first-of-its kind objective evaluation for text-to-image faithfulness metrics that utilizes a high image-to-prompt ratio to organize its reference images along semantic error graphs, through which a faithfulness metric can be assessed by our novel graph-based meta-metrics. Our study reveals a surprising finding: more expensive and recent "state of the art" VLM-metrics actually only have modest gains in performance over simpler and cheaper embedding-based metrics at best. Indeed, these cheap metrics such as CLIPScore and ALIGNScore are actually Pareto optimal along with the vastly more expensive and slightly more performant GPT-4V-based QG/A metrics, even when strictly comparing ordering and separation capabilities (leaving compute cost aside). This underscores the necessity for a more nuanced approach to benchmarking and developing metrics capable of capturing the subtle semantic nuances between prompts and generated images. The establishment of T2IScoreScore as a benchmarking tool is a significant step forward, offering a structured way to rigorously test and improve T2I prompt faithfulness metrics, ensuring they can more accurately reflect the semantic coherence between prompts and generated images, thereby facilitating the development of more reliable and effective T2I models. 
 
 Appendix: Limitations to our work are discussed throughout. For example, in §6.4 we discuss how our metametrics are limited by intrinsic biases of rank-correlation metrics among ties (many of which occur when multiple images occupy one node on a SEG). Additionally, compared to other evaluation sets, our total number of prompts is modest (this is required to achieve a high image-to-prompt ratio, however, which is a core strength of our work). Finally, due to its secretive nature, we are only able to produce rough estimates of the compute cost of GPT-3 and GPT-4 based metrics. We estimate them to the best of our ability using third-party information ( §A.3). This research will steer the development of more effective faithfulness metrics, which in turn will guide T2I model development. T2I models are inherently dual-use: they can be used to produce misinformation and other harmful content in addition to useful and entertaining imagery. Any work that contributes to improving their overall performance necessarily drives a small amount of both positive and deleterious impact in this way.MS checked SEGs, designed the benchmark and meta-metrics, implemented the SEG tree iteration process and evaluation code for the ordering and separation scores, collated the QG/A answers into ID-level scores, and assessed the final scores.FJ produced and annotated the Synth SEGs, produced and annotated a subset of the Real SEGs, and checked all others. FJ collected answers for Fuyu for the QG/A metrics and cleaned and organized the final dataset release.MK produced the Nat SEGs and produced, annotated, and checked the other SEGs. MK generated the TIFA and DSG questions for all prompts, implemented and evaluated CLIPScore and ALIGNScore, collected answers for the QG/A metrics from BLIP, InstructBLIP, GPT-4V and refactored code.YL evaluated LLMScore for the examples and conceived of measuring faithfulness errors in T2I faithfulness metrics. AS collected answers for the QG/A metrics from LLaVA and VIEScore. 50 tokens of LLM or VLM compute to score an image. LLMScore's use of mPLUG to caption is negligible alongside the cost of running GPT-3.Total compute cost of our study. In total, we estimate our study took 9.89 × 10 1 8 FLOPs, mostly through OpenAI's service, but also on lab-owned NVIDIA Titan-X and A-100 GPUs.For more information on the structure of the semantic error graphs (SEGs), we provide examples here. SEG 85 is one example with a more interesting topology than the example in Figure 1 . Figure 5 has a structure including two-error edges, single-parent nodes, single-child nodes, multi-parent nodes, and multi-child nodes in the same graph, corresponding to prompt "guy with umbrella hat sitting at a table with another person with a hat under a red umbrella."Figure 5 : Example of a SEG (85) with a more complex structure. Some nodes have multiple child nodes, and some edges correspond to more than one error (dark red).Figure 7 exemplifies why we choose to only score rank order along walks of the graph, rather than between all pairs of nodes. A priori there's no reason the beach-less images should be worse than the umbrellas, yet metrics consistently rate the beach error more severe.The first few steps of TIFA [16] and Davidsonian Scene Graph (DSG) [6] scoring methods are nearly identical: an LLM generates a set of requirements as questions, and a VQA system answers them. However, the two methods differ chiefly in how the answers are combined into a single image-level score. TIFA simply scores images by the correct answer rate, while DSG uses the graph structure of the requirements to build in some robustness: if an upstream requirement is not met (e.g., is there a boy? : no), then downstream requirements are all also assessed as not being met, regardless of answer. In the example provided, if the question "is the boy's shirt green?" were answered yes, the DSG accumulation technique would still score this requirement as being not met, due to the upstream requirement, while the TIFA accumulation method would score it as being met. No beachAvg Scores: CLIPScore: .82 TIFA-L:.78 TIFA-mP: . 85 DSG-L:.78 DSG-MP: 1.0 LLM-EC: 0.33 Avg Scores: CLIPScore: .69 TIFA-L:.44 TIFA-mP: .78 DSG-L:.22 DSG-mP: .89 LLM-EC: 1.6 Avg Scores: CLIPScore: .37 TIFA-L:.15 TIFA-mP: .24 DSG-L:.20 DSG-mP: .30 LLM-EC: 5.2 Figure 7: Examples from SEG 71 (The beach is crowded with red and white umbrellas). Even though both nodes 1a and 1b have the same error count (1) they systematically differ across all metrics: all metrics punish the images where the umbrellas are just in water (no beach, 1b) more than they penalize an empty beach with no umbrellas (1a). Table 3 : Comparing how using DSG vs TIFA-style accumulation for scoring each image by DSG questions impacts performance along both our metrics. The right half of this table is identical to the DSG section in Table 2 , and bold, italic, and highlighting follows the same rules, except cells in the TIFA half are marked as if they were replacing the right half cells in the DSG section in Table 2 .As a supplementary experiment, we compare how accumulating the DSG questions using the DSG technique compares to accumulating them with the TIFA technique in Table 3 . Interstingly, the impact of this change differs between strong and weak VLMs, between ordering and and separation scores, and between the easier and harder subsets. For example, switching from the DSG to TIFAstyle acculumation consistently improves ordering performance for mPLUG, while it worsens performance for LLaVA, InstructBLIP, and BLIP1. For Fuyu, the weakest model, DSGstyle accumulation significantly improves performance over TIFA. This strengthens the claim from [6] that using the scene graph to check requirements adds robustness; it makes a lot of sense that this robustness benefits the lowest-performing VQA systems the most.For separation scores, TIFA accumulation improves performance of more models. In particular, TIFA accumulation pushes InstructBLIP into the top 3 for separation on the Synth subset, while no DSG metric using DSG accumulation breaks into the top 3 (red highlighted cell).Here we provide full histograms for our Spearman Ordering and Kolmogorov-Smirnov Separation scores, across every SEG, for all metrics we assessed. Here we provide line plots for a set of metrics and SEGs. Note that for normalized _ rank, higher is worse (more errors). High-correlation is assessed when the metric lines (higher better) go down as the metric lines go up. Ordering score across the Synth, Nat, and Real populations, and the two least-correlated (c, d). Note that the two highest-correlated metrics are both QG/A metrics using the same underlying VLM (DSG and TIFA using BLIP1, (a); TIFA using LLaVA with two different system prompts, (b)).Figure 12 and Figure 13 show scatter plots for the Ordering (Spearman) and Separation (KS statistic) scores for every SEG between the most highly-correlated (a, b) and low-correlation (c, d) pairs of metrics under evaluation, respectively.Although this benchmark has 14 pairs of images per prompt, each pair is separately annotated. With no way to compare between pairs, the effective number of images per prompt is2, with many repeated prompts.We use individual nodes ni ∈ S containing pairs of prompt, image (P, I) ∈ ni to make this equation easier to read. 


Decision: **Summary:** This paper addresses the issue of how to evaluate evaluation metrics for text-to-image models. They introduce a dataset of error graphs, which are prompts alongside images of with increasing number of semantic error graphs, such as incorrect color or count of objects. Evaluation metrics are scored by how accurately they rank images. Images that are more consistent with prompt, e.g. higher in the graph, should be ranked higher as well. They find that simpler, embedding space metrics such as CLIPScore tend to outperform more complex, GQ/A based metrics.

**Aggregate Strengths/Weaknesses:** The reviewers find that this is a structured approach to evaluating T2I metrics. They view the datasets as a valuable contribution and the experiments as in-depth and rigorous. The weaknesses are that there is not a single cohesive takeaway when comparing metrics using TS2. It’s not clear how researchers should decide on a single metric and TS2 also does not manage to incorporate other aspects of each metric, such as the fine-grained analysis GQ/A metrics provide. As a smaller point, the reviewers suggest including more models (even GANs/VAE) and more metrics (VQAScore, ImageReward). Time-permitting, it would be nice to see VQAScore in the final version. I'm especially curious about the results given its simplicity (no question generation, the difficulty of GQ/A approaches) while still leveraging a VQA model.

**Recommendation:** Every reviewer recommended this paper for acceptance. This work addresses an important problem of better understanding and selecting our evaluation metrics given the rise of T2I models. The dataset is well-structured and makes comparing two different metrics straightforward. The paper was well-received by all reviewers. I therefore recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces T2IScoreScore (TS2), which aims to evaluate how good newly developed text-to-image (T2I) evaluation metrics/methods are. The authors formalize the task of evaluating t2i metrics as their abilities to *order* images correctly within SEGs.
strength_weakness: strength_weakness: 1. Limited Scope: The evaluation is primarily focused on a specific subset of T2I models (many variants of SD, and Dalle-2) and metrics. Expanding the scope to include a broader range of models and datasets would strengthen the generalizability of the findings. Potentially should consider other synthetic images from models such as OpenMUSE or aMUSEd (https://huggingface.co/blog/amused) with totally different generation architectures than diffusion, etc. Alternatively, text-2-image is an old task, even GAN and VAE can probably have image distribution other hand SD and Dalle-2 which heavily depends on CLIP. 

2. Intrinsic Bias: The reliance on rank-correlation metrics, which have intrinsic biases, might affect the evaluation results. A more thorough discussion of these biases and potential alternatives could enhance the robustness of the conclusions.
questions: questions: Instead of pairwise ranking/comparisons, which might not always be robust, have the authors design multi-images ranking instead of only two images for pairwise comparisons? Something like bradley-terry style ranking could make the evaluation more robust.
limitations: limitations: n/a
paper_summary: paper_summary: The paper proposes an evaluation framework for holistically assessing text-to-image (T2I) evaluation methods. Since most of them are primarily established through simplistic correlational evidence and only compared to the CLIPScore baseline, this approach presents a more detailed way of assessment and also benchmarks existing promising evaluations. While there isn't a single clear winner, results suggest that CLIPScore is still a very competitive candidate and is especially successful when considering the much lower compute costs.
strength_weakness: strength_weakness: - I like the general setup but I'm unsure about the accuracy of the "number of errors" counting system. I'll give two examples from within the paper. Take the last example in Figure 3 with the prompt "A gray elephant and a pink flamingo". An image with two flamingos is categorized as containing one error because there is no elephant. However, if there additionally was an elephant, it would still have an error since there are two instead of one flamingo. So one could argue that there are in fact two errors: a missing elephant and an additional flamingo. Or you say it's one error because there is one animal that is a flamingo but should be an elephant. So this is inherently ambiguous. However, in any ranking solution, this can actually matter quite a lot, so I'm worried that this introduces noise into the analysis process that is hard to reason over. I'm wondering to what extent this is taken into account by the design and how sensitive the results are to this. (Second example to illustrate from Figure 1 SEG: it's noted that when the shirt is not green, it's counted as one error. What if the shirt was additionally also suddenly a hoody. Is that then two errors or still only one? When the boy is gone overall, it's two errors because the shirt isn't green and there is no boy -- but what if there was now a grey shirt in the picture?) 
- Given that the evaluation framework provides many different evaluations for varying setups and (as discussed in the paper) those might come with their own biases, what is the recommendation to those who are thinking about using this framework for when they can call their metric successful? Defining an overall evaluation aggregate might also help with the adoption of the framework.
- (Minor: It's hard to see which numbers are italicized in the results table.)
questions: questions: None
limitations: limitations: Sufficiently addressed.
paper_summary: paper_summary: This paper presents a rigorous evaluation for text-to-image alignment metrics. This is primarily done by introducing a dataset with several images for each prompt, allowing the construction of semantic graphs that can be used to measure the accuracy of the alignment metrics. From the analysis on the benchmark, a major conclusion is that CLIPScore provides an excellent tradeoff (or is at least on the pareto-optimal frontier) between speed and alignment. VQA-based metrics (e.g TIFA, DSG) while improving over CLIPScore in many cases, come with much higher costs (in some cases orders of magnitudes higher), highlighting important considerations for text-image alignment methods.
strength_weakness: strength_weakness: In terms of models/methods evaluated, I see 2 notable omissions: human-preference models such as ImageReward would also be a good addition since they might also capture some notion of text-image alignment while also cheap to use. Another good addition would be VQAScore (much more recent), but seems to show extremely strong results on several image-text matching benchmarks, while not being nearly as expensive as the Question-Generation methods (e.g. TIFA, DSG). 

Minor: While I totally agree with the issue of methods evaluating on their own proposed test set, the column "ad-hoc" in Tab. 1 makes little sense without an explanation (line 85 seems too limited). Either a more complete explanation should be given, or it would be better to replace this column with something more objective/concrete.  


[a] Lin et al. "Evaluating Text-to-Visual Generation with Image-to-Text Generation", 2024
questions: questions: Overall, I really like the paper and would like to see this accepted. I have a couple of questions for my curiosity. 
While 2.8k prompts is still a lot more than TIFA, DSG, is this size still not too small/limited diversity to be able to comprehensively make conclusions about alignment methods?

Is there any idea of what 'human-performance' would look like on these benchmarks, and if existing models are very far off from that?
limitations: limitations: No major concern.
paper_summary: paper_summary: The authors propose T2IScoreScore (TS2), a benchmark and set of meta-metrics for evaluating text-to-image (T2I) faithfulness metrics. Compared to existing relevant benchmarks, TS2 has higher image-to-prompt ratios, which allows users to organize semantic error graphs (SEGs), where each edge corresponds to a specific error with respect to the prompt that a child image set possesses but its parent images do not. Based on SEG, the authors evaluate T2I metrics, including embedding-based (CLIPScore/ALIGNScore), QG/A-based (TIFA/DSG), and caption-based (LLMScore/VIEScore) metrics, by how the metrics properly order and separate images. Different metrics show different advantages, and the authors highlight that simple embedding-based metrics outperform other more computational metrics in separation criteria.
strength_weakness: strength_weakness: **1. Simply treating QG/A metrics as score regressors.**

One of the major motivations behind using QG/A metrics, even though they are computationally expensive, is that they divide multiple aspects of prompts and provide comprehensive skill-specific performances of the T2I model. While the QG/A metrics can be used as a score regressor, comparing them with embedding-based metrics ignores their biggest advantages. This background and limitation needs to be clarified in introduction section, otherwise this could misleading readers who are new to T2I metrics.
questions: questions: See weaknesses
limitations: limitations: There is no significant negative societal impact.

Comments: title: Some Comments/Suggestions
comment: I thank the authors for their clarifications in the rebuttal. They sufficiently addressed my concerns and I updated my scores accordingly.
I appreciate authors' response and decided to keep my current score.
Please make sure to incorporate the points you mentioned in the next version if accepted.
Thank you for clarifying. These are great points:

1. TS2 + and orthogonal quality-only eval might actually tease out the degree to which human annotators attend over each consideration by directly comparing their preference correlations to each metric under different annotation schemes. A really cool idea might be to treat those two considerations as principal components a metric could be interpolated between, or enabling a search for a jointly optimized single "best metric." We will definitely investigate adding ImageReward to the final leaderboard given this.
2. Yes, this is another stimulating direction for future work. Using this human baseline also has implications for direct "VLM-as-a-judge" metrics that ask them to provide Likert scores, as only "superhuman Likert-assigning" VLMs would be sufficient to beat humble CLIPScore.
3. Agreed, we will make sure to clarify that QG/A metrics have this interpretability advantage, and in particular that this consideration, alongside TS2 evaluation, helps users choose a metric based on scenario and needs. Eg., in an interactive app (relatively low analysis throughput rate) cost is less of a consideration, and a human would benefit from interpretable analysis. Whereas for an online reward/feedback model or a supervisory post-filter for image generation, cost is an important consideration and interpretability isn't. Grounding metric selection in all of these considerations is best; TS2's contribution is in capturing one important consideration well.

Thanks for the further stimulating discussion!
Thanks for a quick follow up! To give a more committal answer your question:

Intuitively, the walk-based spearman correlation metric is probably the best choice for an overall score, as it captures the core desideratum of "able to correctly compare similar images by structural differences." For this desideratum, higher is always better.

While the other two scores do matter---it is important that adjacent nodes be statistically significantly separated---it is less clear that higher is always better, vs anything over a threshold being sufficient. Thus a good recommendation might be to rely on the walk ordering score (which is also the main novel contribution here) and to treat the separation scores as a secondary consideration.

In other words, par performance on the separation metrics alongside significant gains on ordering would be a very positive development, whereas significant improvement along separation with a loss in ordering could be negative---exact reverse ordering of the nodes with statistically significant node separation would get high delta scores, but be very bad.

This is the recommendation and justification we will provide in the camera-ready: **a metric is clearly superior to others when it presents significantly higher ordering (particularly over the hard *nat* subset) without a significant drop in separation scores**---equivalent separation is sufficient.

Ultimately, this recommendation is a judgement call and its main grounding is the aforementioned theoretical analysis (high ordering score always captures a good correlation to the scores, whereas high separation score can be present, even when the ordering is reversed). We will use this analysis to justify our recommendation to researchers in the conclusion section of the camera ready.
I thank the authors for their response. 

What you're saying makes sense to me, especially when it comes to the error counting matter. 

I just want to reiterate on one part of my prior review which is on defining when this framework establishes "success". I understand that this framework provides a detailed holistic overview on a range of interesting dimensions (see Table 2). However, I'm wondering whether there is a recommendation for researchers who want to use this framework to choose the best-performing solution. Are there specific rankings/dimensions that are most diagnostic for overall performance? (And for adoption in the broader community, having an overall score that accumulates the individual results in Table 2 might help with adoption in the community. Do you have a suggestion what this score might be?)
I thank the authors for their reply, I have no major concerns left about the paper, and I see that all the other concerns of the reviewers are satisfactorily addressed. That said, I have a few comments that the authors may wish to think about:

1) Human-Preference Models: I agree that human-preference models are not immediately clear about what exactly they evaluate. They also start from CLIP/BLIP models, and then finetune it on data which capture some mixture of visual quality (i.e artifacts), aesthetics, and prompt following all at once. For instance, if a vital object from the prompt is missed in the image, the user would naturally rate it low. Similarly, if the image has a lot of artfiacts but follows the prompt well, it is unlikely to do well on comparisons. Depending on the guidelines, annotation protocol, you can get models that result in very different behaviors. For instance, in the VQAScore paper (disclaimer: I have no connection to it), ImageReward outperforms both CLIP, BLIP on most benchmarks (Tab. 4) and is even doing reasonably well on Winoground, Eqben (Tab.3 ). Therefore, I would not dismiss them as solving an entirely different task, and adding TS2 as an additional eval benchmark for these models would be a good idea.

2) Human-Evaluations: I think the authors make an excellent point that humans doing Likert scoring of image-prompt accuracy might actually not outperform existing metrics. This is a useful pointer (since some prior works recommend Likert scoring of image-prompts as a good strategy to evaluate models[a]) in performing more rigorous user studies for evaluating text-to-image models. 

3) QG/A metrics doing more than a single score regression: To reviewer Xy9h, the authors point out that QG/A metrics are proposed claiming superior correlation with human judgement on various benchmarks. While I agree with this, the authors should look at Fig. 1 of TIFA which clearly makes the claims of "fine-grained", "accurate", "interpretable". Of course, the fine-grained/interpretable aspects are the hardest to evaluate and justify, therefore papers will inevitably resort to maximizing performance/correlation on benchmarks to justify the method. That does not mean the other aspects of the method are invalid/absent, they are just insufficiently evaluated (beyond a few qualitative examples). Therefore, I would suggest the authors that they acknowledge the strength of QG/A methods, while providing a fair assessment of their shortcomings (which is already there in the paper). 

I hope the authors can think about these aspects and make the additions/modifications that they deem fit for the camera ready/benchmark leaderboard. 

[a]: Otani et al. "Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation", CVPR 2023