Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating

Authors: Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Christopher Knutsen, Cyrus Rashtchian, Anant Nawalgaria, Jordi Pont-Tuset, Aida Nematzadeh
Primary area: datasets and benchmarks
Keywords: text-to-image evaluation; text-to-image alignment; human evaluation;

Abstract:
While text-to-image (T2I) generative models have become ubiquitous, they do not necessarily generate images that align with a given prompt. 
While many metrics and benchmarks have been proposed to evaluate T2I models and alignment metrics, the impact of the evaluation components (prompt sets, human annotations, evaluation task) has not been systematically measured.
We find that looking at only *one slice of data*, i.e. one set of capabilities or human annotations, is not enough to obtain stable conclusions that generalise to new conditions or slices when evaluating T2I models or alignment metrics. 
We address this by introducing an evaluation suite of $>$100K annotations across four human annotation templates that comprehensively evaluates models' capabilities across a range of methods for gathering human annotations and comparing models.
In particular, we propose (1) a carefully curated set of prompts -- *Gecko2K*; (2) a statistically grounded method of comparing T2I models; and (3) how to systematically evaluate metrics under three *evaluation tasks* -- *model ordering, pair-wise instance scoring, point-wise instance scoring*.
Using this evaluation suite, we evaluate a wide range of metrics and find that a metric may do better in one setting but worse in another.
As a result, we introduce a new, interpretable auto-eval metric that is consistently better correlated with human ratings than such existing metrics on our  evaluation suite--across different human templates and evaluation settings--and on TIFA160.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730668620721

thread_type:
official_review

<thread_object>
creator: reviewer_dzmn

<content>
summary:
The paper introduces a comprehensive benchmark for text-to-image evaluation. The main contribution is the Gecko dataset: which consists of 2k prompts (1k from existing DSG, and 1k fresh prompts), along with images from 4 T2I models (SD1.5, SDXL, Imagen Vermeer, and Muse). The key insight in the paper is that annotations from different formats (i.e likert scale/absolute rating, pairwise comparisons and DSG type scoring) can end up with different conclusions, and to that end, the paper suggests model ordering evaluations using all these metrics. Finally, the paper also presents Gecko, a evaluation metric based on Q/A with some fixes for improved Q/A generation, which seems to be a better alternative for evaluating T2I models.

strengths:
The paper is quite rigorous in its methodology and seems to comprehensively validate several evaluation methodologies for T2I. In general, this might be one of the most statistically sound evaluations of T2I models and has a lot of insights for future work on this topic. 
Further, the data provided in the paper (assuming public release) would also be useful to evaluate newer reward models/metrics in the future.

weaknesses:
The biggest drawback/weakness that I can see in this paper is that it predominantly evaluates only 4 T2I models on 1/2k prompts (excluding the results with the gecko metric later on). While this already may be an improvement over prior work, I am curious as to would there be benefits to scaling the prompts 10x? Or are the results/conclusions at this level itself sufficient and there would be mostly diminishing returns from increasing the scale of the evaluations?

From the paper, it's not immediately clear whether all the data from the Gecko evaluation shall be released publicly. While the findings of the paper are valuable in itself, the contributions of the paper are significantly determined by the public release (or lack thereof) of the data. 


The authors may also want to acknowledge a concurrent work on similar lines[1]


[1] Saxon et al. "Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)", NeurIPS 2024

questions:
I would like to confirm that I understand correctly: In Fig. 2 an "=" in a grid cell means that for the considered evaluation protocol (e.g Likert, SxS), the difference in the performances of the model are not statistically significant (even if the absolute score is different?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702124638

thread_type:
official_review

<thread_object>
creator: reviewer_gy7m

<content>
summary:
- Automatic evals are an effective alternative to human judgement to evaluate text-to-image (T2I) models. 
- In this work authors perform a rigorous analysis on evaluation prompt sets, metrics and templates to derive reliable conclusions.
- Authors identify that different conclusions can be drawn when the prompt sets or evaluation templates are changed. 
- This work introduces a new prompt set Gecko2K, a statistically grounded method to compare T2I models and a new framework to systematically evaluate metrics under model ordering, pair-wise instance scoring and point-wise instance scoring.
- Based on their analysis, authors propose a new automatic evaluation method that is highly correlated with human judgement across different human templates and evaluation settings.

strengths:
- The authors conduct a rigorous study on the methodologies used for evaluating the text faithfulness of T2I models.
- The insights and takeaways from this paper are highly useful to advance research in both modeling and evaluating generative models.
- The experiments are very exhaustive and support all the claims made in the paper.
- The auto-eval method developed to increase coverage, remove hallucinations and normalize the VLM scores is novel and effective.
- A lot of thought and analysis has gone into designing the Gecko-2k reliable prompt set and future research in T2I can benefit from this work.
- Authors demonstrate that this method is not just limited to T2I models but can be easily extended to any modality generation models.

weaknesses:
- The paper does not read well and requires multiple passes to properly assimilate. 
- The terminology used is not intuitive and have not been used in prior literature making it harder to put them in context. Authors mention several terms like "skills", "reliable prompts" without properly introducing them creating confusion at several places. I would recommend authors first introduce the terms (what are skills? Maybe introduce them in the introduction. 
Providing an example: a) The term "skills" was first introduced in L044 and the readers are directed to Table1 for more details. It has only become clear what those are after reading Table 8 (in the supplementary material). The column names in Table 8 are also different (Prompt Category and Subcategory and not skills and subskills). After finishing the introduction, the reader is still unsure what skills are. I have no issues with the term "skills" but authors can clarify what skills and subskills are in the first 2-3 paragraphs of the introduction(where the term was first introduced). Maybe L190-194 can be copied to the introduction so the readers know what to expect? 

- A few suggestions, Fig. 2 is very hard to understand. Is there another way to convey the same information which is easier to read? There are too many rows making it hard to parse. Can you break the prompt sets to make it less busy? All the reliable prompt sets can be put in a separate figure and moved to the supplementary. From my understanding, the reliable prompts help reduce rater disagreement and not central to the contribution of the work. Also the color scheme was used for relations (red: less than, green: greater than and purple: roughly equal). Can the color scheme instead be used to show disagreement between templates i.e. to convey the information in L284-292?

questions:
- The differentiation between pairwise instance scoring and model ordering is not clear. Authors should spend some time discussing the differences. Concretely, from L402-403, if a metric is a "good indicator on side-by-side comparison" (pair-wise instance scoring) then wouldn't it automatically be a good one to predict the "model ordering"? Do you mean pairwise instance scoring gives a partial ordering but model ordering should give a total order between models? How would a metric give a total order without comparing all models exhaustively? Please elaborate on the differences to highlight the importance of "model ordering" when a metric is already a good "pairwise instance" scorer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730950951674

thread_type:
official_review

<thread_object>
creator: reviewer_kjyy

<content>
summary:
This work introduces a comprehensive evaluation suite with over 100,000 annotations and four human annotation templates to assess T2I model capabilities across diverse conditions. Key contributions include a curated prompt set (Gecko2K), a statistical method for comparing models, and a framework for systematic metric evaluation across three tasks. The results suggest that metrics vary in effectiveness across settings, and to overcome these issues the authors propose an interpretable auto-eval metric that consistently correlates better with human ratings.

strengths:
- Proper comparison with previous works
- Through evaluation of the proposed metric and the dataset
- The authors have proposed a new benchmark dataset to highlight the issue of evaluation across multiple templates
- Furthermore, the authors propose a new metric for evaluating QA/VQA models
- The paper is well-written and easy to follow

weaknesses:
- My primary concern is with the validity of the metric proposed:
  - A central aspect of the proposed metric is its use of NLI to detect hallucinations, making the metric reliant on the accuracy of the NLI model. Errors in the NLI model could undermine confidence in the reliability of this metric.
  - The proposed alignment metric in Equation 2 relies entirely on the negative log-likelihood (NLL) scores generated by the visual question-answering (VQA) model to evaluate alignment. NLL scores measure how well the model predicts a correct answer given an image and a question prompt; lower scores indicate higher confidence in the answer. However, this approach assumes that the VQA model’s confidence is accurate and does not account for situations where the model might be overly confident (assigning low NLL scores even when it’s wrong) or under-confident (assigning high NLL scores even when it’s correct).
If the model is over-confident, it might produce low NLL scores for incorrect or poorly aligned answers, leading to artificially high alignment scores. Conversely, if the model is under-confident, it might assign high NLL scores even for well-aligned responses, causing the alignment score to underestimate the actual alignment. Because Equation 2 relies solely on these NLL scores without adjustments, it doesn’t account for these potential biases in model confidence, which could compromise the accuracy and reliability of the alignment metric.

questions:
- Since the proposed metric depends heavily on the accuracy of the NLI model to detect hallucinations, how do you address potential errors from the NLI model that could impact the reliability of your metric? Have you considered any methods to mitigate this dependency?
- The alignment metric in Equation 2 assumes that the VQA model’s confidence, as measured by NLL scores, is accurate. How do you account for cases where the VQA model might be overly confident in incorrect answers or under-confident incorrect ones?
- Given that low NLL scores could indicate either true alignment or over-confidence in incorrect answers, and high NLL scores could indicate either misalignment or under-confidence incorrect answers, have you considered additional adjustments or alternative metrics to address this bias?
- Since the proposed metric doesn’t currently account for potential biases in the VQA model’s confidence, how might this impact the interpretability and reliability of alignment scores? Would adding a calibration step improve the metric’s robustness?

Post-rebuttal: After reviewing the authors' responses, I am increasing my score from 5 to 6.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739890985650

thread_type:
public_comment

<thread_object>
creator: ~olivia_wiles1

<content>
title:
Updated paper title.

comment:
We have updated the paper title to better highlight the three main contributions in the work. We have made only a handful of superficial changes within the text of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper introduces Gecko2K, a robust benchmark and auto-evaluation metric for text-to-image (T2I) alignment, rigorously validating evaluation tasks, human templates, and metrics. Key findings show significant metric variability across settings and demonstrate superior human correlation with the proposed method.
(b) Strengths: Comprehensive analysis, novel benchmark, new auto-eval metric, and practical insights for T2I research.
(c) Weaknesses: Reliance on VQA/NLI models, limited scale (4 models, 2K prompts), and clarity issues in terminology and visuals.
(d) Decision: Accept. The paper provides strong contributions, sound evaluations, and actionable findings, advancing T2I alignment research despite minor limitations.