Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating
Authors: Olivia Wiles
Chuhan Zhang
Isabela Albuquerque
Ivana Kajic
Su Wang
Emanuele Bugliarello
Yasumasa Onoe
Pinelopi Papalampidi
Ira Ktena
Christopher Knutsen
Cyrus Rashtchian
Anant Nawalgaria
Jordi Pont-Tuset
Aida Nematzadeh
Venue: ICLR 2025 Spotlight (2025)
Keywords: text-to-image evaluation; text-to-image alignment; human evaluation;
Primary area: datasets and benchmarks
TL;DR: We create a large benchmark for T2I alignment to evaluate models and metrics across skills, evaluation tasks, and human annotation templates.

Abstract: While text-to-image (T2I) generative models have become ubiquitous, they do not necessarily generate images that align with a given prompt. 
While many metrics and benchmarks have been proposed to evaluate T2I models and alignment metrics, the impact of the evaluation components (prompt sets, human annotations, evaluation task) has not been systematically measured.
We find that looking at only *one slice of data*, i.e. one set of capabilities or human annotations, is not enough to obtain stable conclusions that generalise to new conditions or slices when evaluating T2I models or alignment metrics. 
We address this by introducing an evaluation suite of $>$100K annotations across four human annotation templates that comprehensively evaluates models' capabilities across a range of methods for gathering human annotations and comparing models.
In particular, we propose (1) a carefully curated set of prompts -- *Gecko2K*; (2) a statistically grounded method of comparing T2I models; and (3) how to systematically evaluate metrics under three *evaluation tasks* -- *model ordering, pair-wise instance scoring, point-wise instance scoring*.
Using this evaluation suite, we evaluate a wide range of metrics and find that a metric may do better in one setting but worse in another.
As a result, we introduce a new, interpretable auto-eval metric that is consistently better correlated with human ratings than such existing metrics on our  evaluation suite--across different human templates and evaluation settings--and on TIFA160.

Published as a conference paper at ICLR 2025
REVISITING TEXT-TO-IMAGE EVALUATION WITH GECKO: ON METRICS, PROMPTS AND HUMAN RAT- ING
Olivia Wiles*
Chuhan Zhang
Isabela Albuquerque*1
Ivana Kajićl
Su Wang
Emanuele Bugliarello
Yasumasa Onoe
Pinelopi Papalampidi
Ira Ktena
Chris Knutsen
Cyrus Rashtchian
Anant Nawalgaria
Jordi Tont-Tuset?
Aida Nematzadeh
ABSTRACT
While text-to-image (T2I) generative models have become ubiquitous, they do not necessarily generate images that align with given prompt. While many metrics and benchmarks have been proposed to evaluate T2I models for alignment, the impact of the evaluation components (prompt sets, human annotations, evaluation task) has not been systematically measured. We find that looking at only one slice of data, i.e. one set of skills or human annotations, is not enough to obtain sta- ble conclusions that generalise to new conditions or slices when evaluating T21 models aligmment metrics. We address this by introducing evaluation suite > 100K annotations across four human annotation templates that comprehen- sively evaluates models' capabilities across range of methods for gathering hu- man annotations and comparing models. In particular, we propose (1) a carefully curated of prompts Gecko2K; (2) a statistically grounded method of com- paring T2I models; and (3) framework to systematically evaluate metrics under three evaluation tasks model ordering, pair-wise instance scoring, point-wise instance scoring. Using this evaluation suite, we compare wide range of metrics and find that given metric may do better in one setting but worse in another. As a result, we introduce new, interpretable auto-eval metric that is consistently bet- ter latrelated with human ratings than existing ones on our evaluation across different human templates and evaluation settings-and and T6FA
INTRODUCTION
Text-to-image (T2I) models (Saharia et al., 2022; Yu al., 2022b; Betker al., 2023; Rombach et al., 2022) generate images of impressive quality, but the images are not necessarily aligned with the prompt. The key comparing T2I models is the dataset of prompts and human annotations we collect. Human annotation is slow and expensive, motivating the creation (Hu et al., 2023; Cho et al., 2023a) of automatic-evaluation (auto-eval) metrics as a replacement. To evaluate both metrics and models, human annotation is the gold standard. However, Clark et al. (2021) show that the template design and annotator knowledge can significantly impact results the text domain. this work, we create comprehensive benchmark to answer the question: how do the choices around prompts and human annotation templates impact our metric and modelling decisions?
There has been limited work analysing the impact model and metric ranking due to these choices. Previous work builds benchmark by collecting annotations across one template and prompts that cover limited distribution of skills (see Table for comparison). A skill refers to generation
Equua contribution. Correspondence to: oawiles@ @ooiles@googll.ommm nematzadeh @ google.com. Google DeepMind, 'Google Research, 'Google Cloud. Github link; https: //gi thub. com/ com/goog. ggoogl deepm hwwpg//coeeeeehhhhnoeeeee

---

Published as a conference paper at ICLR 2025
Likert
Word Level
DSG
Side-by-Siide Side
Model 1 (M1)
Q1: Is there cat?
Q2: the cartoon? X
professor
outfir
writing
Q3: Is the cat in a protessor outhie? V/
1-2-3-4-5
writing a baok?
book
the
title
what
Q5: Is the book title "what if cat
wrote book
wrote book?"
Model 2 (M2)
Q1: there cat? V/
cartoon cat in
Q2: the cut cartoon? X
professor
writing
professor outfit? v/ Q4: Is the cat writing a book?
Q3: the
1-2-3-4-5
book
with
the
tille
what
book title "what if cat wrote book?" X
wrote
Model ordering
MI < M2
MI M2
MI M2
MI M2
Figure 1: Model ordering outcomes for one annotation template do not necessarily generalise to other templates. We generate images for two models using the prompt: cartoon cat professor outfit, writing book with the title "what a cat wrote book.". " By collecting extensive human evaluation, we expose disparities across templates: outcomes between T2I models or auto- eval metric obtained for one template may not generalise to others.
challenge, such as text rendering or generating different colors and shapes and a sub-skill refers to sub-challenges (e.g. generating longer text or Gibberish). Other work does not systematically gather prompts to ensure a wide coverage of skills and properties with the exception Zhu et al. (2023). Cho et al. (2023a) does consider different lengths of prompts and Zhu al. (2023) some skills but this is done for a specific template and does not consider varied challenges for given skill.
By looking at slice data (e.g.. too few too specific prompts plate), we are at risk of drawing conclusions that are specific to that slice and do not generalise. As result, collect comprehensive dataset (Table 1) systematically over different prompt sources and different skills/subskills. Given this prompt set, we generate images from four T2I models and rate them across four human annotation templates. By considering model rankings across annotation templates for a given prompt, we can further determine how reliably a prompt measures alignment.
human annotation tem-
Similarly, the choice of task may impact our results. Auto-eval metrics are typically evaluated using correlation with human judgement. However, in practice, we would want to use metrics for three tasks: (1) model ordering, ranking T2I models based on significant relationships; (2) pair-wise instance scoring, choosing whether given output is better than another and (3) point-wise instance scoring, an estimation of samples' overall alignment. Evaluating metrics on one task not enough: may think are choosing the best metric for all three but we show that conclusions for one task do not necessarily generalise. An overview our contributions follows:
Gecko: An evaluation suite for T2I alignment which includes a comprehensive set of 2K prompts, 4 human templates to evaluate 4 T2I models to give ~ 100K human annotations (Table 1). We get predictions from a wide range auto-eval metrics and evaluate under realistic settings (model ordering, pair-wise instance scoring, point-wise instance scoring).
Using our suite, we demonstrate limitations of looking at a single slice of data currently done in the literature: different metrics and models show different results depending on the prompt slice or template.
Based on our analyses, we introduce an interpretable state-of-the-art QA/VQA metric. It gets the most number of model comparisons right, and performs average 40.5% better than interpretable baselines on our dataset in terms of pair-wise instance scoring and point-wise instance scoring respectively, and 10.5% better on TIFA160 (Hu et al., 2023).
2
RELATED WORK
Benchmarking alignment in T2I models. Many benchmarks have been proposed to holistically evaluate model capabilities within T21 alignment. Early benchmarks small scale and created
2

---

Published as a conference paper at ICLR 2025
Liert
Word Lovel
DSGUH)
SXS
prouppe arnutated
Skills (ATI
4 Sub-SLill
Categories
(Cle- al, 3023a]
105K
[1(13)
Drawillnnhh (Saharia al., 2022)
200
25
25K
7(111
PaniP: (Ya 20226)
1,6
16K
9(23)
TIEAI60 (it st. 2023)
150
148
8(12)
Printkilll
20236)
150
2.25K
2023)
24K
15,20)
HEIM al., 2014)
-34
-150K
GeckoZK Gackoe)) GeckorS)
2K
~105K
12(12)
1K
<:4KK
11(1
IK
~54K
12012)
Table 1: Comparison of a'annttteed alignment datasets. We report the amount of human annotation and skill division for each dataset. We can see that many datasets include only a handful of annotated prompts a small number of annotations (anns) per image overall. No dataset besides Gecko2K collects ratings across multiple different human annotation templates. We also include the number of skills and sub-skills in each dataset. Again, Gecko includes the most number of sub-skills, allowing for a fine-grained evaluation of metrics and models. When datasets do not include skills, we map their categories into Akills/sub-skill as appropriate.
alongside model development to perform side-by-side model comparisons (Saharia et al., 2022; Yu et al., 2022b; Betker et al., 2023). Later work (e.g., TIFA (Hu et al., 2023), DSGIK (Cho et al., 2023a) and HEIM (Lee et al., 2024)) focuses on creating holistic benchmarks by drawing from ex- isting datasets (e.g.. MSCOCO (Lin et al., 2014), Localized Narratives (Pont-Tuset et al., 2020) and CountBench (Paiss et al., 2023)) to evaluate a range of capabilities including counting, spatial re- lationships, and robustness. Other datasets focus on specific challenge such as compositionality (Huang et al., 2024a), contrastive reasoning (Zhu et al., 2023), text rendering (Tuo et al., 2023), reasoning (Cho et al., 2023b), spatial reasoning (Gokhale et al., 2022), or specifically image order- ing given an increasing number of errors (Saxon et al., 2024). The Gecko2K benchmark is similar in spirit to TIFA and DSGIK in that it evaluates set of skills. However, in addition to drawing from previous
taaneess-hiie may be biased or poorly representative of the challenges of partic- ular skill- we collate prompts across sub-skills for each skill obtain discriminative prompt set. Moreover, we gather human annotations across multiple templates and many prompts (see Table I).
Automatic metrics measuring T2I alignment. Inspired by work in image captioning, widely used auto-eval metric is CLIPSSore (Hessel et al., 2021). However, such metrics poorly capture finer-grained aspects of images (Bugliarello et al., 2023; Yuksekgonul et al., 2022). Motivated by work in NLP on evaluation using entailment or QA metrics (Maynez et al., 2020; Kryšciński et al., 2019; Honovich et al., 2021), similar metrics (Yarom et al., 2024) have been devised for T21 align- ment. However, such a metric may not generalise to new settings and is not interpretable one cannot diagnose why an alignment score is given. Visual question answering (VQA) methods such as TIFA (Hu et al., 2023), VQ Yarom et al. (2024) and DSG (Cho et al., 2023a) do not require task-specific finetuning and give an interpretable explanation for their score. These metrics create QA pairs which are then scored with VIM given an image and aggregated into a single score. However, the performance of such methods is conditional on the behaviour of the underlying LLMs used for question generation, and VLMs used for answering questions.
3 Gecko2K: THE GECKO BENCHMARK
We curate a fine-grained skill-based benchmark, Gecko2K, with good coverage by curating two sets of prompts: one created systematically based on a set of skills and subskills (Gecko(S)) and one generated by combining existing datasets but tagging them and resampling to ensure good coverage over those tags (G)) We generate Gecko(R) by extending the DSGIK (Cho et al., 2023a) benchmark creation approach to use automatic tagging and improve the distribution of skills and linguistic properties (see App. for details on the automatic tagging). However, due to the automatic tagging and nature of the underlying datasets, Gecko(R) is limited in the skills/sub-skills it covers.
To generate our systematic set, we propose a hierarchical method combined with LLM generation in order to ensure systematic distribution across skills (c.g., counting) and subskills (e.g., simple modifier: '1 cat' vs additive: cat and dogs'). This notion of sub-skills ensures we are capturing wide distribution of prompts and not just one easy slice (e.g.generating counts of 1-4 objects).
3

---

Published as a conference paper at ICLR 2025
3.1 GR): RESAMPLING DAVIDSONIAN SCENE GRAPH BENCHMARK
The recent DSGIK benchmark (Cho et al., 2023a) curates a list of prompts from existing image- text datasets" but does not control for the coverage or complexity of a given skill. The authors randomly sample 100 prompts and limit the prompt length to 200 characters. The resulting dataset imbalanced in terms of the distribution of skills. Also, as T2I models take in longer and longer prompts, the dataset will not test models on that capability. We take principled approach in creating Gecko(R) by resampling from the base datasets in DSGIK for better coverage and lifting the length limit. After this process, there are 175 prompts longer than 200 characters and maximum length of 570 characters. Also, this new dataset has better coverage over a variety of skills than the original DSGIK dataset (see Fig. 6).
While resampling improves the distribution of skills, Gecko(R) has the following shortcomings. Due to the limitations of automatic tagging, it does not include all skills we wish to explore (e.g., language). It also does not include sub-skills: e.g., text rendering prompts do not focus on numerical text, or longer text (see Fig. 7). Finally, automatic tagging can be error prone.
3.2 GECKO(S): A CONTROLLED AND DIAGNOSTIC PROMPT SET
The aim of Gec is to generate prompts a controllable manner for skills that are not well represented in previous work. We divide skills into sub-skills to diversify the difficulty and content of prompts. We take inspiration from psychology literature where possible (e.g., colour perception) and known limitations of current T2I models.
Curating a controlled set of prompts with an LLM. To generate a set of prompts semi- automatically, we use an LLM. We first decide on the sub-skills we wish to test for. For exam- ple, for text rendering, we may want to test for (1) English vs Gibberish to evaluate the model's ability to generate uncommon words, and (2) the length of the text to be generated. We then create a template which conditions the genera-
Sub- skill
Example
Numbers / sym- bols
equation of "3e4 7" esched into rock
Leagth
with words "the furure dlready
here.." reflected many street. (n=29)
Gibberish
grafflii made with hright pink paint on the con-
crese, soying "fuff floop floo!""
Typography
Jove you" written send four grass
Table 2: Subcategories and corresponding mo-
tivations for the text rendering skill.
tion on these properties. Note that as we can generate as much data as desired, we can define distribution over the properties and control the number of examples generated for each sub-skill. Finally, we run the LLM and manually validate that the prompts are reasonable, fluent, and match the conditioning variables (e.g., the prompt has the right length and is Gibberish/ English). A sample template is given in App. B.3.
Gecko(S) make up. Using this approach and also some manual curation, we focus on twelve skills falling into five categories (Fig. 5): (1) NAMED ENTITIES; (2) TEXT RENDERING; (3) LINUUSICELICGUISTICC COMPLEXITY; (4) RELATIONAL: ACTION, SPATIAL, SCALE; (5) AT- TRIBUTES: COLOR, COUNT, SURFACES (TEXTURE/MATERIAL)) SHAPE, STYLE. For sub-skills, give full breakdown for all skills in App. B.4. In Table 2 we give examples of the sub-skills and corresponding prompts for TEXT RENDERING. Using this approach, we get better coverage over the given sub-skills than other datasets (including Gecko(R)) as shown in Fig. 7.
4 COMPARING ANNOTATION TEMPLATES FOR MODELLING
We examine how the choice of human annotation template impacts results when comparing four models: SD1.5 (Rombach et al., 2022), SDXL. (Podell et al., 2023), Muse (Chang et al., 2023), and Imagen Vermeer (Vasconcelos et al., 2024). We consider absolute comparison templates (i.e., Likert, Word Level from Liang et al. (2023), and DSG(H) from Cho et al. (2023a)) which evaluate models individually, and template for relative comparison of two models (side-by-side or SxS). A
"TIFA(Hu et al., 2023), Stanford (Krause et al., 2017), Localized (Pont Tuseet et al., 2020), Count- Bench (Paiss et al., 2023), VRD (Lu al., 2016). DiffusionDB (Wang et al., 2022). (Ture & Nemade, 2023), PoseScript (Delmas et al., 2022), Whoops (Bitton et al., 2023), TratTexx--reativee (Liu et al., 2022). Muse based on the original model, but trained different data sources.
4

---

Published as a conference paper at ICLR 2025
Muse
SDXT
SDLS
SDXI.
SDLS
SDI
WL
DIED
DEE
WL
SxS5
WIL
S65
WT.
DXED
Imugen
G(S)
Muse
SDXL
Figure 2: Comparing models using human annotations. We compare model rankings on (R))o(S)(R). Each grid represents comparison between two models. Entries in the grid depict results for WI., Likert (L), DSG(H) (D(H)), and side-by-side (SxS) scores. The >> sign indicates the left-side model is better, worse (<), or not significantly different (=) than the model on the top.
high-level visualisation of each template is in Fig. and details in App. D. We further introduce principled method to determine significant model orderings based on human judgements.
4.1 DATA QUALITY
We validate the reliability of each template and examine the choice of the template impacts the quality of the collected data. Given the collected human ratings across the three templates, we compute inter-annotator agreement (IAA) for each generative model by measuring Krippendorff's a, &a 1, (Hayes & Krippendorff, 2007), where value of I indicates perfect agreement and O chance (Zapf et al., 2016). Results reported in Table 3 show that agreements are all high, with > 0.5, except for the Likert SD1.5 pair for Gecko(R); we conjecture this is due to the lower quality images of SD1.5. Overall we find that fine-grained templates (WL and DSG(H)) are more reliable ge. have higher IAA) and WL achieves the highest IAA for the diverse Gecko(R). We also measure IAA for the SxS template in Table 11. We see lower IAA for the SxS template (though still far above chance) compared to the fine-grained ones. The IAA is < 0.5 for 6 out of 12 model comparisons. It seems, given the same number of annotators, the SxS template is less reliable.
Reliable prompts. Upon manual investigation, we find that differences in human ratings across templates can arise when prompts difficult judge with respect to alignment (and not due the choice of the template): for example, when a prompt contains domain specific knowledge such as "A bottle of Irn-Bru sitting on a shelf" or subjective notions such "a fuluristic sculpture". To understand how this impacts our results, we consider a subset of the prompts that achieve high IAA across templates and models, For each model and absolute template, we select the prompts for which inter-rater disagreement is < 50% of the maximum disagreement observed across all prompts for that model-template pair. The intersection of these prompts across models and templates gives our reliable prompts. We additionally remove instances where all Likert ratings are Unsure to get 531 and 725 reliable prompts for Gecko(R) and Gecko(S), respectively. We first validate that using reli- able prompts increases IAA on the SxS template (which was not used in the selection process) and find that it increases the average K., from 0.45 to 0.47 on Gecko(R), and 0.49 to 0.54 on Gecko(S) (see App. 2 for details). In the next sections, demonstrate how this subset of prompts increases agreement among templates, but at the expense of removing some potentially meaningful prompts.
4.2 ABSOLUTE ANNOTATION TEMPLATES: COMPARING T21 MODELS
Average ratings. For the absolute annotation templates, previous work compares T2I models by comparing the average ratings across examples. We report these values in Table 3 and find that the chosen prompt set impacts which model is best (e.g. SDXL for Gecko(R) and Muse for Gecko(S)). Moreover, the model with the lowest rating depends on both the prompt and the template: given Gecko(R), Imagen is worse using Likert, but SD1.5 is worse if using DSG(H). This highlights the importance of examining models in various conditions. When using T2I models in practice, we need to make conclusions about model ordering with high confidence. We argue that this evaluation is not enough: does not measure if the difference between models significant, which is particularly important as models start to saturate. As a result, we introduce the model ordering task.
Model ordering. We verify the significance of outcomes by performing the Wilcoxon signed-rank test with p < 0.001. Where results indicate the null-hypothesis is rejected (i.e., the distribution
Defined the variance across image, word, and question ratings for Likert, WT. and DSG(H) respectively.
5

---

Published as a conference paper at ICLR 2025
Inter anmotator agreement
Scores
Gen. model
R)
Gecko(S)
Gecko(R)
GeckotS)
WL
Likert
DSG(H)
WL
Likert
DSG(H)
WL
Likert
DSG(H)
WL
Likert
DSG(H)
Imagen
0.64
0.68
0.72
0.57
0.75
0.74
0.60
0.80
p.59
g.78
Muse
0.78
0.72
0.69
0.58
0.72
0.84
0.61=
0.83
0,88
0.63-m2
0.84
SDXL
0.76
0.57
0.67
0.56
0.70
a.87
0.86
0.80
0.60:121
0.79
SD1.5
0.36
0.66
0.69
0.86
0.67
0.76
0.61
0.49
0.68
Table 3: Inter-annotator agreement and ratings for all models and templates. We measure inter-annotator agreement for each human evaluation template with Krippendorff's ss a. Higher values indicate better agreement. We also show the mean and std. deviation for the annotated judgements of all templates after mapping the ratings to the [0, 1] interval, with indicating perfect alignment.
of ratings is significantly different), can say that one model is better than another. To deter- mine which model is best, we compare the mean values of their ratings. In Fig. 2 we visualise the outcomes for all model pairs across all templates. We that Muse is not worse than any of the contenders across all templates and prompt sets, except for 2 out of the 12 comparisons involving Muse for Gecko(R); we determine is the best overall model. In contrast with the results presented in Table 3, where SDXL is identified as the best model for Gecko(R) across all the templates, we observe that the significance results reveal that Muse and SDXL actually have similar performance, showcasing the importance of determining significance before drawing conclusions.
Reliable prompts. Constraining Gecko(R) using the reliable subset decreases the number of con- flicts between the different templates, but at the potential expense of comparing models on fewer, potentially easier, prompts. Considering the two prompt sets, we observe that when using the syn- thetic prompts, Gecko(S)-rel, all templates agree in Fig. 13 in Appendix D.2. We hypothesise this is because the skills (e.g., color or shape), while hard to generate, are easy to evaluate within gen- eration. For Gecko(R)-rel, we see disagreements between templates, where surprisingly, DSG(H) often result in a different relation than the two other templates. We also consider the full prompt-set, Gecko2K rrel, as it better captures the overall use cases of T2I models: we find that there is always majority agreement, and the two fine-grained templates (WL and DSG(H)) always agree.
Results by skill. We explore how human judgements vary by skill and template; average ratings for each absolute template are shown in Fig. 23-Fig. 26 in the appendix. lower average ratings per skill across templates indicates how 'challenging' a given skill is: we can see that 'lang compositional", 'lang complexity' "count" and 'text' are consistently difficult across templates.
4.3 RELATIVE ANNOTATION TEMPLATE: COMPARING T21 MODELS
Model ordering. For the SxS template, a model is considered better is chosen as preferred more often than the competitor and the Unsure rating. To assess statistical significance, a similar procedure as for the absolute annotation templates using binary scores for the ratings: when there was a tie, +1 when a model was preferred by the majority of raters, and -1 otherwise.
perform
We also compare the SxS template with the considered absolute annotation templates by computing the accuracy obtained by each absolute template when predicting the preferred model given by SxS on Gecko2K-rel. Results presented in Table 12 in A.p.D.2 show that all absolute annotation templates predict SxS judgements with similar average accuracy of around 70%, with DSG being the overall best, This shows that, although the results of pairwise model comparisons are the same in many cases for Gecko2K-rel as shown in Fig. 13, absolute and side-by-side annotations do not necessarily correspond to the same model ordering at the datapoint level.
Fine- fine-grained templates (i.e. ones that require multiple annotations per example), WL and DSG(H), yield the highest inter-annotator agreement. Takeaway 2: All three absolute annotation templates achieve similar, but not perfect, accuracy when predicting relative comparison annotations for each datapoint. Takeaway 3: To compare models reliably, we need to measure the significant model ordering. Model ordering depends on the human template and prompt set, but some prompt sets lead to consistent agreement across templates (e.g., are discriminative) such as our skill-based Gecko(S) the larger, reliable set Gecko2k-Rel.
Takeaway 1:

---

Published as a conference paper at ICLR 2025
GeckorR)
GeckotS)
Metries
Zero
WL Liken DSG(()
SXS
WL
Likent EXS(())
SXS
SpearmanR
Acc
SpearmanR
Interpretable (QA/QQA)
TIENAA 2/eaL1
0.26
0.34
0.28
41.7
0.39
0.32
0.39
53.2
DSG 2IBALE
0.47
0,42
0.45
Geckogaut
0.41
0.55
0.46
62.1
0.47
0.52
0.45
74.6
Geckoogsin
0.43
0.58
0.48
72.2
0.54
0.59
0.56
78.8
Uninterpretable (single score)
CLIP
0.14
0.16
0.13
54.4
0.25
0.18
0.26
67.2
PynmidCIIIP
0.26
0.27
0.26
64.3
0.22
0.25
0.23
70.7
VQAScorGaestas Flash
0.54
73.1
0.51
0.57
76.5
VNI.I
0.37 0.49
0.42
54.4
0.45
0.55
0.45
72.7
Table 4: Correlation between auto-eval metrics and human ratings across annotation templates on Gecko2K. With the same backend, Geeko outperforms all other QA/VQA metrics across all evaluations and Gecko with GeminiFlash performs even better; it performs better or similar to the strongest single-score approach (VQAScore). Bold: Top results. Underlined: Top results by category.
THE GECKO METRIC
An auto-eval metric is more useful if it is (1) interpretable-iit it reports where model fails in ad- not require a reference distribution, and (3)
dition to its overall goodness, (2) reference-free- modular-can easily leverage better pretrained models for improved performance. As a result, focus on improving recent work using a two-stage QA/VQA metric (Hu et al., 2023; Cho et al., 2023a; Yarom et al., 2024) that matches this criteria (as opposed to metrics such as VNLI (Yarom et al., 2024) and CLIP (Radford et al., 2021)). However, the QA/VQA pipelines are impacted by the shortcomings of the pretrained models used. In particular, we identify two main limitations of these pipelines and address them: the QA generation is not always grounded in the prompt as the generated questions might not necessarily cover all key parts of the prompt and also there might be hallucinated questions that not related to the prompt. Morcover, at the VQA stage. the highest scoring answer might still be low probably but is treated as the "right" answer- we model this un- certainty in the VQA responses. Finally, we simplify the previously proposed methods by removing complexities (such as scene graph generation in DSG) and show that our simplified and improved setup is significantly better across the board.
A standard QA setup (e.g.. Hu et al. (2023)) consists of three steps: (1) QA generation: prompting an L.MM generate a set of binary question-answer pairs (Q:, A. on given T2I text description T. (2) VQA assessment: employing VQA model to predict answer {A)) for the generated questions given the generated image 1. (3) Scoring: computing the alignment score by assessing the VQA accuracy using Eq. (1):
Σ D[A = A.J. N
1
Groundedness: inereasing coverage. To ensure the coverage of questions over the key elements in text sentence T, we split the QA generation into two steps. We first prompt the LLM to index the visually groundable words in the sentence. For example, the sentence "A red colored dog." transformed into "A {red colored] [2)g.." Subsequently, using the text with annotated keywords {W/J as input, we prompt the LLM again to generate a QA pair {q:, a;} for each word labelled {w[} in an iterative manner (see App. C for the prompting details). This two-step process ensures more comprehensive and controllable QA generation process, particularly for complex or detailed text descriptions where the prompted LLM often selectively generates questions for specific segments of the text while overlooking others.
Groundedness:
removing hallucination. L.LMs can hallucinate (Bang et al., 2023; Guerreiro et al., 2023), leading to the generation of low-quality, unreliable QA pairs. We filter out hallucinated QA pairs by taking inspiration from previous work in NLP (Maynez et al., 2020; Kryšciñisii et al., 2019); employ a Natural Language Inference (NLI) model (Honovich et al., 2022) model for measuring the factual consistency between the text T and QA pairs (Q..4,) QA pairs with a consistency score lower than threshold r are removed, ensuring that the remaining QAs are about the prompt.
Uncertainty: VQA score normalisation. Finally, we improve aggregation of scores from the VQA model. The reliance binary judgement-strrctly matching A and A, without considering the
7

---

Published as a conference paper at ICLR 2025
GedkarR
GerkanSi
TIFA
63.35
1025
11.00
6312
0.10
contrige
0.25
0.34
0032
0.41
0:35
0.40
+ Secre pocss
0.32
0.42
0.37
0.43
037
0.41
Birning
6.51
0.46
0.53
5:41
0.55
6.43
0.54
0.61
3:50
Table Validation of each component of the proposed Gecko metric on Gecko2K. We evaluate the utility of the three proposed improvements by adding them to the TIFA baseline one by one. They all bring higher correlation with human judgement across the board on Gecko2K.
P
0.71
8:34
0x2
1.43
0:55
0.69
0:45
5:44
0.50
0:63
0:45
Figure 3: Per skill results of different metries. Likert correlation for each skill; square black in- dicates p- p-values > 0.05. Full results in App. F.
predicted probability of A oveoooolooks the inherent uncertainty in the predictions; VQA model can predict a very similar score for two answers. we simply take the max, then we lose this notion of uncertainty reflected in the scores. As a result, we normalise the scores follows.
N
8a
Aligmment (T,
N
Si
where the negative log likelihood of answer A is 8; and the correct answer is A' with score 8a-
EXPERIMENTS ON AUTO-EVAL METRICS
We evaluate metrics across multiple prompt sets and templates to determine how they fare on the three tasks: (1) Do they give good numeric measurement of overall alignment - point-wise in- stance scoring; (2) Are they good indicators side by side comparisons pair-wise instance scoring; (3) Can they predict model ordering. We demonstrate that the task can impact rankings but that our Gecko metric consistently performs best for (eC) across tasks and TIFA 160. App. F.1 gives a thorough description of each task and intuitive examples for how they differ.
6.1 EXPERIMENTAL SETUP
Metrics. We benchmark two types of metrics. First, metrics that give single score, including contrastive models (CLIP (Radford et al., 2021), PyramidCLIP (Gao et al., 2022) and 16 variants in Sec. F.4); (2) VNLI (Yarom al., 2024); and (3) VQAScore (Lin al., 2024). Second, interpretable QA/VQA based methods: TIFA (Hu et al., 2023), DSG (Cho et al., 2023a) and our metric Gecko.
a VIT-B/32 (Dosovitskiy al., 2020) CLIP model and ViT- B/16 (Dosovitskiy et al., 2020) PyramidCLIP model. For VQAScore, we use a GeminiFlash (Reid al., 2024) backend. For all the VQA-based metries, we use PaLM-2 (Anil al., 2023) as the LLM and PaLI (Chen et al., 2022) as the VQA models in all the metrics for fair comparison. When evaluating the Gecko metric, apart from using the LLM and VQA models above, we utilise a T5-11B model from Honovich et al. (2022) for NLI filtering and set the threshold r at 0.005. This threshold was determined by examining QA pairs with NLI probability scores below 0.05. We observed that QAs with scores below 0.005 are typically hallucinations. We re-use the original prompts from TIFA for generating QAs, and add coverage notation to their selected texts as described in Sec. 5. We additionally explore how the performance of point-wise instance scoring changes for the Gecko metric we swap out the QA/VQA models for a stronger Gemini Flash model. Finally, some baseline models are trained with maximum text input length L,e.g. LeLi 77 and LYNLI 82. For these models, we only take the first L tokens from the text as input.
Back-end models. For CLIP,
6.2 COMPARING AUTO-EVAL METRICS ON POINT-WISE INSTANCE SCORING
We first evaluate how well metrics measure T2I alignment at instance level. We compute the Pearson and Spearman Ranked correlation between the auto-eval scores and human scores on all the instances in prompt set. The evaluations are done the Gecko benchmark and TIFA 160.
8

---

Published as a conference paper at ICLR 2025
Metries
Q4
VQA
Spearman's.
Kendall's 7
WL
Likert
SxS
RODGET,
METEOR
SPICE
NA
NIA
0.33
0.34
0125
VIdeoCLIP
Metries
0.18
Pearson
0.21
28.0
Acc
CLIP
0.33
0.23
VQAScoreeetnt Flash
0.30
0.33
52.0
TIFA
GPT3
GPE3
MPLUG
BLIS1
0.36
0.60
0.7
0.44
GeckO Flash 0.43
0.45
55.8
Pala
DSG
PALM
0.57
0.16
Correlation between auto-eval metries and human ratings for text-to- video evaluations. Gecko outperforms other auto-eval metrics on VBench overall consis- tency prompts. demonstrating the generality of the approach to other modalities.
Table 7:
Oooka
PALM
PAL
0.6
0.50
Table 6: Comparing different metries by their correlation with human Likert rat- ings on T6FA160. The Gecko metric out- performs the others by a significant margin.
Component validation on proposed Gecko metric. We validate the utility of the three key improvements we proposed: coverage, linear normalisation, and NLI filtering. Starting from our baseline TIFA, we include the improvements one at a time. The results in Table 5 uniformly demon- strate a positive impact. NI.I filtering brings the largest boost among the three, underscoring the limitation of the PaLM-2 LLM in reliably generating high-quality and accurate QA pairs.
Results on Geeko benchmark We next compare auto-eval metrics. We start with metrics (CLIP and its variants, TIFA, DSG, Gecko) that do not rely on fine-tuning, As shown in Table 4, the Gecko met- ric outperforms other QA/VQA metrics using the same backend by a wide margin. Swapping out the backend of Gecko with stronger GeminiFlash model leads to large improvements across the board. Contrastive models (e.g. CLIP variants) are worse than QA-based metrics, but VQAScore is strong baseline. Finally, we compare Gecko with VNLI, our supervised baseline, as it fine-tuned for text-image alignment on mixed dataset containing COCO (which is used in Ge), while other metrics are zero-shot. is worth noting that the correlation scores of different auto-eval metries are generally higher on Gecko(S) than on (R). This validates that our skills-based benchmark has more objective and balanced measure of alignment. We observe similar conclusions on the Gecko2K Reliable Prompts (Gecko2K-Rac Rell) subset; results are in App. F.3.
TIFA160 results. We compare the Gecko metric with other metrics on TIFA160 (Hu et al., 2023), set of 160 text-image pairs, each annotated with two Likert ratings. In Table 6, we list the results reported Hu et al. (2023) and Cho et al. (2023a), and compare them with Gecko as well as our re-implementation of TIFA/ DSG, Gecko has the highest correlation, with an average correlation 0.07 higher than that of DSG, when using the same QA and VQA models. This shows that the power of our proposed metric is from the method itself, not from the advance of models used.
Skill-based evaluation with Gecko. To better understand the differences between auto-eval metric- s/annotation templates with respect to various skills, we visualise breakdown of skills in Gec in Fig. 3 and App. 1, F.5. The metrics have different strengths: e.g., we see that while Gecko, VQAScore, VNLI metrics are consistently good across skills, the Gecko metric is better on more complex and compositional language, DSG is best on compositional prompts, and VNLI is better on named entities. As with the overall results, these per skill conclusions seem to hold across templates.
Qualitative examples. We visualise examples in Fig. 4. For the negation example, the reason DSG(H) gives inconsistent results with WL/Likert here is that the question generation is confused by the negation (asking if there are cars as opposed to no cars). We can also see that VNLI and DSG mistakenly think none of the images are aligned. VNLI and DSG perform better on the shape prompt but VNLI scores Imagen incorrectly and DSG gives hard scores per question (0 or 1) and so is sometimes not able to capture subtler differences in the human ratings.
6.3 COMPARING AUTO-EVAL METRICS ON WAIR-WISE INSTANCE SCORING
We measure how well an auto-eval metric is able to select between two generations given a prompt. We compare metrics' predictions with the human choices we collected by computing accuracy- percentage of times the metric gets the comparison right. Results are in the SxS column in Table 4. Although Gecko was the clear winner on point-wise instance scoring, single-score metrics are gen- erally very good at SxS comparison. PyramidCLIP was worse than TIFA and DSG on point-wise instance scoring, but it has much higher SxS accuracy, showing that different human annotation templates do not always give the same result, and single-score metrics can be a good estimator on the pair-wise instance scoring task. While VQAScore is better than the Gecko metric on SxS com-
9

---

Published as a conference paper at ICLR 2025
(subskdil::
Iaggloosplexitt (negation) bridge with
Shage: (hlerarchical) number
Prompt:
cincles
300
Imsagen
Muse
SDXL
SD15
Imagen
Muse
SDXL
SD15
I.
0.
Lakert
DSGGl)
0.5
0,5
0.92
0.
Goeko:
0.96
0.94
0.93
0.91
0.9
0.95
0.55
0.25
1.
0.36
Figure 4:
Qualitative results. Image generations of the four '21 models prompts in Gecko(S), with the human annotation ratings and auto-eval scores.
parison Gecko(S), Gecko is better Gec and the Gecko metric is the only interpretable metric that has better or comparable performance with single-score metrics on SxS comparisons.
6.4 COMPARING AUTO-EVAL METRICS ON MODEL ORDERING
A good auto-eval metric should be able to give overall model ordering for set of prompts. To decide on a ground-truth ordering, we use Gecko2K-rel as it is the largest subset that has highest agreement across templates. We take the majority vote relationship in Fig. 13 as the ground truth. We compare these results to the significant relationships found using the auto-eval metrics in App. F22 (we only use PaLM/PaLI-2 backends if there is a choice). We find that CLIP performs poorly, confusing wins with losses. All other auto-eval metrics perform well, never confusing a win with loss but sometimes not finding significant relations when there one or vice versa. Gecko correctly finds and predicts all significant relations, unlike the other metrics.
6.5 EXTENDING GECKO TO OTHER MODALITIES
To explore the generality of our approach on different modalities, we validate on text-to-vides video generation. We choose prompt set from VBench (Huang et al., 2024b) and compare the following models: Lumiere (Bar-Tal et al., 2024), Phenaki (Villegas et al., 2022) and WALT (Gupta et al., 2023). For human evaluation, consider absolute (i.e., Likert, Word Level) and side-by-side templates. For automatic evaluation, we benchmark contrastive models (i.e., VideoCLIP: Xu et al. 2021) and VQA-based metries. For VQA-based metries, we extend the VQAScore and our fine- grained Gecko metric on videos using Gemini Flash, which can process long context multimodal inputs. We present the results in Table 7 and find that the Gecko metric agrees more closely with human judgement across all human templates than other metrics. See Appendix G for more details.
Although Gecko is the best metric on different human templates and modalities, we find that the ranking of different auto-eval metrics can change depending on whether they are evaluated on an instance-level template (e.g., Likert or DSG(H)), a comparative template (e.g. SxS) for model ordering. It is important to evaluate metrics across range of settings and in particular on one relative and one absolute template under budget constraints.
Takeaway:
7 CONCLUSIONS
We introduce the Gecko evaluation suite, a comprehensive set of prompts, human ratings across templates, and tasks evaluate T2I models and alignment metrics. We find that looking single slice of the data (e.g., one annotation template, or one evaluation task) can give misleading obser- vations of the relative benefits of one model or metric. Instead, we show that we need to use comprehensive prompt set (or manually evaluated "reliable prompts") to achieve consistent model orderings and thereby confidence in model rankings. Given this evaluation suite, we demonstrate that our Gecko metric performs consistently best across three tasks, measuring how metrics per- form in scoring each image- text instance with respect to their alignment as well as ranking models. Our work highlights the importance of standardising the evaluation framework with respect to the prompt sets, the annotation templates, and metrics used. This is crucial when conducting research models and metrics, and also to make informed decisions.
10

---

Published as a conference paper at ICLR 2025
8 ETHICS STATEMENT
When gathering our dataset, we ensure that raters are compensated and provide consent as described in App. D.I.I. We also run safety filters over the generated images before giving them the raters. This work is step towards better evaluation of texl-to-image models which are known to halluci- nate. It gives tools others developers and practitioners properly understand and evaluate T21 models in the future.
9 REPRODUCIBILITY STATEMENT
We give extensive details of our setup in the Appendix. For human annotation, we visualise the templates used and give extensive detail on how these raw ratings are aggregated in App. D.1. For the dataset collation, we give the few shot prompts used to generate tags and templates: the few shot prompt for Gecko(R) is given in Listing For Gecko(S), we give our full decomposition of skills in Table 8 with examples and explanation of how generated prompts for each specific skill in App. B.4 with sample few shot prompts. For metrics, we give full details of the baselines in Sec. 6.1 and the additional CLIP baselines in Sec. F.4. For Gecko metrics, give the few shot prompt for generating coverage in Listing 3 and for generating the QAs in Listing 4.
We thank Zi Wang, Miloš Stanojević, and Jason Baldridge for their feedback throughout the project. We are grateful to Andrew Zisserman for his feedback on the manuscript. We thank Aayush Upadhyay and the rest of the Podium team for their help in running models.
Acknowledgements

---

Published as a conference paper at ICLR 2025
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1.74991--44 1971-4980, 2018. 26
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprini ar 2rXiv:2305. /04003 2023. 8
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302. 00000.00022... 2023." 7
Omer Bar-Tal Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12940 2024. 10, 45
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dalle-3. pdf, 2(3):8, 2023. 1,
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-languagee bench- mark of synthetic and compositional images. In ICCV, 2023.
Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Ne- matzadeh. Measuring progress in fine-grained vision-and-languagee understanding. arXiv preprint 2705002300. 2023. 3
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Computer vision and pattern recognition (CVPR), 2018 IEEE conference on. IEEE, 2018. 24
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 4055-4075. PMLR. 23-29 Jul 2023. URL https: / /prcceed ngs.. wtw.mmmr.ppeee. press v202 2hang2233 2.3b htm1.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Sori- cut. Pali: A jointly-scales multilingual language-ing image model. arXiv preprint NiXv:22090066944 2022. 8
Xinyi Chen, Raquel Fernández, and Sandro Pezzelle. The bla benchmark: Investigating basic lan- guage abilities of pre-trained multimodal models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5817-5830, 2023. 24
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tused and Su Wang. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-image generation. arXiv preprint arXi::2310. 18235, 2023a. 1, 3, 4, 7, 8, 9, 17, 18, 19, 26, 27, 40
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, 2023b. 3
12

---

Published as a conference paper at ICLR 2025
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A. Smith. All that's 'human' not gold: Evaluating human evaluation of generated text. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference Natural Language Processing (Volume I: Long Papers), 2021.
Dean C. Delis, Lynn C. Robertson, and Robert Efron. Hemispheric specialization of memory for visual hierarchical stimuli. Neuropsychologia, 24(2):205 1986. ISSN 0028-3932. 23
Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Mloreno-Noguce and Grégory Ro- gez. Posescript: 3d human poses from natural language. ECCV, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020. 8
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramid- clip: Hierarchical feature alignment for vision-language model pretraining. In NeurlPS, 2022. 8, 40
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by- trial behaviour of cnns and humans by measuring error consistency. Advances in Neural Informa- fion Processing Systems, 33:13 1390--13902, 2020.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL, 9: 346-361, 2021. 23
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in exxt-to-image generation. arXiv preprint 2rXiv:2212.10011 2022. 3
Nuno M Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. Hallucinations in large multilingual translation models. TACL, 11:1500-1517, 2023. 7
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 24
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint MrXiv:2312.06622 2023, 10, 45
Andrew F Hayes and Klaus Krippendorff. Answering the call for standard reliability measure for coding data. Communication methods and measures, 1(1):77-89, 2007. 5
Cindy Yoonjoung Heo, Bona Kim, Kwangsoo Park, and Robin M Back. A comparison of best-worst scaling and likert scale methods on peer-to-peer accommodation attributes. Journal of business research, 148:368-377, 2022. 27
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A
reference-free evaluation metric for image captioning. In EMNLP, pp. 7514-7528, 2021. 3
Matthew Honnibal and Ines Montani. Spacy 2: Natural language understanding with bloom embed- dings, convolutional neural networks and incremental parsing. spacy.io, 2017. 24
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. Q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. arXiv preprint arXiv:2104.082209 2021.
13

---

Published as a conference paper at ICLR 2025
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation. arXiv preprint ar arXiv:2204.0499) 2022. 7.8
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. arXiv preprint arXiv:2303. 2023. 1, 3, 4, 7, 8,9, 9,,,0 40
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-worid compositional text-to-image generation. NeurIPS, 36, 2024a. 3
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024b. 10,4; 45
Gabriel Uharco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL zenodo. 5143773. If you this software, please cite it as below. 40
5281/
Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene Ilyushin. Uncertainty-aware evalua- tion for vision-language models. arXiv preprint arXiv:2402.1 2441:22002.4418., 2024. 26
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In CVPR, 2017. 4
Wojeiech Kryšcinsski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. arXiv preprint 1910v:1910..288440 2019. 3,
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text- to-image models. NeurlPS, 36, 2024. 3
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JMLR.org. 2023. 40
Weixin Liang, James Zou, and Zhou Yu. Beyond user self-reported likert scale ratings: A compar- ison model for automatic dialog evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1363-1374, 2020. 27
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image genera- tion. arXiv preprint 2xXiv:2312./024:: 2023. 27
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint 2441v2240.00/29,, 2024. 8
Rosanne I Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212. 10562, 2022.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Ticyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22139-22144 2024. 45
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016. 4
14

---

Published as a conference paper at ICLR 2025
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.0060 2020. 7,7
Shinri Ohta, Naoki Fukui, and Kuniyoshi L. Sakai. Computational principles of syntax the re- gions specialized for language: integrating theoretical linguistics and functional neuroimaging. Frontiers in Behavioral Neuroscience, 2013. 24
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel.
Teaching clip to count ten. arXiv preprint arXiv:2302. 12066, 2023. 3,4
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0/952 2023. 4
Jordi Pont-Tuset, Jasper Uijlings. Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connect- ing vision and language with localized narratives. In ECCV, 2020. 3,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 7,8 8
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555 05530, 2024. 8
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 1,4
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-3644 2022. 1, 3, 19
Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image immee prompt coherence met- rics with (2iscorescore (ts2). arXiv preprint arXiv:2404.044 04.5514 2024.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. Gen- crating semantically precise scene graphs from textual descriptions for improved image retrieval. In Workshop on Vision and Language (VL.5), Lisbon, Portugal, September 2015. Association for Computational Linguistics. 24
Robyn Speer. Python library: rspeer/wordfree:: v3.0, September 2022. URL httpsss/////oioorrg 10.5281 / 71 199437. 20
J Ridley Stroop. Studies of interference in serial verbal reactions. Journal of experimental psychol- ogy, 18(6):643, 1935. 23
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 1r3.v 2023. 40
Yuxiang Tuo, Wangmeng Xiang, Jun- Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multilin- gual visual text generation and editing. arXiv preprint arXiv:23/1.03000 2023. 3
Tulia Turc and Gaurav Nemade. Midjourney user prompts & generated images (250k), 2023.
Cristina N. Vasconcelos, Abdullah Rashwan, Austin Waters, Trevor Walker, Keyang Xu, Jimmy Yan, Rui Qian, Shixin Luo, Zarana Parekh, Andrew Bunner, Hongliang Fei, Roopal Garg, Mandy Guo, Ivana Kajic, Yeqing Li, Henna Nandwani, Jordi Pont-Tuset, Yasumasa Onoe, Sarah Rosston, Su Wang, Wenlei Zhou, Kevin Swersky, David J. Fleet, Jason M. Baldridge, and Oliver Wang. Greedy growing enables high- nigh-resolution pixel-based diffusion models, 2024. URL https:// arxiv arxiv.org wwwii.oorg/bb///a/e 16759.
15

---

Published as a conference paper at ICLR 2025
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Heman Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022. 10, 45
ZijieJ J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:22100 1102210..489900 14896, 2022. 4
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 200--2555555 2020. 23
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zeutlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pp. 6787-6800, 2021. 10, 45
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment
evaluation. NeurlPS, 36, 2024. 3, 7,7,8, 8, 40
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL fttp:///////eeeeoeoeo////oooooo Be277P3AYC. 40
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprini arXiv:2206. 10789, 2(3):5, 2022b. 1, 3, 19
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- lision-language models behave like lags-of-wordsh and what to do about it? In ICLR, 2022.
Antonia Zapf, Stefanic Castell, Lars Morawietz, and André Karch. Mcasuring inter-rater reliability for nominal data-which coefficients and confidence intervals are appropriate? BMC medical research methodology. 16:1-10, 2016. 5
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In Kamalika Chaudhuri. Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25994-26009, PMLR, 17-23 Jul 2022. URL https: / http///oooednggeeeeeeee mlr. press / /v1 62/ html. 40
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training arXiv preprint arXiv:2303.15344 2023, 40
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang. A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to- image fidelity metrics. arXiv preprint 0rXiv:23/2.023388 2023. 3,3
16


Reviews:
[{"review_id": "cCfO8udssz", "reviewer": "Reviewer_kjYy", "summary": "This work introduces a comprehensive evaluation suite with over 100,000 annotations and four human annotation templates to assess T2I model capabilities across diverse conditions. Key contributions include a curated prompt set (Gecko2K), a statistical method for comparing models, and a framework for systematic metric evaluation across three tasks. The results suggest that metrics vary in effectiveness across settings, and to overcome these issues the authors propose an interpretable auto-eval metric that consistently correlates better with human ratings.", "questions": "- Since the proposed metric depends heavily on the accuracy of the NLI model to detect hallucinations, how do you address potential errors from the NLI model that could impact the reliability of your metric? Have you considered any methods to mitigate this dependency?\n- The alignment metric in Equation 2 assumes that the VQA model’s confidence, as measured by NLL scores, is accurate. How do you account for cases where the VQA model might be overly confident in incorrect answers or under-confident incorrect ones?\n- Given that low NLL scores could indicate either true alignment or over-confidence in incorrect answers, and high NLL scores could indicate either misalignment or under-confidence incorrect answers, have you considered additional adjustments or alternative metrics to address this bias?\n- Since the proposed metric doesn’t currently account for potential biases in the VQA model’s confidence, how might this impact the interpretability and reliability of alignment scores? Would adding a calibration step improve the metric’s robustness?\n\nPost-rebuttal: After reviewing the authors' responses, I am increasing my score from 5 to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Proper comparison with previous works\n- Through evaluation of the proposed metric and the dataset\n- The authors have proposed a new benchmark dataset to highlight the issue of evaluation across multiple templates\n- Furthermore, the authors propose a new metric for evaluating QA/VQA models\n- The paper is well-written and easy to follow", "weaknesses": "- My primary concern is with the validity of the metric proposed:\n  - A central aspect of the proposed metric is its use of NLI to detect hallucinations, making the metric reliant on the accuracy of the NLI model. Errors in the NLI model could undermine confidence in the reliability of this metric.\n  - The proposed alignment metric in Equation 2 relies entirely on the negative log-likelihood (NLL) scores generated by the visual question-answering (VQA) model to evaluate alignment. NLL scores measure how well the model predicts a correct answer given an image and a question prompt; lower scores indicate higher confidence in the answer. However, this approach assumes that the VQA model’s confidence is accurate and does not account for situations where the model might be overly confident (assigning low NLL scores even when it’s wrong) or under-confident (assigning high NLL scores even when it’s correct).\nIf the model is over-confident, it might produce low NLL scores for incorrect or poorly aligned answers, leading to artificially high alignment scores. Conversely, if the model is under-confident, it might assign high NLL scores even for well-aligned responses, causing the alignment score to underestimate the actual alignment. Because Equation 2 relies solely on these NLL scores without adjustments, it doesn’t account for these potential biases in model confidence, which could compromise the accuracy and reliability of the alignment metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aj1lVPuYM3", "reviewer": "Reviewer_GY7M", "summary": "- Automatic evals are an effective alternative to human judgement to evaluate text-to-image (T2I) models. \n- In this work authors perform a rigorous analysis on evaluation prompt sets, metrics and templates to derive reliable conclusions.\n- Authors identify that different conclusions can be drawn when the prompt sets or evaluation templates are changed. \n- This work introduces a new prompt set Gecko2K, a statistically grounded method to compare T2I models and a new framework to systematically evaluate metrics under model ordering, pair-wise instance scoring and point-wise instance scoring.\n- Based on their analysis, authors propose a new automatic evaluation method that is highly correlated with human judgement across different human templates and evaluation settings.", "questions": "- The differentiation between pairwise instance scoring and model ordering is not clear. Authors should spend some time discussing the differences. Concretely, from L402-403, if a metric is a \"good indicator on side-by-side comparison\" (pair-wise instance scoring) then wouldn't it automatically be a good one to predict the \"model ordering\"? Do you mean pairwise instance scoring gives a partial ordering but model ordering should give a total order between models? How would a metric give a total order without comparing all models exhaustively? Please elaborate on the differences to highlight the importance of \"model ordering\" when a metric is already a good \"pairwise instance\" scorer.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The authors conduct a rigorous study on the methodologies used for evaluating the text faithfulness of T2I models.\n- The insights and takeaways from this paper are highly useful to advance research in both modeling and evaluating generative models.\n- The experiments are very exhaustive and support all the claims made in the paper.\n- The auto-eval method developed to increase coverage, remove hallucinations and normalize the VLM scores is novel and effective.\n- A lot of thought and analysis has gone into designing the Gecko-2k reliable prompt set and future research in T2I can benefit from this work.\n- Authors demonstrate that this method is not just limited to T2I models but can be easily extended to any modality generation models.", "weaknesses": "- The paper does not read well and requires multiple passes to properly assimilate. \n- The terminology used is not intuitive and have not been used in prior literature making it harder to put them in context. Authors mention several terms like \"skills\", \"reliable prompts\" without properly introducing them creating confusion at several places. I would recommend authors first introduce the terms (what are skills? Maybe introduce them in the introduction. \nProviding an example: a) The term \"skills\" was first introduced in L044 and the readers are directed to Table1 for more details. It has only become clear what those are after reading Table 8 (in the supplementary material). The column names in Table 8 are also different (Prompt Category and Subcategory and not skills and subskills). After finishing the introduction, the reader is still unsure what skills are. I have no issues with the term \"skills\" but authors can clarify what skills and subskills are in the first 2-3 paragraphs of the introduction(where the term was first introduced). Maybe L190-194 can be copied to the introduction so the readers know what to expect? \n\n- A few suggestions, Fig. 2 is very hard to understand. Is there another way to convey the same information which is easier to read? There are too many rows making it hard to parse. Can you break the prompt sets to make it less busy? All the reliable prompt sets can be put in a separate figure and moved to the supplementary. From my understanding, the reliable prompts help reduce rater disagreement and not central to the contribution of the work. Also the color scheme was used for relations (red: less than, green: greater than and purple: roughly equal). Can the color scheme instead be used to show disagreement between templates i.e. to convey the information in L284-292?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mhukfe1oBA", "reviewer": "Reviewer_dzMN", "summary": "The paper introduces a comprehensive benchmark for text-to-image evaluation. The main contribution is the Gecko dataset: which consists of 2k prompts (1k from existing DSG, and 1k fresh prompts), along with images from 4 T2I models (SD1.5, SDXL, Imagen Vermeer, and Muse). The key insight in the paper is that annotations from different formats (i.e likert scale/absolute rating, pairwise comparisons and DSG type scoring) can end up with different conclusions, and to that end, the paper suggests model ordering evaluations using all these metrics. Finally, the paper also presents Gecko, a evaluation metric based on Q/A with some fixes for improved Q/A generation, which seems to be a better alternative for evaluating T2I models.", "questions": "I would like to confirm that I understand correctly: In Fig. 2 an \"=\" in a grid cell means that for the considered evaluation protocol (e.g Likert, SxS), the difference in the performances of the model are not statistically significant (even if the absolute score is different?)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is quite rigorous in its methodology and seems to comprehensively validate several evaluation methodologies for T2I. In general, this might be one of the most statistically sound evaluations of T2I models and has a lot of insights for future work on this topic. \nFurther, the data provided in the paper (assuming public release) would also be useful to evaluate newer reward models/metrics in the future.", "weaknesses": "The biggest drawback/weakness that I can see in this paper is that it predominantly evaluates only 4 T2I models on 1/2k prompts (excluding the results with the gecko metric later on). While this already may be an improvement over prior work, I am curious as to would there be benefits to scaling the prompts 10x? Or are the results/conclusions at this level itself sufficient and there would be mostly diminishing returns from increasing the scale of the evaluations?\n\nFrom the paper, it's not immediately clear whether all the data from the Gecko evaluation shall be released publicly. While the findings of the paper are valuable in itself, the contributions of the paper are significantly determined by the public release (or lack thereof) of the data. \n\n\nThe authors may also want to acknowledge a concurrent work on similar lines[1]\n\n\n[1] Saxon et al. \"Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)\", NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)