SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation
Authors: Shrikant Kendre
Austin Xu
Honglu Zhou
Michael S Ryoo
Shafiq Joty
Juan Carlos Niebles
Venue: Accepted by TMLR ()

Abstract: Traditional evaluation metrics for textual and visual question answering—like ROUGE, METEOR, and Exact Match (EM)—focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation.

Published in Transactions on Machine Learning Research (01/2026)
SMILE: A Composite Lexical-Semantic Metric for Question- Answering Evaluation
Shrikant Kendre*
shendre@@sdeesfove.com
Salesforce AT Research
Austin Xu" Salesforce AI Research
austim.snGeeddesjore.coom
Honglu Zhou Salesforce Research
tonglu..hon@@aleefoee.com
Michael S. Ryoo Salesforce Research
maryoo@salesooree.com
Shafiq Joty! Salesforce AT Research
sjoty@salesforee.co
Juan Carlos Niebles
jniebles@salesforee.comm
Salesforce AT Research
Reviewed on OpenReview: https: /ttps://openiee opwnr/view.eeeee.. net/ formm id- <dd Inpovuuu
Abstract
Traditional evaluation metrics for textual and visual question answering, like ROUGE, ME- TEOR, and Exact Match (EM), focus hcavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While mea- sures like BERTScore and MoverScore leverage contextual embeddings to address this lim- itation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive eval- uation. Extensive benchmarks across text, image, and vidco QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation. Code and evaluation scripts are available qna-evanaavallll
at http:p//gghhhhooooo//ssfeeeeeter
1 Introduction
Question answering (QA) is an essential task used to measure the progress of language-based models. Across text, image, and video domains, the primary measure of model performance on QA benchmarks is accuracy, which typically computed via ezact (or easy) match (EM): A model response is deemed correct the ground-truth answer, typically annotated by humans, exactly matches (or can be found within) the model response. As recent models have grown be more capable language generators, model answers have grown
"Equal contribution.
Equal senior contribution.

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
x
Low-latency, high-performing
evaluation metrics
Pearson correlation w/ human annotations
0.6
x
0.4
ROUGE-L1
METEOR
BERTScore
SBERT
GPT-4o
GPT-3.5
SMILE (ours)
0.0
0
50
100
150
200
250
300
Inference time (ms/sample)
Figure SMILE offers high-performance, low-latency QA evaluation, breaking the trade-off be- tween cost and performance. Performance averaged across human-annotated samples from benchmarks for natural language (x), image (4), and video (o) domains.
more nuanced, making EM overly stringent (Wang et al. 2023a). A reasonable atlempl mitigale these issues is to employ N-gram based metrics typically used for text generation evaluation, such as ROUGE (Lin 2004h or METEOR (Banerjee & Lavie 2005), or embedding- metrics like BERTScore (Zhang et al.
2019), MoverScore (Zhao et al.) 2019), and BLEURT (Sellam et al. 2020), to assess similarity between
the predicted response and the ground-truth (Rajpurkar et al., 2016 Bajaj et al. 2016: Dunn et al. 2017 Kočiský et al. 2018 Yang et al. 2018a). While such metrics capture high-level similarity between the model response and ground-truth, they may miss fine ana-ggrinnd details crucial to answer correctness (c.g., "The cat
is on chair" vs. "The cat is under a chair") that result in lower correlation with human judgments MMañas
ct al. 2024
Concurrently, due to their strong language comprehension abilities, large language models (LLMs) have been deployed as automatic evaluators for text generation. This approach, broadly known as LLM-as-judge functions by either prompting more capable LLMs, like GPT-4o, or finctuning smaller LLMs specifically for evaluation. LLM-as-judge is appealing as LLMs can adapt to different evaluation criteria and generate explanations. Consequently, recent. methods and benchmarks (Jacovi et al. 2025 Wang et al. 2024a now employ judges as evaluators in QA settings.
However, using judge models for evaluation increases costs. For practitioners and developers with limited re- sources, repeatedly querying pay-per-use APT models to evaluate large datasets (5K+ samples) or dedicating limited compute to hosting an evaluation server can be impractical for rapid development, which may drive them to use lesser but faster metrics. Beyond resource demands, generative evaluators also exhibit relatively high latency (see Figure [ and are susceptible to hallucinations, as we qualitatively show Section 3
This work revisits embedding-based approaches for automatic QA evaluation and introduces Semantic Metric Integrating Lexical Exactness (SMILE), lightweight yet high-performing framework for grading QA tasks. SMILE aims to retain the efficiency of embedding based evaluators while addressing their limitations, such lack of fine-grained response understanding. To do so, SMILE compriscs two subscores: A semantic subscore to assess overall response content, and a keyword subscore to reward lexical exact Overall, SMILE offers a best-of-botte both-worlds evaluation solution: As Figure [ shows, it correlates with human annotators as strongly as GPT-4o. Additionally, like other embedding- based metrics, SMILE core components can be precomputed for fast lookup, resulting in 9x speedup compared to LLM-as-judge API queries. SMILE's lightweight. design allows it to run CPU during evaluation, requiring minimal GPU VRAM to perform
2

---

Published in Transactions on Machine Learning Research (01/2026)
one time evaluation dataset preprocessing step. Additionally, SMILE offers interpretability through its composite structure with two subscores. While SMILE can be applied in many settings, our study focuses on factoid QA tasks. Our contributions are:
(1) We revisit the promise of embedding- based automatic evaluation metrics for QA tasks and propose SMILE, which utilizes both sentence and keyword level similarity scores to evaluate based on holistic and fine-grained content.
(2) We construct 225-sample human annotated test set from nine (text, image, video) QA datasets, labeled by 4 domain experts. This set is used to benchmark SMILE and other metrics based on their correlation with human judgments.
(3) We demonstrate that SMILE can serve as a lightweight drop-in replacement for more powerful LLM-as- judge models across modalities.
(4) Extensive ablation studies demonstrate the necessity of each of SMILE's components.
In all, our experiments demonstrate that SMILE is a lightweight yet higl-performing automatic eval- uation metric for QA settings. Code and evaluation scripts are available at https://github...
2 Background and related work
This work addresses three QA modalities: Natural language QA (NLQA), visual QA (VQA), and video QA (VidQA). Across these, a model ʃ receives an input (g,c) and generates a textual answer = [(;; The input consists of a question and context c, where e is text for NLQA, an image for VQA, and a video for VidQA. The task of the model is to produce a natural language answer y to the question q basod on the given context c.
For model evaluation, adopt reference-based protocol, assuming human-annotated ground-truth answer y" is available for each input (q, c) Given a model response y. the goal is to determine its correctness. Specifically, we aim to design an evaluator j that produces an evaluation score 8 = jiy,,'''' based on y and y"- This source-free setup, where evaluation occurs without access to (q, aligns with the original exact match (EM) evaluation setup. Source- evaluation may represent an easier evaluation setting, as prior work indicates LLM-based evaluators struggle when context is included (Xu et al. 2025).
For practitioners prioritizing accuracy, the evaluation score s be binary correct/ /incore/nnccreet label. For detailed failure analysis, a finer-grained score (e.g. 0-5 scale) may be preferred. Regardless of the format, the score should be convertible to binary label, typically via straightforward threshold (e.g., scores < 3
as incorrect, scores 2 4 as correct for a 0-5 scale) (Maaz et al.) 2024 Wang et al. 2024a) We now review cxisting metrics and evaluators.
Text generation metrics. QA benchmarks have typically used EM or ROUGE [Lin 2004 to assess model
outputs, e.g., (Yang et al. 2018b) Rajpurkar al. 2016 Dunn al.) 2017 Fan et al. 2019). As model responses grew more nuanced, n-gram metrics such BLEU
(Papineni al. 2002 and METEOR (Banerjec) 2016). Despite better correlation with human annotations than EM, n-gram metrics have been shown to be insufficient for modern QA tasks (Chen et al. 2019h.
& Lavie 2005 were adopted in QA settings (Bajaj et al.
Ribeedding-aseed metrics. Embedding models, like BERT (Devlin et al. 2019 and finetuned variants, o.g.; [Reimers & Gurevych, 2019 Gao et al., 2021), are trained to mcasure semantic similarity. As such, embedding- based metrics are natural step in overcoming the limitations of EM and n-gram-based met- rics (Chen et al.] 2019), with notable methods like BERTScore (Zhang et al. 2019). BARRTScor (Yuan) et al. 2021), BLEURT (Selamm et al.] 2020) being used as evaluators in benchmarks (Ao et al. 2024 Re-
cent work (Bulian et al. 2022 Risch et al. 2021 Lee et al. 2020 developed metrics specifically ly for the QA
setting.
LLM-as-judge evaluators. The LLM-as-judge paradigm initially utilized frontier LLMs for automatic
evaluation (Wang al.] [202bb] Liu et al., 2023c in prompted evaluators hold were soon identified
Fu al.) 2024 Chiang & Lee 2023). However, biases
(Panickssery et al. 2024 Wang et al. 2023c Park et al.
3

---

Published in Transactions on Machine Learning Research (01/2026)
TGIF
DocVQA
TextVQA
HotpotQA
WEM Retoaad Plas
motrument and the soniigee estart it coneciorr ts p sxttumnt sutting and has cnntenderee to a checsay boor vilegg due to is tooddtoon rnder apprrssaion
driigreeter
the Esaoo Hills eseding 330mm 1977 in showe bengrec in Bupkinghum mt the Kahriday tatale sox al logie of didhury sini Gandeeseen Hertforten monoged andloweed National Thua. E--]
Question: what goes of the track while attermpting to take a corner at speed? ? Answer:
Question: What the "Convension Rane' for Eventt?
Question: Is this a grocery store? Answer: yes
Question: What was the name the hill that was the soat of an estate uphill Aldibury
8nswer:8
Synthotic Answer: incked grocery store
Synthetic Answer The truck tends to go off the track while atterppting to
Synthetic Answer: The conversion rate
Synthetic Answver The Chiltern Hills served as the seat an estate uphil in Aldbury
for an event is 8.
Model output: Yes, the mage shows a grocery store oustomers ety produce
take corner speed.
output:
Model output: The truck in the video is attempting to taker a comer at ligh speed and loses control cousing to spin cut of track and crash into the dir.. This issurs doametiic scene with dind and
Column
Average
Answer: Chiltern Hills
[-]
Model output: Bridgewater monument
Conversion Rote
col 33.18
provides range food houschold
"Conversion Rate" "Bvent""
ieems for custormers purchase. presence of customer su suggests thar
33.28%, This besee on cal [..]
Human: 4o
SMILE
Human:
4PT
SMILE:0
Human:
aBERT
SMILE:
Human:
SMILE:
Judge models may hallucinate even on a simple example
Judiee models may hatlacinate when pusput is verbose
Eribediting based metric miss short resporses in long outputs
Fribedding metrics often overrate relevant but wrong answvers
SMILE subscores:
SMILE subscores:
SMILE subscores:
SMILE subscones:
sutscoreu,
subscore 8,1
0.721
2
Keyword subssore 5
Seyword subscore 4
Keyword
Reyword subscore fy
SMBLE Sc0rD Pouus
0.905
SMILE score Yours"
0.505
SMILE scort Suar
0,850
SMILE score houra
SMILE aineed scone:
SMILE bimned scuree
SMILE birned woone
SMILE bimned scuree
Max sim words. truck
Max slmwwords 3318
Max sim words yes
Max sim words, bricgewater monument
Figure 2: Example failure cases for existing methods. Columns 1-2 illustrate LLM-as-judge fail- ures: hallucination even on simple verification (column 1), and incorrect yet concrete responses (column 2). Columns 3-4 illustrate embedding- based model failures with lengthy (column 3) and relevant but incorrect (column 4) responses. SMILE correctly identifies each case and exposes expanded, interpretable subscores (bottom): semantic alignment (sa), keyword/faithffunes faithfulnesss (s1), overall score (SSMILE), binned decision score, and the top max-similarity words, providing transparent evidence for why response is accepted or rejected. Scores are shown on a 0/1 scale for comparison. [...] denotes omitted content for brevity.
2024h. leading to the finetuning of smaller models for evaluation (Kim et al. 2024 Li et al. 2023a Zheng
et al. 2024 Wang et al., 2023d Shiwen et al. 2024). Recent efforts focus on training for diverse eval evaluation
protocols (Vu et al. 2024 Wang et al.] 2024b), such as pairwise, single rating, and binary classification. Applying ALM-as-judges specifically to QA grading is a recent development {Mañas et al. 2024), with many
benchmarks (Krishna et al. 2024 Jacovi et al. 2025 and studies (Maaz et al. 2024
Liu et al. 2023b
employing API models.
3 When do existing evaluation methods and metrics fail?
Before introducing SMILE, we present a qualitative case study highlighting the failure modes of LLM-as- judge and embedding- metrics in QA cvaluation. Our analysis of 225 human-cvaluated model responses reveals common failure point: verbose or generic model outputs, consistent with Mañas et al. (2024); Luo et al. (2021). By "verbose," we refer to predictions significantly longer than the ground-truth answer, often containing tangential information, hedging language, or hallucinated content that obscures the core answer cases where keyword matching effectively identifies missing critical terms while semantic similarity alone scores highly due to topical relevance.
Surprisingly, we also find that using a powerful LLM like GPT-4o does not guarantee accurate evaluations (Yan et al. 2025 Thakur et al. 2025). We highlight two representative failure modes in Figure 2 (columns 1- 2). GPT-4o was prompted to generate binary accuracy label, as well 1-5 score (see Section for prompt).
4

---

Published in Transactions on Machine Learning Research (01/2026)
For ease of comparison, we convert SMILE scores to binary accuracy indicator. A primary concern with using LLMs as judges is hallucinations. Figure 2 (column 1) shows that even for relatively simple samples, GPT-40 may hallucinate an incorrect label. Figure 2 (column 2) is an example of concreteness bias, a known judge model bias (Park et al.) 2024). Here, GPT-4o is tricked by a response that includes concrete artifacts, like the table the model generated, even it. incorrect response.
Seeking an efficient evaluator, we also analyzed failure modes of embedding based approaches. Semantic similarity metrics like BERTScore exhibited well-kmown limitations (Zhang et al. 2019). Figure 2 (columns 3-4) highlights two key examples: column 3 shows how overly verbose model responses can easily misled semantic similarity metrics, as much of the output is irrelevant to the simple "yes" response. This can be viewed this as distributional misaligrement (Agrawal et al., 2022): increasingly high-quality model outputs are often lengthier, contrasting with the typical short answers in factoid QA benchmarks. Conversely, when model responses are short but semantically relevant, these metrics are prone to false positives, as illustrated in Figure 2 (column 4). The limitations of embedding- based and LLM-as-judge methods motivated SMILE-a hallucination-ffre evaluation metric presented next.
4 The SMILE metric
Our analysis in Section 3 pinpointed two critical limitations of embedding- based approaches: (1) distribu- tional gap between verbose model responses and concise ground-truth answers, and (2) a lack of fine-gruined understanding due to their semantic focus. SMILE directly addresses these issues with two key innovations: (1) Synthelic answer gemeration to bridge the stylistic distribution gap, and (2) targeted sub-scores capturing both semantic and lexical similarity between model responses and ground-trutle
Bridging the stylistic distribution gap. As shown in Section 3 assessing directly based on the ground truth y", which is typically short for short-form (factoid) QA (e.g., a single word short phrase), may be sub-optimal, as model responses tend to be more verbose. Motivated by past work that have used LLMs to perform other kinds of zero-shot distribution alignment (Gao et al. 2023 Xu et al. 2024), we utilize an LLM to generate synthetic model response from the ground-truth. Our key insight is that for short-form QA tasks, a lightweight model (e.g., 3B parameter) can be deployed as a synthetic answer generator g. Specifically, the generator g takes as input the original question and ground truth answer y" and outputs a synthetic answer ğ = =gy"" 9), which aligns stylistically with model responses, but reflects the ground- truth answer content. As concrete example from our evaluation setup, for input question "What is the Conversion Rate for Event?" and ground-truth "8", generated synthetic answer is "The conversion rate of event is 8". We emphasize that synthelic answer generalion is independent of the model being evaluated and is performed only once, prior to test-time, per evaluation set. As a result, synthetic answers may be stored and used for any subsequent cvaluations.
Integrating semantic and lexical similarity. The core idea of SMILE is to measure both semantic and lezical similarity between the model response and the ground-truth using an embedding model We caleulate a semantic similarity score, which we denote Sa, as
(v e))) sim(s e()),,
(1)
where, sim(x, v) (1+ /=/2>>//1 which is a linearly transformed cosine similarity that lies within an
interpretable interval of [0, 1]. As we show in Section 5.4 generating synthetic answers bridges the stylistic distribution gap between ground-truth answers and model responses enough to make semantic similarity meaningful. However, Section 3 shows that this semantic similarity score alone is insufficient to capture the nuances of evaluation. As a result, we additionally compute a lexical similarity score, which we denote
s, E [0, 1]. as
1 (EM(y, ( w²) sr v() -
+ max (sim (())
(2)
where EM(y.'") E 10 score between the prediction y and ground-truth y² and N, denotes the i-th n-gram of response y. In computing st, we take advantage of the fact that y" typically a short phrase to compute two complementary scores. The easy match sub-score F.M serves a preliminary check for lexical
5

---

Published in Transactions on Machine Learning Research (01/2026)
Embedding
Friendly fares
Exact match KW score (1)
Keyword
Embedding KW score (1)
(1)
Gold answer
?
What does this jet advertise having?
Friendly low fares
Evaluated
says
the jet body friendly low fares
SMILE
(0.98)
Synthetic
Cosine similarity
answer
genarator
The advertisement the
says "Friendly low fares"
The advertisement on the jet
Semantic score (0.96)
says "Friendly
Synthesic
(1) Preprocessing: Distributional
answer
alignment SLM
(2) Evaluation: Computing SMILE with semantic keyword scores
Figure Overview of SMILE. SMILE evaluates QA outputs in two steps using a synthetic answer generator and an embedding model. (1) One-time preprocessing generates stylistically aligned synthetic answers using a small language model (SLM). (2) SMILE computes two sub-scores: semantic score based the syuthetic answer, and word subscore that combines exact match with embedding comparisons of model response N-grams to the gold answer. This subscoring balances semantic and lexical evaluation.
answer correctness. However, as noted in prior work Waan et al. 2023a Luo et al. 2021), string matching may be too stringent for synonym- like answers (e.g., "cat" vs. "kitten") As result, we loosen the necessity for string matches via the maximum n-gram embedding similarity score, which serves as a continuous-valued
mcasure of lexical exactness.
Evaluation with SMILE. With our semantic and lexical scores compuled, we can now compute the SMILE score, denoted SSMILE E 1];
ASMILE (V. = (w sy 0: + (1 w) se(v.v':e))
where 10, is some user-specified weight to balance the two subscores. This weighting mechanism allows practitioners to express their preferences: Those who are more inclined towards exact match may place higher weight on 8t, whereas those who value higher responses whose meaning is closest with the ground-truth may place a higher weight on 85.
4.1 Optimizations for test-time speed-up
SMILE offers significant speed advantage over LLM-as-judge methods, as extracting representations from lightweight embedding models like BERT (Devlin et al., 2019) is far faster than generating natural lan- guage outputs. Like other embedding- based metrics, this speed advantage can further enhanced by pre-compuling and storing representations for synthetic answers e(ÿ) and keyword representations e(y") be- fore evaluation. By storing e(ğ) and e(g"), only the model response representations e(y) and e(Ni(y)) need to be calculated during test time. Notably, synthetic answer generation is one-time preprocessing cost per dataset; once generated, synthetic answers are reused across all model evaluations, amorlizing this cost to negligible overhead when benchmarking multiple models.
4.2 Interpretability of SMILE scores
SMILE's semantic and lexical subscores provide practitioners with more interpretable and actionable feedback than other metrics. These subscores enable monitoring of model performance along two complementary axes: semantic conlent, a holistic measure of response relevance, and lexical exactness, a finer-grained measure of response quality. Importantly, SMILE allows evaluation not only at the instance-level but also at the population-level. Aggrogating 8, and 8q across all test samples revcals a model's general strengths and weaknesses. This contrasts with LLM-as-judge methods, which offer more specific instance-level natural language feedback, making it hard to extract overall insights. See examples in Section C

---

Published in Transactions on Machine Learning Research (01/2026)
5 Experiments and results
Benchmarks and generator models. We assessed SMILE on established benchmarks across three do- mains: NLQA, VQA, and VidQA. To ensure diverse evaluation, we included three benchmarks per domain:
MRQA (Fisch et al.] 2019) HotpotQA (Yang et al. 2018b) and MuSiQue (Trivedi et al. 2022) for NLQA,
VQA (Singh et al. 2019), Doc (Mathew et al. 2020), and POPE (Li et al. 2023b) for VQA, and TGIF (Jang el al. 2017], MSVD (Xu et al. 2017], and MSRVTT (Xu et al. 2016] for VidQA. For HotpotQA and MuSiQue, we used the standardized setup from ContextualBench (Nguyen et al. 2024b. We generated responses using the following models for each domain: GPT-4o (Hurst et al., 2024] for NLQA,
LL-V5-1.5 for VQA 7B (Liu et al., 2023a|b b) and Qwen2.5-VI. 3B Instruct (Bai et al.
2025 for VidQA.
These models were selected for their strong capabilities in producing high-quality textual responses, forming the basis of our analysis. We also evaluate on QA-Eval (Wang et al. 2023a), large-scale
NLQA datasct (~10k samples) based on Natural Question (NQ) and TriviaQA (TQ), with responses from GPT-3.5 and GPT-4o. This setup enables robust. comparison of SMILE against. LLM judges al scale.
Data annotation efforts. To evaluate QA metrics, we assessed their aligmment with human judgments using a golden evaluation set. We constructed this set by sampling model outputs from the nine benchmarks (three per domain), randomly selecting 25 input-oulput pairs per dataset for annotation. Four annotators (authors of the paper with native level English) evaluated the generated outputs based on a predefined rubric, considering correctness, relevance, and clarity. To ensure unbiased evaluation, all annotations were conducted in a double-blinded manner annotators were not informed which model generated each answer nor which dataset each question originated from, and questions and outputs were shuffled randomly across all benchmarks. Given potential ambiguity, annotators used 3 point scale: clearly incorrect, unclear, clearly correct. To check annotation quality, we calculated KrippendorlI's alpha. (Krippendorf]] 2011). achieving a score of 0.71, indicating substantial inter-annotator agreement. This high agreement confirms the reliability of our annotations, so we procoed with it as the basis of our evaluation.
Baselines and metrics. We compared SMILE with established metrics, including traditional NLP mea- sures: ROUGE-L, METEOR and Exact and Easy Match; alongside embedding- based similarity metrics: BERTScore (with Roberta-largo) and sBERT cosine similarity Following Maaz al. (2024), we also em- ployed GPT-4o and GPT-3.5-Turbo as judge models, prompting them for 0-5 score and binary yes/no no prediction. For all baselines, we provide detailed implementation details in Section A. including judge model prompts.
SMILE implementation. We choose Llama-3.22 3.2-3B Instructt as our synthetic answer generator and ember-V as our embedding model e. This combination is computationally lightweight: the 335M parameter ember-vl can run inference on a CPU, and generating responses with the 3B Llama model requires < 10GB of VRAM. Furthermore, our ablation study (Section 5.4) shows that larger models offer only marginal performance improvements, highlighting SMILE's inherent lightweight, nature. SMILE scores, similar to GPT-4o, are discretized into six bins (0-5), with scores 2 considered correct. The N-gram value is dynamically set based ground truth answer length, and the parameter fixed to 0.5.
5.1 Main experimental results
Using our golden evaluation set, we compare SMILE against existing baseline metrics. To holistically assess evaluators-human agreement, we computed Pearson correlation and Kendall's Tau-b from Human Accuracy. Pearson correlation and Kendall's Tau measure agreement with human amnotations on the instance level, ranging from -1 (perfect disagreement) to +1 (perfect agreement). Kendall's Tau-b, focuses on ranking consistency and accounts for ties in the data. We report results in two evaluation settings to separate formatting effects from motric behavior: Orig uses the original ground-truth references (the common practical setup), and Syn uses SMILE's synthetic canonicalized references.
Pearson correlation results are presented in Table D SMILE achieves the best overall correlation among all evaluation metrics, consistently outpeffrming traditional NLG metrics (ROUGE-L, METEOR, BERTScore,
7

---

Published in Transactions on Machine Learning Research (01/2026)
Metric
Ref.
Video QA: Qwen2.5
Visual QA:
llava 1.5 7B
Language QA:
GPT-4o
TGIF
MSVD
MSRVTT
Tex
DocVQA
POPE
MRQA HotposQA
MUSIQUE
Overall
Orig
0.099
0.109
0.147
0.118
Exact Match
Syn
nan
nan
nan
nan
nan
0.099
nan
0.109
0.147
0.118
Orig
0.793
0.379
0.290
0.795
0.375
0.451
0.676
0.657
0.890
0.590
Easy Match
Syn
nan
nan
nan
nan
nan
0.143
0.129
0.185
0.185
0.161
Orig 0.603
0.442
0.217
0.596
0.705
0.001
0.361
0.603
0.445
0.441
ROUGE-L
Syn
0.625
0.372
-0.340
0.081
0.228
0.518
0.284
0.042
-0.053
0.283
Orig 0.663
0.488
0.438
0.667
0.747
0.086
0.528
0.664
0.611
0.544
METEOR
Syn
0.617
0.451
-0.311
0.264
0.314
0.414
0.283
0.051
-0.046
0.306
Orig 0.421
0.331
0.110
0.398
0.677
0.164
0.310
0.620
0,411
0.382
BERTScore
Syn
0.674
0.497
-0.247
0.206
0.345
0.434
0.223
0.094
0.194
0.324
Orig 0.472
0.592
0.315
0.602
0.852
-0.164
0.352
0.664
0.363
0.486
SBERT
Syn
0.806
0.579
0.020
0.419
0.426
0.541
0.037
0.203
0.195
0.358
Orig -0.101
0.256
-0.004
0.483
0.705
-0.488
0.354
0.558
0.454
0.378
BLEURT
Syn
0.692
0.517
-0.093
0.517
0.504
0.366
0.174
0.158
0.136
0.351
Orig 0.613
0.390
0.278
0.338
0.671
0.053
0.242
0.379
0.231
0.355
Moverscore
Syn
0.629
0.388
-0.331
0.296
0.426
0.414
0.190
0.001
0.074
0.305
Orig 0.825
0.609
0.350
0.663
0.803
0.422
0.810
0.668
0.525
0.631
GPT-3.5
Syn
0.742
0.845
0.059
0.584
0.702
0.210
0.389
0.234
0.310
0.453
Orig 0.778
0.687
0.627
0.859
0.761
0.699
0.294
0.678
0.764
0.683
GPT-4o
Syn
0.898
0.766
0.358
0.844
0.904
0.874
0.944
0.774
0.690
0.784
Orig 0.800
0.564
0.552
0.784
0.914
0.469
0.860
0.952
0.950
0.761
SMILE
Syn
0.847
0.607
0.496
0.796
0.909
0.763
0.839
0.902
0.948
0.790
Table 1: Pearson correlation with human judgments (1) across Vidco, Visual, and Language QA. For each metric we report, two settings: Orig (evaluated against original ground-truth references) and Syn (evaluated against SMILE's synthetic references), shown for all datasets and Overall. SMILE achieves the strongest overall correlation across modalitics. NaN values indicate zero variance in metric scores, making correlation undefined.
SBERT, BLEURT, Moverscore) and GPTT-355 by substantial margins. While GPT-4o achieves slightly higher correlation in some multimodal settings (Video QA, Visual QA), SMILE demonstrates superior performance in Language QA and attains the highest overall correlation, indicating agreement with human evaluations.
Kendall's Tau-b results, presented in Table 2 further establish SMILE's superior ranking agreement with human judgments. SMILE achieves the highest Kendall's Tau-b in 5 out of 6 domain categories, surpassing both GPT-4o and GPT-3.5. This dominance in ranking agreement underscores SMILE's exceptional ability to order generated responses in a way that closely mirrors human annotated rankings.
Finally, we evaluate SMILE and GPT-3.5/40 as evaluators on QA-Eval, a human-annotated benchmark containing manually annotated correctness judgments for question-answer pairs. We use two prompting variants: (1) original prompt (based on Maaz et al. (2024)), and (2) extract-style prompt (asks LLM extract short answer first). As shown in Table 3 SMILE consistently outperforms GPT-3.5 and closely matches GPT-4o. Notably, LLMs degrade under the extract prompt, highlighting SMILE's robustness and prompt independence.
Impact of Synthetic Canonicalized References. To analyze the impact of canonicalized references, we compare results using Orig and Syn answers. Across datascts and modalities, traditional metrics stagnate or degrade under Syn, whereas SMILE remains stable and typically strongest. Canonicalization therefore does not "rescue" baseline metries; SMILE's agreement with humans is robust. to the choice of reference.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Metric
Ref.
Video QA: Qwen2.5
Visual QA:
llava 1.5 7B
Language QA:
GPT-4o
TGIF
MSVD
MSRVTT
TextVQA
DocVQA
POPE
MRQA HotpotQA
MUSIQUE
Overall
Exact Match
Orig
nan
0.100
0.109
0.147
0.119
Syn
nan
nan
nan
nan
nan 0.100
nan
0.109
0.147
0.119
Orig
0.765
0.414
0.295
0.773
0.361
0,420
0.676
0.657
0.890
0.583
Easy Match
Syn
nan
nan
nan
nan
nan
0.145
0.129
0.185
0.185
0.161
Orig 0.598
0.509
0.248
0.614
0.696
0.162
0.390
0.496
0.554
0.474
ROUGE-L
Syn
0.591
0.296
-0.264
0.052
0.182
0.381
0.223
0.014
-0.070
0.230
Orig 0.592
0.511
0.446
0.644
0.686
0.161
0.396
0.458
0.582
0.497
METEOR
Syn
0.522
0.339
-0,255
0.261
0.218
0.294
0.222
0.028
-0.046
0.243
Orig 0.347
0.240
0.060
0.322
0.526
0.125
0.289
0.455
0.440
0.312
BERTScore
Syn 0.618
0.411
-0.187
0.131
0.290
0.408
0.171
0.043
0.185
0.272
Orig 0.393
0.505
0.230
0.435
0.662
-0.102
0.289
0.455
0.382
0.384
SBERT
Syn 0.674
0.394
-0.043
0.305
0.336
0.374
0.051
0.142
0.231
0.283
Orig 0.019
0.231
-0.034
0.566
0.526
-0.238
0.323
0.469
0.520
0.325
BLEURT
Syn
0.571
0.419
-0.068
0.435
0.463
0.306
0.170
0.028
0.104
0.285
Orig 0.506
0.342
0.170
0.226
0.426
-0.011
0.341
0.355
0.104
0.276
Moverscore
Syn
0.646
0.300
-0,221
0.226
0.327
0.328
0.187
-0.057
-0.012
0.256
Orig 0.738
0.487
0.291
0.625
0.690
0.441
0.638
0.439
0.570
0.547
GPT-3.5
Syn
0.668
0.759
0.033
0.549
0.642
0.209
0.341
0.261
0.300
0.418
Orig 0.686
0.580
0.575
0.780
0.658
0.676
0.281
0.488
0.767
0.610
GPT-4o
Syn
0.797
0.638
0.272
0.728
0.754
0.660
0.614
0.688
0.589
0.638
Orig 0.765
0.559
0.485
0.730
0.841
0.607
0.805
0.765
0.729
SMILE
Syn
0.773
0.595
0.463
0.652
0.920
0.607
0.805
0.765
0.981
0.729
Table 2: Kendall's Tau-b (f) ranking agreement with human judgments across Video, Visual, and Language QA. For each metric we report two settings: Orig (original ground-truth references) and Syn (SMILE's synthetic references). Across all datasets and Overall SMILE attains the highest ranking agrocment. NaN values indicate zero variance metric scores, making corrclation undefined.
GPT 3.5
GPT-4o
NQ
TQ
TQ
Overall
GPT-3.5, original prompt
0.756
0.849
0.713
0.706
0.756
GPT-4o, original prompt
0.865
0.913
0.815
0.806
0.850
GPT-3.5, extract prompt
0.478
0.572
0.413
0.440
0.476
GPT-4o, extract prompt 0.831
0.898
0.783
0.774
0.821
SMILE
0.829 0.889
0.786
0.760
0.816
Table 3: Pearson Correlation with human judgment on QAEval (~10k human-annotated sam- ples). SMILE shows shows strong agreement with human annotations, outperforming GPT-3.5 and roughly match- ing GPT-4o.
Short-answer examples illustrate the Syn-induced lexical-overlap bias: under Syn, traditional metrics tend to over-score verbose or even contradictory responses because increased surface overlap with the canonicalized reference can mask factual errors. On TGIF, generic non-answers receive higher scores; on POPE, fluent contradictions (c.g., 'no bowl" versus 'contains a bowl') are similarly over-scored. These cases show that canonicalization alone cannot fix metrics that privilege surface overlap and fluency over semantic correctness.

---

Published in Transactions on Machine Learning Research (01/2026)
Video QA: Qwen2.5
Visual QA:
llava 1.5
Language QA:
GPT-4o
TGIF
MSVD
MSRVTT
Tex
DocVQA
POPE
MRQA
HotpotQA
MuSiQue
Overall
GPT-4o
0.705
0.657
0.503
0.436
0.191
0.783
0.920
0.909
0.700
0.645
Exact Match
-0.705
-0.657
-0.503
-0.424
-0.187
-0.782
-0.877
-0.884
-0.681
0.633
Easy Match
-0.025
-0.217
-0.106
-0.056
-0.044
-0.021
-0.083
-0.152
-0.135
0.093
ROUGE-L
-0.705
-0.657
-0.503
-0.413
-0.179
-0.773
-0.648
-0.493
-0.553
0.541
METEOR
-0.705
-0.657
-0.503
-0.405
-0.166
-0.783
-0.592
-0.509
-0.454
0.530
BERTScore
0.294
0.340
0.497
0.562
0.805
0.216
0.008
0.091
0.300
0.354
sBERT
-0.705
-0.657
-0.503
-0.379
-0.145
-0.780
-0.566
-0.399
-0.504
0.515
SMILE
-0.032
-0.241
0.132
0.021
0.104
-0.041
-0.008
-0.016
0.005
0.067
Table 4: Deviation from GPT-4o accuracy across Video, Visual, and Language QA tasks, using complete.
test sets. SMILE exhibits the smallest deviation among evaluators, closcly aligning with GPT-4o.
Metrics
Checkpoint 1 Cost($)
Checkpoint 2
Rank
Rank
Cost(S)
GPT-4o
1
12
2
11.99
GPT--3.-turbo
12
12.00
METEOR
1
2
sBERT
2
SMILE
1
2
Table 5: Chockpoint selection TGIF video QA (Video model): Rank (1=best) and approximate evaluation cost. (USD) per metric. SMILE ranks checkpoints similarly to GPT metrics, but without inference costs. f_" denotes methods without API inference cost.
SMILE mitigates this by combining sentence-level semantic similarity with keyword-level exactness and an optional exact-match component in a weighted aggregation that captures both intent and factual content. Qualitative examples in Figure illlstrate these patterns across TGIF, DocVQA, TextVQA, and HotpotQA
5.2 SMILE as drop-in replacement for GPT-4o
Building SMILE's alignment with human judgment, now demonstrate its capability to supplant GPT- 4o evaluation metric. To do so, compare model accuracy derived from SMILE scores against that from GPT-4o-based evaluation each benchmark's complete test-set. We find that SMILE exhibits the lowest overall deviation among all tested methods, as summarized in Table 4 This compelling result strongly suggests SMILE a reliable and direct alternative to resource-intensive LLM-as-judge approaches like GPT- 4o.
5.3 Cost effective Model selection with SMILE
Selecting optimal chockpoints during ML model training crucial for maximizing performance on down- stream lasks. Traditionally, this selection process relies on not-so reliable metrics like METEOR and SMILE
ROUGE, expensive metrics such LLM-bbsed judge evaluations. In this experiment, to identify the best checkpoint. Specifically, we select two intermediate checkpoints(with similar perfor- mance) from the Video model and evaluate their performance on the TGIF benchmark. The evaluation is conducted using five metrics: GPT-4o, GPT-3.5-turbo, METEOR, sBERT and SMILE.
As per Table 5 our findings demonstrate that SMILE selects the same optimal checkpoint (i.e. checkpoint 1) GPT-40 and GPT-3.5-ta 5.5turbo.. This alignment highlights SMILE's effectiveness, emphasizing its capability to provide reliable checkpoint selection without incurring additional evaluation cost.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Video QA
Visual QA
Language QA
Overall
SMILE
0.650
0.823
0.896
0.790
Component Ablation
w/o semantic scores
0.628
0.775
0.941
0.782
w w/o keyword scores
0.463
0.533
0.249
0.415
w/o synthetic answers
0.639
0.722
0.921
0.761
Model Ablation
SMILE
0.650
0.823
0.896
0.790
Embedding: GTEBB
0.647
0.824
0.947
0.806
Syn. answer: CPT-335-Turbo
0.636
0.802
0.930
0.790
Table 6: Component and model ablations. Performance is assessed by Pearson correlation. Keyword scores are the primary contributor, highlighting the importance of lexical exactness. Embedding model
scaling yields marginal (< 2%) gains.
Frequency
Gold answers
0.2
Model outputs
Synthetic answers
0.1
0.0
0
100
200
300
400
500
Length of responses
Figure 4: Length of gold answers, model outputs, and synthetic answers, across all domains and benchmarks in characters. Synthetic answers more align better with model outputs in terms of output length, enabling belter semantic evaluation.
An advantage of SMILE is its substantial reduction in evaluation costs compared to GPT-based models. GPT-4o and GPT-3.5 cost's around $12 for each checkpoint evaluation TGIF, and the cost increases more checkpoints and evaluation benchmarks
added. In contrast, SMILE has almost extra cost. Therefore, adopting SMILE not only maintains performance accuracy but also significantly lowers monetary overhead, making it highly efficient and scalable solution for checkpoint selection.
5.4 Ablations
This section presents an ablation study of SMILE centered on three koy perspectives: (1) Component analysis, systematically removing steps (synthetic answer generation, semantic similarity score, keyword score) demonstrate their individual importance, (2) Model scaling, examining the impact of using larger models for both synthetic answer generation and embedding, (3) Hyperparameter tuning, analyzing the effect of the weight in SMILE. Results are detailed in Table and Figure
Component analysis. SMILE comprises three key components: (1) semantic similarity, (2) lexical ex- actness, and (3) distribution alignment via a lightweight language model. Table 6 (top) summarizes the contribution these components to SMILE's robust performance. Experiments demonstrate that both key- word and semantic scores are essential. Removing keywork scores significantly reduces Pearson correlation, underscoring the critical role of lexical exactness in QA evaluation. Conversely, relying solely on keyword scores neglects global structure, degrading performance notably in VidQA and VQA. Synthetic answers pro- benefits: Visual QA sees +14% improvement (gold answers often single words
vide domain--ependder while model outputs are verbose), while Language QA sees minimal change since predictions are alroady concise. Figure ! illustrates this effect, showing how synthetic answers bridge the length mismatch between short gold answers and longer model outputs. Combining semantic scores, keyword scores, and synthetic answers yields robust and accurate evaluation across domains.
Model scaling. A key advantage of SMILE is that offers the ability to efficiently run evaluation. Our model choices in Section demonstrate this: SMILE at inference time requires only 355M parameter
11

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
Pearson correlation
0.6
0.4
0.2
NLQA
VidQA
VQA
Overall
0.0
0.2
0.4
0.6
0.8
Weight
Figure Sweep w, which trades off lexical exactness for semantic similarity as w increases. SMILE exhibits relatively stable aggregate performance for S 0.5.
bedding model, and pre-generating synthetic answers requires only 3B generative model. Table (bottom) further establishes SMILE's lightweight nature: increasing model capacity yields minimal performance gains, if at all. Our model ablation focused on the synthetic answer generation model and the embedding model. Using GPT-3.5- tion with human judgments, indicating that effective synthelic onswer generation is achievable with smaller lightweight models. Replacing ember-v1 with the substantially larger GTE-7B (Li et al., 2023cb embedding model resulted in only a marginal performance gain of less than 2%, despite a 20x increase in model size. This indicates that SMILE remains effective even with tightweight embedding models.
instead of clama-3.2-3B- Inssruuttt for synthetic answers yielded comparable correla-
Hyperparameter tuning. SMILE's hyperparameter allows practitioners to preciscly decide the impact of the semantic and keyword subscores. Specifically, as w increases, more importance is given to the semantic subscore. As we show in Figure 5] overall performance is relatively stable for w < 0.5 before smoothly decreasing. This aligns with results from our component ablation study in Table G The keyword subscores alone exhibited relatively strong performance, while the semantic subscore fared worse. However, SMILE hyperparameter choice relatively forgiving, with any choice of w that slightly upweights the weyword subscore likely perform well. For practitioners selecting w: use w=0.0 0.3 for factual/entity QA verbose models (keyword matching critical); -=0.5-0.6 for OCR/document QA binary yes/no questions; w=0.7-00 0.8 for visual QA with contexl-dependent answers; and w=0.5 robust general-purpose default.
6 Conclusion
We introduce SMILE, a light QA evaluation metric that integrates semantic and lexical analysis address the high cost, biases, and inconsistencies of based evaluators. Benchmarking across text, image, and video QA demonstrates SMILE's strong correlation with human judgment, surpassing traditional metrics and LLM judges like GPT-4o, while ablation studies validate the importance of its components. SMILE provides an efficient, interpretable, and robust alternative to costly LLM evaluations, effectively balancing lexical precision with semantic relevance across modalities.
Limitations
Although SMILE offers lightweight, interpretable, and scalable alternative to LLM-based evaluators, it has limitations. (1) SMILE is source-free and does not access context, which may cause it to miss context- dependent errors. (2) The metrie relies on synthetic answers to align ground-truths with model outputs; their quality can affect scoring, especially for lorg-form or open-ended responses. (3) Our evaluation is limited to factoid QA; effectiveness on complex reasoning, multi-hop, or conversational QA remains unexplorod. (4) SMILE's weighting parameter for balancing lexical and semantic components may require tuning for specific tasks domains.
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Aishwarya Agrawal, Ivana Kajić, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, and Aida Nematzadeh. Reassessing evaluation practices in visual question answering: A case study on out-of- distribution generalization. arXiv preprint 1xX1::2205.1191, 2022.
Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. arXiv preprint 2000 19940, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint or orxi::2002. 13923, 2025.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension datasct. arXiv preprint ar arxiv:1611 1.09268, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved corre- lation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation und/or summarization, pp. 65 72, 2005.
Jannis Bulian, Christian Buck, Wojeiech Gajowski, Benjamin Bocrschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. arXiv preprinl arXiv:2202.0766 2022.
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Evaluating question answering eval- uation. In Proceedings of the 2nd workshop machine reading for question answering, pp. 119-124, 2019.
Cheng- Han Chiang and Hung- Yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapler of the association for compulational linguislics: haman language lechnologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint vrXiv:1704.0511 2017.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09. 09100,, 2019.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. Mrga 2019 shared task: Evaluating generalization in reading comprehension. arXiv preprint orXiv:1910. 09753, 2019.
Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556-6576, 2024.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Lânguistics (Volume 1: Long Papers), pp. 1662-1777, 2023.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence emboddings. arXiv preprint arXiv:2104.00 088021 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint 2r 21276 2024.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et al. The facts grounding leaderboard: Benchmarking llms' ability to ground responses to long-form input. orXiv preprint Sxi::2501.032000 2025.
Yunscok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2758 2766, 2017.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungiae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. Xiv preprinl or orXiv:2405.01533 2024.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativega reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317 328, 2018.
Klaus Krippendorff. Computing krippendorff's alpha-reiiability, 2011.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upad- hyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-auggeented generation, 2024. URL bettps://arxiv.orgge wtgggbs/////...... 12941
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Joongbo Shin, and Ky- omin Jung. Kpqa: A metric for generative question answering using keyphrase weights. arXie preprint arXiv:2005.00199 2020.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXa::2100005 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXie preprinl ar 2al::3300.0355 2023b.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXio preprint arXim:2308.0322 2023c.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction luning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2023b.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods Natural Lunguage Processing, pp. 2511 2522, 2023c.
Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, and Chitta Baral. 'just because you are right, doesn't mean i am wrong': Overcoming bottleneck in development and evaluation of open-ended vqa tasks. In Proceedings of the 16th Conference of the European Chapter of the Associalion for Compulational Linguistics: Main Volume, 2021.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (ACL 2024), 2024.
Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. Improving automatic vqa evaluation using large language models. In Proceedings of the A AAAI Conference on Artificial Intelligence, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Docvqa: A dataset for vqa document images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2077.03398, 2020.
Xuan-Phi Nguyen, Shrey Pandit, Semthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zixuan Ke, Silvio Savarese, Caiming Xong, and Shafiq Joty. Sfr-rag: Towards contextually faithful 1mms. arXiv preprint arXiv:2409.0900 2024.
Arjun Panickssery, Samuel Bowman, and Shi Feng. Lim evaluators recognize and favor their own genera- tions. Xiv preprint arXiv:24044 04. 19076, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machinc translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0653) 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of lext. arXiv preprint orXiv:1666: 660.05250, 05250, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint xrx:i:1908 10084, 2019.
Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. Semantic answer similarity for evaluating question answering models. arXiv preprint arXiv:2108. 100:2100000100, 2021.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.0440 2020.
Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. https: //huggingface. thnngiingface..c/kkkywox September 2024. URL httpps://uuggingfacee /ttpss//mmngiagaaac...///yoooo co/kkwoork.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa. models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8322 8326, 2019.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating aligmment and vulnerabilities in llms-as-judges. In Proceedings of the Fourth Workshop on Generation, Evalualion and Metrics (GEM)), pp. 404 430, 2025.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop ques- tions via single-hop question composition. Transactions of the Association for Computational Linguistics. 10:539 554, 2022.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foun- dational autoraters: Taming large language models for better automatic evaluation. arXiv preprinl arXiv:2407.10817, 2024.
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-qa evaluation. Advances in Neural Information Processing Systems, 36:77013-77060 2023a.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. In Proceedings of EMNLP Workshop,
pp. 1, 2023b.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. Xix preprint 4x7:02440,,0633, 2024a.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement preference optimization, 2024b.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghnai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint Xxio2205517992 2023c.
Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. In The Twelfth International Conference on Learning Representations, 2023d.
Austin Xu, Will Monroe, and Klinton Bicknell. Large language model augmented exercise retrieval for personalized language learning. In Proceedings of the 14th Learning Analylics and Knowledge Conference, pp. 294, 2024.
Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joly. Does context matter? contextu- aljudgebench for evaluating lim-based judges in contextual settings. arXiv preprint arXiv:2503.15620, 2025.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yucting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Mullimedia, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288 5296, 2016.
Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, et al. Verifybench: Benchmarking reference- aefrrence-based reward systems for large language models. Xixiv preprint arXi::2505. 15801, 2025.
An Yang, Kai Liu, Jing Liu, Yajuan Lyu, and Sujian Li. Adaptations of rouge and bleu to better evaluate machine reading comprehension task. arXie preprint arx: 03578, 2018a.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018b.
Weizhe Yuan, Graham Ncubig, and Pengfci Liu. Bartscore: Evaluating generated text as text generation. Advances in newral informalion processing systems, 34:27263 27277, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. Xix preprint ar 1xXxx:1904.06697 2019.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. orXiv preprint arXiv:1909. 02622, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judge with MT- -rnBhench and Chatbot Arena. Advances in Newral Information Processing Systems, 36, 2024.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Additional SMILE and baseline implementation details
A.1 Prompt templates
As described in Section | we prompt the synthetic answer generator with the original question and ground- truth answer and task it with generating a synthetic answer. We provide the full prompt below.
Synthetic
generation prompts
#* Systen prompt:
You are an intelligent chatbot designed for generating anover as sentence from question-anzee ansxxr pairs. Your task to generate singla sentenca using the quest:in and ansver already providad. Hare'a how you can accomplish the taak:
#AINSTRUCTIONS::
provided ansver
Generate short single sentence reaponse using the question and the answer.
Response SHOULD ALMAYS USE THE WORDS FRON ANSWER provided. RESPONSE.
USE QUESTION
Return only the response and nothing elae.
#* User prompt
Please phrase short singla sentence ansver uaing queation-anaeer pair only:
Question: {<question>>
Answar: [<answar>}
PROVIDE
FROM SINGLE SENTENCE.
To prompt GPT-4o and GPT-3.5 judge models, utilize prompls adopted from Maaz et al. (2024).
described in Section 5
We provide full prompts below.
VMA Original pronpt: GPT-4c/GPT-3. T.rbo prompte
Syston prompts
You intelligent chathot dasignad for evaluating the
generativa outputs for correct ansver
question-answer pairs.
compare predicted ansver
they reaningfully.
Here's now can accomplish the task:
#NINSTRUCTIONS:
Pocus an the neaningful match betwean the pradicted ansver and the correct ansger.
Conaider synonyиa paraphrasea
natches.
Evaluate the correctness of the prediction compared to the anewer.
Veer pronpt
Please evaluate the following video-based question-aneet ansver pair:
Question: (<question>)
Corract Ansver: [Sansver>)
Predicted
{ondeelouttuut>
Provide your evaluation only as yee/no and score where the score is an integer value between and 5, with 5 indicating the highest neaningful natch. Please
the reeponee in the forn
with 'pred' and 'score', where a string of 'yes' or 'no' and value of 'score' is in INTEGER, not STAING.
Python dictionary
value of 'pred" Do NOT PROVIDE AKY OTHER OUTPUT TEXT EXPLANATION. Only provide the Python dictionary string. For example, your responae should look like this: (f' (t'prdd?: !yes', "score": 4)).
#*8 Extract prompt: OPT-40/GPT-3.6- judga prompts System proupts
You are an expert evaluator for video-baeed question anevering systeme. Your taak is to judge the factual accuracy of predicted anaver by comparing it corract ansver. will follow structured avaluation approach to ensure consistency:
INSTRUCTIONS:
Step 1: Extract the kay facts fron the Corract Ansver.
Step
list, Step 3: Extract the key facts from the Predicted Ansver.
natches Predicted
Step Compare
they are.
Consider parsphrasing, synonyus, and partisl overlape.
Ignore gramnstical errors.
Panalize hallucinated contradicted information.
Step 4: Based on the comparison, assign factual accuracy score betseen D and (INTEGER only), shere:
17

---

Published in Transactions on Machine Learning Research (01/2026)
6 Fully accurate and aligned
Mostly accurata,
paraphrasing
3 Partially correct but with notable nissing or incorrect info
2 Limited accuracy, mostly incorrect or unrelated
Completely inaccurate
No relation or total hallucination
Reapand strictly
following format:
{'score': X, 'pred': :Y) where is an integer betveen 0 and and Y in either 'yes' (I>3) or 'no (X<=3). Do not include any explanation or extra text.)
Ueer prompt
Eraluate the follewing videa-based
Question: {<question>]
Correct Anaver: (<anaver>)
Predicted Answer: {<model_onne >
Return your evaluation following instructions above.
DO NOT PROVIDE AKY OTHER OUIPUT TEXT OR EXPLAMATION. Only provide the Python dictionary string.
A.2 SMILE text processing
As a part of text pre-processing, we perform standard text normalization on words present in ground truth answers and predictions. We first convert each string to lower case and remove all punctuation. Then, each word lemmatized using POS-aware lemmatization capture accurate basc forms. If the resulting processed word is empty after these steps, the original lower-case word retained.
A.3 Metrics conversion to accuracy
For all evaluated baselines and metrics, we must convert from scores binary correct or incorrect accuracy labels. ROUGE, METEOR, BERTScore, sBERT, and SMILE all output continuous-valued scores between 0 and 1. We apply threshold of 0.67, considering anything above the threshold to be correct and anything below to be incorrect. The choice of 0.67 is the same considering anything with score of or above to be correct after converting the continuous [0,1] score to 0-5 scale with uniform binning. For CPT-3.5-Turbb and GPT-4o, the model is prompled to output a yes/no nn label for correctness, which we use directly.
B Data annotation details
B.1 Annotation instructions
Annotators were given with detailed instructions on how to annotate responses. We adopted a point scale: clearly incorrect, unclear, and clearly correct. We defined each of these categories as follows:
Clearly incorrect: The model definitively produces response that is incorrect.
Unclear: The model response cannot be confirmed correct from the ground-truth answer.
Clearly correct: The model response can be explicitly verified as correct using the ground-truth answer. We also defined edge case behavior:
Extrancous information: If the model response corrcctly answers the question, but includes other infor- mation that may may not be factual, we consider the response clearly correct. As concrete example, for question "What brand of soda is in this picture" with bouund-truth "Coca-Cola"" response "Coca-Cola is in this picture. It is the most popular soda in the world by unit sales and has over 60 different flavors" to be correct, even though it contains extraneous factually verifiable information.
consider the model
Synonyms or ambiguous subjects: We consider a model response that answers the question using an ambiguous subject to be unclear. As concrete example, for question "who describes a vidco game??" with ground-cruth "man", we consider the model response "person" " be unclear, it does describe in sufficient detail the person.
18

---

Published in Transactions on Machine Learning Research (01/2026)
B.2 Annotation aggregation and conversion to accuracy labels
We collected responses from four annotators. To aggregate individual annotations into a single label, we utilized majority vote, employing random tie-breaking as needed. To form final accuracy labels, we consider clearly correct responses to be accurate and consider unclear and clearly incorrect responses to be inaccurate.
C SMILE Interpretability Examples
As discussed in Section 4.2 we provide detailed SMILE subscores in Figure In the TGIF example from Figure 2 the model output shows a high semantic score 9, (Equation ① reflcctng strong relevance to the synthetic answer. The lexical relevance score 8t (Equation (2) is also high, indicating perfect overlap with the ground truth. To elarify which word contributes most to the keyword score, also return the word(s) with the maximun similarity ("max sim words"). These components together offer actionable insights into model strengths and weaknesses, helping guide targeted improvements.
D Supplement ablation results
In this section, we present additional plots to supplement our Model Ablation described in Section 5.4 Specifically, we include scatter plots and distribution plots to further illustrate the performance difference when varying model choices for synthetic answer generation and embedding.
D.1 Synthetic answer generation ablation
Referring to Figure 0 we sec very strong lincar correlation between the two sets of generated synthetic answers and thus backs our claim that generating synthetic answers is fairly simple task as mentioned in Section 5.4 Figure 7 further bolster our claim, and highlights that the 'avg score' distribution remains very similar, hence we see a marginal difference in the performance as reported in table G
D.2 Embedding model ablation
Figure and Figure provides insight into the performance variation observed in Table 6 highlighting that keyword scores exhibit greater sensitivity to the choice embedding model compared to sentence scores.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset: TextVQA
Dataset: DocVQA
Dataset: POPE
sentence score (Llama-3 1ataa..2-3B--im 2-38-Instruct)
1.1
sentence score (Llama-3.2-33 2-38-lmm (8-IInstruct)
1.
sentence score (Llama-3.2 1aama-3.2-38-lmm-. (8--nntt
1.1
1.0
1.0
0.8
0.8
0.8
0.7
0.7
0.6
0.5
0.5
0.5
0.5
0.6
0.7
0.8
0.9
1.0
11
0.5
0.6
0.7
0.8
0.9
1.0
11
0.5
0.6
0.7
0.8
0.9
1.0
11
sentence score (GPT-3.5-Turbo)
sentence score (GPT-3.5-Turbo)
sentence score (GPT-3.5-Turbo)
Dataset: MRQA
Dataset: HotpotQA
Dataset: MUSIQUE
sentence score (Llama-3.2-33) (2-3B-instruct)
1.1
1.1
sentence score (.. 3.2338-nsttt)
1.1
sentence score 3amma-32-38-Insttuue
0.9
0.8
0.8
0.8
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
11
sentence score (GPT-3.5-Turbo)
sentence score (GPT-3.5-Turbo)
sentence score (GPT-3.5-Turbo)
Dataset: TGIF
Dataset: MSVD
Dataset: MSRVTT
sentence score (Ulama-3. 1aamm-3.2-3BB (.2-3B-instruct
1.1
sentence score (Uama-3.2-3B-1l (.2-3ut-nsttruct
sentence score 1llmm-3.223B-Instruue
0.9
0.9
0.9
0.8
0.8
0.6
0.6
0.6
sentence (GPT-3.5-Turbo)
sentence (GPT-3.5-Turbo)
sentence (GPT:3.5-Turbo)
Figure 6: Distribution analysis of SMILE sentence embedding scores across different synthetic answer sets.
A strong linear relationship is observed between the two synthetic answer sets, indicating that synthetic answers can reliably be generated using any state-of-tho-art generation model.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset: TextVQA
Dataset: DocVQA
Dataset: POPE
90
GPT-3.5-Turbo
90
Jama-332-38-innst 3.--stinstttt
Llama-32-33B-insin 2-38-instruct
Jamm-3.2-3B-Instee
80
70
70
70
frequency
frequency
frequency
60
50
50
50
40
40
30
30
20
20
20
10
0
0
0.5
0.6
0.7
0.8
0.9
1.0
2.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SMILE (avg scores)
SMILE (avg scores)
SMILE (avg scores)
Dataset: MRQA
Dataset: HotpotQA
Dataset: MUSIQUE
90
90
GPT-3.5-Turbo
GPT-3.5-Turbo
GPT-3.5
Llama-3.2 3/2-38-insttruct
80
Llama- 1oma-33.2-25 3.2-3B-inttruct
80
Llama-3.2 3ama-3.2-38--mmm 2-23t-tnsttrct
70
80
60
60
frequency
frequency
frequency
40
40
50
20
20
10
10
0.5
1.0
SMILE (avg scores)
SMILE (avg scores)
SMILE (avg scores)
Dataset:
Dataset:
Dataset:
GPT-3.5-Turbo
GPT-3.5-Turbo
GPT-3.5-Turbo
80
80
Uama-3.2-38-Inne
70
70-
60
frequency
frequency
frequency
50
50
40
30
30
20
10
10
10
0
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
SMILE (avg scores)
SMILE (avg scores)
SMILE (avg scores)
Figure 7: Distribution analysis of 'SMILE avg scores' across different synthetic answer sets. We see a very similar score distribution, highlighting the fact the performance remains very similar.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset: TextVQA
Dataset: DocVQA
Dataset: POPE
1.1
1.1
1.1
1.0
1.0
sentence score (GTE-7B)
sentence score (GTE-78)
sentence score (GTE-7B)
0.9
0.8
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
sentence score (ember-v1)
sentence score (ember-v1)
sentence score (ember-v1)
Dataset: MRQA
Dataset: HotpotQA
Dataset: MUSIQUE
1.1
1.1
1.2
sentence score (GTE-7B)
sentence Score (GTE-7B)
sentence score (GTE-7B)
0.9
0.9
0.9
0.8
0.7-
0.6
0.6
0.5
1.0
0.9
sentence score (ember-v1)
sentence score (ember-v1)
sentence score (ember-v1)
Dataset: TGIF
Dataset:
Dataset: MSRVTT
1.0
1.0
1.0
sentence score (GTE-7B)
sentence score (GTE-7B)
sentence score (GTE-7B)
0.9
0.9
0.8
0.7
0.7
0.7
0.5
0.5
0.6
0.7
0.8
10
1.1
0.6
0.7
0.9
1.0
0.5
0.6
0.8
0.9
1.0
11
sentence score (ember-v1)
sentence score (ember-v1)
sentence score (ember.v1)
Figure 8: Analyzing Sentence score distributions using different embedding models. Sentence scores show stronger linear correlation, indicating that robust to change in embedding model.
22

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset: TEXTVQA
Dataset: DOCVQA
Dataset: POPE
1.1
1.1
keyword (GTE-7B)
keyword score (GTE-78)
keyword score (GTE-7B (B)
0.9
0.9
0.9
0.8
0.6
0.6
0.6
keyword score (ember-v1)
keyword score (ember-v1)
keyword score (ember-v1)
Dataset:
Dataset: HOTPOTQA
Dataset: MUSIQUE
1.0
1.0
1.0
keyword (GTE-7B)
keyword score (GTE-76)
keyword score (GTE-7B)
0.8
0.8
0.7
0.7
0.7
0.5
0.5
0.5
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.5
0.6
0.7
0.8
0.9
1.0
1.1
keyword score (ember-v1)
keyword score (ember-v1)
keyword score (ember-v1)
Dataset: TGIF
Dataset: MSVD
Dataset: MSRVTT
1.1
1.1
1.1
1.0
1.0
keyword (GTE-7B)
keyword score (GTE-7B)
keyword score (GTE-7B)
0.9
0.9
0.9
0.8
0.8
0.8
0.6
0.6
0.6
0.5
0.7
1.0
0.5
0.6
0.9
0.8
1.
keyword score (ember-v1)
keyword (ember-v1)
keyword (ember-vl)
Figure 9: Analyzing Keyword score distributions using different embedding models. Keyword scores show linear correlation, but has added noise, indicating that it is more sensitive to changes in embedding models.
23


Reviews:
[{"review_id": "r459e7SGyS", "reviewer": "Reviewer_cS6p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces SMILE (Semantic Metric Integrating Lexical Exactness) as a new benchmarking method that effectively addresses key weaknesses of traditional n‑gram–based metrics such as ROUGE, METEOR, and CIDEr, which often fail when semantically correct answers use different surface forms from the reference. On the other side, the LLM-based benchmarking kind of introduces the LLM-based hallucination in defining the actual correctness of a finetuned model. SMILE’s core contribution is to jointly model sentence‑level semantic similarity and keyword‑level semantic relevance while preserving explicit lexical matching, so it can appropriately reward both paraphrases and the accurate inclusion of critical content words rather than just local n‑gram overlap. This composite design leads to scores that correlate better with human judgments, particularly in realistic settings like textual, visual, and video question answering, where the correct sentence may appear anywhere in a longer paragraph and be phrased flexibly, and it offers these benefits with substantially lower computational cost than LLM‑as‑a‑judge evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Few links seem to be broken or weren't able to access:t https://anonymous.4open.science/r/SMILE-Artifact-2025/", "audience_yn": "Yes", "audience_explanation": "Benchmarking metrics for LLM and VLM models are still evolving, and existing metrics do not always adequately justify the confidence in fine-tuned models. As a result, the research community working on fine-tuning these models can significantly benefit from this work.", "requested_changes": "Correcting links in the paper. On the granularity of the LLM/VLM benchmarking, for very high-stakes or ultra-fine-grained evaluation (tiny numerical differences, subtle negations, hedging, or partial credit over many subfacts, example 1.2cm vs 1.3cm ), usually used in spatial AI use cases, how well can SMILE fit in those usecases?", "broader_impact_concerns": ".", "additional_comments": ".", "extra_scores": {}, "extra_text": {}}, {"review_id": "xmWcIvh54x", "reviewer": "Reviewer_sy4h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes \"SMILE,\" a composite evaluation metric for Question Answering (QA) tasks. The metric combines lexical exactness (keyword scores), semantic similarity (sentence embeddings), and similarity to synthetic answers generated by a Small Language Model (SLM, specifically a 3B model). The authors claim that this composite approach correlates better with human judgment than existing metrics while maintaining computational efficiency compared to LLM-based evaluators.", "claims_and_evidence_yn": "No", "claims_explanation": "The submission suffers from critical methodological flaws, factual errors in data interpretation, and reproducibility issues that fundamentally undermine the claims:\n\n1. **Lack of Novelty and Trivial Linearity:**\nThe proposed method is essentially a trivial linear combination of existing metrics (Sentence Embeddings + Keyword Matching). The only \"novel\" component—the SLM-generated synthetic answer—is shown by the authors' own ablation study (Table 6) to have marginal impact (removing it only reduces correlation from 0.790 to 0.761). Without this component, the method reduces to a standard ensemble of established metrics, lacking significant technical contribution or innovation.\n\n\n2. **Direct Contradiction Between Data and Claims (Table 6):**\nThere is a fatal factual error in the ablation study analysis. The text states that *\"Keyword scores are the primary contributor,\"* yet the data in Table 6 shows that removing keyword scores results in a negligible drop in correlation (0.790 -> 0.782). Conversely, removing **Sentence scores** causes a massive degradation (0.790 -> 0.415). The authors have fundamentally misinterpreted their own results, attributing importance to the least significant component while ignoring the actual dominant factor.\n\n3. **Lack of Reproducibility:**\nThe code and evaluation scripts link provided in the abstract (`https://anonymous.4open.science/r/SMILE-Artifact-2025/`) is broken/inaccessible. This prevents any verification of the data selection process, implementation details, or results, which is a disqualifying factor for claims of empirical improvement.\n\n4. **Misleading Efficiency Claims (Figure 1):**\nThe inference time comparison appears to be flawed. The SMILE metric requires generating a synthetic answer using a 3B parameter SLM. However, the reported inference time in Figure 1 does not seem to account for the latency of this generation step. Excluding the dominant cost factor (SLM generation) creates an unfair comparison against baseline metrics.\n\n5. **Statistically Insufficient Evidence:**\nThe evaluation relies on only **225 data points** across 9 datasets (averaging 25 samples per dataset). This sample size is too small to establish statistical significance for Pearson correlation comparisons. Furthermore, without accessible code, the selection process for these specific 225 points is opaque, raising concerns about potential selection bias (cherry-picking).\n\n6. **Circular and Unverified Methodology:**\n    * **Table 1 Circularity:** The \"Syn\" column reports the correlation between the metric and synthetic answers. Since the SMILE score explicitly calculates similarity to these synthetic answers as a component, this high correlation is tautological and offers no evaluative value.\n    * **Unverified Quality:** The paper assumes that integrating SLM-generated answers improves evaluation, but fails to validate the *quality* of these synthetic answers. If the SLM hallucinates, the metric measures similarity to noise.", "audience_yn": "Yes", "audience_explanation": "The general idea of combining lexical and semantic signals for QA evaluation is of interest to the TMLR audience, particularly given the growing concerns around cost, bias, and instability of LLM-as-judge methods. Researchers working on evaluation metrics and benchmarking would likely find the problem setting relevant.\n\nHowever, due to the methodological and experimental shortcomings outlined above, the current findings do not provide sufficiently reliable or generalizable insights. As a result, while the topic is relevant, the present version of the paper does not yet deliver conclusions that the community can confidently build upon.", "requested_changes": "Given the identified issues, substantial revisions are necessary to align the submission with TMLR's acceptance criteria. I recommend the authors address the following points:\n\n**Methodology & Reproducibility:**\n1. **Define the Weights:** Explicitly state the values of the weights ($w$) used in Equation 3 for the reported experiments. Additionally, provide guidance on how these weights should be selected or tuned for new datasets.\n2. **Fix Reproducibility:** Provide a working link to the code and evaluation scripts.\n3. **Fair Latency Benchmarking:** Report the full end-to-end inference time, strictly including the SLM synthetic answer generation time.\n\n**Empirical Rigor:**\n\n4. **Correct Ablation Analysis:** Rewrite the analysis of Table 6 to accurately reflect the data (acknowledging \"Sentence Score\" as the dominant factor) and justify the inclusion of the SLM component given its marginal contribution (<2%).\n5. **Statistical Support for Observations:** Figure 2 currently lists only 4 anecdotal examples. This observation should be supported by a statistical analysis on a larger sample size to prove these are systematic issues rather than cherry-picked outliers.\n6. **Expand Sample Size:** The correlation study (Table 1) relies on only 225 data points, which is statistically weak. The sample size should be significantly expanded.\n7. **Explain Missing Data:** Explain the presence of \"NaN\" values in the \"Exact Match\" and \"Easy Match\" columns in Table 1.\n\n**Presentation & Clarity:**\n\n8. **Fix Visual Accessibility (Figure 1):** The use of a green box to highlight red data points (SMILE) against brown baseline points (GPT-3.5) is visually ambiguous and unfriendly to readers with red-green color blindness. Please use distinct shapes or high-contrast colors.\n9. **Clarify Terminology:** Ensure consistency in terminology. The text refers to \"semantic score,\" but Table 6 uses \"sentence score.\"\n10. **Clarify Claims (Section 3):**\n    * The statement regarding \"verbose or generic model outputs\" based on 225 samples is vague; provide specific examples or metrics (e.g., length distribution) to define what \"verbose\" means in this context.\n    * The claim *\"using a powerful LLM like GPT-4o does not guarantee accurate evaluations\"* is presented as a novel finding but is common knowledge in the field. Please add appropriate citations to existing literature on LLM-as-a-Judge limitations.", "broader_impact_concerns": "No specific broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2lXWnOTY3Y", "reviewer": "Reviewer_dJJy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary\nThe paper proposes Semantic Metric Integrating Lexical Exactness (SMILE), a lightweight composite metric for evaluating question-answering (QA) systems. The metric consists of two sub-scores: a semantic subscore computed via embedding similarity between the model output and a synthetic answer, and a lexical subscore combining exact-match and maximum n-gram similarity. Experiments in text, image, and video QA domains demonstrate the superiority of the proposed metric.\n\nStrengths\n- The motivation is clear, as existing metrics fail due to either surface-level matching or LLM-judge biases and cost.\n- Extensive experiments on nine QA datasets and three domains.\n- Comprehensive ablation studies validate the design choices.\n\nWeaknesses\n- The paper states that \"SMILE consistently outperforms other evaluation metrics across tasks, achieving the highest overall correlation with human evaluations. Notably, SMILE significantly surpasses GPT-4o and GPT-3.5, despite their prominence as LLM-as-judge evaluators.\" However, from Tables 1 and 2, SMILE only achieves best result in 3/9 datasets in Pearson correlation and 4/9 datasets in  Kendall’s Tau-b metric. The overall result is the best mainly because of the superior performance on Language QA, but the method is frequently inferior to other baselines on the other two domains, Visual QA and Video QA, which requires more discussion.\n- While the paper emphasizes SMILE’s computational efficiency and attributes a large portion of its speed advantage to precomputing embeddings for synthetic answers and gold references, but many other embedding-based baselines evaluated could also precompute embeddings with minimal effort, achieving the same reduction in per-sample inference cost. The paper should clarify that SMILE’s speed advantage holds primarily against LLM-as-judge baselines, which cannot benefit from precomputation.\n- More detailed analysis of evaluation efficiency should be provided beyond what is shown in Figure 1, such as the computational overhead of each component. Also, is SMILE the only method that used precomputation in Figure 1 for speed up? More explanations on the inference time derivation scheme can be provided for clarity.\n- The approach is limited to source-free QA evaluation and does not catch context-dependent factual errors.\n- The human evaluation process needs more clarifications. Especially, whether annotators know which model generated each answer should be clarified to avoid bias in evaluation.\n- The human benchmark of 225 samples is limited.\n- There are multiple nan numbers in the tables, which require some explanations.", "claims_and_evidence_yn": "No", "claims_explanation": "There are a few claims not fully supported by experiment results.\n- For example, the manuscript states that SMILE “consistently outperforms” other metrics and “significantly surpasses GPT-4o and GPT-3.5,” yet Tables 1 and 2 show that SMILE achieves the best performance on only a minority of the datasets. The limitation of SMILE on Visual and Video QA benchmarks is not discussed.\n- In addition, the claim of computational superiority is not convincingly demonstrated.\n- The reliability of the human judgment benchmark—central to validating SMILE’s correlation with human preferences—requires further clarification.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a central bottleneck in modern NLP evaluation: how to fairly, cheaply, and reliably evaluate QA models without resorting to expensive and inconsistent LLM-as-judge pipelines, which is highly relevant to the TMLR audience.", "requested_changes": "The authors could refer the the aforementioned weaknesses for improving their manuscript.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)