CLAIR: Evaluating Image Captions with Large Language Models
Authors: David Chan
Suzanne Petryk
Joseph E. Gonzalez
Trevor Darrell
John Canny
Venue: EMNLP 2023 Main (2023)
Keywords: Image Captioning
Evaluation
Metrics
Large Language Models
TL;DR: Asking large language models to score generated captions against reference scores correlates strongly with human judgement.

Abstract: The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.

CLAIR: Evaluating Image Captions with Large Language Models
David M. Chan, Suzanne Petryk, Joseph E. Gonzalez, Trevor Darrell, John Canny
University of California, Berkeley
jearidchannspttry._jjeooozaa trevordarrell Xannyyeeeereleey edu
Abstract
You are trying to tell if a candidate set of captions is describing the same image reference set of captions. Candidate set: (candidate captions) Reference set:
The evaluation sechine--geneatee image captions poses interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, specificity. Existing eegineeen measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, propose CLAIR', novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K- Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over ingge-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available wit.tttthhh.... io/clair/.
(reference captions) On a precise scale from to 100, how likely is it that the candidate set is describing the same image the reference set? (JSON format, with key "score", value between 0 and 100, and key "reason" with string value.)
Figure 1: CLAIR: (surprisingly simple) large language model-based measure for image caption evaluation. We find that CLAIR not only correlates strongly with human judgments of caption quality but can also generate interpretable reasons for the generated scores.
et al., 2021; I; Hu et al., 2023; Yarom et al., 2023; Cho et al., 2023) directly incorporate visual information. Unfortunately, while significant strides have been made in automated evaluation, human preference studies remain the most reliable (yet costly) source of caption evaluation.
1 Introduction Background
Automatically evaluating the quality of image captions challenging. There are many dimensions to consider, such as grammatical quality, semantic relevance, correctness, and specificity, among others. To ensure fair evaluations, most image captioning works employ suite of measures, each capturing different aspects. For instance, n-gram-based measures like BLEU (Papineni al., 2002) CIDEr (Vedantam al., 2015) broadly measure content overlap. SPICE (Anderson al., 2016) compares scene graph structures, and CLIPScore, TIFA, SecTrue and VPEval (Hessel
Fortunately, recent advances in large language models (LLMs) have opened new avenues for automatic evaluation. Models trained with reinforcement learning from human feedback (RLHF, Christiano al. (2017)) or similar methods are particularly useful for open-ended evaluation tasks, including image captioning, due to their explicit training to align with human preferences.
In our work, paralleling several recent works which find that LLMs can act effective "judges" for selecting the better answer from two candi- dates (Bubeck et al., 2023; Dettmers al., 2023; Chiang et al., 2023), we explore the ability of LLMs to evaluate caption quality in the multimodal setting.
Mcaaning "clear" in French, in line with other colorful metric names (Papineni al., 2002: Lin, 2004; Lita al., 2005).

---

We introduce CLAIR (Criterion using LAnguage models for Image caption Rating), measure which scores a candidate caption based on the likelihood that it describes the same image as a set references by directly asking an L.M to produce a numeric rating. We further query the LLM to provide a reason for its score, providing a level of interpretability to the scalar rating. As far as are aware, this is the first paper to explore replacing measures of semantic text quality with directly obtained LLM judgments, however concurrently, Zheng et al. (2023) have shown that directly provid- ing an answer rating can align highly with human preferences on a range of standard language-based tasks, such as conversational instruction following.
experimental implementation is surprisingly simple: uses no in-context examples (is entirely zero- shot), and default inference parameters for the APls. See Appendix B for further implementation details.
The choice of language model directly affects the quality of the CLAIR measure more accurate mod- els should produce evaluations that align better with human judgment. We explore three language mod- els: GPT-3.5 (ChatGPT) (OpenAI, 2022), Claude (Instant) (Bai ct al., 2022), and PaLM (Chowdhery et al., 2022). Unfortunately, we found for several open-weight language models including Koala (Geng et al., 2023) and Vicuna (Chiang et al., 2023) that CLAIR aligned poorly with human judgment.
As CLAIR is language model-agnostic, we can leverage the different distributions learned by each model and combine their decisions an ensemble approach we term CLAIRE. We calculate individ- ual CLAIR scores for each model and compute an unweighted average to obtain the ensemble score.
Through several experiments captioning datasets such as MS-COCO (Xu al., 2016), Flickr8k (Mao et al., 2014), and PASCAL- 50S (Vedantam et al., 2015), we find that CLAIR correlates surprisingly well with human preferences, outperforming prior captioning measures, We additionally propose CLAIR,, where we Ensemble the outputs of several LLMs by taking the average score, leading further improvements.
Benchmark measures: We benchmark against several existing measure of caption similarity. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Agarwal and Lavie, 2008) and CIDEr (Vedantam et al., 2015) all primarily measure n-gram overlap (however have different weighting schemes between n-grams, and across precision/re- call). We also compare against SPICE (Anderson et al., 2016), which compares caption parse trees and focuses on matching perceived action and object relationships in addition to n-grams. While the aforementioned measures are commonly re- ported in image captioning works, we also compare against several more modern measures, including BERT-Score (Zhang et al., 2020) (which measures the distance between BERT embeddings in the text), (Yi et al., 2020) (which fine-tunes BERT for for image captioning. LEIC (Cui et... 2018) and NUBIA (Kane al., 2020) (which are custom trained models for image caption evaluation). TIGEr (Jiang et al., 2019) (which is a model trained for caption evaluation which takes into account the original image context), and CLIP-Scon (Hessel et al., 2021) which uses the recent CLIP (Radford et al., 2021) model for reference- free evaluation.
Despite a simple pipeline using an LLM prompt with minimal output parsing, CLAIR's strong correlation with human preferences suggests that it captures multiple dimensions of caption similarity at once a feature that prior measures struggle to achieve alone. More generally, CLAIR demon- strates how language-only models evaluate vision-language tasks. We show LLMs can provide not only reliable scalar ratings but also correspond- ing reasoning for a given rating, offering a valuable combination of accuracy and interpretability.
2 CLAIR: LLMs for Caption Evaluation
In CLAIR, we adapt the zero-shot in-context learning approach described in Brown et al. (2020) to score candidate captions with large language models (LLMs). This involves converting the caption evaluation problem into human-readable text completion task which is solved by the LLM. Using the prompt in Figure 1, CLAIR first generates completions from the LLM and then parses those completions into both candidate scores and an explainable reason for the score. We use a greedy sampling method 0t= to encourage reproducibil- ity in the results, while acknowledging the inherent nondeterminism in LLMs (see section 4). CLAIR's
Evaluation & Discussion
To evaluate the quality of the measure, we run several evaluations that compare scores generated

---

Human
CLAR
BLEU4
BLEU4
ROUSE
Human
BLEU4
CIDer
ROUSE
3/5
0.7
0.0
0.55
0.61
2.7/5
0.5
0.0
0.01
0.43
2/5
0.4
0.0
0.00
0.14
Candidate
Candidate
Candidate
boy junips into the blue pool water.
A cyclist in hemmet and yellow jersey.
Person snowboarding at ski slopn.
CLAIR Reason
CLAIR Reason
CLAIR Reason
candidate
The candidate only mentions syclist and yellow jersey, describes man green shirt carrying clink while noing bicytle Altoough thera are some similanties, such as the fact that both sets montion riding
The candidate
mention
mention a boy jumping into water, but the dandidate set does not provide as much detail about the bey's appsarance or the on
reference
per son snowicer singg reference captions mention multiple paople skingg or climbing ng a mourtain. Additionally the candidate caption does mention details about surroundings, presant the rafarenca captions.
the
paol. Additionally the reference set includes castions about diving off a pier and holding are's nose, which are not mentioned in the candi date set.
the details undertain whether they describing the same image or nol.
differenoe
References
"A References
group References
1. stande wearing slucll cost ahorrss jancoe chetoerdi
colles"
rnsurias:
toot lese"
"A taighi grean and surgg rang Maysle imosgg creets man bright groon start ding a/ cych in poweo courtyord cary
oue stow couurry care Four slicts wclling uo-3 show CONOIC3 INI""
taad
shikking
7A boy Muc 8h ungs F10 the wattr
boy inholiing Ni note and jerping yff rivling boom!
man ridex tél skk snts onw hand and holdo rinie with othwet dsing wirlina
Figure 2: Several qualitative examples of CLAIR from the Flickr8K -xxpert dataset. CLAIIR not only correlates better with human judgments of caption quality but also provides detailed explanations for its score. CLAIR scores normalized by 100.
by CLAIR to both human judgments of caption quality and other image captioning evaluation mea- sures. We additionally provide several qualitative examples in Figure A unique benefit of CLAIR is that it provides not only numeric scores but is also introspectable, as it can identify which details in the candidate caption set match the reference set.
Table I: Samplc-level correlation (Kendall's 7) with human judgments. All p-values < 0.001. *: Model has access to additional visual context, Results for LEIC, BERT-S++, TIGEr, and NUBIA are drawn from their original work.
Dataset
Measure
COMPOSITE
Flickr8K
MS-COCO
BLEU@
0.313
0.323
0.265
BLEU@4
0.306
0,308
0.215
Sample-level human correlation: We first ask the question, how well does CLAIR correlate with human judgments of caption quality at a sample level? We do so by exploring the performance on three datasets, COMPOSITE, BiickrK--Exper,, and MS-COCO (See Appendix B for details).
ROUGE-L
0.324
0.323
0.221
BERT-S
0.301
0,392
0.163
METEOR
0.389
0.418
0.239
CIDEr
0.377
0.439
0.262
SPICE
0.403
0.449
0.257
BERT-S+
0.449
0.467
NUBIA
0,495
LEIC*
0.466
The results of sample-level correlation experiments are shown in Table 1. We can that CLAIR outperforms language-only measures (c.g.. 0.604 to 0.449 for BERT-S++), and most cases, outperforms vision-augmennt augmented measures. CLAIR achieves strong sample-level correlation on all datasets; for instance, CLAIRg closes the gap to inter-human agreement by 0.097 over vision-based measures and 0.132 over language-based measures. The improvements of CLAIR over CLAIR sug- gest that each language model may have some bias (similar to each human), yet the ensemble of models correlates more strongly with human judgments. A reasonable concern might be that the models underlying existing approaches are significantly smaller than those in CLAIR, and trained on less
TIGE+
0.454
0.493
CLIP-S*
0.538
0,512
0.217
R
0.554
0,530
0.305
RefCLIP-X*
0.523
0,549
0,274
CLAIR
+ GPT3.5
0.604
0.616
0.296
+ Clande
0.542
0.563
0.320
+ PaLM
0.580
0.546
0.355
CLAIR
0.592
0.627
0.374
Hume-
0.736
data. To address this, we introduce and compare against RefCLIP-X. which replaces the CLIP model in RefCLIP with a CLIP ViT-bigG/14 model trained on LATON 2B (Ilharco al., 2021). Even in this case, CLAIR demonstrates significantly improved performance.

---

Table 2: System- levell correlation between the average CLAIR score and human model evaluation for 5 models trained and evaluated on MS-COCO. All p-values < 0.05.
Table 4: Pearson correlation with human judgments when evaluating sets of captions on MS-COCO (N 794).
Measure
Coverage--ratee
Correctness
Measure
Kendall's
Spearman's
Pearson
BLEU@4
0.004
0.816
0.003
BLEU@1
0.399
0.600
0.706
ROUGE-L
0.011 I 0.583
0.038
0.184
BLEU@4
0.799
0.899
0.910
METEOR
0.016
0.398
0.006
0.785
ROUGE-L
0.600
0.700
0,792
CIDEr
0.004 0.844
0.026
0.173
METEOR
0.600
0.700
0,666
TRM-METEOR
0.128cc..m
0.108 c0.001
CIDEr
0.399
0.600
0,856
TRM-BLEU
0.127
0.151 co.000
SPICE
0.399
0.600
0.690
MMD-BERT
0.129 1029.20.011
0.124
CLAIR
FID-BERT
0.081 0.011
0.098
c0.001
0.799
0,899
0.869
CLAIR
Claude
1.000
1.000
0.868
PaLM
1.000
1.000
0.954
+ GPT3.5
0,195 0.011
0.187
0.014
CLAIRF
1.000
1.000
0.903
+ Claude
0.110
0.099
0.124
0.145
+ PaLM
0.129
0.081
0.085
0.172
CLAIR
0.183
0.027
0.156
0.018
Table 3: Accuracy of measures when matching human decisions for PASCAL-50S (5 reference captions). s; Model has access to additional visual context.
Inter-Human
0.225kc.000
0.274.co.001
tive analysis. The PASCAL-50S dataset (Vedantam et al., 2015) contains set of 4000 human-annotated caption pairs. For each pair of captions, humans 72.88
Measure
HC
III
HM
MM
All
BLEU@1
51.20
95.70
91.20
58.20
74.08
BLEU@4
53.00
92.40
86.70
$9.40
ROUGE-L
51.50
94.50
92.50
57.70
74.05
label which caption in the pair is closest to the reference set for the image. The caption pairs fall into four groups: "HC:" two human--ritte cap- tions matching the image, "HI:" one human caption, and one machine geeerrted caption, with only one matching the image, "HM:" a matching human cap- tion and matching machine-generated caption and "MM:" " two matching machine -eeerineeggerateed captions. See Appendix B for more dataset information.
METEOR
56.70
97.60
94.20
63.40
77.98
CIDEr
53,00
98.00
91.50
64,50
76.75
SPICE
52.60
93.90
83.60
48.10
69.55
TIGEr
56.00
99.80
92.80
74.20
80.70
CLIP-S*
56.50
99.30
96.40
70.40
80.70
RefCLIP-S*
64,50
99.60
95.40
72.80
83.10
CLAIR
+ GPT3.5
52.40
99.50
89.80
73.00
78.67
+ Claude
57.90
98.50
91.30
62,90
77.65
PaLM
54.70
98.30
87.30
64.00
76.08
CLAIR
57.70
99.80
94.60
75.60
81.93
The performance on PASCAL-50S is given in Table 3. We can that CLAIRg outperforms all existing text-only measures (e.g.. by 5.18% overall score over CIDEr), and in many cases, even outperforms measures that have access to the image at test time. Note that is relatively weaker than image-augmented models in the HC setting; however, since both captions are correct, the model often cannot judge which is better purely the text. Models such as RefCLIP-S that have access the image are naturally better discriminators in this case. We suspect that CLAIR's discriminative performance could be further improved by giving the LLM a choice between the two captions; however, we leave this optimization to future work.
System-level human correlation: In addition computing the sample-level correlation on the MS-COCO dataset, we use the annotations from the five models considered by Rohrbach et al. (2018) to compute the system-level correlation. For each of the methods, we compute the mean human score on the test samples, and mean metric score on the test samples, followed by the Kendall's 's rank correlation coefficient (Kendall's 's tau, strength of ordinal asso- ciation) between these values (the of five mean human scores, and the set of five metric scores). Our results, given in Table 2, demonstrate that CLAIR ranks the five methods in a novel way that is more accordant with human rankings of the methods. These results further suggest that CLAIR has the potential to redefine which methods are preferable to humans compared to existing n-gram approaches.
Groups of Captions: While CLAIR is capable of comparing a single candidate caption set of reference captions, is also capable of comparing sets of candidate captions sets of reference captions. This task is necessary when evaluating the ability of a model to generate captions that are diverse and that fully describe the conditional text
Decision Making: In addition to evaluating the correlation with human judgments, we also evaluate the capability of the measure to perform discrimina-

---

distribution. We evaluate on the COCO-Sets dataset (Chan et al., 2022), 794 caption sets rated by AMT workers on two scales: how closely a candidate set matches the reference set in terms of both correctness and content coverage (See Appendix B for details). The results of this experiment are given in 4. We can see that CLAIR outperforms well when measuring the quality of group of captions, and approaches the inter-human correlation on the (very) challenging task. CLAIR also outperforms TRM-METEOR and TRM-BLEU (Chan et al., 2022), suggesting that LLMs can judge both the content and diversity the caption sets.
this drops to $0.000113 per sample. We hope that over time, advances in LLM inference (such as quantization and distillation), coupled with improvements in architecture will continue to yield lower-cost alternatives with strong performance on the caption evaluation task.
While CLAIR does suffer from potential hallucination, we strongly believe that this weakness does not diminish the fact that CLAIR still correlates strongly with human judgment. CLAIR, hallucinations in the score manifest as "incorrect" judgements of similarity, while hallu- cinations in the explanations manifest as poorly grounded explanations of the score/quality. Hal- lucinations in the score should be considered false negatives (blind spots instead of hallucinations). In the case of hallucinations in the explanations, such hallucinations may lead to misinterpretation, but arguably less misinterpretation than a black box method, and may even indicate misunderstandings in the model. Hallucination is well-known chal- lenge of current LLMs and is the subject of a great amount of research instruction-tunin RLAIF, and other methods. As hallucination and instruction-following performance of the base mod- els improves, CLAIR inherit similar improvements.
Hallucination:
4 Limitations
While CLAIR correlates well with human judg- ments of caption quality, it has several limitations:
Non-Determinism and Parsing Errors: Because CLAIR depends on the output of a language model, the measure can be non-deterministic and noisy. For instance, it may fail to elicit judgment (e.g., "As an AT language model, cannot see, and thus, cannot de- termine if the image captions match the references"), or rarely, generate malformed JSON output. To ad- dress these issues, we perform multiple queries to the I.L,, sometimes at higher temperatures nec- essary. As consequence, the measure may differ between runs, although we found the variance to be relatively insignificant (< 0.01 in many of the exper- iments). Additionally, since the language models used are not open-source, the models are subject to arbitrary change, replacement, or removal, which limits the efficacy of the measure as a long-term com- parable measurement. We hope that increasing open access to language models with efforts such as Koala (Geng et al., 2023) and Vicuna (Chiang et al., 2023), will help to alleviate these challenges in the future. Increased Cost: CLAIR relies on language models which contain many billions of parameters. These language models have not only monetary cost but also human and environmental costs (Bender et al., 2021) which can reduce its utility as a target during training, such as for ceiticcitical sequence training (Rennie et al., 2017). While API-based LLMS may be considered costly, even open-source LLMs have a cost (which can often be hard quantify). CLAIR on the MS-COCO dataset uses an average of 226. 148 tokens per sample (on OpenAl's APT), representing a cost of $0.0067 per sample (GPT-4), or $0.00033 per sample (GPT 3.5). For PALM,
RLHF,,
Explainability: While CLAIR generates explana- tions for cach rating, CLAIR has no strict scoring rubric. Much like human judgments, there is no direct way of attributing changes in score to changes in caption quality. For similar reasons, it is difficult to evaluate the quality of the generated explanations. Qualitatively, the explanations are generally reasonable and consider multiple axes of judgment.
5 Conclusion
This work introduces CLAIR, an LLM-based evaluation measure for image captioning. CLAIR's ss superior performance compared to highly- engineered measures indicates remarkable fact: LLMs are well aligned with human judgments of caption quality, even more so than some measures designed specifically for semantic similarity. CLAIR is only glimpse into how LLMs can be used for evaluation tasks, and image captioning is only the beginning. We hope that our work will inspire further exploration of similar measures in other vision and language domains, such as visual storytelling (Huang et al., 2016), where human eval- uation of generated text remains a challenging task.

---

References
Eric Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yianmis Aloimonos. 2015. From images to sentences through scene description graphs using commonsense reasoning and knowledge. ArXiv
Jaemin Cho. Abhay Zala, and Mohit Bansal. 2023.
Visual programming for text-to-image generation and evaluation. arXiv preprint arXiv:2305.
preprint,
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311.
Abhaya Agarwal and Alon Lavie. 2008. Meteor, M-BLEU and M-TER: Evaluation metrics for high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 115-118. Association for Computational Linguistics.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299-4307.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
30: Annual Conference
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.000 05862.
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. 2018. Learning to evaluate image captioning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018. Salt Lake City, UT, USA, June 18-22, 2018, pages 5804-5812. IEEE Computer Society.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? Proceedings the 2021 ACM conference
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Effficient finetuning of quantized Ilms. ArXiv preprint, abs/2305. 14314.
on fairness, accountability, and transparency, pages 610-623.
Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. Blog post.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, lla Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12 12, 2020, virtual.
Jack Hessel, Ari Holzzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image caption- ing. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pages 7514-7528. Association for Computational Linguistics.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description a ranking task: Data, models and evaluation metrics. Joumna of Artificial Intelligence Research, 17:853-899.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early exper- iments with gpt-4. ArXiv preprint, 12222
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint 118.
David M Chan, Yiming Ni, Austin Myers, Sudheendra Vijayanarasimhan, David A Ross, and John Canny. 2022. Distribution aware metries for conditional natural language generation. ArXiv preprint, abs/2209.07518.
Ting Hao Kenneth Huang. Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239,
Wei-Lin Chiang, Zhuohan Li, Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Z Joseph E. Gonzalez, lon Stoica, and

---

San Diego, California, Association for Computational Linguistics.
language supervision. Proceedings of the 38th International Conference on Machine Learning, ICMI, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8766 PMLR.
Gabriel Ilbarco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt, 2021. Openclip. If you use this software. please cite as below.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self- celticcttical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1 1979-199999 IEER Computer Society.
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jian- feng Gao. 2019. TIGEr: Text-to-image grounding for image caption evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP) pages 2141-2152. Association for Computational Linguistics.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object ballucination in image captioning- In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 4035 4045. Association for Computational Linguistics.
Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020. NUBIA: NeUral based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 28-37. Association for Computational Linguistics.
Ramakrishma Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015. Boston, MA, USA, June 7-12, 2015, pages 4566 4575. IEEE Computer Society.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. Hoi. 2022. BLIP: bootstrapping language-imag pre-training for unified vision- Lagiugla understand- ing and generation. In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: unifying architectures, tasks, and modalities through simple requence-to-sequeee learning framework. Inter- national Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 1--8833340. PMLR.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81. Association for
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR- VTT: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. Las Vegas, NV. USA, June 27-30, 2016, pages 5288 5296. IEEE Computer Society.
Computational Linguistics.
Lucian Vlad Lita, Monica Rogati, and Alon Lavie. 2005. Blane: Learning evaluation metrics for mt. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 740-747.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo. Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Sypektor. 2023. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXiv:2305. 10400.
Junhua Mao. Wei Xu, Yi Yang, Jiang Wang, and Alan I. Yuille. 2014. Explain images with multi- modal recurrent neural networks. arXiv preprint arXiv:1410.1 1090.
Yanzhi Yi, Hangyu Deng, and Jinglu Hu. 2020. Im- proving image captioning evaluation by considering inter references variance. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 985-994. Association for Computational Linguistics.
OpenAl. 2022. Introducing chatgpt.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: method for automatic evaluation machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318. Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generatiin with BERT. In 8th International

---

Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
In most cases (99.997% for GPT-3, 99.991% for Claude, and 99.94% for PaLM during the course of our experiments), this is successful, and the score and reason are returned. In the case that the ISON output is malformed, we attempt to extract any sequence of digits from the LLM to use as a score, and set the reason to "Unknown." When this fails, can be the case when the models produce an output such as "As an Al language model, cannot see, and thus, cannot determine the image captions match the references", we retry the prompt at a higher temperature (t=1.0) several times. Failing this (which occurred only three times in the entire evaluation of this paper, across several hundred thousand calls), we set the score to O for the caption.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. et al. 2023. Judging Imm-as-a an--ss---udge with mt- mt-bench and chatbot arena. ArXiv preprint, abs/2300.05665.
Appendix
A Acknowledgements
We thank Suhong Moon and Kate Saenko for their helpful comments on the work. Authors, as part of their affiliation with UC Berkeley, were supported in part by the NSF, DoD, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program, well as gifts from Anyscale, As- tronomer, Google, IBM, Intel, Lacework, Microsoft, Mohamed Bin Zayed University of Artificial Intel- ligence, Samsung SDS, Uber, and VMware.
B.3 Datasets
In this section, we provide additional detail regard- ing the datasets used in the evaluations in section 3.
COMPOSITE: The COMPOSITE dataset (Aditya et al., 2015) contains muchine--gneratee test captions for 3995 images spread across the MS-COCO (Xu al., 2016), Flickr&K (Mao et al., 2014) and Flickr30k (Young al., 2014) datasets. Each image has three test captions, one written by a human, and two that are model generated. The candidate captions are graded by annotators on Amazon Mechanical Turk (AMT) on a scale of I (not relevant) to 5 (very relevant). Inter-human correlations are not available for this dataset.
B Additional Experimental Details
In this section, we provide several additional details for the experiments in section run with the CLAIR measure.
B.1 Input Prompt Formatting
The CLAIR prompt is given in its entirety in Figure 1. During run-time, candidate and reference captions are prefixed with "_ and inserted into the prompt, one per line. The resulting query is passed to the large language model. addition, for models which were not RLHF-tuned to perform conversation (such as PaLM), we found that it was helpful to append an additional prefix { ("socre": to the beginning of the output, to encourage the correct output formatting. CLAIR is surprisingly simple: it uses no in-context examples (is entirely zero-shot), and default inference parameters for the APIs. The model checkpoint metadata is generally unknown (as the APIs are somewhat fluid and evolving).
The Flickr8K-Experi dataset (Hodosh et al., 2013) 3) contains 5822 captions asso- ciated with 1000 images. The dataset is annotated with expert human judgments of quality, where images are rated from (caption is unrelated to the image) to 4 (caption describes the image without errors). Unlike the composite and MS-COCO datasets, the captions here are selected using image retrieval system, instead of generated using learned image captioning model. Following Jiang et al. (2019), we exclude any candidate captions that overlap the reference set.
Flickr8K-Expert.
B.2 LLM Output Post- Processoo
experiments by Rohrbach et al. (2018), we compute the sample- level correlation between our method and human ratings on a 500-image subset of the MS-COCO Karpathy test set. Each image in the subset contains candidate captions generated by 5 models, and each caption is labeled with the average three human
MS-COCO:
Following
Because CLAIR relies on LLM to produce output, there no guarantee that the output will be in the format we expect (i.e. valid, parsable JSON). To extract both the score and the reason, we first extract the first set of paired braces from the output the LLM and attempt to parse the result as ISON.

---

ratings generated by AMT workers which range from (very bad) (very good). Inter-human correlations are not available this dataset.
PASCAL-50S: PASCAL-50S contains 1000 images drawn from the PASCAL sentence dataset. Each image is associated with at least 50 (and as many as 120) reference captions. In addition to the reference captions, PASCAL-50S contains set of 4000 human annotated image/caption pairs containing an image, and two candidate captions. The caption pairs fall into four groups:
1. HC: In the HC group, both captions in the pair are human written, and describe the content of the target image correctly.
2. HI: In the HI group, both captions in the pair are human written, but one caption correctly describes the content of the image, and the other caption describes the content of different image.
3. HM: In the HM group, one caption is written by a human, and one caption is written by machine, but both correctly describe the content of the image.
4. MM: the MM group, both captions are written by a machine, and both correctly describe the image content.
In PASCAL-50S, the task is to decide which caption in the pair humans prefer more (a subjective task, hopefully indicating caption quality). Following previous work (Jiang et al., 2019; Hessel al., 2021), we limit the number of reference sentences to five during evaluation.
COCO-Sets: The COCO-Sets dataset (Chan al., 2022) is set of samples that designed evaluate the correlation of distribution- image captioning measures with human judgments distributional distance. In this dataset, humans were presented with two candidate caption sets (two image captioning models, OFA (Wang et al., 2022) and BLIP (Li et al., 2022) using different temperatures), and asked which candidate caption set correlated better with reference caption set measures: how much they overlapped factually (correctness), and how much information they pro- vided about the references (coverage). It consists of 794 AMT worker teekerrgeee judgments caption quality for images in the MS-COCO dataset.


Reviews:
[{"review_id": "CyFlqkQdJU", "reviewer": "Reviewer_kPSe", "summary": "", "questions": "Q1. How will you tackle hallucinations in CLAIR as it uses LLMs.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our novel, and powerful methods for evaluation of the image captioning task. We'd like to take this opportunity to respond to some of the weaknesses discussed by the reviewer.\n\n**Hallucinations in CLAIR:** While CLAIR does suffer from potential hallucination, we strongly believe that this weakness does not diminish the fact that CLAIR still correlates strongly with human judgment (even more so than traditional reference-based natural language evaluation measures).  In CLAIR, hallucinations in the score manifest as “incorrect” judgements of similarity, while hallucinations in the explanations manifest as poorly grounded explanations of the score/quality. Hallucinations in the score should be considered false negatives (blind spots instead of hallucinations). In the case of hallucinations in the explanations, such hallucinations may lead to misinterpretation, but arguably less misinterpretation than a black box method, and may even indicate misunderstandings in the model. Further, hallucination is a well-known challenge of current LLMs and is the subject of a great amount of research on instruction-tuning, RLHF, and other methods. As hallucination and instruction-following performance of the base models improves, we should inherit similar improvements in CLAIR. To further address this concern, we will add additional discussion about hallucination, and indicate some of the steps that can be taken to reduce hallucination.\n\n**LLMs are Costly:** While API-based LLMs may be considered costly, even open-source LLMs have a cost (which can often be hard to quantify). CLAIR on the MS-COCO dataset uses an average of 226.148 tokens per sample (on OpenAI’s API), representing a cost of \\\\$0.0067 per sample (GPT-4), or \\\\$0.00033 per sample (GPT 3.5). For PALM, this drops to \\\\$0.000113 per sample. We hope that over time, advances in LLM inference (such as quantization and distillation), coupled with improvements in architecture will continue to yield lower-cost alternatives with strong performance on the caption evaluation task. Such a trend is already evident: models such as Llama-2 (released after the submission of this short paper) demonstrate performance on par with Chat-GPT, while remaining free and accessible. Thus, while currently LLMs are a costly tool, we do not believe that long term LLM cost is a fundamental weakness of CLAIR. We will make this clear in the limitations section of the paper, where we already discuss the weaknesses of relying on LLM approaches.\n\n**Generated Explanations are Hard to Evaluate:** We agree that evaluating the explanations generated by CLAIR can be challenging. CLAIR’s rate of hallucination is largely tied to the performance of the underlying LLM As LLMs evolve, we hope that the incidence of hallucination will be dramatically reduced. While one potential avenue for evaluation is human studies of the quality of the explanations, it’s not immediately obvious what constitutes a hallucination in a generated explanation. For example, if we define hallucination as “a description of something that is not grounded in the inputs”, a hallucination in the explanation could be an invaluable indicator of the uncertainty or trustworthiness of the actual score, and raise red flags that may otherwise go unnoticed if a reviewer was just presented with a black box number. Understanding how CLAIR hallucinates, and performing a deep dive into the ways that LLM hallucinations can indicate confidence and uncertainty is an important direction for future research. We will make this clear in the limitations and future work sections of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The following are the reasons to accept the paper --\n\n1. The paper is well written and easy to follow.\n\n2. The authors proposes a novel approach for evaluation using LLM for image-captioning task.\n\n3. CLAIR achieves relative correlation improvement over SPICE of 39.6% on Flickr8k dataset and over image-augmented methods such as RefCLIP of 18.3%\n", "reasons_to_reject": "The following are the weakness of the paper -\n\n1. LLM's are costly to run and evaluate and are constrained by resources. So, in real scenarios it might be difficult to use this approach.\n\n2. Evaluation of generated explanations by LLM's are difficult to evaluate.\n\n3. The evaluation might suffer from hallucinations produced by LLM's.", "questions_to_authors": "Q1. How will you tackle hallucinations in CLAIR as it uses LLMs.", "paper_topic_and_main_contributions": "This paper proposes CLAIR, an LLM based novel evaluation measure for image captioning. The authors found that CLAIR demonstrates a stringer correlation with human judgements of caption quality compared to existing measures. It also shows that how language models can evaluate vision-language tasks. The LLM's can provide not only scalar ratings but also corresponding reasoning for a given rating, offering a valuable combination of accuracy & interpretability. \n\n", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wVLqEy1Upg", "reviewer": "Reviewer_kf2G", "summary": "", "questions": "- Q1: Can you clarify Pascal-50S dataset description (L189-204)? I checked the appendix A L537-557, but I still can’t fully understand how the 4 groups are constructed. Showing some examples and high-level explanation of the characteristics of 4 groups would be help readers.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our extensive experiments and simple and effective method for evaluation of the image captioning task. We'd like to take this opportunity to respond to some of the weaknesses and questions discussed by the reviewer.\n\n**Clarifying Pascal 50S Experiments:** We apologize for any confusion with the PASCAL-50S dataset, and will seek to clarify the dataset in the camera ready version of the paper. PASCAL-50S is a dataset consisting of images paired with at least 50 reference captions. In addition to the reference captions, there are 4000 human annotated image/caption pairs containing an image, and two candidate captions. There are four types of caption pair:\n\n- HC: In the HC group, both captions in the pair are human written, and describe the content of the target image correctly.\n- HI: In the HI group, both captions in the pair are human written, but one caption correctly describes the content of the image, and the other caption describes the content of a different image. \n- HM: In the HM group, one caption is written by a human, and one caption is written by a machine, but both correctly describe the content of the image.\n- MM: In the MM group, both captions are written by a machine, and both correctly describe the image content.\n\nIn PASCAL-50S, the task is to decide which caption in the pair humans prefer more (a subjective task, hopefully indicating caption quality). We will add examples, as well as seek to clarify this in the camera ready version of the paper. We are also happy to answer any further questions during the discussion period. \n\n**API Cost:** While API-based LLMs may be considered costly, even open-source LLMs have a cost (which can often be hard to quantify). CLAIR on the MS-COCO dataset uses an average of 226.148 tokens per sample (on OpenAI’s API), representing a cost of \\\\$0.0067 per sample (GPT-4), or \\\\$0.00033 per sample (GPT 3.5). For PALM, this drops to \\\\$0.000113 per sample. We hope that over time, advances in LLM inference (such as quantization and distillation), coupled with improvements in architecture will continue to yield lower-cost alternatives with strong performance on the caption evaluation task. Such a trend is already evident: models such as Llama-2 (released after the submission of this short paper) demonstrate performance on par with Chat-GPT, while remaining free and accessible. Thus, while currently LLMs are a costly tool, we do not believe that long term LLM cost is a fundamental weakness of CLAIR. We will make this clear in the limitations section of the paper, where we already discuss the weaknesses of relying on LLM approaches.\n\n**Missing Ablations:** While we do not provide any additional experiments besides our extensive human correlation experiments, we believe that human experimentation remains the gold standard for evaluating semantic similarity measures. We agree that exploring additional axes of the problem can lead to interesting effects: for example, we do not use any in-context learning (i.e. no examples), however biasing the model with additional context could lead to different results. While we do not ablate the choice of prompt structure, the proposed structure (Figure 1) demonstrates that even with simple requests, we are able to achieve strong correlation – we don’t believe that an ablation of the prompt structure would lead to materially interesting results, as there would be no way to scientifically demonstrate models are using particular parts of the prompt effectively. We agree that additional experimentation with open-source models would be an interesting augmentation, and we will aim to include as many as possible in the camera ready version of the paper, however we strongly believe that the primary contribution of CLAIR lies outside the specific LLM implementation.\n\n**Weak Baselines:** While the LLMs that we leverage for CLAIR are larger than those used for BERT-Score/CLIPScore, we don’t believe that developing a entirely new metric leveraging a larger LLM in the style of BERTScore/CLIPScore is within the scope of this short paper. Furthermore, CLAIR represents a novel way of using LLMs, i.e. just ask the LLM to produce a token-level score, as opposed to using an implicit embedding in the LLM’s latent space, which we believe sets the approach apart from existing LLM-based methods.\n\nWe do agree that some recent reference-free measures (published in May of this year, less than three months before the submission deadline) such as TIFA, VPEval, and SeeTRUE are interesting potential comparisons. These reference-free measures serve as interesting potential concurrent work, and we will add them to the related work section, and aim to generate comparisons and human evaluations of the method if possible. We don’t believe, however, that the existence of additional reference-free measures diminish the fact that CLAIR represents a surprisingly simple and effective measure for measuring semantic similarity in a reference-based setting (none of TIFA, VPEval, or SeeTrue leverage reference captions), and as we discuss in the paper, we strongly believe that there is a place for both reference-based and reference-free evaluation measures.\n\n**Weak Technical Novelty:** While CLAIR is primarily a prompting paper, we believe that this short paper is of significant interest to the community. CLAIR is the first paper which demonstrates (with extensive human studies and experimentation) that LLMs correlate stronger with human judgment than any existing reference-based measure when evaluating image captioning tasks. Furthermore, CLAIR is one of the first papers to demonstrate that LLMs are capable of producing a token-based score of captions, meaning that methods such as CLAIR are usable even when underlying embeddings are unobtainable (such as is often the case with proprietary algorithms/models). Finally, CLAIR is, to our knowledge, the only text similarity measure which is capable of producing an interrogative explanation of its score, a hugely important tool which can be used for diagnostic and analysis purposes. \n\n**Missing Details:** CLAIR is surprisingly simple: it uses no in-context examples (is entirely zero-shot), and default inference parameters for the APIs. The model checkpoint metadata is generally unknown (as the APIs are somewhat fluid and evolving). We will clarify this in the main paper.\n\n[1] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv preprint arXiv:2104.08718 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- S1: simple but effective implementation.\n\n- S2: strong empirical demonstration of higher human correlation than the existing metrics.\n\n- S3: providing quantitative experiments results on multiple different evaluation benchmarks.", "reasons_to_reject": "- W1: weak technical novelty. There are already captioning evaluation metrics using LM (e.g., BertScore).\n\n- W2: weak baselines. The sizes of the backbone models used in the baseline model-based evaluation (e.g., BertScore, CLIPScore) are much smaller than the LLMs used in CLAIR. As the authors used large LMs for CLAIR, we can also compare with larger components for the baseline metrics like CLIPScore. Also recently proposed text-to-image evaluation metrics that are using LLMs such as TIFA/VPEval/SeeTRUE can be directly used as baselines.\n\n- W3: missing ablation studies: The authors do not provide any studies other than human correlations. Different design choices (e.g., different in-context examples, different prompt structure, use of reasoning, different reasoning) can affect different results. It would be also interesting how well smaller open-source language models (e.g., llama1/2, flan-t5, vicuna) can perform in CLAIR.\n\n- W4: API cost. As mentioned in L277-289, evaluation of these models involves payment to their LLM host companies. In addition to the RL use cases mention by authors (L284-286), usually image captioning pipeline developments involve a lot of evaluation runs (e.g., evaluating on a validation set for each epoch). It would be hard for many academic researchers to repeatedly use CLAIR during model development.\n\n- W5: missing details: It would be hard to replicate the authors experiments given the information in the paper. Missing details include: number of in-context examples, LLM inference hyperparameters, model checkpoint metadata.", "questions_to_authors": "- Q1: Can you clarify Pascal-50S dataset description (L189-204)? I checked the appendix A L537-557, but I still can’t fully understand how the 4 groups are constructed. Showing some examples and high-level explanation of the characteristics of 4 groups would be help readers.", "paper_topic_and_main_contributions": "The paper introduces CLAIR, a new image captioning evaluation metric.\n\nCLAIR uses  frozen LLMs to evaluate the generated caption by comparing with reference captions. Specifically, they give a prompt to LLMs includes 1) candidate captions, 2) reference captions, and 3) task description, where they ask LLM to predict a score within 1-100 range along with text reasoning for the scoring.\n\nThey experiment with 3 LLMs: GPT-3.5 / Claude / PaLM and compare with existing metrics such as text-only metrics (e.g., BLEU/ROUGE/BertScore) and image-text metrics (e.g., CLIPScore).\n\nThey provide quantitative analysis including sample-level correlation (Composite/Flickr8k/COCO), system level correlation (COCO), decision making (Pascal-50S), and group of captions (COCO-sets). Overall, the proposed CLAIR metric achieves higher correlation metrics then the baseline metrics.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XLfvmcZQBH", "reviewer": "Reviewer_2NfD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our human-aligned, transferable, and easy to understand methods for evaluation of the image captioning task. We'd like to take this opportunity to respond to some of the weaknesses discussed by the reviewer.\n\n**Ranking of methods:** While we do not explicitly evaluate the ranking of the method on all of our datasets, in Table 2/Section 3 we provide system-level correlations with human judgment on the MS-COCO measure. The system-level correlation results, which indicate the correlation between the methods’ rankings of different models and human rankings of the same models, show that CLAIR significantly outperformed the baseline measures with respect to human judgements of model quality. In general, we also find that sample-level correlation is well correlated to human judgment (Table 1), which we strongly believe is evidence that CLAIR is capable of ranking methods, in addition to sample captions.\n\n**Easy Datasets:**  We agree that the datasets we evaluate on are not specialized to fine-grained image differentiation. As the reviewer mentions, both Winoground and BISON are designed for evaluating a model’s ability to distinguish fine-grained image concepts (they consist of one or two images, and a challenging single caption to match to the image). CLAIR, however, is intended as a replacement for reference-based measures of caption quality, which means that evaluating CLAIR on these datasets would measure the distance between the ground truth COCO captions, and the candidate captions in Winoground/BISON. In a way, this measures the performance of the “ground truth” model, which is likely somewhat divergent from the quality of the metric, as it is generally understood that even reference image captions do not cover the full range of content present in the images [4,5,6]. \n\n**Clearly wrong captions:** We understand the concern that a soft measure such as CLAIR will fail to recognize clearly incorrect captions (as this is often a weakness of measures based on embeddings, such as BERT-Score or CLIPScore). To evaluate the quality of CLAIR on such replacements, we evaluated the accuracy of CLAIR when applied to the FOIL dataset of hallucinations. FOIL is a dataset where candidate captions are constructed by replacing single nouns in correct captions with semantically close mismatches. We found that CLAIR (on average) detected 93.6% of the semantic mismatches, which is very close to human performance of 94.52%, and above RefCLIPScore [3] (92.6%) and even CHAIRs [2] (92.5%, a metric designed explicitly for hallucination detection in captions)\n\n**LLM Scores:** In general, we found that the quality of the CLAIR scores produced by the LLM is closely related to the language model’s ability to perform reasoning tasks, such as question answering, as well as the model’s ability to respond to questions in the requested format. We strongly believe that it is largely a coincidence that this set of models overlaps with the set of API-based models (compared to free models such as Llama/Vicuna). A detailed investigation of why CLAIR works is certainly interesting: what parts of the training datasets correlate strongly with CLAIR performance, and why is it that CLAIR is able to approximate human judgements of semantic correlation so well? These are challenging questions to answer without significant control of the underlying language model, and while fundamental, we believe they are outside the scope of this short paper, which takes the first step towards quantifying and demonstrating the performance of LLMs in a zero-shot manner on semantic judgment tasks for image captioning applications. \n\n[1] Shekhar, Ravi, et al. \"Foil it! find one mismatch between image and language caption.\" arXiv preprint arXiv:1705.01359 (2017).\n[2] Rohrbach, Anna, et al. \"Object hallucination in image captioning.\" arXiv preprint arXiv:1809.02156 (2018).\n[3] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.\" arXiv preprint arXiv:2104.08718 (2021).\n[4] Caglayan, Ozan, Pranava Madhyastha, and Lucia Specia. \"Curious case of language generation evaluation metrics: A cautionary tale.\" arXiv preprint arXiv:2010.13588 (2020).\n[5] Yeh, Yi-Ting, Maxine Eskenazi, and Shikib Mehri. \"A comprehensive assessment of dialog evaluation metrics.\" arXiv preprint arXiv:2106.03706 (2021).\n[6] Chan, David M., et al. \"Distribution aware metrics for conditional natural language generation.\" arXiv preprint arXiv:2209.07518 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has demonstrated that their approach provides more human aligned scores than on several popular benchmarks\n\nThe set of related tasks to image captioning is large and this work could potentially provide a good metric for many of them\n\nThe paper is relatively easy to follow and understand", "reasons_to_reject": "1.  It isn't clear that the proposed metric will provide a more accurate ranking of methods.  In particular, I would wonder if the ranking of models produced by the proposed metric is different than the metrics we already use for image captioning\n\n2. The authors use relatively easy datasets for evaluating their model. However, if they demonstrated their approach could also be used to evaluate models on datasets like Winoground or BISON, where there are meant to be similar images with captions that have a lot of similar words, then the metric would be far more beneficial for future work on an area where image captioning methods are known to work poorly.\n\n3. The authors note that some LLMS do not provide good scores, but to not provide any insight that would enable us to understand the reason for this discrepancy \n\n4. It isn't clear that the proposed metric would be useful for clearly wrong captions given an image.  For example, if one were to replace \"dog\" for \"cat\" in a caption, would the metric notice? This would be somewhat addressed by the second weakness, but it isn't clear even in settings that the current metrics would be able to distinguish between models that the proposed one would. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach for evaluating image captioning methods by taking advantage of a large language model.  Specifically, they formulate the evaluation protocol for image captioning as a question that they pose to the LLM to answer.  The authors demonstrate that their approach provides more human-aligned scores than metrics proposed by prior work.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RyeI2QXJrs", "reviewer": "Reviewer_Cmgu", "summary": "", "questions": "(1) As CLAIR scores generations solely based on texts, I am wondering what makes it unique and applicable to image captioning. Could it be extended to other text-based generation tasks? The paper should strengthen the motivation of using CLAIR specifically for the image captioning task.\n\n(2) Is it applicable to other image-to-text generation tasks? Could it score well with image-to-text generations with long text targets, such as visual storytelling [3]? Judging long texts is still a challenging problem.\n\n(3) As the authors mentioned in the introduction section, each evaluation metric measures different aspects of captions. Have the authors tried using different prompts that align with the same aspects as other metrics to enable a fairer comparison?\n\n(4) Have the authors considered pair-wise ranking? Some papers [4,5] have shown that pair-wise ranking is more stable than ranking through many candidates at once.\n\n(5) Have the authors experimented with other instructions (e.g., as shown in Figure 1) to observe the differences?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our extensive experiments, and easy to understand methods for evaluation of the image captioning methods. We'd like to take this opportunity to respond to some of the weaknesses and questions discussed by the reviewer.\n\n**Motivation for Image Captioning/Could CLAIR be extended to other tasks:** While we do believe that CLAIR is applicable to domains outside of image captioning, in this short paper we decided to focus on image captioning as it is well within the scope of the authors’ expertise, as well as clearly indicates the benefits of a flexible, LLM-enabled evaluation method. Further, automatically evaluating the quality of image captions is a sufficiently challenging problem, and a good forum for understanding the potential of such LLM-enabled measures. There are many dimensions to consider, such as grammatical quality, semantic relevance, correctness, and specificity, which all lead to a good forum to showcase the potential approach. \n\nWe strongly believe that introducing CLAIR as an image captioning measure does not preclude the application of CLAIR to other tasks such as visual storytelling (which we will discuss as a potential application, as well as mention related references for, in the camera ready paper), and instead, we hope that introducing CLAIR inspires the use of LLM-enabled measures across a wide range of potential applications!\n\n**Other Prompts for CLAIR/Directed Alignment to other measures:** In this short paper, we primarily focused on CLAIR as a replacement for/augmentation to existing n-gram measures for semantic similarity. While we did experiment somewhat in terms of finding an optimal prompt for this task, we did not have any significant interpretable findings regarding the choice of prompt. Our experiments show that with the existing prompt (Figure 1), we are already able to outperform general measures in terms of overall human correlation, which we do believe is a fair approach when comparing the metrics as general purpose measures.\n\nWe do believe that flexibly adapting the prompt can serve as an interesting tool to use for targeted evaluation, for example, we could ask only about certain facets of the image, such as the background of the image, or only ask the model to measure the semantic distance between nouns (ignoring adjectives). We strongly believe that it is interesting and exciting future work to explore “directed” versions of the CLAIR measure, however such work is outside the scope of the current introductory work. \n\n**Cost of LLMs:** While API-based LLMs may be considered costly, even open-source LLMs have a cost (which can often be hard to quantify). CLAIR on the MS-COCO dataset uses an average of 226.148 tokens per sample (on OpenAI’s API), representing a cost of \\\\$0.0067 per sample (GPT-4), or \\\\$0.00033 per sample (GPT 3.5). For PALM, this drops to \\\\$0.000113 per sample. We hope that over time, advances in LLM inference (such as quantization and distillation), coupled with improvements in architecture will continue to yield lower-cost alternatives with strong performance on the caption evaluation task. Such a trend is already evident: models such as Llama-2 (released after the submission of this short paper) demonstrate performance on par with Chat-GPT, while remaining free and accessible. Thus, while currently LLMs are a costly tool, we do not believe that long term LLM cost is a fundamental weakness of CLAIR. We will make this clear in the limitations section of the paper, where we already discuss the weaknesses of relying on LLM approaches.\nPairwise Ranking: In this short paper we did not consider a pairwise ranking approach to stabilize the performance of the metrics (either CLAIR or other measures). We believe that this is an excellent direction for further research: like humans, do LLMs also benefit from the reduction in context/specificity provided by pairwise ranking. \n\n**Other Comments:** We will include the gold references for the captions In Figure 2 in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is very well-written and easy to follow. The organization of the manuscript effectively follows their contributions.\n\n(2) This paper presents interesting aspects of using LLMs to assess the quality of generations.\n\n(3) The authors conduct extensive experiments to demonstrate the improvements of this new evaluation method.\n\n(4) The method not only provides a score for the image caption but also offers a reason for why this judgment is made.\n\n(5)T hough the metric does not consider images as input, it is interesting to see that the results outperformed other metrics that do consider images", "reasons_to_reject": "(1) Though the paper introduces an interesting usage of LLM and demonstrates its superiority, the motivation for considering the image captioning task should be strengthened since the judgment of their method is solely based on texts.\n\n(2) Developing captioning models based on this metric could be a bit costly for users.\n\n(3) The paper should consider mentioning other multimodal generation tasks and evaluation tasks [1,2] (see missing references). ", "questions_to_authors": "(1) As CLAIR scores generations solely based on texts, I am wondering what makes it unique and applicable to image captioning. Could it be extended to other text-based generation tasks? The paper should strengthen the motivation of using CLAIR specifically for the image captioning task.\n\n(2) Is it applicable to other image-to-text generation tasks? Could it score well with image-to-text generations with long text targets, such as visual storytelling [3]? Judging long texts is still a challenging problem.\n\n(3) As the authors mentioned in the introduction section, each evaluation metric measures different aspects of captions. Have the authors tried using different prompts that align with the same aspects as other metrics to enable a fairer comparison?\n\n(4) Have the authors considered pair-wise ranking? Some papers [4,5] have shown that pair-wise ranking is more stable than ranking through many candidates at once.\n\n(5) Have the authors experimented with other instructions (e.g., as shown in Figure 1) to observe the differences?", "paper_topic_and_main_contributions": "This paper introduces a novel usage of LLMs, wherein they are employed to evaluate model-generated image captions. Extensive experiments with strong baselines demonstrate the superiority of this method, and the results indicate that their proposed method, CLAIR, strongly aligns with human judgment.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The gold reference could be included in the examples of Fig2."}, "extra_text": {"Missing_References": "[1] Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations, EMNLP 2020.\n\n[2] Learning to Rank Visual Stories from Human Ranking Data, ACL 2022.\n\n[3] Visual storytelling, NAACL 2016.\n\n[4] No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling, ACL 2018.\n\n[5] Knowledge-enriched visual storytelling, AAAI 2020."}}]

Author rebuttal:

Meta-review: Evaluating machine-generated image captions against reference captions has proven to be a difficult task. This paper proposes simply asking LLMs to score machine-generated captions versus reference captions, and shows that LLM performance aligns closely with human judgments.

Pros:
- Focused problem statement, appropriate scope for a short paper.
- Much better alignment with human judgment than previous metrics. Evaluating image captioning is an important task, and this work brings a methodology that outperforms existing metrics by a significant margin when it comes to aligning with human judgement.
- These findings are also relevant for other text evaluation challenges; the approach here is could be generalized beyond image captioning in future work (to be clear this is not necessary for this short paper). This is exciting.
- Well-written, easy to follow.

Cons:
- Using an LLM in the loop can be costly, so this metric may not be something that can practically be applied at training time. The authors  argue, sufficiently convincingly, that this will change over time and indeed is already changing with recent open-source models that have become available after the EMNLP deadline (and ACL's contemporaneous work policy).
- The authors also propose having the LLM explain its scores, but it's unclear if these explanations are trustworthy. This is ultimately somewhat irrelevant in the sense that the alignment with human judgment is what matters, however, as long as the explanations are not relied upon for anything.
- Depending on whether the LLM used to score is local or black-box+API-based, reproducibility of this metric may be concerning; for example, the model underlying an API call to a black-box model may change, even during scoring for a large set of captions, which could wreak havoc upon the results. This concern can be mitigated by using a stable model for every run.
Decision: Accept-Main