GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Authors: Ting-Yao Hsu
Chieh-Yang Huang
Ryan A. Rossi
Sungchul Kim
C. Lee Giles
Ting-Hao Kenneth Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: text generation
scientific figure caption
caption evaluation

Abstract: There is growing interest in systems that generate captions for scientific figures.
However, assessing these systems' output poses a significant challenge.
Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions.
This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions.
We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures.
We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs.
Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by computer science undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students' rankings.

GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Ting-Yac Yao Hsu, Chieh- Yang Huang, Ryan Rossi, Sungchul Kim,
Clyde Lee Giles,' Ting-Hao 'Kenneth' Huang' Pennsylvvnni State University, University Park, PA, USA.
( (txh357, chiehyang, clg20, Exh710)@psu. edu 2 Adobe Research, San Francisco, CA, USA.
(ryrossi, www.iuuaadbbe.ooommm com
Abstract
Humon-wr/itten Machine-made
Figure Caption
There is growing interest in systems that gen- erate captions for scientific figures. However, assessing these systems' output poses signif- icant challenge. Human evaluation requires academic expertise and costly, while auto- matic evaluation depends on often low-quality author-written captions. This paper investi- gates using large language models (L.LMs) as cost-effective, reference-free method for eval- uating figure captions. We first constructed SCICAP-EVAL, human evaluation dataset that contains human judgments for 3,600 scien- tific figure captions, both original and machine- made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as zero-shot evaluator, out- performed all other models and even surpassed assessments made by Computer Science and In- formatics undergraduates, achieving Kendall correlation score of 0.401 with Ph.D. students' rankings.
Figure-Mentioning Paragraphs
LLM
His ine rommrre
le roult haan Lagpet optire badget
Caption Quality Rating
Scientific Paper
Figure cost-effective, reference-free evaluation scientific figure captions using LLMs. We prompted LLMs to score (1-6) a caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs.
domain-specific Ph.D. students did not prefer cap- tions rated highly by automatic scores. Fortunately, recent findings in this area may provide solutions to the challenge. Figure captions were found to be similar summaries of igure-mentioning para- graphs (Huang et al., 2023), i.e., most words in captions can be semantically traced back to these paragraphs, and captions can be effectively gener- ated using text-summarization models. missing piece from previous research is the use of this find- ing for evaluation, which we address in this paper.
1 Introduction
There has been increasing interest in automating caption generation for scientific figures in scholarly articles. The SCICAP dataset (Hsu et al., 2021), an extensive collection of scientific figures and cap- tions from arXiv papers, encouraged more work in this area and led to several innovative approaches (Yang et al., 2023; Aubakirova et al., 2023; Tang et al., 2023; Li and Tajbakhsh, 2023). However, de- spite advances in caption generation for scientific figures, evaluating the results is still challenging. Human evaluation is costly as only domain experts can assess captions, making crowdsourced evalu- ation impractical. Meanwhile, automatic evalua- tion is not reliable. Huang et al. (2023) found that com/
This paper explores the cost-effective, reference-free evaluation of scientific figure captions using pre-trained large language models (LLMs)). Figure overviews the process. The intuition is that since scientific figure's caption in scholarly articles often functions condensed summary of all figure-mentioning paragraphs (Huang et al., 2023), LLMs, when given the appropriate context, can effectively evaluate how well the caption captures the essential information from these contexts (Chiang and Lee, 2023; Shen et al., 2023; Wu et al., 2023).
We first introduced SCICAP-EVAL, human evaluation dataset for scientific figure captions that contains 600 figures from arXiy papers. For ev-
'SCICAP-EVAL. is accessible at Crowd-AT-Lab/See AT--aaaScciaaa-Evalll

---

ery figure, we collected six captions: one authored by the original paper authors and five machine- generated. Two participant groups independently assessed these captions. The first group, Ph.D. stu- dents from the figure's s domain, ranked the six cap- tions for each figure. The second group, undergrad- uate Computer Science and Informatics students, rated each caption's s helpfulness. We used Ph.D. students for their in-depth understanding but rec- ognized their limited availability. Undergraduates were easier to recruit, but their knowledge might be insufficient. We aimed to examine if undergrad- uates could replace Ph.D. students in building a hu- man evaluation dataset for scientific figure captions. Equipped with SCICAP-EVAL, we then prompted LLMs like GPT-4 and GPT-3.5 to assign a score (1- 6) to each caption, based on how helpful they could be for readers understand the paper. The results showed GPT-4's effectiveness as zero-shot evalu- ator. It achieved 0.401 Kendall correlation with Ph.D. students' rankings, surpassing all other mod- els and even exceeding evaluations by Computer Science and Informatics undergraduates.
(2023) could only utilize 2 human annotators evaluate 100 figure captions. Automatic evalua- tion, on the other hand, has been more broadly used for its cost-effectiveness and ease. Met- rics such as BLEU,, ROUGH, BLEURT, CIDEr, BERTScore,and MoverScore are common (Kan- tharaj et al., 2022; Huang et al., 2023; Yang et al., 2023; Chen et al., 2019, 2020; Sellam et al., 2020; Zhang* * al., 2020), using human-written cap- tions as references. However, these metries are limited, including favoring longer captions (Sun et al., 2019) and relying on often poorly-written human- human-generated captions. For example, Huang et al. (2023) found that models preferred by humans often scored lower in automatic evaluations. These challenges in evaluating scientific figure captions motivate our pursuit of cost-effective, reference- free evaluation techniques.
SCICAP-EVAL Dataset
SCICAP (Hsu et al., 2021) is made of over 416,000 line charts (which were referred to as "graph plots" in (Hsu et al., 2021) and 133,543 of them are single- panel) extracted from arXiv papers in Computer Science (cs.*) and Machine Learning (stat.ML). In our paper, focused SCICAP'S single-panel figures the NLP, CV, and HCI domains. Most of these figures are 2-dimensional and annotated with axes labels, legends, and other texts. By Arunk- umar et al. (2023)'s definition, these figures are "information" visualization rather than "image".
This paper presents three contributions. Firstly, we have validated the effectiveness of using LLM to evaluate figure captions, exploring an avenue pre- viously untouched with such techniques. Secondly, we have developed a new dataset, cultivated over a few months with an investment of over $3,300, poised to serve as benchmark for subsequent re- search. We recruited twenty undergraduates, each handling 20 batches, with a completion time of 30 minutes to an hour per batch. Additionally, fifteen Ph.D. students participated, each completing four batches, each batch spanning 30 minutes to an hour. Lastly, our work resonates with the broader NLP community's 's drive to discover fresh applications for LLMs. In cases where human evaluation is costly and reference-based automatic evaluation is unreliable, LLMs offer an efficient alternative.
We randomly sampled 600 figures from SCICAP, with each cs.CV, CV, cs. CL, cs CS domain providing 200 figures. We then obtained six distinct cap- tions for each figure, primarily sourced from Huang et al. (2023) (approach ii to v), as follows: (i) Au- tritten captions; (ii) Pegasus,,, the captions created by the fine-tuned Pegasus model (Zhang et al., 2020), using figurementioning paragraphs as the input; (iii) Pegasusp+0, same as Pegasusp but additionally incorporating figures' OCR in- put: (iv) Pegasuso, same as Pegasusp but solely relied on OCR as input; (v) TrOCR (Li al., 2023), the fine-tuned vision-to-language model; (vi) Template-basee approach, which filled pre- defined templates with identified axis names.
2 Related Work
Assessing scientific figure captions is both costly and time-consuming due to the need for special- ized knowledge and understanding of the entire scholarly article. This has previously limited eval- uations small scale, typically performed by graduate students in the field. For instance, Huang et al. (2023) only managed to employ 3 Ph.D. students with NLP backgrounds to evaluate 90 figures' captions. In a similar vein, Yang et al.
We then recruited the following two groups of participants independently annotate the quality of these 3,600 captions. Thus, each caption was assessed twice. The entire study took a few months and cost over $3,300. We engaged 20 undergrad-

---

uate participants, each responsible for 20 batches. Each batch consisted of 30 figures, and the time taken to finish each batch ranged from 30 minutes to an hour. Additionally, 15 Ph.D. students were re- cruited, with each student tasked to rank batches. Each batch comprised 10 figures, each having 6 dif- ferent captions. The time taken by Ph.D. students for each batch varied from 30 minutes to an hour. The authors' institute's Institutional Review Board (IRB) approved this study.
per batch, with annotation times ranging from 30 minutes one hour. To ensure annotation quality, we provided each participant with tutorial using an additional 30 figures.
4 Methods
In this section, we describe different approaches used for caption rating, including zero-shot, few- shot, and Chain-of- -hoouhhh prompting with LLMs,, and fine-tuning a classifier using the data collected.
Human assessments by Ph.D. students on a six- item ranking task. We recruited 15 Ph.D. stu- dents (who are not coauthors of this paper) spe- cializing in CV, NLP, and HCI to rank six captions for each of the 40 figures in their respective fields. Specifically, we formed a ranking task with six items, where each Ph.D. student manually ranked 6 different captions for figure. These rankings were based on how helpful the captions were un- derstanding the figures. The annotation interface is shown in Figure 2 in Appendix A. Participants were compensated at $20 per hour, with annotation times varying per individual. To measure data quality, we recruited two non-coauthor Ph.D. students to anno- tate specific batch for assessing inter-annotator agreement, which yielded a Kendall of 0.427.
Zero-Shot and Few-Shot Prompting with LLMs. We included both the target caption and the figure- mentioning paragraphs from the paper (e.g., "As shown in Figure 4,...",, and then asked the LLM to evaluate the caption's 's quality. For the few-shot setting, we included three randomly selected cap- tions with the highest and lowest rankings from Ph.D. students' evaluations in the prompt. See Ap- pendix D for the actual prompts.
Chain-of- -haon-of-Thought Prompting (Wei et al., 2022) (QA, QA-YN). Our intuition is that an effec- tive figure caption distills the essential information from pertinent paragraphs. Acting this intuition, implemented Chain-of- Thouuhh approach to prompt LLMs to calculate evaluation scores. We first provided the LLMs with figuremmetiining paragraphs, asking them to generate questions that a suitable caption should answer. Following this, the LLMs were presented with a caption and asked to identify whether it could answer each question (Yes/No). The final evaluation score of the cap- tion was then derived from the percentage of "Yes" responses. In this paper, we explored two ques- tion types: open-ended (QA) and yes/no questions (NNY. See Appendix for the prompts used.
Human assessments by undergraduate students on a three-class labeling task. We recruited 20 STEM-focused undergraduate students, primarily Computer Science and Informatics majors, to rate the helpfulness of each caption with a "Yes," "No," or "Unsure" rating. We turned to the three-class la- beling task realized that the six-item ranking task was challenging for undergraduate stu- dents, which resulted in a long completion time. We understand using different annotation tasks would raise concerns of comparability between Ph.D. and undergraduate students' assessments, but this is the acceptable trade-off to mcaningfully col- lect broader set of undergraduate students' feed- back. For each participant, we assigned five batches covering three different domains, with each batch comprising 30 unique figure captions. In addition the helpfulness rating, students were asked to identify errors in the extracted figure or caption and label caption features. Full instructions, the user interface, and statistics of the annotation can be found in Appendix B. Compensation was $15
Classifiers learned from SCICAP-EVAL data. We fine-tuned the SciBERT classifier (Beltagy et al., 2019) with SCICAP-EVAL data to predict figure captions' helpfulness (Yes/No).
5 Experimental Results
Experimental setups. We conducted experi- ments with four LLMs: GPT-4 (OpenAI, 2023), GPT-3.5 (Brown al., 2020), Falcon (Almazrouei et al., 2023), and LLaMA-2 (Touvron et al., 2023), across various settings. We parsed the prediction scores from the outputs. When the L.MMs failed to predict, we allocated the lowest score of 1. For the SciBERT classifier, we split the dataset into
*Computer Science: 5, Biochemistry: 1, Biology: 1, Statis- tics: 1, Information Sciences and Technology: 12.

---

Setting/
(a) Correlation with
Correlation with PhD Students' Reciprocal Rank
Model
Training
PhD Students' Reversed Rank [Good to Bad Caption:
(b) Original Order
(c) Reversed Order
Data
(Picking good samples)
(Spotting bad samples)
6, ....
[Good to Bad Caption:
[Good Bad Caption:
1
B
P
P
Zero-Shot
.501
.401
.491
.391
-.492
Few-Shot
.531
.429
.523
.425
.513
GPT-4
CoT (QA)
.283
.270
.315
.277
-2223
CoT (QA-YN)
.357
.334
.400
.324
-307
Zero-Shot
.655
.370
.622
.383
-.433
Few-Shot
462
.371
.622
400
-.402
GPT-3.5
CoT (QA)
.270
.276
.347
.249
-226
CoT (QA-YN)
.365
.329
.404
.317
-3327
Zero-Shot
.342
.413
.326
-3392
LLaMA 2-70B
Few-Shot
424
.353
.430
.335
-405
Zero-Shot
026
.048
.055
.018
-000
Malcon-7B
Few-Shot
044
.048
.058
.044
-003
Zero-Shot
.169
.137
.167
.152
-.136
Falcon-40B
Few-Shot
.150
.119
.143
.126
-,137
Undergrad
.329
.290
.329
.261
-3319
SciBERT*
PhD)
.372
.308
.379
.372
-,722
Human
Undergrad
.221
.195
.221
.206
-.196
Table 1: Correlation coefficients (Pearson p. Kendall and Spearman r.) of model output ratings versus Ph.D. students' assessments (N=3,155 159, excluding error cases), based on different rank conversions: (1) Reversed Rank, (2) Reciprocal Rank, and (3) Reversed Reciprocal Rank SciBERT's performance was evaluated on the test split, comprising only 10% of the entire dataset.
80/10/10 for train/validation/eest and used the best
GPT-4 better at spotting bad examples than se- lecting good examples. We conducted additional analysis to determine whether LLMs as automatic caption evaluators are more effective at identify- ing good bad examples. Table displays the Pearson (p) correlation coefficients between each model's ratings and the reciprocal rank of Ph.D, students. The original order of reciprocal rank (b) places more weight on top selections, with higher scores indicating a model's effectiveness at identi- fying top examples. Conversely, the reversed order of reciprocal rank (c) prioritizes bottom selections, with higher absolute scores signifying the model's proficiency pinpointing poor examples. Table shows that GPT-4, in either zero-shot or few-shot prompting, excels at identifying low-quality over high-quality examples. This suggests its utility in cleaning training data by detecting poor samples.
model from the validation set for final testing. We excluded data marked as errors in undergrads' an- notations, resulting in 3, 1.59 valid samples. We fine-tuned SciBERT undergraduate and Ph.D. annotations, respectively. Captions ranked bottom three by Ph.D. students were labeled "No" and the top three as "Yes". We treated "No" and "Un- sure" as "No" for undergraduate labels. The model training details and decoding configuration are de- scribed in Appendix C.
GPT-4 prompting achieved the highest corre- lations with Ph.D. students' assessments. Ta- ble 1(a) shows that GPT-4, prompted in either a zero-shot or few-shot manner, exhibited the strongest correlations with Ph.D. students' judg- ments, surpassing all other models and even un- dergraduate students' ratings. Meanwhile, despite Yes-No questions yiclding better results than open- ended questions, Chain-of-Thoon Thought approaches generally underperformed. We hypothesize that an- swering questions may only partly determine the helpfulness caption. More research is needed to develop improved workflows.
Discussion
Do we need paragraphs that mention figures for this approach? To delve deeper into GPT-4's capability to evaluate captions, we conducted an ablation study to examine the influence of figure- mentioning paragraphs in the prompt. The results,

---

Reversed Rank
Reciprocal Rank
Fine-tuning LLMs to predict Helpfulness. We do believe that fine-tuning LLMs presents a promis- ing direction. However, our preliminary experi- ments with fine-tuning LLaMA-2 with LoRA (Hu et al., 2022) (both the 7B and 13B models) have not yielded encouraging results thus far. Pearson correlation with Ph.D. Students' Reversed Rank is 0. 0.164 (7B) and 0.183 (13B) respectively, See Ap- pendix C for training details. We believe it requires more exploration in the future.
T-Tsst
(a)
(b)
(c)
P
T
7s
All
.487
.399
.493
.396
-.460
<0.001
First
.478 .392
.484
.387
+,455
<0.001
Random
.479
.394
.486
.395
+,422
<0.001
Caption
.158
.125
.150
.193
-0078
Table 2: Ablation study illustrating the correlations be- tween various inputs and Ph.D. students' rankings. All, First, and Random represent all paragraphs, the first paragraph, and a randomly selected paragraph, respec- tively. (a), (b), (c) are defined Table The T-Test assesses the difference in scores when using different inputs versus using only the caption. The results indi- cate providing paragraphs is necessary.
How did the appearance and data domain of the images impact perceived utility? This work focuses on the utility of varied caption text of fig- ures rather than the utility of the figures' images. We have chosen to focus on captions as we believe captions provide the most feasible entry point for technologies to intervene and boost the utility of figures in scholarly articles. Captions, often stan- dalone like the caption() label in I8TEX, can be vised or replaced without altering the figure image. No raw chart data is strictly needed for improving captions. Using technology to customize or im- prove caption text is more achievable than altering figure images automatically, especially given the capabilities of LLMs. In the long run, we agree that the utility of the captions can be considered jointly with the figure images- as good caption would not save a poorly designed visualization- but this is future direction and beyond this paper's scope.
OCR
Visual Stats Relation Takeaway
Ph.D.
.299
.120
.110
.089
.195
Undergrad .323
.279
.227
.378
.479
Table 3: Pearson correlation coefficients (p) between caption features and the helpfulness ratings.
presented in Table 2, indicate that GPT-4's perfor- mance drops lot when paragraphs are not pre- sented, highlighting the need for paragraphs.
Toward personalized caption generation. A goal of having undergraduate students to rate cap- tions was to assess their potential to replace Ph.D. students in constructing human evaluation datasets for scientific captions, given their easier recruit- ment. However, their "helpfulness" ratings did not correlate well with those of Ph.D. students, suggesting different reader groups may perceive "helpfulness" differently. We further probed this by correlating caption features (annotated by under- graduates, see Appendix B) with the helpfulness assessments by Ph.D. and undergraduate students. Table 3 shows that for Ph.D. students, caption's helpfulness hinges most on its reference to terms and text from the figure (OCR), while undergradu- ate students prioritize captions that state the figure's s takeaway messages (Takeaway). Furthermore, a brief post-study survey revealed variations among Ph.D. students' assessment criteria. While the accu- racy of information was a priority for most, some students focused the ability of the caption to summarize the figure image, and others evaluated the volume of information conveyed within the cap- tion. These observations suggest that future caption generation systems could focus on personalization to meet the needs of various reader groups.
7 Conclusion and Future Work
In conclusion, we have shown the capacity of LLMs, specifically GPT-4, to evaluate scien- tific figure captions effectively. This work pro- duced SCICAP-EVAL. a human evaluation dataset of 3,600 captions, both original and machine- generated, for 600 arXiv figures. Using SCICAP- EVAL, we confirmed GPT-4's ability to assess caption quality when supplied with context from fgure-mentioning paragraphs. GPT-4 even outper- formed Computer Science and Informatics under- graduates, achieving a Kendall correlation score of 0,401 against Ph.D. students' ratings. Future work will leverage this evaluation capability to cleanse noisy training data, well-known challenge in fig- ure caption generation (Huang al., 2023). Fur- thermore, we plan investigate the personalized captions generation to cater to individual reader re- quirements. Last, we will explore ways to consider caption factuality in the evaluation metrics.

---

Acknowledgements
classification. In 2023 IEEE Visualization and Visual Analytics ( (VIS).
We are grateful to the anonymous reviewers for their insightful feedback and to all the participants in user study. We would also like to thank Tong Yu from Adobe for his feedback on the draft. This research was made possible through support from Adobe's gift funds and seed funding from Penn- sylvania State University's College of Information Sciences and Technology (IST).
Dana Aubakirova, Kim Gerdes, and Lufei Liu, 2023, Patfig: Generating short and long captions for patent figures. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pages 2843- 2849.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-JJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguisties.
Limitations
Despite its potential, our work is subject to sev- eral limitations. Firstly, similar to the method of Huang et al. (2023), our approach requires figure- mentioning paragraphs to act as the context for effective LLM-bbsed caption evaluation. However, mention identification is not always straightfor- ward in real-world data; for example, no mentions were identified for 18.81% % of figures in the original dataset. Secondly, our best results were achieved with a closed-source LLM, implying we inherit limitations such as restricted understanding of the model's training and data sources. Thirdly, our evaluation approach does not prioritize verifying the accuracy of the information within a caption, potentially leading to high ratings for inaccurate captions. Finally, as our methodology is solely text- dependent, it cannot capture any figure's visual elements not mentioned in the text.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Charles Chen, Ruiyi Zhang, Sungchul Kim, Scott Co- hen, Tong Yu, Ryan Rossi, and Razvan Bunescu. 2019. Neural caption generation over figures. In Adjunct Proceedings the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Com- puting and Proceedings of the 2019 ACM Inter- national Symposium on Wearable Computers. Ubi- Comp/ISWC '19 Adjunct, page 482-485, New York, NY, USA. Association for Computing Machinery.
Ethics Statement
The proposed technology is considered low-risk as inaccurate figure caption assessments are un- likely to harm readers significantly. However, it is worth noting that our text- based approach inher- ently overlooks visual content, which could poten- tially influence the accessibility of the technology in evaluating figure captions.
Charles Chen, Ruiyi Zhang, Funyee Koh, Sungchul Kim, Scott Cohen, and Ryan Rossi. 2020. Figure captioning with relation maps for reasoning. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1537-1545.
Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be alternative human evalua- tions? arXiv preprint arXiv:2305.01937
References
Ting- Yao Hsu, C Lee Giles, and Ting-Hap Hao Huang. 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli. Ruxandra Cojocaru. Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B; an open large language model with state-of-the-ard performance.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Anjana Arunkumar, Lace Padilla, Yeul Bac, and Chris Bryan. 2023. Image or information? examin- ing the nature and impact of visualization perceptual

---

Chich-Yang Huang, Ting- Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk- Yin Chan, Eu- nyee Koh, C Lee Giles, and Ting-Hao Huang. 2023. Summaries as captions: Generating figure captions for scientific documents with automated text summa- rization. In Proceedings of the 16th International Natural Language Generation Conference, pages 80- 92, Prague, Czechia. Association for Computational Linguistics.
Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Shankar Kantharaj. Rixie Tiffany Leong. Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Shart-to-text: A large-scale bench- mark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 4005-4003.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. 2022. Chain-of ofain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. Trocr: Transformer-basee optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13094-13102.
Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023. Less is more for long document summary evaluation by Ilms. arXiv preprint 209.030.
Shengzhi Li and Nima Tajbakhsh. 2023. Seigraphqa: large-scale synthetic multi-turn inestion-anssering dataset for scientific graphs. arXiv preprint 0rXiv:2308.03349
Zhishen Yang. Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2023. Scicapt: A knowledge augmented dataset to study the challenges of scientific figure captioning. arXiv preprint arXiv:2306 23..099....
OpenAl. 2023. Gpt-4 technical report.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguisties.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339, PMLR.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. 2023, Are large language models good evalua- tors for abstractive summarization? arXiv preprint arXiv:2305.1 13091.
Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Evalu- ating Neural Language Generation, pages 21-29.
Benny Tang, Angie Boggust, and Arvind Satyanarayan. 2023. Vistext: A benchmark for semantically rich chart captioning. In Proceedings of the s1 Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7268-7298.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almabairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan

---

A Annotating Interface for Ph.D. Students
Error Type
Total
Image Extraction Error
102
Text Extraction Error
242
Figure shows the ranking task interface used by the Ph.D. students in Section 3.
Not Line Chart
101
Compound Figure
B Annotating Instruction for Undergraduate Students
Table For all 3,600 figure-caption pairs, there are 3,159 valid samples. The remaining 441 figure-captions contain at least one eror.
Figure 3 shows the rating task interface used by the undergraduate students in Section We inquired whether the figure-caption combination contained any of the following four errors (Yes/No):
Takeaway: Does this caption describe the high-level takeaways, conclusions, insights the figure tries to convey?
The image we ex- tracted from the PDF (shown above) has obvi- ous errors (e.g., not containing the complete figure, containing parts that are figures. damaged image, etc.)
Image Extraction Error:
Helpfulness: Does this caption help you un- derstand the figure?
C Training and Decoding Details
Text Extraction Error: The text we extracted from the PDF (shown above) has obvious er- rors (e.g., not containing the complete caption, containing extra that is not the caption, incorrect text recognition, etc.)
We describe the model training details and the de- coding configuration used in Section 5.
Training Details for Classification models. We fine-tune SciBERT checkpoint from HuggingFace for 100 epochs, using batch size = 32, learning rate 5e-5 with a linear decay scheduler, warmup steps 500, weight decay = 0.01. We evaluate every 100 steps, and the checkpoint with the highest fl validation set is kept and used to predict final result.
Not Line Chart: This figure is not line chart.
Compound Figure: This figure a compound figure that contains multiple subfigures.
The resulting error counts are shown in Table 4. We also asked participants to annotate whether the caption contains any of the following six aspects (Yes/No):
For our fine-tuning LLaMA-2, we modeled as text generation task. The input includes informa- tion from figure- munureenntioning paragraphs and target captions, and the model outputs evaluation for each aspect in specific format (e.g., [helpfulness: yes]). The model was trained with fora_ranks 8, learn- ing rate 5e-5, batch size 16, and we trained the model for 50 epochs. We kept the last checkpoint for evaluation.
OCR (Optical Character Recognition): Does this caption mention any words or phrases that appear in the figure? (Examples include the figure title, X or Y axis titles, legends, names of models, methods, subjects, etc.)
Visual: Does this caption mention any visual characteristics of the figure? (Examples in- clude color, shape, direction, size. position, or opacity of any elements in the figure.)
Decoding Details for Open Large Language models. Parameter settings for Falcon-7b, Falcon-40b, and LLaMA 2-70b6 the same: do __ssamllee True, top_p 0.95, min _et__ne__to__ tokens 10, max_new_ jos__e__tokesss 200, repetition_ penallt 1.0, temperature = 0. num _eams = Note that all the models used in the experiment are the instruction-following model.
Stats: Does this caption mention any statistics or numbers from the figure? (For example. "20% of..." or "The value of 0.33.....)
Relation: Does this caption describe rela- tionship among two or more elements or sub- jects in the figure? (For example, "A is lower than B." "A is higher than B," or "A is the lowest/highest in the figure.")
We used allemna//ccbbertsscivcbbb.
"We used tiiuae/fallcom77-iinsttuu.... We used tiiuee/falcon-44bbinssrrtt.t WWe used meta-1llamm/Ll1mm=222766

---

D Prompts used for LLMs
question? Please answer Yes No one by one and explain why or why not. Do not repeat the question."
In this section, we provide the prompt we used in Section 4. [PARAGRAPHS] and [CAPTION] are placeholders for figure-mentioning paragraphs and the target caption.
Zero-shot prompting. "Given the paragraphs and caption below, please rate the level of useful- ness of the caption from I to 6 based on how well the caption could help readers understand the im- portant information. 6 is the highest; is the low- est. Please also explain your rating. Paragraphs: [PARAGRAPHS], Caption: [CAPTION]"
Few-shot prompting. "Given the paragraph and caption below, please rate the level of usefulness of the caption from 1 to 6 based on how well the caption could help readers understand the impor- tant information. 6 is the highest; 1 is the lowest. Please also explain your rating. The following are 3 examples of high-quality captions: Best-1, Best-2, Best-3. The following are 3 examples of low-quality captions:
Worst-l, Worst-2, Worst-3, Paragraphs: [PARAGRAPHS], Caption: [CAPTION]"
Chain-of- Thouught prompting. Here the prompts used for generating questions and obtain- ing answers for each question. We explore two types of questions, open-ended (QA) and yes/no questions (QA-YN), and prompts only difference including yes or no in question generation:
Open-ended Question Generation: "The following are paragraphs from paper that mentioned figure-inder Based on these para- graphs, please generate at most five questions that the caption of figure-index should be able to answer. These questions quite be interest- ing and useful to the readers of the paper, who are mostly researchers in domain and Al."
Yes/No Question Generation: "The follow- ing are paragraphs from a paper that men- tioned figure-index. Based on these para- graphs, please generate at most five yes or no questions that the caption of figure-index should be able to answer. These questions quite be interesting and useful to the readers of the paper, who are mostly researchers in domain and Al."
Answer:
"The following the caption of figuee-index. Does this caption answer each

---

Figure ID: 11gun7-1pp peg
Paper Now carn catoon Pryeical Activty large Arabyu eng Challangas Model Appl
Poper Abstreet: Gaveliction reprosents effecine nomriaa use behwvir nome compuring kuations Howeres despite the human pryeical scrvery itew compatiions phytical actityy
caertial healthy Hobyle tie ebout catin and pertosulr ccmpet shapp
thallange mobie
application where rusiipll
olke
prodefied cumber of systhrdre Fouding ovler
greatly plysica sign licantty Futhhrmons
fiighty usequai
coual mee wornen effective incrasiing acthiiees loveagge insigh develop
predict wehtthr compet taen particularly eagazing signifionn ecunaey Our modkk serve guiciline desigr moce engaging compettiions: and moce netidal beboviorsl shanges.
Your ID:
Total Number ber Captions: 10
Number Captions Brren:
Number Captions That Need Eankasted:
Your Prograss This
Pesdoos
Fgure 1.
When papar caption help understand the mensage ge that the figure tries to cammey..
these captions were generated computers which might cone errors
When obviqusly and almost
information in lnleomeed, it fire
When
0.32
83.00
24.00
24.00
Average daaly eriification diffcreet
figure
persentegeaa of make
onyour undestanding caption cuclity
Figuro
Meeely
Fiee
Foard
Fiuan
CDF of mumber of artice clister
Uenweb Deagent diver Coptoon
Figees 910
The
Previous
Fguee
0.32 $3.0 2400 24.00
CA.2
perticiuants
competitinn by acrmas gender diversity
Precdrus Caption
Neet Exprior
Figure 2: The interface used by the Ph.D. students to rank the captions.

---

Figure Captioning Annotation
In this HIT, you will see 10 figure-caption pairs about scientific articles. Read each figure and select the best answer for each question. Please spend at least 30 seconds on each one.
Please look example figure. Below example captions with sample questions you will asked about the pairing of the figure and various captions.
00:3
200
20.5
Mays
Does this caption mention any words or phrases that aapaaaaarinnn the figura? (Examples include the figure title, X Y axis titles, lagends, of models, methods, subjects,
caption:
Accunccy
Anower: caption contains words the figure tlighiigted in yellow).
Example caption: Fgure 6: There are theee lines grapholot
Answen No The
Does this caption mention any sisual characteristics of the figure? (Examples include color, shape, direction, size, opacity any
figure.)
Example captione Figure
line) jagged
figure.
Anower: The caption mentions color, shape and postion the Agure.
Example caption: Fiuure 6 The relaton between acouracy and rumber ways.
No The
Does this caption mention statistics or numbers from the figure? (For example, "20% of.." or "The value of
Example captions igure 40-way
sccurney.
Answer: les The caption mentions statitics in the fg.ur.
Example caption: Fgure 6 The like line (10 shott hos the highest accurccy.
No There
Does this caption deeccribe relationship amone two more elements or subblects in the fiqure? (For example, lowest/highest figure.")
higher D,"
Example caption: Figure
highest accuracy compared
Answer: Yea The caption desoriess relat onchip among dements in the figure.
Example caption: Figure 6. There theee line the grapholos.
amang
caption.
Peggns
Prov ous Caption
Nest Caption
Please read following caption and figure image carefully and answer the questions on side.
image caption
any
issues?
Image danaaed Irape
Text Exteette Eneer The extrcter hrom roe bhown anovid hay codoccanon 3. not cantainng complne ciptice contaiting wita not that not the caction nconect aod rocgretinn etc)
Not Line Cuart Sgare nst
Compoand Pigure: Syone compontd Eganc marias
1. Does this caption mention any wards or ahrases that appear the figurs? (Examples include the figure title, titles, legends, models, methods, subjects, etc.)
Plats the query time construction time tradeoff for Herrrrhisal delasck.
caption mention any visual characteristics
of the figure? (Examples include color, shape, direction, size, position, opecity elements in the figure) Yer O Mo O
Figure 3: The interface used by the undergraduate students to rate the captions.


Reviews:
[{"review_id": "DoY4PUNUk0", "reviewer": "Reviewer_EmJc", "summary": "", "questions": "Could you discuss how the appearance and data domain of the images impacted perceived utility more thoroughly? These references might be useful for the same:\nWhat Makes a Visualization Memorable? Borkin et al., 2016 (https://vcg.seas.harvard.edu/publications/what-makes-a-visualization-memorable/paper/)\nImage or Information? Examining the Nature and Impact of Visualization Perceptual Classification. Arunkumar et al., 2023. (https://arxiv.org/pdf/2307.10571.pdf)", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EmJc for pointing out these very interesting papers, which we will cite in our final draft.\n\n#### **[Attributes of Figures Used in the Paper]**\nSciCap [1] is made of over 416,000 line charts (133,543 are single-panel) (which were referred to as “graph plots” in [1]) extracted from arXiv papers in Computer Science (cs.*) and Machine Learning (stat.ML). In our paper, we further focused on SciCap’s single-panel figures in the NLP, CV, and HCI domains. Our impression is that most of these figures are 2-dimensional with axes and some text annotations like legends. These figures should be better characterized as “information” visualization than “image” [2]. We will add more details on our figures' common design elements and attributes in the final draft.\n\n#### **[Utility of Figures vs. Figure Captions]**\nOur paper focuses on the utility of varied caption text of figures rather than the utility of the figures' images. We have chosen to focus on captions as we believe captions provide the most feasible entry point for technologies to intervene and boost the utility of figures in scholarly articles. Captions, often standalone like the \\caption{} label in Latex, can be revised or replaced without altering the figure image. No raw chart data is strictly needed for improving captions. Using technology to customize or improve caption text is more achievable than altering figure images automatically, especially given the capabilities of LLMs.\n\nIn the long run, we agree that the utility of the captions can be considered jointly with the figure images &ndash;as a good caption would not save a poorly designed visualization&ndash; but we view this as a future direction and beyond this paper's scope.\n\n\n#### **[References]**\n[1] Hsu, T. Y., Giles, C. L., & Huang, T. H. (2021, November). SciCap: Generating Captions for Scientific Figures. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 3258-3264).\n\n[2] Arunkumar, A., Padilla, L., Bae, G. Y., & Bryan, C. (2023). Image or Information? Examining the Nature and Impact of Visualization Perceptual Classification. IEEE Vis 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper aims to solve an interesting problem, particularly the use of scientific figures also opens up avenues to fine-tune scientific figure generation. ", "reasons_to_reject": "The approach used is straightforward, however I would have liked more discussion on the attributes of the dataset used, particularly because in data visualization, it has been shown that the presence and absence of design elements, particularly narrative elements in charts heavily impact how the chart utility varies.", "questions_to_authors": "Could you discuss how the appearance and data domain of the images impacted perceived utility more thoroughly? These references might be useful for the same:\nWhat Makes a Visualization Memorable? Borkin et al., 2016 (https://vcg.seas.harvard.edu/publications/what-makes-a-visualization-memorable/paper/)\nImage or Information? Examining the Nature and Impact of Visualization Perceptual Classification. Arunkumar et al., 2023. (https://arxiv.org/pdf/2307.10571.pdf)", "paper_topic_and_main_contributions": "This paper investigates using large language models for evaluating figure captions. 600 scientific figures were used from SciCap and 6 captions for each figure were generated using different methods. Caption quality was then assessed in a user study with PhD students and undergraduate students. Evaluation of GPT-4, GPT-3.5, and SciBERT was done. GPT-4 prompting achieved the highest correlations with Ph.D. students’ assessments. However, undergraduate “usefulness” ratings did not correlate well with those of Ph.D. students. The authors discuss how this might arise from the difference in perceived utility. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "BHqpERP1nU", "reviewer": "Reviewer_Tt8o", "summary": "", "questions": "- How did you sample figures from SciCap? What criteria did you use? Was it just random?\n- Details for the models: What size of Falcon? Was this instruction-tuned?\n- You fine-tuned SciBERT, but why didn't you fine-tune Falcon? There are a lot of ways to fine-tune large models now, especially with tools like 8-bit and LoRA\n- Is there a reason you chose to score from 1 to 6? 6 seems like a rather arbitrary number. \n\nOther comments:\n- I think some of the questions asked to the undergraduate students (e.g. types of errors) are very interesting! It would be nice if you can also add some statistics to the Appendix so we can get an idea of what the dataset is like.\n- Any thoughts on how to evaluate caption factuality? I understand it's challenging but I think it's an interesting and important issue especially for scientific captions, I was just curious if you had any thoughts.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Tt8o for the valuable feedback, which we will incorporate into our final draft. We will add the references and also hire an editor for proofreading.\n\n#### **[Contribution and Limitations of Text-Only Approaches]**\nThe rationale behind a text-only approach is that figure captions (in arXiv papers) were recently found to be similar to summaries of figure-mentioning paragraphs [1], i.e., most words in captions can be semantically traced back to these paragraphs (Section 4 of [1]), captions can be effectively generated using text-summarization models (Section 6 of [1].) One missing piece from prior work is to exploit this interesting finding for evaluation, which we proposed in this paper. We will revise the paper to make this contribution more clear. \n\nWe are aware of the limitations of text-only approaches for a vision-and-language task, as stated in the Limitation section. We believe our proposed approach has value to the research community regardless of its limitations because it is very easy to use and gives strong results. Exploring the new uses of LLMs is also a pressing topic in the current NLP community. We will incorporate these considerations into the Limitation section.\n\n#### **[Different Labeling Schemes Between PhD and Undergraduate Students]**\nIn fact, we initially used the identical 6-item ranking task for both undergraduate and Ph.D. students. However, we soon found that comparing six different captions while comprehending the figure image and the paper content was too challenging for undergraduate students, who performed the task very slowly. After some design iterations, we eventually settled on the current 3-class labeling task for the undergrads.\n\nWe understand using different annotation tasks would raise concerns of comparability between Ph.D. and undergraduate students’ assessments, but this is the trade-off we are willing to make in order to meaningfully collect a broader set of undergraduate students' feedback. We will make this decision clear in the final draft.\n\n#### **[Questions (Q)]**\n\n\n#### **[Q1 - Figure Sampling]**\nYes, we randomly sampled 200 figures from the SciCap dataset’s CV, NLP, and HCI domain (according to the arXiv category,) respectively. \n\n#### **[Q2 - Details of the Models]**\n\n== **Scibert** ==\n\nWe finetune “allenai/scibert_scivocab_uncased” checkpoint from HuggingFace for 100 epochs, batch size = 32, learning rate = 5e-5, warmup steps = 500, weight decay = 0.01, we use the best f1 checkpoint on validation set to predict final result. \n\n== **Falcon & LLama2** ==\n\nGiven the limited resources, we report Falcon-7b zero-shot and few-shot results before the deadline. But we did try Falcon-40b and Llama2-70b afterward. Parameter settings for both Falcon-7b, Falcon-40b and Llama2-70b are the same. The only difference is the pretrained model. All the model used in the experiment is the instruction-following model.\n  \nFalcon-7b: tiiuae/falcon-7b-instruct\n\nFalcon-40b: tiiuae/falcon-40b-instruct\n\nLlama2-70b: meta-llama/Llama-2-70b-chat-hf\n\nParameter settings: do_sample=True, top_p=0.95, min_new_tokens=10, max_new_tokens=200, repetition_penalty=1.0, temperature=0.1, num_beams=1\n\n== **Correlation Scores** ==\n\n| Model      | Setting | (a) Corr. to PhD Reversed Rank, (6~1)      | (b) Corr. to PhD Original (1~⅙) | (c) Corr. to PhD Reversed (⅙~1) |\n|------------|-----------------------|---------------------------------------|:---------------------------------|:---------------------------------|\n|            |                       | Pear. &nbsp;&nbsp; Kendall. &nbsp;&nbsp; Spear.                              | Pear.                           | Pear.                           |\n| Falcon-40b | zero-shot             | .169 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .137 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .167    | .152                            | -.136                           |\n|            | few-shot              | .150  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  .119   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   .143                                | .126                            | -.137                           |\n| Llama2-70b | zero-shot             | .407  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .342  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;   .413                               | .326                            | -.392                           |\n|            | few-shot              | .424      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  .353  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  .430                         | .335                               | -.405                           |\n\n\n#### **[Q3 - Fine-tuning LLM]**\nWe do believe that fine-tuning LLMs presents a promising direction for improving evaluation performance in the future. However, our preliminary experiments with fine-tuning LLaMA-2 (both the 7B and 13B models) have not yielded encouraging results thus far.\n\nFor our fine-tuning process, we modeled it as a text generation task. The input includes information from figure-mentioning paragraphs and target captions, and the model outputs evaluation for each aspect in a specific format (e.g., [[helpfulness: yes]]). The model was trained with a lora_rank of 8, a learning rate of 5e-5, batch size of 16, and we trained the model for 50 epochs. We kept the last checkpoint for evaluation. The scores were as follows:\n\n| Model       | Training data | (a) Corr. to PhD Reversed Rank, (6~1)     | (b) Corr. to PhD Original (1~⅙) | (c) Corr. to PhD Reversed (⅙~1) |\n|-------------|-----------------------|---------------------------------------|---------------------------------|---------------------------------|\n|            |                       | Pear. &nbsp;&nbsp; Kendall. &nbsp;&nbsp; Spear.                              | Pear.                           | Pear.                           |\n| *llama2-7b  | Undergrad             | .164 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .144 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .164 | .162                            | -.138                           |\n| *llama2-13b | Undergrad             | .183 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .162 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  .183 | .138                            | -.193                           |\n\n*: finetune llama2 classifier performance was evaluated on the test split, comprising only 10% of the entire dataset.\n\nOur computation resources do not allow us to fine-tune LLaMA-2 70B as the training requires several hundreds of hours to run using LoRA and 8-bit quantization. While the initial results for LLaMA-2 7B and 13B were not as we expected, we still believe it is a promising direction to explore. We will add this to our future work.\n\n#### **[Q4 - Range of the Scores]**\nTo clarify, it was a ranking task with 6 items, where each Ph.D. student manually ranked 6 different captions for a figure.\n\n#### **[Other Comments (C)]**\n**[C1 - Statistics of the Annotation]**\n\nWe will release the student annotation dataset in the final version. For all 3,600 figure-caption pairs, there are 3,159 valid samples. The remaining 441 figure-captions contain at least one error. We give the detailed error analysis below. \n\n|       | ImageError | CaptionError | ClassificationError | SubfigureError |\n|-------|:------------:|:--------------:|:---------------------:|:----------------:|\n| Total | 102        | 242          | 101                 | 23             |\n\n**[C2 - Evaluating Caption Factuality]**\n\nWe believe that human evaluation is a promising way to check the caption factuality, but it’s extremely hard to find domain experts to do evaluation.\n\nAnother way to check the caption factuality is to align the information between captions and figures/paragraphs. For information whose source information can be successfully identified, its factuality can be guaranteed. Huang et al. [1] used awesome-alignment to extract information shared between captions and paragraphs, which we believe is also a promising way for evaluating factuality. However, the alignments between figures and captions remain unexplored. Although it might be possible to directly align the patches and captions (using the transformer-based visual models), we are not sure whether such small patches can capture concepts such as “the blue line outperforms the red line”. We will add this to our discussion and leave it for future work.\n\n#### **[References]**\n\n[1] Huang, C. Y., Hsu, T. Y., Rossi, R., Nenkova, A., Kim, S., Chan, G. Y. Y., ... & Huang, T. H. K. (2023). Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization. INLG 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A lot of papers right now are using GPT-4 as a ground truth oracle evaluator (rather than using human annotations), especially for generative tasks. There is a big question mark surrounding this practice because we don't fully know whether this is something GPT-4 is capable of performing well. This paper investigates that topic specifically for scientific captions. As this trend of \"oracle GPT-4\" continues, I think we need more papers to really show us what GPT-4 can or cannot do, and I think this paper would make a really nice addition to that list.\n\nEven though there are a few gaps in the paper/methodology, I think the main points are solid, and I don't doubt any of the conclusions. I think it makes sense that GPT-4 and GPT-3.5 would perform the best on this task, and it's nice to see it verified.", "reasons_to_reject": "The biggest elephant in the room is that the paper's \"captioning\" isn't really captioning -- it's more of summarizing. The models aren't evaluating a caption based on the figure. They're evaluating a caption based on some text, which can potentially miss a lot of information, especially if the text is not very descriptive. I also feel that using a text-only method somewhat takes away from the excitement and difficulty of the task since evaluating a caption based on a scientific figure is a lot more challenging than evaluating a one-sentence summary of a piece of text.\n\nComparisons in the main table (Table 1) aren't entirely fair. For instance, GPT4 was asked to rate models from 1 to 6, while the undergraduate students were asked to give qualitative discrete values (\"Yes\"/\"No\"/\"Unsure\"), so it is a bit difficult to compare. \n\nWriting is not very thorough. There are a bunch of missing citations, and a lot of the more specific experimental details (e.g. hyperparameters, model sizes) seem to be omitted. Some experimental choices don't seem particularly justified (see Questions below).", "questions_to_authors": "- How did you sample figures from SciCap? What criteria did you use? Was it just random?\n- Details for the models: What size of Falcon? Was this instruction-tuned?\n- You fine-tuned SciBERT, but why didn't you fine-tune Falcon? There are a lot of ways to fine-tune large models now, especially with tools like 8-bit and LoRA\n- Is there a reason you chose to score from 1 to 6? 6 seems like a rather arbitrary number. \n\nOther comments:\n- I think some of the questions asked to the undergraduate students (e.g. types of errors) are very interesting! It would be nice if you can also add some statistics to the Appendix so we can get an idea of what the dataset is like.\n- Any thoughts on how to evaluate caption factuality? I understand it's challenging but I think it's an interesting and important issue especially for scientific captions, I was just curious if you had any thoughts.", "paper_topic_and_main_contributions": "The paper proposes SciCap-Eval, which is a labelled dataset which contains scores for captions taken from SciCap (Hsu et al, 2021). SciCap-Eval contains both human and machine-generated scores for 3,600 scientific captions (ranging over 600 figures). The main issue SciCap-Eval is addressing is the difficulty of evaluating scientific captions, and they solve this by providing annotated scores. \n\nEach figure in SciCap-Eval comes with 6 captions (1 human ground truth and 5 machine-generated). These were then assigned the PhD students (who are experts in the field of the figures) to score the captions from 1 to 6. They treat these PhD captions as the new ground truth.\n\nThe authors then evaluate various captioning methods such as GPT-4, GPT-3, Falcon, SciBERT (fine-tuned), and undergraduate students. They measure the correlation of these scores with the scores given by the PhD students and conclude that GPT-4 (both zero-shot and few-shot) has the highest correlation, outperforming even fine-tuned models and undergraduate students.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Line 53 -- ROUGE score citation\n\nLine 131 -- Pegasus\n\nLine 135 -- TrOCR\n\nLine 195 -- Chain-of-thought", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 66 -- missing punctuation\n\nLine 127 -- \"provides\" --> \"providing\""}, "extra_text": {}}, {"review_id": "P7yEtnZzVQ", "reviewer": "Reviewer_whcN", "summary": "", "questions": "I also have the following questions for the authors:\n1. You hire Ph.D students from the expert domain according to the specific image, while undergraduate students only from CS department. I understand that undergraduate students may not have the specific research area. But I still think this setting might be a little unfair to achieve the final conclusion. In order the get the general conclusion, is it better to hire undergraduate students from diverse departments to achieve more general conclusion?\n2. There seems no consistent conclusion. The author claimed they explore cost-effective evaluation, but GPT4 is expensive to use. While for their own tuned, cost-free SciBERT, it seems the performance lays far behind GPT4. Is there any final conclusion for this paper to give that which evaluator achieves high performance and is really cost-effective?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer whcN for the valuable feedback and for recognizing our approach as an excellent caption evaluator.\n\n#### **[Contribution]**\nWe believe the proposed approach has value to the NLP community because it is easy to use and gives strong results. At a higher level, our work responds to the NLP community’s pressing research theme of exploring new uses of LLMs. Furthermore, our work responds to a set of newly-emerged findings: Figure captions were recently found to be similar to summaries of figure-mentioning paragraphs [1], i.e., most words in captions can be semantically traced back to these paragraphs (Section 4 of [1]), captions can be effectively generated using text-summarization models (Section 6 of [1].) One missing piece from prior work is to exploit this finding for evaluation, which we proposed in this paper. We will revise the paper to make this contribution more explicit.\n\nIn addition to the technical contribution, we also made artifact contributions by creating a new dataset, which will be released publicly upon publication. The recruiting and annotation process alone took about 4-5 months and cost over $3,300. Other researchers will be able to build new work on top of this dataset.\n\n#### **[Questions (Q)]**\n**[Q1 - Background of the Undergraduate Students]**:\n\nTo clarify, the undergraduate students we recruited were from several STEM majors, just that many of them were in CS or Informatics. Please see the table below for the detailed distribution.  \n\n| Major                                  | Number of Undergrad Students |\n|----------------------------------------|:------------------------------:|\n| Computer Science                       | 5                            |\n| Biochemistry                           | 1                            |\n| Biology                                | 1                            |\n| Data Science                           | 3                            |\n| Security and Risk Analysis             | 1                            |\n| Cybersecurity Analytics and Operations | 5                            |\n| Statistics                             | 1                            |\n| Enterprise Technology Integration      | 1                            |\n| Information Sciences and Technology    | 2                            |\n\n\n**[Q2 - Cost of GPT-4]**: \n\nBy “cost-effective,” we meant that our method, though not free like some automatic evaluation methods, remains more economical than human evaluations. Evaluating scientific figure captions requires participants to comprehend scientific papers, making it hard to recruit competent participants (e.g., can not use MTurk) and generally expensive to run human evaluations. In our work, we paid each undergraduate student \\\\$15 to label 30 figures (30 figure-caption pairs); we paid each Ph.D. student \\\\$20 per hour, and it took them 1.5 to 4 hours to rank 6 captions for each of 40 figures (40*6 = 240 figure-caption pairs). GPT-4, with OpenAI’s current price rate, costs \\\\$40&ndash;\\\\$50 to evaluate 3,600 figure-caption pairs in our study. While we agree that human evaluation is important and should not be ignored or replaced by automatic evaluations, GPT-4 offers an easy and affordable option for quick sanity checks, allowing practitioners to iterate their caption system quickly before having humans test it. \n\n#### **[References]**\n\n[1] Huang, C. Y., Hsu, T. Y., Rossi, R., Nenkova, A., Kim, S., Chan, G. Y. Y., ... & Huang, T. H. K. (2023). Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization. INLG 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proved that GPT4 serves as an excellent image caption evaluator even in zero-shot setting.", "reasons_to_reject": "There are no too much novel techniques in this paper, and the whole pipeline seems like a experimental report without any technical contribution. ", "questions_to_authors": "I also have the following questions for the authors:\n1. You hire Ph.D students from the expert domain according to the specific image, while undergraduate students only from CS department. I understand that undergraduate students may not have the specific research area. But I still think this setting might be a little unfair to achieve the final conclusion. In order the get the general conclusion, is it better to hire undergraduate students from diverse departments to achieve more general conclusion?\n2. There seems no consistent conclusion. The author claimed they explore cost-effective evaluation, but GPT4 is expensive to use. While for their own tuned, cost-free SciBERT, it seems the performance lays far behind GPT4. Is there any final conclusion for this paper to give that which evaluator achieves high performance and is really cost-effective?", "paper_topic_and_main_contributions": "The paper investigates using LLM to evaluate the quality of generated image caption. Specifically, the authors first collected a image caption dataset incorporating 3,600 <image, caption> pair from both human written and machine produced ones. Then they applied LLM to score the quality of caption, which is compared with the one from Ph.D. students and undergraduates, and found that GPT4 performs well as it achieves highest correlation score with Ph.D. students' assessments in all settings. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Bertscore: Evaluating text generation with bert. Zhang et al. 2019.\nBLEURT: Learning Robust Metrics for Text Generation. Sellam et al. 2020."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an investigation into the use of large language models (LLMs) as evaluators for figure captions, specifically scientific ones. Using the SCICAP-EVAL dataset, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures, the authors find that GPT-4, used as a zero-shot evaluator, outperforms all other models and even surpasses assessments made by computer science undergraduates. 

The reviewers have shared diverse views on the paper. Overall, the reviewers acknowledge the paper's merits in investigating an interesting problem, providing a new dataset (SCICAP-EVAL), and demonstrating the effectiveness of GPT-4 as a caption evaluator. The use of GPT-4 as an evaluator, especially in the zero-shot setting, is noteworthy. However, the reviewers also express concerns: One primary concern is the lack of novel techniques in the paper, and the overall approach is seen as more of an experimental report.  There's also concern about the methodology discussed in the paper on figure captioning, which could potentially be misleading as the model doesn't utilize visual input (i.e., the figure) to perform this task. Comparison between the GPT-4 scores and undergraduate students' evaluations is another area of concern due to differing scoring scales and a lack of detail about the experimental design and the rationale behind certain choices. The authors are urged to provide more clarity on these aspects.

In conclusion, while the paper provides valuable insights into the use of LLMs as evaluators for figure captions, there are points raised by the reviewers that need to be addressed, particularly in terms of technical contributions and experimental details. The paper could significantly benefit from another round of revision to address these concerns and questions. However, the reviewers are generally in agreement that the paper has merits and potential contributions to the filed.
Decision: Accept-Findings