Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
Authors: Andrea Sottana
Bin Liang
Kai Zou
Zheng Yuan
Venue: EMNLP 2023 Main (2023)
Keywords: evaluation
human evaluation
LLM
summarization
simplification
grammatical error correction
ChatGPT
GPT-4
Sequence to sequence
TL;DR: An evaluation study of Large Language Models using automatic metrics, human evaluation, and model-to-model evaluation with GPT-4.

Abstract: Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape, and it is becoming clear that the quality of automatic evaluation metrics is not keeping up with the pace of development of generative models. We aim to improve the understanding of current models' performance by providing a preliminary and hybrid evaluation on a range of open and closed-source generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction (GEC), using both automatic and human evaluation. We also explore the potential of the recently released GPT-4 to act as an evaluator. We find that ChatGPT consistently outperforms many other popular models according to human reviewers on the majority of metrics, while scoring much more poorly when using classic automatic evaluation metrics. We also find that human reviewers rate the gold reference as much worse than the best models' outputs, indicating the poor quality of many popular benchmarks. Finally, we find that GPT-4 is capable of ranking models' outputs in a way which aligns reasonably closely to human judgement despite task-specific variations, with a lower alignment in the GEC task.

Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
Andrea Sottana' Bin Liang' Kai Zou' AT
Zheng Yuan
Department of Informatics, King's College London
santaa, bin. binng, kz)@netmind.
mheng-yuan@kcl... ac.
Abstract
2022). Human evaluation has often been consid- ered as the trusted evaluation method, though is- sues with human evaluation have also been widely acknowledged (Iskender et al., 2021), e.g. it can be difficult to reproduce (Cumbicus-Pineda et al.. 2021). Nonetheless, human evaluation study re- mains one of the best tools to sensibly assess any bias limitation with automatic metries (Liang et al., 2022).
Large Language Models (LLMs) evaluation is patchy and inconsistent landscape, and it is be- coming clear that quality automatic eval- uation metrics is not keeping up with the pace of development of generative models. We aim improve the understanding of current mod- els' performance by providing preliminary and hybrid evaluation on range of open and closed-source generative L.MSs three NLP benchmarks: text summarisation, text simplifi- cation and grammatical error correction (GEC), using both automatic and human evaluation. We also explore the potential of the recently released GPT-4 to act as an evaluator. We find that ChatGPT consistently outperforms many other popular models according to human re- viewers on the majority of metrics, while scor- ing much more poorly when using classic auto- matic evaluation metrics. We also find that hu- reviewers rate the gold reference much worse than the best models' outputs, indicating the poor quality many popular benchmarks. Finally, we find that GPT-4 is capable of rank- ing models' outputs in a way which aligns rea- sonably closely to human judgement despite task-specific variations, with lower alignment in the GEC task.
Recent evaluation work has often focused on a single task (Zhang et al., 2023; Coyne et al., 2023), single model (Bang et al., 2023), single dataset (Gilardi al., 2023) automatic evaluation (Liang et al., 2022). In this work, carry multi- dataset, multi-model, multi-task hybrid evaluation using automatic metrics, human evaluation, and model-to-model evaluation with GPT-4 (OpenAI, 2023). We explore the open and closed-source LLMs space to sample the current landscape of available models and evaluate them on the fol- lowing sequence-to-sequence tasks, reframed text generation tasks without the requirement for task-specific fine-tuning: text summarisation, text simplification, and grammatical error correction (GEC).
These are main findings: firstly, we show how traditional reference- evaluation metrics are inadequate at predicting or replacing human judgement. It unclear whether this is due the limitations of the metrics or to the poor qual- ity of references of large open source datasets, or both. While automatic metrics might have been an adequate proxy to evaluate previous models, they seem unable to reliably capture the performance of latest-generation LLMs which now generate
1 Introduction
In recent years, Large Language Models (LLMs), particularly Transformer based (Vaswani et al., 2017; Devlin et al., 2019), have shown remarkable abilities across a wide range of NLP tasks. With the recent advances in capabilities of general-purpose generative models (Brown et al., 2020; Touvron et al., 2023), range of NLP tasks can be reformu- lated as generation tasks.
Robust evaluation is still unsolved prob- lem and established automatic evaluation metrics have been found be poor surrogates, correlat- ing weakly with human judgement (Coyne al., 2023). There is often no clear consensus on how these models should be evaluated (Mousavi et al.,
'When preparing the manuscript, the authors have noticed that some recent work has also explored model-to-model eval- uation, e.g. Chiang and Lee (2023); Liu et al. (2023); Fu et al. (2023). This paper makes a significant contribution by using an extensive set of metrics to provide comprchensive evalun- tion of cach model on several different scpuence-to-scquene tasks.

---

ceptable output that is significantly different from the gold reference. Secondly, we prove that even open-source models outperform the gold standard reference large and well-established datasets ac- cording to human evaluators. This shows how data quality is now one of the main bottlenecks in eval- uation research. Finally, we reveal how GPT-4 has reasonable alignment with human judgement when ranking different models most tasks and met- ries; we did however observe some variations, with lower alignment in some metries than in others. Our code available at https://gtt thub. com/
details, prompt engineering and hyper-parameter tuning, refer to appendix B.
Evaluation Metrics
We analysed models' outputs using both automatic metrics and human evaluation, and assessed the ability of the recently released GPT-4 model to act as reviewer.
3.1 Automatic Evaluation
We used the most widely adopted reference-bacte metrics for each of the tasks. For text simplification, report the SARI score (Xu et al., 2016). For text summarisation, report the ROUGE score (Lin, 2004); following Phang et al. (2022), we compute the geometric mean of ROUGE- (1, 2, L. F1 scores. For GEC, we report the Fo.s score computed using the ERRANT toolkit (Bryant et al., 2017).
2 Experimental Setup
2.1 Datasets
For text simplification, we used the Newsela test set (Xu et al., 2015), in particular the version used by Jiang et al. (2020). We randomly selected 3,000 samples after removing samples redundancy. For summarisation, experiments were rum on 3,000 random samples taken from the CNN MailyMail test set (Hermann et al., 2015; Nallapati et al., 2016). For GEC, we used the BEA-2019 Shared Task (Bryant et al., 2019) development set compris- ing of 4,384 samples.?
3.2 Human Evaluation
Due to budgetary and time constraints, we recruited human reviewers through the Prolific platform and asked them to review the quality of the mod- els' outputs, as well as the gold reference on 100 randomly selected samples per dataset. All three reviewers were asked to annotate the same 100 samples for each of the three tasks. The studies were conducted open-source POTATO annotation tool (Pei et al., 2022). For human evaluation of text summarisa- tion, we followed the evaluation criteria and their definitions as adopted in Fabbri et al. (2021): Rel- evance, Fluency, Coherence and Consistency, on 5-point Likert scale (Likert, 1932) from to 5. For text simplification, followed the evaluation criteria and their definitions adopted in Grabar and Saggion (2022): Semantics, Fluency and Sim- plicity, on 5-point Likert scale. For GEC, we adopted the Over-correction criterion from Fang et al. (2023) and introduced two new criteria: Se- mantics and Granmaticality. The definitions and assessment scales for these GEC criteria are de- tailed in Appendix C. The full set of instructions given to human reviewers for all tasks can be found in our GitHub repository linked above.
2.2 Models
customised version of the
All experiments were performed on zero-shot un- supervised basis, without any additional fine-tuning in-context learning, using range of open- source LLMs and OpenAl commercial models." We experimented with the HuggingFace imple- mentation of the following source models: (Chung et al., 2022); TOpp (bigscience/Topp) (Sanh et al.,
Flan-T5 ggogle//lann-t5-x
2021); OPT-IML (facebook/optiiml-mmx--- (lyer et al., 2022); Flan-UL2 (google/flan-u12) (Tay al., 2022). The OpenAl models used were GPT-3 (ext-davinci-0003) (Brown al., 2020); InstructGPT (davinci-instruct-oe (Ouyang et al., 2022) and ChatGPT 1ett-33.-turbo-0003 0301). For implementation
MMor details are given in Appendix
"We did not use the test set as was not fully disclosed. 'It is worth noting that some of the models used are already fine-tuned to follow instructions on wide range of NLP tasks, some of which include the tasks above.
We only accepted reviewers based in UK, with En- glish as their first language and who had at least Degree or Master's level education English Language or English Literature, and 100% Prolific approval rate with least 200 prior submissions. "htps://www.poolifi..ooo
http://huggingfacc.co
'The open-source models were run on local servers with up to 6 NVIDIA GeForce RTX 3090 GPUs each.
http:///ooppaaa.oo//hthcttg

---

Open source
Score (main subset)
Score (human eval. subset)
Task
Model
Temperature
TOpp
Yes
0.01
28.82
31.62
summarisation (ROUGE score)
GPT-3
No
24.22
27.19
ChatGPT
No
23.76
25.72
Simplification (SARI score)
Flan-T5
Yes
0.01
44.98
44.61
InstructGPT
No
44.79
43.25
ChatGPT
No
37.55
35.01
OPT-IML
Yes
0.01
39.05
44.97
GEC (Fo.5 score)
GPT-3
No
38.40
41.75
ChatGPT
No
0.2
39.54
37.97
Table 1: Automatic evaluation of the best are shown both on the main subset and the small subset used for human evaluation. iDue to the specifies of HuggingFace implementation. temperature of 0.0 cannot used, we therefore used a value of 0.01 for such cases.
model and two commercial models from OpenAL Results
3.3 GPT-4 as a Reviewer
4.2 Human and GPT-4 Evaluation Results
We used GPT-4 as an additional reviewer assess whether can be reliably deployed in place of hu- man reviewers. The definition of the evaluation criteria and their assessment scales were included in the GPT-4 prompt together with the input text for each sample." 10 GPT-4 was also asked to an- notate the same 100 samples that were shown to human reviewers for each of the three tasks. The full prompts given to GPT-4 for all tasks can also be found in our GitHub repository linked above.
Human reviewers and GPT-4 were shown 4 outputs per sample: the outputs from the models in Table and the gold standard, and were asked score each model's 's output on the metrics and scales described in section 3.2. We then converted their scores to rankings for each model and each reviewer from best (1) to worst (4) and took the average. 12 The rankings from human evaluation and GPT-4 evalu- ation (in brackets) shown in Table 2, alongside the interval Krippendorff a coefficient (Krippen- dorff, 2011) to express inter-annotator agreement. The raw scores and a more detailed set Krippen- dorff a coefficients based on individual annotator pairs are shown in Appendix E. As it can be clearly seen, there is generally very good inter-annotator agreement, with an average Krippendorff of 0.88 across all metrics, with the lowest being 0.62.
4 Results and Discussion
4.1 Automatic Evaluation Results
Results are shown in Table 1. order to allow a comparison between open-source and paid-for models' performance, for each task, we report the best open-source model and two commercial mod- els from OpenAI." For text summarisation, T0pp significantly outperformed GPT-3 and ChatGPT (with p 0.001). For text simplification, Flan- T5 and InstructGPT yield the best results, sig- nificantly outperforming ChatGPT (p < 0.001). For GEC, ChatGPT and OPT-IML perform best with a very similar distribution, significantly out- performing GPT-3 (p < 0.001).
On text summarisation, most reviewers scored ChatGPT as the best for Relevance and Fluency, and all reviewers scored ChatGPT as best model for Coherence and Consistency, while ChatGPT had a worse ROUGE score compared to other mod- els when using automatic evaluation (see Table 1). Interestingly, all human reviewers scored the gold reference summaries the worst all met- rics. This reveals the poor quality of reference sum- maries when compared most models' outputs, and therefore reference-based automatic metrics could produce unreliable results. It is therefore not surprising that ChatGPT outputs were ranked the worst by automatic metrics in text summarisation and simplification, but the best when using human evaluators.
We also observed that for each task, the same prompt seemed to perform best for all models and temperature settings, with only one exception, sug- gesting that the quality of prompts is almost model- invariant. See Appendix D for more details.
10 "Occasionally GPT-4 returned a score of 4.5, we con- verted 4.5 4 for evaluation purposes out 3,000 cases). More detailed results are in Appendix D.
Thiss is to remove subjectivity and individual differences as human reviewers and GPT-4 might employ different mark- ing criteria.

---

Average human annotator rankings (GPT-4 rankings in brackets)
RELEVANCE
FLUENCY
COHERENCE
CONSISTENCY
Summarisation
(a) 0.88, af 0.81)
(01 0.88, 02 0.82)
(0) 1.00, 02 = 0.91)
(a1 0.97, 0.86)
Gold reference
4.00 (3.00)
4,00 (3.00)
4,00 (3.00)
4,00 (3.00)
TOpp
3.00 (4.00)
3,00 (4.00)
3,00 (4.00)
2.83 (4.00)
GPT-3
1.67 (2.00)
1.67 (1.50)
2,00 (2.00)
2.17 (2.00)
ChatGPT
1.33 (1.00)
1.33 (1.50)
1.00 (1.00)
1.00 (1.00)
SEMANTICS
FLUENCY
SIMPLICITY
Simplification
(01
1.00, og = 0.722)
(01 = 1.00, 02 = 0.50)
(01 = 0.63, 02 = 0.63)
Gold reference
4.00 (4.00)
4.00 (1.50)
3.33 (2.00)
Flan-T5
1.00 (2.00)
3.00 (3.00)
3.33 (4.00)
InstrucIGPT
2.00 (3.00)
2.00 (4.00)
2.33 (3.00)
ChatGPT
3.00 (1.00)
1.00 (1.50)
1.00 (1.00)
SEMANTICS
GRAMMATICALITY
OVER CORERCORNNCT
GEC
(c11 = 0.88, 0.34)
(a1 = 1.00, 0q 0.83)
(e1 = 0.62, 02 = 0.58)
Gold reference
3.33 (2.00)
3.00 (2.50)
2.50 (1.00)
OPT-IMI.
1.00 (4.00)
4,00 (4.00)
1,00 (2.00)
GPT-3
2,00 (1.00)
2,00 (1.00)
3,00 (3.00)
ChatGPT
3.67 (3.00)
1.00 (2.50)
3,50 (4.00)
Table 2: Average human evaluation rankings per model, task and metrics, where 1.00 means best model and 4.00 means worst model. GPT-4 rankings in brackets. When two models were ranked the same, results are shown as average between lower and upper bound (e.g. two best models are shown as 1.50 each). 'an represents the interval Krippendorff coefficient based on the human annotators rankings, while ag includes GPT-4 rankings.
For text simplification, ChatGPT was rated the best model by all reviewers for Fluency and Sim- plicity, while was rated poorly for Semanties, with the best model being Flan-T5. We observed that this was due to Flan-T5 returning a lot of out- puts which were identical to the inputs, therefore the semantics was obviously fully preserved, but without any inherent text simplification. The gold standard was scored as worst according all re- viewers.
did not observe the systematic positional bias when using GPT-4 as a reviewer as reported by Wang et al. (2023). However, we postulate that averaging the scores across the samples and using rankings instead of absolute scores helped to dampen this effect. If we include GPT-4 evaluation, the average Krippendorff a is 0.70 across all metrics, with the lowest being 0.34.
Conclusion
We had substantially different results for GEC, where ChatGPT rated the best model by hu- man reviewers for Grammaticality (meaning all or most errors were fixed) but was rated as worst or second worst model for Semantics and Over- correction, for which the best model was OPT- IML. This underlines how ChatGPT tends to over- correct, and in doing so might add information to the sentence which were not originally present, which is consistent with recent findings (Fang et al., 2023; Wu et al., 2023). The gold reference was scored mostly as second worst on most metrics and by most reviewers.
Model evaluation is a topic which is attracting in- creasing interest from the community. Liang et al. (2022) have recently published an extensive eval- uation report on LLMs, however they mostly fo- cused on automatic evaluation. Prompted by the recent advances in generative capabilities of the latest LLMs, we conducted this study explore the drift between human judgement and automatic, reference-based evaluation of zero-shot model per- formance. We also explored model-to-model evalu- ation with GPT-4. The study was conducted using large, open source datasets often acting as bench- marks for their respective tasks.
For both text summarisation and simplification, GPT-4 used as reviewer produced surprisingly good results which correlate well, albeit not per- fectly, with human reviewers. We observed a stronger disagreement between human reviewers and GPT-4 in GEC. It is also worth noting that we
Our work reveals systematic misalignment be- tween reference-based automatic metrics and hu- man evaluation on range of generative tasks, high- lighting the inadequacy of the gold reference in the public NLP benchmarks. is not clear whether this misalignment is purely due to the limitations

---

of automatic metrics, or whether poor reference quality makes using any reference-based compar- ative metrics unreliable. Despite ChatGPT being rated one of the best models on most metrics by human reviewers, the best open-source LLMs also consistently outperformed the reference outputs. We also explored the potential of GPT-4 to act as reviewer and found it has strong correlation with human judgement for summarisation and simplifi- cation tasks, and moderate correlation for GEC.
settings we chose for human evaluation were not the most appropriate, which means the
study may have suffered from some bias in- directly introduced by using automatic met- rics for selection of outputs for the human evaluation study. This is further aggravated by traditional open source datasets only pre- senting one gold reference output per sample when multiple equally valid outputs could ex- ist, leading to unreliable scores; for example, two summaries of the same story can be both very good but contain few common bi-grams, leading to a poor ROUGE score when doing automatic evaluation.
Future work will look at improving the quality of prompts, providing few-shot in-context learning (Brown et al., 2020), or exploring the potential of chain-of thought prompting (Wei et al., 2022) in improving models' outputs. Given the misalign- ment mentioned above, extending human evalua- tion to larger datasets and to wider range of model settings will also be of particular future interest, so as to minimise the bias introduced when us- ing automatic metrics to select a subset for human evaluation. Finally, introducing multiple automatic evaluation metrics (e.g. reference-less) for each task might help deepen our understanding of the re- lation between such metrics and human judgement.
Given the wide variety of the text corpora on which most of the models we used were pre- trained on, is very likely that at least some of the models may have been trained on some of the open-source datasets we used to eval- uate them. While it is difficult to mitigate for this (for example OpenAl did not publish a list of datasets used to train their models). our results might have been affected by this, and using new unreleased datasets would have been preferable to reduce this bias. However, this was not possible due to the highly expen- sive and time consuming nature of the task of creating high quality large datasets from scratch, which is a well known issue across the research community.
Limitations
This paper suffers from the following limitations:
A limited amount of prompt tuning and prompt space investigation was carried out. Between 2 and 5 different prompts per task tried, therefore a more focused study on prompt engineering could potentially bring significant improvements, however this is stand-alone exploration topic, which we leave for future work.
While we did not use the same model for both inference and evaluation, we used GPT-4 for evaluation of all models, including the out- puts from CharGPT. Considering they belong the same family of OpenAl models, GPT-4 might have a bias for rating ChatGPT's out- puts higher than other models. However, our results were not able to validate or refute this, as human reviewers also rated ChatGPT out- puts as the best across most metrics.
We did not perform any in-context learning or chain-of-thought prompting, which have been shown to significantly improve the per- formance of generative models. As such, there may be margin for improving the quality of models' outputs, while the quality of gold references will remain unchanged until new datasets become available.
Due to time and budgetary constraints, we were only able to hire 3 reviewers (not includ- ing GPT-4), and asked reviewers to annotate 100 samples per dataset, which is small pro- portion of each dataset. Due to the small num- ber of reviewers and reviewed samples, the noise-to-signal ratio may affect the strength and generalisability of our findings. Further- more, using human evaluation as gold stan- dard is also prone to introducing bias. How-
We used automatic metries (SARI, ROUGE and Fo.s) determine the best combination of settings (model, prompt, temperature) for each task. However, since this study revealed poor correlation between human judgement and such metrics, we cannot exclude that the

---

ever, we found that in most cases all annota- tors agreed that the gold standard was worse than the best models' outputs, so we do be- lieve this is a valid conclusion, given how consistent was across different tasks and annotators.
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Christopher Bryant, Mariano Felice, Distein E. Ander- sen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NIPP for Building Educational Applications, pages 52-75, Florence, Italy. Association for Computational Lin- guistics.
Ethics Statement
Our work makes use of LLMs, and there are known concerns associated with such models (Bender et al., 2021), including data bias, toxicity of train- ing content or outputs, their environmental impact, the lack of explainability for their outputs, and the potential to replace human workers with resulting job losses. We did not perform any fine-tuning part of this project, and only used open-source datasets. Some of the OpenAl's models used are not open-source, and their overall impact on society only starting to become apparent. Over- all we believe this research does not increase the risk of harm caused by these models or datasets as we only explored their limitations and perfor- mance. We employed 3 human annotators through the Prolific platform for a 16-hour study. Reviewers were paid 1.320 per hour, not including Prolific's !3 We did not collect any personal informa- tion beyond demographic data provided by Prolific, including age, profession, gender amongst others. While Prolific does provide such data, we did not use them as screening criteria, and only adopted the screening criteria mentioned in section 3.2. All an- notators were provided with a detailed description of the study before committing to take part.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 793-805, Vancouver, Canada. Association for Computational Linguistics.
Cheng-Han Chiang and Hung-yi Lee, 2023. Can large language models be alternative human evalua- tions? In Proceedings of the 61st Annual Meeting of the Association for Compitational Linguistics ( (Vol- 1: Long Papers), pages 15607-15631, Toronto, Canada. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Fed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoc Le, and Jason Wei. 2022. Scaling instruction- eestruttionffintuueed language mod- els.
Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa. Michael Zock, and Kentaro Inui. 2023. Analyzing the performance of gpt-3.5 and gpt-4 in grammatical error correction. arXiv preprint arXiv:2303.1 1.4.22433...4.....
References
Oscar M Cumbicus- Pineda, Itziar Gouzalez- Dios, and Aitor Soroa. 2021. Linguistic capabilities for a checklist-based evaluation in automatic text simplifi- cation. CTTS@ SEPLN.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie. Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, al. 2023. A multi- task, multilingual, multimodal evaluation chatgpt on reasoning, ballucination, and interactivity, arXiv preprint arXi::2302.0444 04023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM confer- ence on fairness, accountability, and transparency, pages 610-623.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Alexander R Fabbri, Wojciech Kryšciński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
The base pay was f12 with 10% bonus if they completed the full 16-hour study, All 3 reviewers completed the study and were awarded this bonus.

---

Tao Fang. Shu Yang, Kaixin Lan, Derek F Wong, Jin- peng Hu, Lidia S Chao, and Yue Zhang. 2023. Is chatgpt highly fluent grammatical error correction system? a comprehensive evaluation. arXiv preprint arXiv:2304.0174.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nig evaluation using gpt-4 with better human align- ment. arXiv preprint Stiv:2333.663..
Seyed Mahed Mousavi, Gabriel Roccabruna, Michela Lorandi, Simone Caldarella, and Giuseppe Riccardi. 2022. Evaluation of response generation models: Shouldn't 't it be shareable and replicable? In Pro- ceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 136 166-147,20 Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166
Fabrizio Gilardi, Meysam Alizadeh, and Mael Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint ar arXiv:2303.1505:
Natalia Grabar and Horacio Saggion. 2022. Evalua- tion of automatic text simplification: Where are we now, where should we go from here In Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume I: conférence principale. pages 453-463.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulşehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequenee RNNs and beyond. In Proceedings of the 20th SIGNLL Conference Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015, Teaching machines to read and comprehend. In Proceedings of the 28th Interna- tional Conference on Neural Information Processing Systems- - Volume I, NIPS' 15, page 1693-1701, Cam- bridge, MA, USA. MIT Press.
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27144.
Neslihan Iskender. Tim Polzehl, and Sebastian Möller. 2021. Reliability of human evaluation for text sum- marization: Lessons learned and challenges ahead. In Proceedings of the Workshop on Human Evalua- tion of Systems ( (HumEval), pages 86-96, Online. Association for Computational Linguistics.
Jiaxin Pei, Aparna Ananthasubramaniam,, Xingyao Wang. Naitian Zhou, Apostolos Dedeloudis, Jack- son Sargent, and David Jurgens. 2022. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lanuage Processing: System Demonstrations.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang. Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization.
Jason Phang. Yao Zhao, and Peter J Liu. 2022. Investigating efficiently extending transformers for long input summarization. arXiv preprint arXiv:220..04344
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong. and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960. On- line. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shamya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandcy, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan. Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multi- task prompted training enables zero-shot task gener- alization.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wa, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint 221::2211009111111
Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology, 140:5-55.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint 031.v.2205.05131.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 113977
Source: Ron Bee, a professor at San Diego State University. is worried that so few Americans serve in the military. Destinaaiio: Ron Bee is professor in California and he is worried.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Original dataset. sample 2
Source: Ron Bee a professor at San Diego State University, is worried that so few Americans serve in the military. Destination: Very few people join the mil- itary now.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunho Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 130.92266..
Our merged sample
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, al. 2022. Thain-of-thougll prompting elicits soning in large language models. Advances Neural Information Processing Systems, 35:24824-24833
Source: Ron Bee a professor at San Diego State University, is worried that so few Americans serve in the military.
Destination: Ron Bee is a professor California, and he is worried. Very few people join the military now.
Haoran Wu, Wenxuan Wang. Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint ar33v:2303. 2.t.::233.1364444 13648.
Implementation Details
Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 3:283-297.
Due to time and budgetary constraints. the full scale experiments
performed using the most promising settings after preliminary study con- ducted on a subset of each dataset (which consists of 100 samples) on much broader range of set- tings. We experimented with a range prompts and temperature values to better explore the ca- pabilities of each model. The final settings are task dependent; for example, we empirically ob- served that lower temperature values always gave the best outcomes for text summarisation and sim- plification, whereas for GEC it was beneficial use higher values for some models.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Gallison-Burche 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401 415.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Perey Liang, Kathleen McKcown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301. 13848.
138.48
A Newsela Dataset Processing
B.1 Prompt Engineering
We observed that the ACL 2020 version (Jiang al., 2020) of the Newsela dataset (Xu et al., 2015) contains number of samples where either the source (input) or the destination (reference) were duplicated. In such cases, based on our ob- servations, was appropriate to merge them into a single sample. If the source was duplicate but the destination wasn't, I, we kept the source without duplication, and created the destination by merging the two original destination samples, in the order in which they appear in the dataset. Likewise if the destination was duplicate but the source wasn't See example below
The following prompts were used, where In indi- cates a newline and [...] indicates the input sam- ple; for each of the three tasks, we report the best prompt, i.e. the prompt whose output was used for our evaluation work, at the top (prompt (a)). The same prompt yielded best results regardless of model and temperature, with extremely limited exceptions.
1. Text summarisation
(a) Summarize the following text.
[...] An The summary is:
(b) [...] An Summarize the text above.
Original dataset, sample

---

(c) Summarize the following text.
temperatures of 0.2 0.5, despite the best overall combination being at a temperature of 0.0 even for GEC. For all other hyper-parameters, we used the default settings for each model without modifica- tions.
[...] An The very short summary
is:
(d) This is the main story: [...] \n The summarized version of the story is:
B.3 Tokenization and Truncation
2. Text simplification
While the Newsela and BEA-2019 dataset samples are all below 512 tokens, 14 the samples from CNN DailyMail have broader distribution, with 80.6% exceeding 482 tokens and 9.8% exceeding 1506 kens. Different models and implementations have different maximum sequence lengths. Furthermore, while OpenAl models count the total number input and output tokens towards their maximum sequence length, HuggingFace models have two separate limits for input and output tokens respec- tively. In order to facilitate the inference process, used the following heuristics tailor different design decisions each model to try to maximise performance:
(a) Simplify the following text.
[...] An The simplified version
is:
(b) This is the main story: [...]Nn The simplified version of the story is:
(c) Simplify the following text. [...]
(d) Explain this to 5 year old. [...]
(e) Explain this to 5 year old. [...] An The explanation to year old could be:
For GPT-3, which accepts up to 4000 com- bined input and output tokens, we did not per- form any truncation, as the longest sample had 2,571 tokens.
3. Grammatical Error Correction
(a) Reply with a corrected version of the input sentence with all grammatical and spelling errors fixed. If there are no errors, reply with a copy of the original sentence. AnAn Input sentence: [...] An Corrected sentence: (b) Correct the following to standard English: \nAn Sentence: [...] An Correction:
For InstructGPT, which accepts up to 2049 combined input and output tokens, we trun- cated the input after 1506 tokens. This leaves 512 tokens for the generated output, as well as a further 31 tokens for the prompt (it is imper- ative not truncate the portion of the prompt the end of the input)
For HuggingFace models accepting inputs up 512 tokens (excluding the output), trun- cated at 482 tokens leave space for the prompt; for HuggingFacc models accepting inputs up to 2048 tokens we truncated 2018 tokens.
When using GPT-4 as reviewer, we prompted GPT-4 to output the text following strict json for- mat rules its output could be processed scale programmatically. When it failed do so, re- run the evaluation on that specific sample until the output was in the desired format, which happened mostly at the first attempt and occasionally after 2-3 attempts as GPT-4 output is non-deterministic.
C Human Evaluation Criteria for GEC
The criteria and their definitions and assessment scales given to reviewers for the GEC task are re- ported below.
B.2 Hyperparameter Tuning
We experimented with the following temperature values: 0.0 (we used 0.01 for HuggingFace mod- els due implementation requirements), 0.2, 0.5, 0.7. We observed that for text simplification and summarisation, the lowest value always yielded the best results, whereas for GEC, some combinations models and prompts yielded better results for
Semantics. This assesses whether the mean- ing the text is preserved following the GEC, Semantic preservation is assessed on a 5-point
"We used text-davinci-eee as tokenizer with the tiktoken python library, however we observed negligihle differences when using HuggingFace tokenizers.

---

Likert scale from (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalise corrections which change the meaning unnecessarily. For example, the sentence wentt at Rome for my birthday" should be corrected to "I went to Rome for my birthday". A correction such as "I went to Rome for my anniversary" should be pe- nalised in this category as they introduce necessary changes the meaning.
meaning of the sentence, but it nonetheless repre- sents a clear case of over-correction as "wentt at" should have been corrected "went to" instead of "decided to go to". As such we felt there would be value in keeping these two metrics separate.
D Detailed Automatic Evaluation Results
Table 3 shows the average results of the experi- ments we run on the summarisation dataset, for each model, temperature and prompt. Refer to Appendix B.1 for prompt details. Table shows the average results of the experiments we run on the simplification dataset. Table 5 shows the aver- age results of the experiments we run on the GEC dataset.
Grammaticality. This assesses the quality of the correction and answers the question "How many errors are left in the corrected sentence?". Please provide a count of the remaining errors, regardless of whether they were present in the source or they were newly introduced errors in the supposed corrected version. The three options "0", "1", "2 more".
Summ. Prompt
ROUGE score
Model
Temp.
ChatGPT
23.76
24.22
23.28
GPT-3
23.24
Over-correction. Since there can be multi- ple ways to correct sentence, this assesses whether the correction is unnecessarily ver- bose or makes unnecessary syntax changes. The best correction should be done with the minimum number of edits. For exam- ple, if the sentence "I wentt at Rome for my birthday" is corrected to "I decided to go to Rome for my birthday" this should be penalised under this category because contains unnecessary syntax changes, even though the final sentence grammatically cor- rect. This metric answers the question: the system over-correcting or making unneces- sary syntax changes? The answers should be "No", "Minor over-correction", "Mod- erate over-correction" "Substantial over- correction".
0.7
lb
22.46
18.60
InstructGPT
la
19.60
0.7
1b
19.04
28.82
0.01
28 28.80
TOpp
la
26.31
lb
26.19
23.77
0.01
18.61
Flan-UL.2
la
21.83
lb
17.25
Table 3: Detailed automatic evaluation results the text summarisation task.
Note that a correction which results in a change of meaning will most likely also an over-correction. Therefore we expect that if a correction is given a poor score in the Semantics category, it will also receive
score in the Over-correction cate- gory, and as such there may be some overlap be- tween these two metries. However, the reverse is necessarily true, as you could easily have over-correction without a change of meaning. For example, correcting a sentence from "I wentt at Rome for my birthday" to decided to go to Rome for my birthday" doesn't significantly affect the

---

Simp. Prompt
SARI
Model
Temp.
ChatGPT
37.55
GEC Prompt
Fo.5
2a
36.03
Model
Temp.
score
36.60
3a
39.48
GPT-3
0.5
36.47
35.81
ChatGPT
3b
22.47
3a
39.54
Za
35.73
0.2
3b
22.38
2b
36.42
3a 38.40
44.79
3b
33.39
43.44
3a
38.31
InstructGPT
2a
43.51
GPT-3
0.2
3h
33.28
2h
42.53
3a
37.86
0.7
42.63
0.5
3b
32.22
40.84
3a
40.44
0.01
2a
41.33
0
3h
38.92
2h
38.21
3a 40.00
OPT-IMI.
0.5
40.79
InstructGPPT
0.2
3b
37.56
37.99
3a
36.77
0.7
2a
40.45
0.5
3b
31.52
2b
37.52
3a
39.05
0.01
44.98
0.01
3h
36.04
38.11
3a
38.36
Flan-TS
36.97 2b
2a
43.98
OPT-IML
0.3
3b
35.52
35.84
0.7
42.79
0.6
3b
33.70
35.59
3a
29.89
0.01
2a
43.87
0.9
3b
27.38
2b
40.63
3a
19.17
TOpp
0.5
42.48
0.01
3b
12.84
39.59
3a
20.64
0.7
2b
41.00
38.94
Flan-TS
0.3
3b
13.83
3a
22.33
43.43
0.6
0.01
3h
15.54
35.75
3a
20.62
Flan-U112
0.5
2a
42.11
0.9
3b
15.40
2h
34.87
0.7
2a
41.00
Table Detailed automatic evaluation results on the GEC task.
2b
33.94
Table 4: Detailed automatic evaluation results on the text simplification task.

---

E Detailed Human and GPT-4 Evaluation Results
Table 6 shows the average scores and standard deviation for each human reviewer and GPT-4 for each task, metric and model across the three 100-sample subsets. Table 7 shows the same data by ranking, instead of absolute scores. It also shows the interval Krippendorff coefficient expressing inicr-annotator agreement for all annotator pairs, including each annotator and GPT-4.
Std. dev. annot.
Std. dev. annot.
Std. annot.
Std. dew. GPT-4
Avg. annot. 3
Avg. unnon
Avg. annot,
Avg. GPT-4
Metric
Model
Gold reference
3.06
0.97
3.57
1.11
3.89
1.18
4.46
0.61
TÜpp
3.44
0.98
3.89
0.87
4,08
1.02
3.94
0.54
RELEVANCE
GPT-3
4.61
0.68
4.47
0.68
4.67
0.60
4.68
0.47
CharGPT
4.91
0.38
4.79
0.43
4.59
0.63
4.84
0.37
Text sunmarisation
Gold reference
2.24
1.14
3,40
0.98
1.87
0.69
4.98
0.14
TOpp
3.44
1.20
4.66
0.78
4.30
1.28
4.93
0.26
FLUENCY
GPT-3
4.62
0.70
4.92
0.44
4.84
0.48
5.00
0.00
ChatGPT
4.85
0.52
4.96
0.24
4.82
0.55
5.00
0.00
Gold reference
1.43
3.64
1.32
3.51
4.50
TOpp
3.90
1.18
0.88
4.20
1.18
3.72
COHERENCE
GPT-3
0.53
0.43
4.78
0.67
4.52
ChatGPT
4.90
0.41
4.87
0.36
4.87
0.48
4.81
0.39
Gold reference
4.48
0.87
3.73
1.15
4.05
1.37
4.74
0.46
Topp
4.90
0.41
4.38
0.85
4.83
0.69
4.21
0.57
CONSISTENCY
GPT-3
4.96
0.20
4.68
0.61
4.83
0.65
4.76
0.43
ChatGPT
4.98
0.20
4.76
0.45
4.89
0.44
4.88
0.32
Gold reference
2.78
1.31
1.31
1.91
1.43
4.03
Flat-TS
4.97
0.17
0.53
4.83
4.55
SEMANTICS
InstructGPT
4.78
0.72
0.60
4.67
0.94
4.48
Text Simplification
CharGPT
4.42
0.83
4.58
0.85
3.98
1,30
4.66
0.57
Gold reference
3.70
0.69
3.67
1.33
2.10
0.70
5.00
0.00
Flat-T5
4.64
0.57
4.64
0.79
3.80
1.25
4.99
0.10
FLUENCY
InstructGPT
4.76
0.74
4.80
0.68
4.33
0.87
4.95
0.41
ChatGPT
4.77
0.65
4.93
0.32
4.47
0.92
5.00
0.00
Gold reference
3.10
1.05
3.81
1.17
3.35
1.28
3.91
0.60
Flat-T5
3.39
0.63
3.89
0.79
3,09
0.35
3.19
1.21
SIMPLICITY
InstructGPT
3.41
0.72
4.04
0.79
3.14
0.58
3.27
1.16
ChatGPT
4.52
0.70
4.69
0.70
4.15
1.06
4.25
0.62
Grammatical Error Correction
Gold reference
4.91
0.38
4.74
0.59
4.77
0.66
4.86
0.51
OPT-IML
5.00
0.00
4.94
0.37
4,97
0.22
4.37
0.73
SEMANTICS
GPT-3
4.95
0.26
4.75
0.65
4.88
0.41
4.88
0.35
ChatGPT
4.89
0.58
4.68
0.86
4.79
0.74
4.79
0.73
Gold reference
0.64
0.77
0.38
0.61
0.42
0.72
0.12
0.38
OPT-IML
1.00
0.76
0.58
0.67
0.83
0.85
0.62
0.58
GRAMMATICALITYI
GPT-3
0.54
0.71
0.18
0.41
0.26
0.56
0.04
0.24
ChatGPT
0.47
0.67
0.15
0.38
0.24
0.55
0.12
0.41
Gold reference
0.05
0.30
0.45
0.75
0.40
0.79
0.00
0.00
OPT-IMI.
0.00
0.00
0.14
0.55
0.01
0.10
0.12
0.35
OVERCORRECTION
GPT-3
0.04
0.28
0.56
0.86
0.46
0.84
0.18
0.41
CharGPT
0.06
0.34
0.61
1.01
0.40
0.82
0.19
0.52
Table 6: Average and standard deviation of the scores given by each human annotator and GPT-4 per model, task and metrics, across analysed samples (100 per task). iAll metrics on Likert scale (worst) to (best) except Grammaticality scale (best) (worst) and Over-Corection scale (best) to (worst).

---

Kripp. annot. and GPT-4
Kripp annot, and GPT-4
Kripp a annot, and GPT-4
Kripp- a annot. lad 2
Kripp. a annot. land 3
Kripp. annot. 2 2nnd
Avg. annot. (rank)
Avg. annot. 2 (rank)
Avg. annot. 3 (rank)
Avg. GPT4 (rank)
Metric
Model
Gold reference
TOpp
RELEVANCE
1.00
0.83
0.83
0.83
0.83
0.65
GPT-3
ChatGPT
summarisation
Gold reference
4
4
4
3
TÜpp
4
FLUENCY
GPT-3
2
1.5
1.00
0.83
0.83
0.77
0.77
0.77
CharGPT
1.5
Gold reference
TOpp
3
3
3
COHERENCE
1.00
1.00
1.00
0.83
0.83
0.83
GPT-3
2
2
2
2
ChatGPT
Gold reference
TOpp
2.5
CONSISTENCY
1.00
0.95
0.95
0.83
0.83
0.68
GPT-3
ChatGPT
Gold reference
Flan-T5
2
SEMANTICS
1.00
1.00
1.00
0.48
0.48
0.48
InstructGPT
2
2
3
Text Simplification
ChatGPT
3
3
3
Gold reference
Flan-TS
FLUENCY
1.00
1.00
1.00
0.03
0.03
0.03
InstructGPT
ChatGPT
1.5
Gold reference
4
4
Flan-T5
SIMPLICITY
1.00
0.48
0.48
0.48
0.48
1.00
InstructGPT
ChaiGPT
Grammatical Error Correction
Gold reference OPT-IML
SEMANTICS
1.00
0.83
0.83
-0.05
-0.05
-0.23
GPT-3
ChaiGPT
3
Gold reference
2.5
OPT-IML
4
4
4
GRAMMATICALITY
1.00
1.00
1.00
0.68
0.68
0.68
GPT-3
2
ChatGPT
2.5
Gold reference
3
2
2.5
OPT-IML
2
OVERCORRECTION
0.83
0.40
0.48
0.83
GPT-3
2
3
4
ChaiGPT 4
4
2.5
Table 7: Average rankings given by each human annotator and GPT-4 per model, task and metrics, where means best model and 4 means worst model. Results are based on averaged scores across all analysed samples (100 per task). The 6 columns represent for all annotator pairs, including each annotator and GPT-4. The Krippendorff a coefficients and metric, but aggregated across all models. Krippendorff a can also be computed with more than two annotators. For inter-anmotate agreements based on all human annotators combined, with and without GPT-4 annotations, see Table 2.
interval Krippendorff a coefficients expressing inter-annotator agreement
shown task


Reviews:
[{"review_id": "z0Lkiuth6P", "reviewer": "Reviewer_EMpQ", "summary": "", "questions": "- One thing that was not clear from my reading of the paper is if all the annotators (including GPT) were given the same set of results to evaluate, or if it was different subsets of the test data. I understand the outputs were normalized to fit a ranking from 1 to 4, but did they all annotate the same results? In that case, why did you use Krippendorff alpha instead of Fleiss kappa, that would be a good fit for many raters labeling the same data?\n\n- It is not clear why you did not include in Table 1 the automatic scores for every evaluated open-source model. I think it is related to the outputs taken into consideration for human evaluation (Table 2). Can you clarify that please?\n\n- You mention that GPT-4 might be evaluating input-output pairs that were used for training it. Did you consider the possibility of designing a few new test examples for each dataset, in order to tackle this problem and obtain a stronger conclusion about this?\n\n- In the ethics statement you consider that this research does not increase the risk of harm caused by these models. However, some people might find the conclusion of using GPT-4 as an evaluator a bit risky. What are your thoughts about systematic LLM-based evaluations are performed for new models in the future?\n\n- As far as I understand, the \"Semantics\" category used in GEC human evaluation is introduced by you. However it seems to overlap with the \"Over-correction\" criterion, as described in appendix C using \"I wentt at Rome for my Birthday\" as an example. This can be a deeper problem when asking annotators to distinguish between the two criteria, making the annotation even harder. Can you clarify the differences between the two?\n\nEDIT: The authors clarified these questions, some of these points should be included in the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1 for the useful comments.  Please see below our responses to their main concerns and questions.\n\n- **Reason to Reject 1**: Regarding the lack of reported agreement using annotator pairs (annotator1 and annotator 2, annotator 2 and GPT-4, etc.), we note that we added very detailed annotator-by-annotator results in Table 6 within Appendix E. We weren’t able to include these in the main corpus due to page limitations. While the Krippendorff scores are not reported in Table 6, they can easily be calculated using the results from the Table. Moreover, the Krippendorff scores in Table 2 are split into `alpha_1` and `alpha_2`, where `alpha_1` includes the three annotators and `alpha_2` includes the three annotators and GPT4, so we can already clearly see that the three annotators have a higher agreement with each other than with GPT4 overall; however we are very happy to include annotator pairs Krippendorff coefficients within Appendix E for the camera-ready version. Regarding us not reporting the agreements based on the range of human scores (e.g. what is the agreement between annotator 1 an GPT 4 when annotator 1 gives a 2 score vs when gives a 5 score), we felt that this was outside the main scope of the paper. Moreover, since we only analysed 100 samples using human reviewers and GPT-4, we felt it was not a large enough sample size to provide meaningful insights when looked at such a granular level of detail; for example, a reviewer might consistently give a score of 4 for a standard that a different reviewer considers worth of a score of 5, and as such we felt that looking at the correlation between annotators’ average rankings for each model was more relevant than focusing on individual score-by-score agreement.\n- **Reason to Reject 2**: The reviewer raised another valid point regarding human evaluators not being able to evaluate outputs from models that were discarded in the first evaluation round, potentially introducing bias. We do understand this concern, and as human evaluation is very time intensive and costly, we had to narrow down the scope of this experiment; however, we would like to reassure the reviewer that we (the authors) manually looked at these models’ outputs ourselves before making a decision on whether it was sensible to use automatic metrics for selection of models to be evaluated during human evaluation, and came to the conclusion this was the best course of action.\n- **Question 1**: We can confirm that all three human annotators and GPT-4 annotated the exact same set of 100 samples for each of the three tasks. We apologise for the confusion and will make that clearer in the camera-ready version. While both Fleiss kappa and Krippendorff alpha are suitable for handling many (>2) raters, we chose to use Krippendorff alpha for a number of reasons: 1) Krippendorff seems to be reported more often in similar literature, and in particular we decided to follow (Fabbri et al., TACL 2021)[1] in reporting Krippendorff alpha, 2) while Fleiss kappa can be applied to ordinal (ranked) data, we found it was more commonly used when assessing binary or categorical ratings, 3) Krippendorff alpha is more general and also suited to handle missing data, unlike Fleiss kappa (we didn’t have missing data in our case, but this may explain why we saw it reported more frequently across the literature we reviewed). In the end, both Fleiss kappa and Krippendorff alpha would be suitable and sensible coefficients to use in this case, and we just made a decision to use the latter.\n\n[1] Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9:391–409.\n- **Question 2**: This was mostly due to space constraints, as we experimented with multiple prompts and temperature values for each model and only reported the best combination for each in Table 1, reporting all combinations would have taken up the whole page, and, as the reviewer correctly identified, consistency with Table 2 was another reason for this choice, as we only asked human reviewers to review a subset of the models based on automatic metric results, due to time and budgetary constraints. However, we do report the full set of automatic scores for every evaluated open-source and closed-source model in Tables 3, 4 and 5 within Appendix D.\n- **Question 3**: This is a good and relevant question. We didn’t create new data for evaluation as we made the decision to stick to open-source widely used datasets. We also felt that only adding a few samples created by us might not lead to strong confidence in our conclusions, and might also affect reproducibility. Moreover, generating a substantial amount of high-quality new samples for evaluation (at least 100 input-output pairs per task) would be very time and cost intensive and therefore outside the scope of this work. However, it is certainly an interesting proposition which we could leave for future research work.\n- **Question 4**: LLM-based evaluations certainly have the potential of reducing the need for human evaluators, so this is a very valid observation. However, at the moment traditional automatic evaluation metrics are still widely used (and they also don’t require human evaluators), and as we’ve shown, they’re clearly inadequate at evaluating the latest generation of LLMs. Therefore there is a desperate need for better evaluation strategies. Considering the costs of employing human evaluators, it’s very unlikely that they will ever entirely replace automatic evaluation, and as such there is a strong case to be made in exploring LLMs to replace automatic rule-based evaluation metrics. We also believe the market will ultimately and inevitably adjust to any new technology as it’s been the case so far, and if LLMs can reach human-level judgement in the future (which is still not yet the case), then evaluators’ time could possibly be freed for more complex and engaging tasks.\n- **Question 5**: We agree that an over-correction could lead to the meaning (semantics) being changed. As such, there could be some overlap between these two metrics, and you could reasonably argue that most corrections leading to a change of meaning, are also over-corrections. However, the reverse is not necessarily the case, for example, changing a sentence from “I wentt at Rome for my birthday” to “I decided to go to Rome for my birthday” doesn’t significantly affect the meaning of the sentence, but it nonetheless represents a clear case of over-correction. As such we felt there would be value in keeping them separate to get a better insight into whether and how frequently over-corrections change the meaning of the sentence or not. We are more than happy to clarify this difference in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting approach to reflect about LLM evaluation. NLP practitioners might find aspects of this work useful, since the evaluation dilemma is becoming stronger on a daily basis.\n\nThis work can also trigger interesting discussions, for example about the usage of LLMs to evaluate other NLP models or systems or how they perform in relation to automatic metrics or human evaluators.", "reasons_to_reject": "Although this work is really interesting for the community, it could be much better with a deeper analysis of the presented experiments. For example, one of the objectives of this work is to evaluate how GPT-4 correlates with human evaluators. However, the agreement is not reported using pairs (annotator1 and annotator 2, annotator 2 and GPT-4, etc.) but aggregating all the annotators hence not allowing further discussion. Since they note that GPT-4 does not correlate perfectly with human evaluators, the agreement can also be analyzed depending on the range of human scores (e.g. what is the agreement between annotator 1 an GPT 4 when annotator 1 gives a 2 score vs when gives a 5 score).\n\nFurthermore, as the authors indicate, there could be an important bias regarding model selection based on automatic metrics, which means that human evaluators were not able to evaluate outputs from models that were discarded in the first evaluation round.", "questions_to_authors": "- One thing that was not clear from my reading of the paper is if all the annotators (including GPT) were given the same set of results to evaluate, or if it was different subsets of the test data. I understand the outputs were normalized to fit a ranking from 1 to 4, but did they all annotate the same results? In that case, why did you use Krippendorff alpha instead of Fleiss kappa, that would be a good fit for many raters labeling the same data?\n\n- It is not clear why you did not include in Table 1 the automatic scores for every evaluated open-source model. I think it is related to the outputs taken into consideration for human evaluation (Table 2). Can you clarify that please?\n\n- You mention that GPT-4 might be evaluating input-output pairs that were used for training it. Did you consider the possibility of designing a few new test examples for each dataset, in order to tackle this problem and obtain a stronger conclusion about this?\n\n- In the ethics statement you consider that this research does not increase the risk of harm caused by these models. However, some people might find the conclusion of using GPT-4 as an evaluator a bit risky. What are your thoughts about systematic LLM-based evaluations are performed for new models in the future?\n\n- As far as I understand, the \"Semantics\" category used in GEC human evaluation is introduced by you. However it seems to overlap with the \"Over-correction\" criterion, as described in appendix C using \"I wentt at Rome for my Birthday\" as an example. This can be a deeper problem when asking annotators to distinguish between the two criteria, making the annotation even harder. Can you clarify the differences between the two?\n\nEDIT: The authors clarified these questions, some of these points should be included in the paper.", "paper_topic_and_main_contributions": "In this paper the authors perform a thorough evaluation of LLMs for three NLP tasks: text summarization, text simplification, and grammatical error correction. They use both automatic metrics and human evaluations based on scores.\nThey also experiment with using GPT-4 as if it were a \"human evaluator\". The findings are that ChatGPT outperforms other models according to human reviewers while scoring poorly when using automatic metrics; that the gold reference of popular benchmarks are scored worse than the output models; and that GPT-4 shows a reasonable performance as an evaluator (based on the Krippendorff alpha coefficient), although its agreement with the human raters is always lower. The code and datasets used are public and available.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OSpV8IV91u", "reviewer": "Reviewer_Etgw", "summary": "", "questions": "A. If FLAN-T5 copies the input in text classification most of the time, shouldn't it be rated as the most semantic preserving by GPT4?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 for the comments, to which we have given a lot of thought. Please find below our responses to their concerns and questions in the same order as they were raised, and we hope the reviewer finds them clear.\n\n**Lack of details of experiment settings**\n- On the first point regarding lack of details on GPT4 prompting, as specified in lines 155-158, we included the full evaluation criteria and their assessment scales in the prompt. It is true that we did not report the prompts verbatim in the paper. This is because the prompts were very long. The instructions given the human reviewers and GPT-4 were almost identical, with minor differences (for example, in the human reviewer version we added details on time to spend and pay rate, whereas in the GPT4 prompt we included details on how to return the output in a json format so we could easily parse it). For the avoidance of doubt, we have included all instructions given to human reviewers and GPT4 at the end of this rebuttal, and are happy to add them to an Appendix in the camera-ready version.\n- On the point about lack of clarity on whether the samples rated by GPT-4 and human reviewers overlap, we can confirm that all three human annotators and GPT-4 annotated the exact same set of 100 samples for each of the three tasks. We appreciate this was not explicitly stated in the text, and will make that clearer in the camera-ready version.\n- At the time of writing the paper, we used the latest available version of ChatGPT. Most experiments were run in the first half of 2023. Using `gpt-3.5-turbo` in the API will always yield the latest version, I believe the version we used at the time is the one currently labelled as `gpt-3.5-turbo-0301` which was last updated on 1st March 2023. See here for more details https://platform.openai.com/docs/models/gpt-4. As this is a very rapidly changing landscape, newer versions have since become available. Regarding other openAI models, such as `text-davinci-003` and `davinci-instruct-beta`, they do not seem to have changed since the experiments were run, and as far as we are aware they are not currently versioned.\n- We confirm that the order of appearance of the outputs from the models and the gold reference are fixed in the human evaluation questionnaire, and are also the same in the GPT4 evaluation prompts. Human annotators were informed that one of the four outputs was human-generated, but were not told which one. The human evaluation survey for each of the three tasks had a landing page which contained detailed information about the task and evaluation criteria. Once the reviewer has familiarized themselves they click on a “Proceed” button which takes them to the first sample. Each sample was on a different webpage to avoid confusion, and the users were able to move between samples using browser buttons. On each page, the main input text was shown at the top. The user was then shown four different outputs below each input, and below each output they were asked to rate the output above using the task-specific criteria outlined in the paper. They were able to see all four outputs for the same input on the same webpage, making it easier to give scores which were correct relative to each other. The detailed instructions given are very long and are attached at the end of this rebuttal. As mentioned above, we are happy to add them to an Appendix in the camera-ready version.\n\n**Odd experiment settings**\n- According to OpenAI, `text-davinci-003`’s strengths are “Complex intent, cause and effect, creative generation, search, summarization for audience” and `davinci-instruct-beta`’s strengths are “complex intent, cause and effect”. While we agree that they are similar models, they are not the same model; they indeed show different results. It is true that `text-davinci-003`, which is always mapped to the best model of the GPT3 series, is currently an InstructGPT model, but it wasn’t always the case. It is not correct to say there is no `davinci-instruct-beta` model on OpenAI. However, we have just noticed that `davinci-instruct-beta` has recently been marked as a “legacy” model, which wasn’t the case a few months ago, and therefore may be hidden from the main webpage. In order to find this model on the the OpenAI playground, you can navigate here https://platform.openai.com/playground?mode=complete&model=davinci-instruct-beta and select Mode → “Complete” (not the default of “Chat”) and on the “Models” dropdown, click “show more models”, where you will be able to see `davinci-instruct-beta` amongst the available models. We will make this clearer in the camera-ready version.\n\n**Limited novelty**\n\nWe are aware that poor correlation between automatic metrics and human judgment, as well as the issue of low quality gold references, have been raised before in similar and different contexts. However, we believe we add a significant contribution on evaluating automatic evaluation metrics capability to capture the performance of LLMs, some of which have been recently released, as well as using GPT4 as an evaluator for such models. We have reviewed the referenced work, and we note that [2] explores LLMs evaluation on open-ended story generation and adversarial attacks, which are different tasks to the ones we analysed. One of the questions posed is _\"How grammatically correct is the text of the story fragment? (on a scale of 1-5, with 1 being the lowest?)\"_, which is similar to one of the evaluation criteria we used, but we also looked at many others. Reference [3] assesses the quality of LLMs outputs on text summarization and dialogue generation, whereas [4] runs experiments on four text generation tasks, namely dialogue response generation, text summarization, data-to-text, and machine translation, with very similar evaluation criteria to the ones we used on text summarization (Table 1 in [4]). However, none of these referenced works focus on text simplification or grammatical error correction.  As such we believe the particulars of our work differ from previous and concurrent works and our paper provides new insights, despite some overlap in the general research topic. However, we thank the reviewer for kindly pointing out relevant concurrent and recent work, in particular references [2], [3] and [4] and we would be happy to include those citations in our camera-ready version.\n\n**Questions for the authors**\n- **Question 1**: We observed that GPT4 rated Flan-T5 as second best model for semantic preservation on text simplification. In terms of absolute scores, as it can be seen in Table 6 (Appendix E), GPT4 gave ChatGPT an average score of 4.66 for semantic preservation, and it gave Flan-T5 a score of 4.55, so the absolute difference is not large. We agree with the reviewer that since Flan-T5 copies the input in text simplification most of the time, it should be rated as the most semantic preserving by GPT4, as it was by human annotators. We do not know why this was not the case, with GPT4 being a paid-for black-box model, and we only limited ourselves to reporting the observed results. An investigation into possible explanations behind this behaviour of GPT-4 would certainly be interesting, but is currently beyond the scope of this paper.\n\n\nOn a final note on reproducibility, we hope that the provision of the full instructions below will help alleviate the concerns. We made the decision not to include them in the paper due to their length, but are happy to do so in an Appendix in the camera-ready version.\n\nBelow are the instructions given to human reviewers and GPT4 by task, as requested. The human reviewers instructions contain html tags as they were delivered as a UI.\n\n- HUMAN REVIEWERS INSTRUCTIONS FOR TEXT SIMPLIFICATION\n\n```html\n<big>Human evaluation of text simplification</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction. We will start from text simplification. You will be asked to read some short texts and their simplified versions and rate them according to some criteria which are specified below. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This research is divided into 3 parts taking about 4, 8 and 4 hours respectively (around 16 hours in total). You should only start working on this research if you can at least commit for the first part (4 hours). To make things simpler, we are splitting these 4 hours into 4 separate studies taking about one hour each, which you can take on separate days. All these 4 one-hour studies should be completed by the end of Thursday 4th May. This is study part 1 of 4. A link to the following 3 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the first 4 one-hour studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus and invite you to take part to the two remaining parts of the study which will require a longer time commitment. You can decide not to continue with the research after you have completed the first 4 one-hour studies. Work will be reviewed as it comes through during UK working hours (excluding weekends and bank holidays). If at any point you have any questions or we have anything we'd like to communicate, feel free to message us through the Prolific messaging service, we will also use it if we need to get in touch, and potentially to discuss continuation of your work into the following studies. <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 25 pieces of text taken mostly from older news articles. Alongside each of them, you will be shown 4 different possible simplified versions of the text. The purpose of text simplification can be varied, but include things such as making the text easier to read, more accessible to language learners, or explaining a medical or legal text in plain speak. We ask that you carefully read the original text and rank each of the 4 simplifications according to the following metrics, which are defined below. <div style=\"height: 10px;\" /> A simplification box would look as follows (you will have 4 of these per source text) <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Semantics (or adequacy)</strong> </p> This assesses whether the meaning of the text is preserved further to the simplification. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize simplified text which is not self contained. For example, if the sentence <em>\"Scientists have no idea how eels protect their own brain and muscles from being damaged by electricity\"</em> is simplified as <em>\"Scientists have no idea how they stay safe\"</em>, this should be rated around 2 out of 5 as it is not possible by just reading the simplified version to understand what \"they\" is referring to. Like wise if the simplified text contains hallucinations (facts not contained in the source) it should be penalised. For example if <em>\"He yelled\"</em> is simplified as <em>\"His father Russell shouted\"</em> this should be rated around 2 under this category. The omission of some unimportant details from the source into the simplified version is acceptable at your discretion, if the text meaning is preserved. </li> <div style=\"height: 15px;\" /> <li> <p><strong>Grammaticality (or fluency)</strong></p>This assesses whether the simplified text remains grammatical and understandable. Sentences should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Grammaticality / Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Simplicity</strong></p> This assesses whether the simplified text, considered in its entirety, is simpler than the source text. Simplicity is assessed on a 5-point Likert scale from 1 (text is significantly more complex than the source) to 5 (text is significantly simpler than the source, or no further simplification was possible).  Note that sometimes you may see that the version which is supposed to be simplified is instead exactly identical to the source. In these cases, use your judgment, and if it is a very short sentence which cannot be simplified further, for example <em>\"How are you?\"</em>, then even if the text is not simpler than the source, it may be appropriate to assign a score of 5. A score of 3 is appropriate for cases where the text was not simplified nor made more complex, but potential for simplification exists. </li> <div style=\"height: 15px;\" />  <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong> <div style=\"height: 5px;\" />  This one-hour study will start on the next page and comprises 25 pages.\n````\n  \n\n- HUMAN REVIEWERS INSTRUCTIONS FOR TEXT SUMMARIZATION\n```html\n<big>Human evaluation of text summarization</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction. This is the second study and looks at text summarization. It is not to be confused with the text simplification study you may have taken part in previously, as the two tasks are different. Here you will be asked to read some texts and their summarized versions and rate them according to the criteria specified below. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This summarization research will take about 8 hours, and you should only start working on this research if you can commit to do it in full. To make things simpler, we are splitting these 8 hours into 8 separate studies taking about one hour each, which you can take on separate days. All these 8 one-hour studies should be completed by the end of Thursday 11th May. This is study part 1 of 8. A link to the following 7 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the 8 studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus and invite you to take part to our last study which will look at grammatical error correction. This separate research will take another 4 hours and is optional, you can decide not to continue once you've completed the 8 one-hour studies on text summarization. Work will be reviewed as it comes through during UK working hours (excluding weekends and bank holidays). If at any point you have any questions or we have anything we'd like to communicate, feel free to message us though the Prolific messaging service, we will also use it if we need to get in touch, and potentially to discuss continuation of your work into the following studies. <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 12 or 13 pieces of text taken mostly from older news articles. Alongside each of them, you will be shown 4 different possible summaries of the text. The purpose of text summarization is to make the text shorter and more concise without changing its overall meaning, omitting unimportant details while retaining the key content of the story. We ask that you carefully read the original text and rank each of the 4 summaries according to the following metrics, which are defined below. To ensure the best quality of outcome, we allocated roughly 5 minutes per page (note: this is significantly longer than we allowed for in the previous study on text simplification you may have taken, because the texts in this study are longer and more complex and we do not want you to feel rushed, as we value quality of output). <div style=\"height: 10px;\" /> A summary box would look as follows (you will have 4 of these per source text). The definition of the criteria you will be asked to rate the summaries on is below. Please ensure you familiarise yourself with the criteria before proceeding. <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Relevance</strong> </p>This assesses the quality of selection of important content from the source. The summary should include only important information from the source document. Please penalize summaries which contain redundancies and excess information. Relevance is assessed on a 5-point Likert scale from 1 (Not Relevant) to 5 (Super Relevant) </li> <div style=\"height: 15px;\" /> <li> <p><strong>Fluency</strong></p>This assesses the quality of individual sentences. Sentences in the summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Coherence</strong></p> This assesses the collective quality of all sentences. The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. Coherence is assessed on a 5-point Likert scale from 1 (Not Coherent) to 5 (Super Coherent) </li> <div style=\"height: 15px;\" /> <li> <p><strong>Consistency</strong></p> This assesses the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Summaries that contain hallucinated facts (information which is not present in the source document) should be penalized. Consistency is assessed on a 5-point Likert scale from 1 (Not Consistent) to 5 (Super Consistent) </li>        <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong>\n```\n\n- HUMAN REVIEWERS INSTRUCTIONS FOR GRAMMATICAL ERROR CORRECTION\n```html\n<big>Human evaluation of Grammatical Error Correction</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction (GEC). This is the third study and looks at GEC. Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. You will be asked to read some sentences and their corrected versions and rate them according to the criteria specified below. Note that not all sentences include grammatical errors, if they do not, we would expect the corrected version to be identical to the source. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This GEC research will take about 4 hours, and you should only start working on this research if you can commit to do it in full. To make things simpler, we are splitting these 4 hours into 4 separate studies taking about one hour each, which you can take on separate days. All these 4 one-hour studies should be completed by the end of Wednesday 17th May. This is study part 1 of 4. A link to the following 3 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the 4 studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus.  <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 25 pieces of text written by learners of English taken from an open source GEC dataset. Alongside each of them, you will be shown 4 different possible corrections of the text. The purpose of GEC is to rewrite the source sentence such that the corrected version is free of any grammatical errors which may be present in the source, but at the same time doing it with the minimum number of edits possible, without paraphrasing or making unnecessary syntax changes, unless absolutely necessary. We ask that you carefully read the original text and rank each of the 4 corrected versions according to the following metrics, which are defined below. <div style=\"height: 10px;\" /> A GEC box would look as follows (you will have 4 of these per source text). The definition of the criteria you will be asked to rate the corrections on is below. Please ensure you familiarise yourself with the criteria before proceeding. <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Semantics</strong> </p>This assesses whether the meaning of the text is preserved following the GEC. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize corrections which change the meaning unnecessarily. For example, the sentence <em>\"I wentt at Rome for my birthday\"</em> should be corrected to <em>\"I went to Rome for my birthday\"</em>. A correction such as <em>\"I went to Rome for my anniversary\"</em> should be penalised in this category as they introduce unnecessary changes to the meaning. </li> <div style=\"height: 15px;\" /> <li> <p><strong>Grammaticality</strong></p>This assesses the quality of the correction and answers the question \"How many errors are left in the corrected sentence?\". Please provide a count of the remaining errors, regardless of whether they were present in the source or they were newly introduced errors in the supposed corrected version.</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Over-correction</strong></p> Since there can be multiple ways to correct a sentence, this assesses whether the correction is unnecessarily verbose or makes unnecessary syntax changes. The best correction should be done with the minimum number of edits. For example, if the sentence <em>\"I wentt at Rome for my birthday\"</em> is corrected to <em>\"I decided to go to Rome for my birthday\"</em> this should be penalized under this category because it contains unnecessary syntax changes, even though the final sentence is grammatically correct. This metric answers the question: Is the system over-correcting or making unnecessary syntax changes? The answers should be \"No\", \"Minor over-correction\", \"Moderate over-correction\" or \"Substantial over-correction\".</li> <div style=\"height: 15px;\" /> <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong>\n```\n\n- GPT4 PROMPT FOR SIMPLIFICATION\n```\n You're GPT4 and are about to start a task where you will be shown some pieces of text taken mostly from older news articles, alongside 4 different possible simplified versions of each text, and you will be asked to evaluate the quality of the simplifications based on some metrics defined below. The purpose of text simplification can be varied, but include things such as making the text easier to read, more accessible to language learners, or explaining a medical or legal text in plain speak. We ask that you carefully read the original text and rank each of the 4 simplifications according to the following metrics, which are defined below.\n\n\nSemantics, or adequacy: This assesses whether the meaning of the text is preserved further to the simplification. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize simplified text which is not self contained. For example, if the sentence \"Scientists have no idea how eels protect their own brain and muscles from being damaged by electricity\" is simplified as \"Scientists have no idea how they stay safe\", this should be rated around 2 out of 5 as it is not possible by just reading the simplified version to understand what \"they\" is referring to. Like wise if the simplified text contains hallucinations (facts not contained in the source) it should be penalised. For example if \"He yelled\" is simplified as \"His father Russell shouted\" this should be rated around 2 under this category. The omission of some unimportant details from the source into the simplified version is acceptable at your discretion, if the text meaning is preserved.\n\n\nGrammaticality (or fluency). This assesses whether the simplified text remains grammatical and understandable. Sentences should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Grammaticality / Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)\n\n\nSimplicity. This assesses whether the simplified text, considered in its entirety, is simpler than the source text. Simplicity is assessed on a 5-point Likert scale from 1 (text is significantly more complex than the source) to 5 (text is significantly simpler than the source, or no further simplification was possible). Note that sometimes you may encounter very short sentences which cannot be simplified further and are repeated identically, for example \"How are you?\". In such cases, even if the text is not simpler than the source, it may be appropriate to assign a score of 5. A score of 3 is appropriate for cases where the text was not simplified nor made more complex, but potential for simplifcation exists.\n\n\nWe will pass you the input you need to rank in json format\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source text, 'id' is the unique identifier, and all other keys represent output texts which you need to evaluate.\n{\"original_input\": \"He is the first to market a simplified version of the tool to the public , a project that , for now , is not dangerous .\", \"gold_reference\": \"Zayner is the first to sell a gene-changing tool to regular people . His set is not dangerous .\", \"google/flan-t5-xxl\": \"He is the first to market a simplified version of the tool to the public, a project that, for now, is not dangerous.\", \"davinci-instruct-beta\": \"He is the first to market a simplified version of the tool to the public.\\n\\nHe is the first to market a simplified version of the tool to the public.\", \"gpt-3.5-turbo\": \"He created a safer version of the tool and introduced it to the public before anyone else.\", \"id\": \"9ec83328fbea94d7cdc4b92aba92f64abaa8727f6c0c99cc3b588b58bb8bb3f5\"}\n\n\nYour answer should contain the id and the scores, for example, if you wish to give gold_reference a semantics score of 2, a grammaticality score of 5 and a simplicity score of 4, and you wish to give google/flan-t5-xxl a semantics score of 5, a grammaticality score of 5, a simplicity score of 4, and you wish to give davinci-instruct-beta a semantics score of 5, a grammaticality score of 5, a simplicity score of 4, and you wish to give gpt-3.5-turbo a semantics score of 5, a grammaticality score of 5 and a simplicity score of 5, then you should return the following output (note how the id item needs to be preserved to allowed for identification):\n{\"gold_reference\": {\"semantics\": 2, \"grammaticality\": 5, \"simplicity\": 4}, \"google/flan-t5-xxl\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 4}, \"davinci-instruct-beta\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 4}, \"gpt-3.5-turbo\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 5}, \"id\": \"9ec83328fbea94d7cdc4b92aba92f64abaa8727f6c0c99cc3b588b58bb8bb3f5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```\n\n- GPT4 PROMPT FOR SUMMARIZATION\n```html\nYou're GPT4 and are about to start a task where you will be shown some pieces of text taken mostly from older news articles, alongside 4 different possible summaries of each text, and you will be asked to evaluate the quality of the summaries based on some metrics defined below. The purpose of text summarization is to make the text shorter and more concise without changing its overall meaning, omitting unimportant details while retaining the key content of the story. We ask that you carefully read the original text and rank each of the 4 summaries according to the following metrics, which are defined below.\n\n\nRelevance. This assesses the quality of selection of important content from the source. The summary should include only important information from the source document. Please penalize summaries which contain redundancies and excess information. Relevance is assessed on a 5-point Likert scale from 1 (Not Relevant) to 5 (Super Relevant)\n\n\nFluency. This assesses the quality of individual sentences. Sentences in the summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)\n\n\nCoherence. This assesses the collective quality of all sentences. The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. Coherence is assessed on a 5-point Likert scale from 1 (Not Coherent) to 5 (Super Coherent)\n\n\nConsistency. This assesses the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Summaries that contain hallucinated facts (information which is not present in the source document) should be penalized. Consistency is assessed on a 5-point Likert scale from 1 (Not Consistent) to 5 (Super Consistent)\n\n\nWe will pass you the input you need to rank in json format.\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source text, \"id\" is the unique identifier, and all other keys represent output texts which you need to evaluate.\n{\"original_input\": \"(CNN)Love it or hate it, Jared Leto's interpretation of the Joker is an internet sensation. The Oscar winner put on white makeup (and a lot of tattoos this time) to portray the Clown Prince of Crime in the upcoming movie \\\"Suicide Squad.\\\" Set for release August 5, 2016, \\\"Suicide Squad\\\" is based on the DC Comics series and also stars Will Smith, Margot Robbie and Viola Davis. Twitter users got their first look at Leto in character Friday night, and the memes started almost immediately. From comparisons to \\\"Home Alone\\\" to an imagining of Ben Affleck tatted up, people on social media put their Photoshopping skills to work all weekend. Which is your favorite?\", \"gold_reference\": \"Leto will play the Clown Prince of Crime in 2016's \\\"Suicide Squad\\\"\\nThe first picture of Leto in character led to a series of spoof photos .\", \"bigscience/T0pp\": \"The internet is buzzing about Jared Leto's new look as the Joker. The Oscar winner is playing the Clown Prince of Crime in \\\"Suicide Squad\\\"\", \"text-davinci-003\": \" Jared Leto's interpretation of the Joker in the upcoming movie \\\"Suicide Squad\\\" has become an internet sensation. The movie, set for release August 5, 2016, also stars Will Smith, Margot Robbie and Viola Davis. Twitter users got their first look at Leto in character Friday night, and the memes started almost immediately, with people Photoshopping him in various ways.\", \"gpt-3.5-turbo\": \"Jared Leto's portrayal of the Joker in the upcoming movie \\\"Suicide Squad\\\" has become an internet sensation. Twitter users have been sharing memes and Photoshopped images of Leto's character, who is heavily tattooed, since his first appearance in character on Friday. The movie is set for release on August 5, 2016 and also stars Will Smith, Margot Robbie, and Viola Davis.\", \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nYour answer should contain the id and the scores, for example, using the example given above, if you wish to give gold_reference a relevance score of 2, a fluency score of 5, a coherence score of 4 and a consistency score of 4, and you wish to give bigscience/T0pp a relevance score of 3, a fluency score of 5, a coherence score of 1 and a consistency score of 3,  and you wish to give text-davinci-003 a relevance score of 5, a fluency score of 4, a coherence score of 3 and a consistency score of 4, and you wish to give gpt-3.5-turbo a relevance score of 4, a fluency score of 3, a coherence score of 5 and a consistency score of 3, then you should return the following output (note how the id item needs to be preserved to allow for identification):\n{\"gold_reference\": {\"relevance\": 2, \"fluency\": 5, \"coherence\": 4, \"consistency\": 4}, \"bigscience/T0pp\": {\"relevance\": 3, \"fluency\": 5, \"coherence\": 1, \"consistency\": 3}, \"text-davinci-003\": {\"relevance\": 5, \"fluency\": 4, \"coherence\": 3, \"consistency\": 4}, \"gpt-3.5-turbo\": {\"relevance\": 4, \"fluency\": 3, \"coherence\": 5, \"consistency\": 3}, \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```\n\n- GPT4 PROMPT FOR GRAMMATICAL ERROR CORRECTION\n```html\nYou're GPT4 and are about to start a task where you will be shown some sentences written by learners of English. Some of these sentences will contain errors, and alongside each sentence you will be shown 4 different possible corrections, and you will be asked to evaluate the quality of the correction based on some metrics defined below. This task is called Grammatical Error Correction (GEC), and is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. Note that not all sentences you will see include grammatical errors; if they do not, we would expect the corrected version to be identical to the source. We ask that you carefully read the original sentence and rank each of the 4 corrections according to the following metrics, which are defined below.\n\n\nSemantics. This assesses whether the meaning of the text is preserved following the GEC. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize corrections which change the meaning unnecessarily. For example, the sentence \"I wentt at Rome for my birthday\" should be corrected to \"I went to Rome for my birthday\". A correction such as \"I went to Rome for my anniversary\" should be penalised in this category as it introduces unnecessary changes to the meaning.\n\n\nGrammaticality. This assesses the quality of the correction and answers the question \"How many errors are left in the corrected sentence?\". Please provide a count of the remaining errors, regardless of whether they were present in the source or they were newly introduced errors in the supposed corrected version. The options are \"0\", \"1\", \"2 or more\". Note that, unlike for semantics where a score of 5 is better than a score of 1, here a score of \"0\" is better than a score of \"1\" which is better than a score of \"2 or more\" (this is because if there are 0 errors remaining, the GEC task has been fulfilled).\n\n\nOver-correction. Since there can be multiple ways to correct a sentence, this assesses whether the correction is unnecessarily verbose or makes unnecessary syntax changes. The best correction should be done with the minimum number of edits. For example, if the sentence \"I wentt at Rome for my birthday\" is corrected to \"I decided to go to Rome for my birthday\" this should be penalized under this category because it contains unnecessary syntax changes, even though the final sentence is grammatically correct. This metric answers the question: Is the system over-correcting or making unnecessary syntax changes? The answers should be \"No\", \"Minor over-correction\", \"Moderate over-correction\" or \"Substantial over-correction\".\n\n\nWe will pass you the input you need to rank in json format.\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source sentence, \"id\" is the unique identifier, and all other keys represent the output corrected sentences which you need to evaluate.\n{\"original_input\": \"Travalling by car is also much more convenient.\", \"gold_reference\": \"Travelling by car is also much more convenient.\", \"opt-iml-max-30b\": \"Travalling by car is also much more convenient.\", \"text-davinci-003\": \"Traveling by car is also much more convenient.\", \"gpt-3.5-turbo\": \"Traveling by car is also much more convenient.\", \"id\": \"765d0adf8a8c58e2617a435be0d2451edcfcb782492ac662432da53843607d9a\"}\n\n\nYour answer should contain the id and the scores, for example, using the example given above, if you wish to give gold_reference a semantics score of 5, a grammaticality score of \"0\", an overcorrection score of \"No\", and you wish to give opt-iml-max-30b a semantics score of 4, a grammaticality score of \"1\", an overcorrection score of \"Minor over-correction\",  and you wish to give text-davinci-003 a semantics score of 3, a grammaticality score of \"2 or more\", an overcorrection score of \"Substantial over-correction\", and you wish to give gpt-3.5-turbo a semantics score of 5, a grammaticality score of \"0\", an overcorrection score of \"Moderate over-correction\", then you should return the following output (note how the id item needs to be preserved to allow for identification):\n{\"gold_reference\": {\"semantics\": 5, \"grammaticality\": \"0\", \"overcorrection\": \"No\"}, \"opt-iml-max-30b\": {\"semantics\": 4, \"grammaticality\": \"1\", \"overcorrection\": \"Minor over-correction\"}, \"text-davinci-003\": {\"semantics\": 3, \"grammaticality\": \"2 or more\", \"overcorrection\": \"Substantial over-correction\"}, \"gpt-3.5-turbo\": {\"semantics\": 5, \"grammaticality\": \"0\", \"overcorrection\":  \"Moderate over-correction\"}, \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper has clear and important takeaways: automatic evaluation metrics are insufficient for evaluating LLM outputs; GPT-4 is a good evaluator; ground truth references are bad.\n- This paper shows that GPT4 is not very good as an evaluator for grammar error correction. Such a negative result can help future researchers avoid using it and draw false conclusions.", "reasons_to_reject": "- **Lack of details of experiment settings**\n    - The paper does not show or discuss how GPT-4 was prompted to rate the samples. Whether the instructions given to GPT-4 are identical to those given to human evaluators is not revealed, so it is hard to judge if the difference between human's ranking and GPT-4's ranking is due to instruction difference. Based on my experience, those instructions can have a significant influence on the results.\n    - The number of samples rated by GPT-4 and whether they overlapped with the samples used in the human evaluation are not clearly stated. I assume that the samples should overlap since the paper calculates Krippendorff $\\alpha$.\n    - The version of OpenAI models (or the time of prompting those models) is not specified. \n    - The instructions given to the human evaluators are not specified. The interface is not shown. This raises some questions: are the order of outputs from the models and ground truth fixed in human evaluation? Are humans informed that one of the texts is written by humans and others are model generated? \n- **Odd experiment settings**\n    - The models used for seq2seq generation are very strange. Precisely, the **GPT3** (`text-davinci-003`) and **InstructGPT** (`davinci-instruct-beta`). In fact, `text-davinci-003` is an InstructGPT model; it is fine-tuned using RLHF. Also, there is currently no `davinci-instruct-beta` on OpenAI now. Please refer to the [OpenAI documentation](https://platform.openai.com/docs/models]). Here, it is unclear which models are used in the paper, undermining reproducibility.\n- **Limited novelty**\n    - Automatic evaluation metrics is not well-correlated with human is a well-known issue; see references [55, 45, 6, 33] in [1]. (This is listed as the first contribution in this paper)\n    - Low-quality reference in summarization is also well-known; see the SummEval paper and [1]. (This is listed as the second contribution in this paper.)\n    - The fact that GPT4 (or GPT3.5) can be used for automatic evaluation in NLG is shown in many prior works, and I am just listing two of them: [2], [3], and [4]. [2] and [3] are concurrent, but [4] is not.\n\n\n[1] Stiennon, Nisan, et al. \"Learning to summarize with human feedback.\" Advances in Neural Information Processing Systems 33 (2020): 3008-3021.\n\n[2] Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.\n\n[3] Liu, Yang, et al. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).\n\n[4] Fu, Jinlan, et al. \"Gptscore: Evaluate as you desire.\" arXiv preprint arXiv:2302.04166 (2023).", "questions_to_authors": "A. If FLAN-T5 copies the input in text classification most of the time, shouldn't it be rated as the most semantic preserving by GPT4?", "paper_topic_and_main_contributions": "Paper Topic\n=== \nThis paper evaluates the performance of several LLMs (FLAN-T5, FLAN-UL2, T0pp, OPT-IML, GPT3, InstructGPT, and ChatGPT) on three sequence-to-sequence tasks: text summarization, text simplification, and grammar error correction. They show that automatic evaluation metrics relying on the gold reference cannot give a model ranking consistent with human evaluation rankings. They attribute this to low-quality references. They also show that the output of LLMs mostly outperforms the gold reference. Last, they show that using GPT-4 to rate the LLM outputs can yield model rankings that are highly consistent with the ranking provided by human evaluators.\n\nContribution\n=== \nNLP engineering experiment", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Title: Sequence-to-Sequence\nLine 349 is an incomplete sentence\nLine 217: **was** scored\nLine 153 do not need the comma"}, "extra_text": {"Missing_References": "[2] Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.\n\n[3] Liu, Yang, et al. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).\n\n[4] Fu, Jinlan, et al. \"Gptscore: Evaluate as you desire.\" arXiv preprint arXiv:2302.04166 (2023).\n\n[2] is concurrent to the current paper, but it is highly relevant and should be cited according to the author's guidelines.\n[3] is highly relevant, it also evaluates summarization tasks."}}, {"review_id": "jWRDCKOKUw", "reviewer": "Reviewer_JKmf", "summary": "", "questions": "1. How were the outputs presented to the prolific workers? Could you also include detailed instructions for the same?\n2. How are the rankings induced by automatic metrics from Table 1 different from ChatGPT? I would suggest breaking down Table 2 in two to highlight: a) rankings induced by human eval and b) the ability of automatic metrics to recover system ranking.\n3. How well do automatic metrics correlate with segment-level rankings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3 for the positive comments. Please see below our responses to their main concerns and questions.\n\n- **Reason to Reject 1**: We agree that 100 samples for human evaluation is not a large subset to make very strong claims. Unfortunately human evaluation experiments take a lot of time for the reviewers to carry out, and are also considerably expensive for the researchers. As such we decided to limit the amount of samples used for this study. We are more than happy to add a note to the Limitations section in the camera-ready version regarding potential limitations of the strength of the findings given the sample size. However, we found that in most cases all annotators agreed that the gold standard was worse than some models’ outputs, so we do believe this is a valid conclusion, given how consistent it was across different tasks and annotators.\n- **Reason to Reject 2**: We fully take on-board the point about the limitations of human evaluation. While human evaluation done by experts is the gold-standard in this study, there can be a degree of subjectivity in evaluating language model outputs, and therefore using a much larger number of reviewers (for example 50 instead of 3) would make the findings much stronger and reduce potential noise. This would require substantially more financial expenditures on the human evaluation task. We will happily add a note about this in the camera-ready version to emphasize this aspect.\n\n\n- **Question 1**: The evaluation survey for each of the three tasks had a landing page which contained detailed information about the task and evaluation criteria. Once the reviewer has familiarized themselves they click on a “Proceed” button which takes them to the first sample. Each sample was on a different webpage to avoid confusion, and the users were able to move between samples using browser buttons. On each page, the main input text was shown at the top. The user was then shown four different outputs below each input, and below each output they were asked to rate the output above using the task-specific criteria outlined in the paper. They were able to see all four outputs for the same input on the same webpage, making it easier to give scores which were correct relative to each other. The detailed instructions given are very long and attached at the end of this rebuttal.\n- **Question 2**: There are two issues with using automatic metrics to recover rankings, in particular 1) automatic metrics cannot be used to rank the gold standard references, as they require such references for their ranking, and as a result the gold standard would always be ranked as the best by reference-based automatic metrics, whereas using human evaluation allows us to include the gold standard in the outputs to rank, 2) the automatic metrics we used only return a single score for each task, whereas for human evaluation we used multiple task-specific criteria (such as relevance, fluency, coherence etc.). We could potentially average these criteria and get a single human ranking score for each model and compare that to the rankings generated using automatic metrics, however given we would not be able to include the gold standard in the comparison, we chose not to do this.\n- **Question 3**: Overall automatic metrics correlate poorly with segment-level rankings. First of all, the gold-reference used by automatic metrics ranks worst or second worst on almost all metrics for all tasks, as shown in Table 2. If we exclude the gold-reference from the comparison, we still note that, for example, ChatGPT is rated as the worst model using automatic metrics for both summarization and simplification (Table 1), whereas it’s one of the best models according to human evaluators.\n\nBelow are the instructions given to human reviewers and GPT4 by task (as requested in question 1). The human reviewers instructions contain html tags as they were delivered as a UI.\n\n- HUMAN REVIEWERS INSTRUCTIONS FOR TEXT SIMPLIFICATION\n\n```html\n<big>Human evaluation of text simplification</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction. We will start from text simplification. You will be asked to read some short texts and their simplified versions and rate them according to some criteria which are specified below. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This research is divided into 3 parts taking about 4, 8 and 4 hours respectively (around 16 hours in total). You should only start working on this research if you can at least commit for the first part (4 hours). To make things simpler, we are splitting these 4 hours into 4 separate studies taking about one hour each, which you can take on separate days. All these 4 one-hour studies should be completed by the end of Thursday 4th May. This is study part 1 of 4. A link to the following 3 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the first 4 one-hour studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus and invite you to take part to the two remaining parts of the study which will require a longer time commitment. You can decide not to continue with the research after you have completed the first 4 one-hour studies. Work will be reviewed as it comes through during UK working hours (excluding weekends and bank holidays). If at any point you have any questions or we have anything we'd like to communicate, feel free to message us through the Prolific messaging service, we will also use it if we need to get in touch, and potentially to discuss continuation of your work into the following studies. <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 25 pieces of text taken mostly from older news articles. Alongside each of them, you will be shown 4 different possible simplified versions of the text. The purpose of text simplification can be varied, but include things such as making the text easier to read, more accessible to language learners, or explaining a medical or legal text in plain speak. We ask that you carefully read the original text and rank each of the 4 simplifications according to the following metrics, which are defined below. <div style=\"height: 10px;\" /> A simplification box would look as follows (you will have 4 of these per source text) <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Semantics (or adequacy)</strong> </p> This assesses whether the meaning of the text is preserved further to the simplification. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize simplified text which is not self contained. For example, if the sentence <em>\"Scientists have no idea how eels protect their own brain and muscles from being damaged by electricity\"</em> is simplified as <em>\"Scientists have no idea how they stay safe\"</em>, this should be rated around 2 out of 5 as it is not possible by just reading the simplified version to understand what \"they\" is referring to. Like wise if the simplified text contains hallucinations (facts not contained in the source) it should be penalised. For example if <em>\"He yelled\"</em> is simplified as <em>\"His father Russell shouted\"</em> this should be rated around 2 under this category. The omission of some unimportant details from the source into the simplified version is acceptable at your discretion, if the text meaning is preserved. </li> <div style=\"height: 15px;\" /> <li> <p><strong>Grammaticality (or fluency)</strong></p>This assesses whether the simplified text remains grammatical and understandable. Sentences should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Grammaticality / Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Simplicity</strong></p> This assesses whether the simplified text, considered in its entirety, is simpler than the source text. Simplicity is assessed on a 5-point Likert scale from 1 (text is significantly more complex than the source) to 5 (text is significantly simpler than the source, or no further simplification was possible).  Note that sometimes you may see that the version which is supposed to be simplified is instead exactly identical to the source. In these cases, use your judgment, and if it is a very short sentence which cannot be simplified further, for example <em>\"How are you?\"</em>, then even if the text is not simpler than the source, it may be appropriate to assign a score of 5. A score of 3 is appropriate for cases where the text was not simplified nor made more complex, but potential for simplification exists. </li> <div style=\"height: 15px;\" />  <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong> <div style=\"height: 5px;\" />  This one-hour study will start on the next page and comprises 25 pages.\n````\n\n- HUMAN REVIEWERS INSTRUCTIONS FOR TEXT SUMMARIZATION\n```html\n<big>Human evaluation of text summarization</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction. This is the second study and looks at text summarization. It is not to be confused with the text simplification study you may have taken part in previously, as the two tasks are different. Here you will be asked to read some texts and their summarized versions and rate them according to the criteria specified below. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This summarization research will take about 8 hours, and you should only start working on this research if you can commit to do it in full. To make things simpler, we are splitting these 8 hours into 8 separate studies taking about one hour each, which you can take on separate days. All these 8 one-hour studies should be completed by the end of Thursday 11th May. This is study part 1 of 8. A link to the following 7 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the 8 studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus and invite you to take part to our last study which will look at grammatical error correction. This separate research will take another 4 hours and is optional, you can decide not to continue once you've completed the 8 one-hour studies on text summarization. Work will be reviewed as it comes through during UK working hours (excluding weekends and bank holidays). If at any point you have any questions or we have anything we'd like to communicate, feel free to message us though the Prolific messaging service, we will also use it if we need to get in touch, and potentially to discuss continuation of your work into the following studies. <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 12 or 13 pieces of text taken mostly from older news articles. Alongside each of them, you will be shown 4 different possible summaries of the text. The purpose of text summarization is to make the text shorter and more concise without changing its overall meaning, omitting unimportant details while retaining the key content of the story. We ask that you carefully read the original text and rank each of the 4 summaries according to the following metrics, which are defined below. To ensure the best quality of outcome, we allocated roughly 5 minutes per page (note: this is significantly longer than we allowed for in the previous study on text simplification you may have taken, because the texts in this study are longer and more complex and we do not want you to feel rushed, as we value quality of output). <div style=\"height: 10px;\" /> A summary box would look as follows (you will have 4 of these per source text). The definition of the criteria you will be asked to rate the summaries on is below. Please ensure you familiarise yourself with the criteria before proceeding. <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Relevance</strong> </p>This assesses the quality of selection of important content from the source. The summary should include only important information from the source document. Please penalize summaries which contain redundancies and excess information. Relevance is assessed on a 5-point Likert scale from 1 (Not Relevant) to 5 (Super Relevant) </li> <div style=\"height: 15px;\" /> <li> <p><strong>Fluency</strong></p>This assesses the quality of individual sentences. Sentences in the summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Coherence</strong></p> This assesses the collective quality of all sentences. The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. Coherence is assessed on a 5-point Likert scale from 1 (Not Coherent) to 5 (Super Coherent) </li> <div style=\"height: 15px;\" /> <li> <p><strong>Consistency</strong></p> This assesses the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Summaries that contain hallucinated facts (information which is not present in the source document) should be penalized. Consistency is assessed on a 5-point Likert scale from 1 (Not Consistent) to 5 (Super Consistent) </li>        <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong>\n```\n\n- HUMAN REVIEWERS INSTRUCTIONS FOR GRAMMATICAL ERROR CORRECTION\n```html\n<big>Human evaluation of Grammatical Error Correction</big> <div style=\"height: 15px;\" />IMPORTANT INFORMATION: We are conducting this research on the quality of text simplification, summarization, and grammatical error correction (GEC). This is the third study and looks at GEC. Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. You will be asked to read some sentences and their corrected versions and rate them according to the criteria specified below. Note that not all sentences include grammatical errors, if they do not, we would expect the corrected version to be identical to the source. Some of the content you will see is human generated, other is AI generated, but you will not be told which is which as this is a blind research. As the interface is not mobile / tablet friendly, we ask that you complete this study using a laptop or desktop. This GEC research will take about 4 hours, and you should only start working on this research if you can commit to do it in full. To make things simpler, we are splitting these 4 hours into 4 separate studies taking about one hour each, which you can take on separate days. All these 4 one-hour studies should be completed by the end of Wednesday 17th May. This is study part 1 of 4. A link to the following 3 studies will be sent once you have accepted this commitment. Base pay is £12 per hour. Once you've completed the 4 studies, we will review the work, and if of a satisfactory quality, we will award a 10% discretionary bonus.  <div style=\"height: 15px;\" /> <ul>  In the study you're about to start, which we estimate should take one hour, you will be shown 25 pieces of text written by learners of English taken from an open source GEC dataset. Alongside each of them, you will be shown 4 different possible corrections of the text. The purpose of GEC is to rewrite the source sentence such that the corrected version is free of any grammatical errors which may be present in the source, but at the same time doing it with the minimum number of edits possible, without paraphrasing or making unnecessary syntax changes, unless absolutely necessary. We ask that you carefully read the original text and rank each of the 4 corrected versions according to the following metrics, which are defined below. <div style=\"height: 10px;\" /> A GEC box would look as follows (you will have 4 of these per source text). The definition of the criteria you will be asked to rate the corrections on is below. Please ensure you familiarise yourself with the criteria before proceeding. <head><style>* {margin: 0; padding: 0;} .imgbox {display: grid; height: 100%;} .center-fit { max-width: 100%; max-height: 100vh; margin: auto;}</style></head><body><div class=\"imgbox\"><img class=\"center-fit\" src='surveyflow/screenshot.png'></div></body>  <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong><div style=\"height: 15px;\" /> <ul>           <li>                <p><strong>Semantics</strong> </p>This assesses whether the meaning of the text is preserved following the GEC. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize corrections which change the meaning unnecessarily. For example, the sentence <em>\"I wentt at Rome for my birthday\"</em> should be corrected to <em>\"I went to Rome for my birthday\"</em>. A correction such as <em>\"I went to Rome for my anniversary\"</em> should be penalised in this category as they introduce unnecessary changes to the meaning. </li> <div style=\"height: 15px;\" /> <li> <p><strong>Grammaticality</strong></p>This assesses the quality of the correction and answers the question \"How many errors are left in the corrected sentence?\". Please provide a count of the remaining errors, regardless of whether they were present in the source or they were newly introduced errors in the supposed corrected version.</li>  <div style=\"height: 15px;\" /> <li> <p><strong>Over-correction</strong></p> Since there can be multiple ways to correct a sentence, this assesses whether the correction is unnecessarily verbose or makes unnecessary syntax changes. The best correction should be done with the minimum number of edits. For example, if the sentence <em>\"I wentt at Rome for my birthday\"</em> is corrected to <em>\"I decided to go to Rome for my birthday\"</em> this should be penalized under this category because it contains unnecessary syntax changes, even though the final sentence is grammatically correct. This metric answers the question: Is the system over-correcting or making unnecessary syntax changes? The answers should be \"No\", \"Minor over-correction\", \"Moderate over-correction\" or \"Substantial over-correction\".</li> <div style=\"height: 15px;\" /> <div style=\"height: 10px;\" />  </ul></ul> <strong>NOTE: please take a screeshot of this page for your future reference as you may not be able to access it again once you start the survey.</strong>\n```\n\n- GPT4 PROMPT FOR SIMPLIFICATION\n```\n You're GPT4 and are about to start a task where you will be shown some pieces of text taken mostly from older news articles, alongside 4 different possible simplified versions of each text, and you will be asked to evaluate the quality of the simplifications based on some metrics defined below. The purpose of text simplification can be varied, but include things such as making the text easier to read, more accessible to language learners, or explaining a medical or legal text in plain speak. We ask that you carefully read the original text and rank each of the 4 simplifications according to the following metrics, which are defined below.\n\n\nSemantics, or adequacy: This assesses whether the meaning of the text is preserved further to the simplification. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize simplified text which is not self contained. For example, if the sentence \"Scientists have no idea how eels protect their own brain and muscles from being damaged by electricity\" is simplified as \"Scientists have no idea how they stay safe\", this should be rated around 2 out of 5 as it is not possible by just reading the simplified version to understand what \"they\" is referring to. Like wise if the simplified text contains hallucinations (facts not contained in the source) it should be penalised. For example if \"He yelled\" is simplified as \"His father Russell shouted\" this should be rated around 2 under this category. The omission of some unimportant details from the source into the simplified version is acceptable at your discretion, if the text meaning is preserved.\n\n\nGrammaticality (or fluency). This assesses whether the simplified text remains grammatical and understandable. Sentences should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Grammaticality / Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)\n\n\nSimplicity. This assesses whether the simplified text, considered in its entirety, is simpler than the source text. Simplicity is assessed on a 5-point Likert scale from 1 (text is significantly more complex than the source) to 5 (text is significantly simpler than the source, or no further simplification was possible). Note that sometimes you may encounter very short sentences which cannot be simplified further and are repeated identically, for example \"How are you?\". In such cases, even if the text is not simpler than the source, it may be appropriate to assign a score of 5. A score of 3 is appropriate for cases where the text was not simplified nor made more complex, but potential for simplifcation exists.\n\n\nWe will pass you the input you need to rank in json format\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source text, 'id' is the unique identifier, and all other keys represent output texts which you need to evaluate.\n{\"original_input\": \"He is the first to market a simplified version of the tool to the public , a project that , for now , is not dangerous .\", \"gold_reference\": \"Zayner is the first to sell a gene-changing tool to regular people . His set is not dangerous .\", \"google/flan-t5-xxl\": \"He is the first to market a simplified version of the tool to the public, a project that, for now, is not dangerous.\", \"davinci-instruct-beta\": \"He is the first to market a simplified version of the tool to the public.\\n\\nHe is the first to market a simplified version of the tool to the public.\", \"gpt-3.5-turbo\": \"He created a safer version of the tool and introduced it to the public before anyone else.\", \"id\": \"9ec83328fbea94d7cdc4b92aba92f64abaa8727f6c0c99cc3b588b58bb8bb3f5\"}\n\n\nYour answer should contain the id and the scores, for example, if you wish to give gold_reference a semantics score of 2, a grammaticality score of 5 and a simplicity score of 4, and you wish to give google/flan-t5-xxl a semantics score of 5, a grammaticality score of 5, a simplicity score of 4, and you wish to give davinci-instruct-beta a semantics score of 5, a grammaticality score of 5, a simplicity score of 4, and you wish to give gpt-3.5-turbo a semantics score of 5, a grammaticality score of 5 and a simplicity score of 5, then you should return the following output (note how the id item needs to be preserved to allowed for identification):\n{\"gold_reference\": {\"semantics\": 2, \"grammaticality\": 5, \"simplicity\": 4}, \"google/flan-t5-xxl\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 4}, \"davinci-instruct-beta\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 4}, \"gpt-3.5-turbo\": {\"semantics\": 5, \"grammaticality\": 5, \"simplicity\": 5}, \"id\": \"9ec83328fbea94d7cdc4b92aba92f64abaa8727f6c0c99cc3b588b58bb8bb3f5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```\n\n- GPT4 PROMPT FOR SUMMARIZATION\n```html\nYou're GPT4 and are about to start a task where you will be shown some pieces of text taken mostly from older news articles, alongside 4 different possible summaries of each text, and you will be asked to evaluate the quality of the summaries based on some metrics defined below. The purpose of text summarization is to make the text shorter and more concise without changing its overall meaning, omitting unimportant details while retaining the key content of the story. We ask that you carefully read the original text and rank each of the 4 summaries according to the following metrics, which are defined below.\n\n\nRelevance. This assesses the quality of selection of important content from the source. The summary should include only important information from the source document. Please penalize summaries which contain redundancies and excess information. Relevance is assessed on a 5-point Likert scale from 1 (Not Relevant) to 5 (Super Relevant)\n\n\nFluency. This assesses the quality of individual sentences. Sentences in the summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Fluency is assessed on a 5-point Likert scale from 1 (Not Fluent) to 5 (Super Fluent)\n\n\nCoherence. This assesses the collective quality of all sentences. The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. Coherence is assessed on a 5-point Likert scale from 1 (Not Coherent) to 5 (Super Coherent)\n\n\nConsistency. This assesses the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Summaries that contain hallucinated facts (information which is not present in the source document) should be penalized. Consistency is assessed on a 5-point Likert scale from 1 (Not Consistent) to 5 (Super Consistent)\n\n\nWe will pass you the input you need to rank in json format.\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source text, \"id\" is the unique identifier, and all other keys represent output texts which you need to evaluate.\n{\"original_input\": \"(CNN)Love it or hate it, Jared Leto's interpretation of the Joker is an internet sensation. The Oscar winner put on white makeup (and a lot of tattoos this time) to portray the Clown Prince of Crime in the upcoming movie \\\"Suicide Squad.\\\" Set for release August 5, 2016, \\\"Suicide Squad\\\" is based on the DC Comics series and also stars Will Smith, Margot Robbie and Viola Davis. Twitter users got their first look at Leto in character Friday night, and the memes started almost immediately. From comparisons to \\\"Home Alone\\\" to an imagining of Ben Affleck tatted up, people on social media put their Photoshopping skills to work all weekend. Which is your favorite?\", \"gold_reference\": \"Leto will play the Clown Prince of Crime in 2016's \\\"Suicide Squad\\\"\\nThe first picture of Leto in character led to a series of spoof photos .\", \"bigscience/T0pp\": \"The internet is buzzing about Jared Leto's new look as the Joker. The Oscar winner is playing the Clown Prince of Crime in \\\"Suicide Squad\\\"\", \"text-davinci-003\": \" Jared Leto's interpretation of the Joker in the upcoming movie \\\"Suicide Squad\\\" has become an internet sensation. The movie, set for release August 5, 2016, also stars Will Smith, Margot Robbie and Viola Davis. Twitter users got their first look at Leto in character Friday night, and the memes started almost immediately, with people Photoshopping him in various ways.\", \"gpt-3.5-turbo\": \"Jared Leto's portrayal of the Joker in the upcoming movie \\\"Suicide Squad\\\" has become an internet sensation. Twitter users have been sharing memes and Photoshopped images of Leto's character, who is heavily tattooed, since his first appearance in character on Friday. The movie is set for release on August 5, 2016 and also stars Will Smith, Margot Robbie, and Viola Davis.\", \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nYour answer should contain the id and the scores, for example, using the example given above, if you wish to give gold_reference a relevance score of 2, a fluency score of 5, a coherence score of 4 and a consistency score of 4, and you wish to give bigscience/T0pp a relevance score of 3, a fluency score of 5, a coherence score of 1 and a consistency score of 3,  and you wish to give text-davinci-003 a relevance score of 5, a fluency score of 4, a coherence score of 3 and a consistency score of 4, and you wish to give gpt-3.5-turbo a relevance score of 4, a fluency score of 3, a coherence score of 5 and a consistency score of 3, then you should return the following output (note how the id item needs to be preserved to allow for identification):\n{\"gold_reference\": {\"relevance\": 2, \"fluency\": 5, \"coherence\": 4, \"consistency\": 4}, \"bigscience/T0pp\": {\"relevance\": 3, \"fluency\": 5, \"coherence\": 1, \"consistency\": 3}, \"text-davinci-003\": {\"relevance\": 5, \"fluency\": 4, \"coherence\": 3, \"consistency\": 4}, \"gpt-3.5-turbo\": {\"relevance\": 4, \"fluency\": 3, \"coherence\": 5, \"consistency\": 3}, \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```\n\n- GPT4 PROMPT FOR GRAMMATICAL ERROR CORRECTION\n```html\nYou're GPT4 and are about to start a task where you will be shown some sentences written by learners of English. Some of these sentences will contain errors, and alongside each sentence you will be shown 4 different possible corrections, and you will be asked to evaluate the quality of the correction based on some metrics defined below. This task is called Grammatical Error Correction (GEC), and is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but also orthographic and semantic errors, such as misspellings and word choice errors respectively. Note that not all sentences you will see include grammatical errors; if they do not, we would expect the corrected version to be identical to the source. We ask that you carefully read the original sentence and rank each of the 4 corrections according to the following metrics, which are defined below.\n\n\nSemantics. This assesses whether the meaning of the text is preserved following the GEC. Semantic preservation is assessed on a 5-point Likert scale from 1 (Meaning Not Preserved) to 5 (Meaning fully preserved). NOTE: You should penalize corrections which change the meaning unnecessarily. For example, the sentence \"I wentt at Rome for my birthday\" should be corrected to \"I went to Rome for my birthday\". A correction such as \"I went to Rome for my anniversary\" should be penalised in this category as it introduces unnecessary changes to the meaning.\n\n\nGrammaticality. This assesses the quality of the correction and answers the question \"How many errors are left in the corrected sentence?\". Please provide a count of the remaining errors, regardless of whether they were present in the source or they were newly introduced errors in the supposed corrected version. The options are \"0\", \"1\", \"2 or more\". Note that, unlike for semantics where a score of 5 is better than a score of 1, here a score of \"0\" is better than a score of \"1\" which is better than a score of \"2 or more\" (this is because if there are 0 errors remaining, the GEC task has been fulfilled).\n\n\nOver-correction. Since there can be multiple ways to correct a sentence, this assesses whether the correction is unnecessarily verbose or makes unnecessary syntax changes. The best correction should be done with the minimum number of edits. For example, if the sentence \"I wentt at Rome for my birthday\" is corrected to \"I decided to go to Rome for my birthday\" this should be penalized under this category because it contains unnecessary syntax changes, even though the final sentence is grammatically correct. This metric answers the question: Is the system over-correcting or making unnecessary syntax changes? The answers should be \"No\", \"Minor over-correction\", \"Moderate over-correction\" or \"Substantial over-correction\".\n\n\nWe will pass you the input you need to rank in json format.\nPlease reply with the scores in json format.\nThis is an example json query where \"original_input\" is the source sentence, \"id\" is the unique identifier, and all other keys represent the output corrected sentences which you need to evaluate.\n{\"original_input\": \"Travalling by car is also much more convenient.\", \"gold_reference\": \"Travelling by car is also much more convenient.\", \"opt-iml-max-30b\": \"Travalling by car is also much more convenient.\", \"text-davinci-003\": \"Traveling by car is also much more convenient.\", \"gpt-3.5-turbo\": \"Traveling by car is also much more convenient.\", \"id\": \"765d0adf8a8c58e2617a435be0d2451edcfcb782492ac662432da53843607d9a\"}\n\n\nYour answer should contain the id and the scores, for example, using the example given above, if you wish to give gold_reference a semantics score of 5, a grammaticality score of \"0\", an overcorrection score of \"No\", and you wish to give opt-iml-max-30b a semantics score of 4, a grammaticality score of \"1\", an overcorrection score of \"Minor over-correction\",  and you wish to give text-davinci-003 a semantics score of 3, a grammaticality score of \"2 or more\", an overcorrection score of \"Substantial over-correction\", and you wish to give gpt-3.5-turbo a semantics score of 5, a grammaticality score of \"0\", an overcorrection score of \"Moderate over-correction\", then you should return the following output (note how the id item needs to be preserved to allow for identification):\n{\"gold_reference\": {\"semantics\": 5, \"grammaticality\": \"0\", \"overcorrection\": \"No\"}, \"opt-iml-max-30b\": {\"semantics\": 4, \"grammaticality\": \"1\", \"overcorrection\": \"Minor over-correction\"}, \"text-davinci-003\": {\"semantics\": 3, \"grammaticality\": \"2 or more\", \"overcorrection\": \"Substantial over-correction\"}, \"gpt-3.5-turbo\": {\"semantics\": 5, \"grammaticality\": \"0\", \"overcorrection\":  \"Moderate over-correction\"}, \"id\": \"49f45c17ee34e2aa420ef557d944a6f2df96f5226c2793e4a9d379c517fde9a5\"}\n\n\nIs this clear? Do you have any questions or are you ready to start?\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents an important data point for the re-consideration of popular benchmarks as the output quality of references is judged lower than model-generated outputs.\n2. The paper presents a systematic evaluation of multiple LLMs on three tasks that can be useful for development in human and automatic evaluation.\n", "reasons_to_reject": "1. The scale of the evaluation (100 examples) is small to make strong claims (e.g. l76: we prove that...). It might be useful to note the scale limitation in the paper when mentioning the findings.\n2. While authors note that selecting models for human eval via automatic metrics could have biased the evaluation in limitations, it is equally important to acknowledge the limitations of the human evaluation as a gold standard itself. It would be useful to provide some discussion in this direction either in the main text or the limitation section.", "questions_to_authors": "1. How were the outputs presented to the prolific workers? Could you also include detailed instructions for the same?\n2. How are the rankings induced by automatic metrics from Table 1 different from ChatGPT? I would suggest breaking down Table 2 in two to highlight: a) rankings induced by human eval and b) the ability of automatic metrics to recover system ranking.\n3. How well do automatic metrics correlate with segment-level rankings?", "paper_topic_and_main_contributions": "The paper presents a hybrid evaluation of recent LLMs on three NLP tasks: text simplification, text summarization, and grammatical error correction. They ask human evaluators to judge the output quality across task-specific and task-agnostic dimension criteria and find a discrepancy between the rankings of automatic metrics and human evaluation and argue that automatic metrics are not reliable even for system development. Further, they show that rankings induced by ChatGPT as an evaluator better approximates human judgments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors of this paper evaluate generative LLMs on three NLP tasks, namely text summarization, text simplification, and grammatical error correction, using both automatic metrics and human evaluations. They also experiment with using GPT-4 as if it were a "human evaluator".

The reviewers highlighted the following reasons to accept:
- a systematic evaluation of multiple LLMs on three tasks, leading to clear and important takeaways: automatic evaluation metrics are insufficient for evaluating LLM outputs; GPT-4 is a good evaluator; ground truth references are bad
- interesting negative result that GPT4 is not very good as an evaluator for grammar error correction

The following reasons to reject were mentioned:
- A more detailed analysis of inter-annotator agreement would have been desirable. The authors noted that some of the requested information is available in an appendix, and that other parts of the requested discussion are either not possible or not meaningful with the available data.
- Lacking details of the experimental settings leading to poor reproducibility. The authors gave complementary information in the rebuttal that addressed most issues. One reviewer decided to keep a low reproducibility score due to the confusing and changing OpenAI model naming conventions, which are however totally beyond the control of this paper's authors.
- Limited novelty, in the sense that similar conclusions about human vs. automatic evaluations have been drawn before. The authors responded that there is no prior work on text simplification or grammatical error correction evaluation in connection with LLMs.
- Some limitations are not presented sufficiently prominently. The authors promised to make the requested changes in the camera-ready version.
Decision: Accept-Main