Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various applications, fundamentally reshaping the landscape of natural language processing (NLP) research. However, recent evaluation frameworks often rely on the output probabilities of LLMs for predictions, primarily due to computational constraints, diverging from real-world LLM usage scenarios. While widely employed, the efficacy of these probability-based evaluation strategies remains an open research question. This study aims to scrutinize the validity of such probability-based evaluation methods within the context of using LLMs for Multiple Choice Questions (MCQs), highlighting their inherent limitations. Our empirical investigation reveals that the prevalent probabilitybased evaluation method inadequately aligns with generation-based prediction. Furthermore, current evaluation frameworks typically assess LLMs through predictive tasks based on output probabilities rather than directly generating responses, owing to computational limitations. We illustrate that these probability-based approaches do not effectively correspond with generative predictions. The outcomes of our study can enhance the understanding of LLM evaluation methodologies and provide insights for future research in this domain.

text: Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 109–131 August 16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Large Language Models (LLMs) have significantly advanced the field of natural language processing (NLP), reshaping the paradigms in NLP research and application (Ouyang et al., 2022; Wei et al., 2022; Sanh et al., 2022; Chung et al., 2022; OpenAI, 2023; Anil et al., 2023; Touvron et al., 2023a,c; Jiang et al., 2023). As the scale of model parameters of language models expands from the million to billion or even trillion levels, a proficient LLM is expected to exhibit a broad mastery across
†equal contribution
various tasks. Recent works aim to assess LLMs comprehensively by aggregating a substantial array of NLP benchmarks (Srivastava et al., 2022; Sanh et al., 2022; Liang et al., 2022; Longpre et al., 2023). Additionally, there exists a line of research that curates human exam questions to challenge LLMs (Hendrycks et al., 2021; Huang et al., 2023; Li et al., 2023b; Koto et al., 2023). The collected questions and NLP benchmarks are adapted into prompts via standardized templates.
Due to computational constraints, recent evaluation frameworks commonly adopt the approach of selecting the option with the highest probability as the prediction of LLMs, as illustrated in Figure 1. These frameworks employ either labelbased prediction, which assesses the probability of the next token output, or sequence-based prediction, which evaluates the probability of an entire option, ultimately selecting the option with the highest probability as the LLM’s prediction. However, these probability-based evaluation methodologies introduce a misalignment between evaluation procedures and real-world application scenarios, where LLMs are typically tasked with generating responses to user queries. This misalignment raises an important question: Is the probability-based evaluation method sufficient to accurately assess
109
the capabilities of LLMs? In this position study, we argue that the current
LLM evaluation and leaderboard misalign the actual LLM capabilities. We examine three prediction methodologies: generation-based, label-based, and sequence-based predictions. We conducted extensive experiments across LLMs with varying model sizes on three prominent benchmarks: MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022), and Belebele (Bandarkar et al., 2023). Our findings reveal a significant disconnect between probability-based methods and generation-based predictions. Even when predictions are correct, the consistency between probability-based methods and generation-based predictions remains notably low. We additionally find that many of these multiple-choice NLP benchmark rankings do not agree with human preference for free-text generation output. Consequently, these results raise serious doubts about the reliability of evaluation outcomes derived from popular benchmarks reliant on probability-based methods. In conclusion, our research emphasizes the urgent need for an evaluation approach that ensures accurate and reliable assessments of LLM capabilities, more closely aligned with real-world usage scenarios. In next section, we will discuss the course of the development and paradigm of the evaluation of LLMs.
2 Evaluating Large Language Models
2.1 Challenges in Evaluating Large Language Models
The advancement of LLMs has substantially broadened their capabilities, transcending conventional NLP tasks. They now demonstrate proficiency in tackling intricate prompts and a wide spectrum of open-ended inquiries. However, unlike tasks with definitive solutions, open-ended questions lack a single correct answer, making it difficult to gauge the LLM’s performance.
Recently, human evaluators have been deployed to appraise responses to open-ended questions using two primary methods. Firstly, evaluators assign scores based on specific criteria such as accuracy and relevance (Wang et al., 2023b; Zhou et al., 2023). Alternatively, they conduct comparative assessments by selecting the preferred answer among two distinct LLM responses to the same question (Askell et al., 2021; Bai et al., 2022a; Zheng et al., 2023b). However, manual evaluation faces significant scalability challenges due to the
high costs associated with human judges. Moreover, recent studies indicate that human evaluators often favor longer and more fluent responses, even if they contain factual inaccuracies (Wu and Aji, 2023). Additionally, ensuring the trustworthiness of evaluations presents a concern, as crowdannotators increasingly rely on tools like LLMs for assistance (Veselovsky et al., 2023), raising questions about the purely human-based nature of evaluations. Moreover, maintaining consistent evaluation quality across a large team of evaluators necessitates extensive coordination and rigorous standardization. Recent research highlights low consistency among human evaluators when assessing LLM responses to open-ended questions.
Another approach to evaluating generative LLMs involves utilizing a stronger LLM as the evaluator, offering greater scalability compared to human judges (Zheng et al., 2023b; Wu and Aji, 2023; Liu et al., 2023). However, LLM judges may exhibit biases in their assessments, influenced by factors such as the order and length of answers, as well as their fluency. Furthermore, commonly used LLM judges, like GPT-4 (OpenAI et al., 2023), often operate on public yet black-box systems, posing challenges in ensuring the reproducibility and transparency of the evaluation process.
2.2 Multiple Choice Question as a Proxy Due to the challenges discussed in Section 2.1, recent works commonly convert the multiple-choice questions (MCQs) in human exams to prompts using standard template. The responses generated by the LLMs are then compared against the humancrafted ground truth, allowing for an assessment of the model’s accuracy. This process streamlines the evaluation and provides a clear metric for understanding the capabilities of LLMs.
Recent frameworks frequently utilize the output probabilities from LLMs across various options for making predictions, to ensure that the prediction from the LLM is among these options, given the unpredictability of the text generated by LLMs. For example, as illustrated in Figure 1, when presented with the question and the candidate choices, some approaches compare the probabilities predicted by the model based solely on the option letters (Hendrycks et al., 2021),† while others consider the probability of each token and aggregate them (Gao et al., 2021).†
†https://github.com/hendrycks/test †https://github.com/EleutherAI/
2.3 Misalignment between MCQ and User-Facing Interaction
We argue that MCQ-proxy might not always reflect the actual performance of LLM under user-facing free-text generation. In MCQ, LLM output is restricted to a limited set of answers; hence, their answer might be different under unrestricted generation. MCQ benchmarks also often only look for a short and direct answer, whereas user-facing interaction expects the LLM to provide a verbose answer; especially after preference tuning. Hence, MCQ benchmarks are not suitable for measuring the nuanced answers of LLMs.
Additionally, prior studies have shown LLM’s brittleness under MCQ benchmarks, e.g., on how the option order is presented (Zheng et al., 2023a; Pezeshkpour and Hruschka, 2023; Alzahrani et al., 2024). Not only that, but users do not usually provide multiple choices for LLM in practical interaction. Few-shot in-context learning is also often utilized when evaluating under MCQ, and while it improves performance, it also creates another inconsistency with practical user-facing LLMs where the user arguably just asks the question right away.
Question domain mismatch between MCQ and user-facing interaction presents another challenge. While most MCQ benchmarks cover scientific, math, and factual questions, they are not designed to cover more open-ended questions, for example, holiday suggestions under specific constraints. They do not cover creative-type questions such as story-writing. Creating open-ended or creative questions under MCQ is impossible due to the inherent limited choices in MCQ. Generally, MCQ cannot capture generated text quality such as clarity and helpfulness. Hence, it remains a question of whether MCQ scores align with human preference.
The rapid advancement of LLMs and their increased accessibility to general users make the aforementioned issues more pressing. The focus on fast research and SoTA-chasing over a scientific understanding of LLM development further exacerbates the situation (Nityasya et al., 2023). Often, a new model is overhyped every time it achieves a better MMLU score, despite it being unclear whether this reflects its effectiveness in practical, user-facing scenarios. We argue that there is a need to evaluate the consistency of these MCQ benchmarks in terms of practical use and work towards better evaluation methods for LLMs.
lm-evaluation-harness
In Section 3, we demonstrate empirical evidence verifying whether these evaluation methodologies faithfully reflect the capability of LLMs.
3 Empirical Evidence
In this section, we empirically show that MCQ performance does not reflect free-text generation performance.
3.1 Experiment Setup In this section, we describe our experimental setup, including the benchmark datasets, models, and prediction methods.
Datasets In this work, we conduct our experiments on three popuplar benchmarks: MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022), and Belebele (Bandarkar et al., 2023). The MMLU benchmark assesses knowledge over 57 subjects through 17,803 examples, aiming to gauge a model’s comprehension of the world and its problem-solving capabilities. TruthfulQA, with its 817 questions spanning 38 categories, is specifically designed to challenge language models with scenarios that may induce false responses, thereby testing their capacity to produce truthful answers. Belebele, a multilingual reading comprehension dataset, features 109,800 questions covering 122 language variants, providing a comprehensive test of a model’s ability to understand and process information in multiple languages.
Models In this study, we undertake comprehensive experimentation across a range of LLMs, including LLaMA-1 (Touvron et al., 2023b), Vicuna (Chiang et al., 2023), LLaMA-2 (Touvron et al., 2023c), and Mistral (Jiang et al., 2023). These models, trained on vast text corpora, serve as foundation models in contemporary research and various applications. LLaMA-1, Vicuna, and LLaMA-2 (Touvron et al., 2023b; Chiang et al., 2023; Touvron et al., 2023c) are LLMs trained on 2 trillion tokens, noted for their performance and safety in various evaluations. Mistral 7B (Jiang et al., 2023) features 7.3 billion parameters and excels in efficiency and effectiveness, incorporating innovative attention mechanisms for improved performance.
Prediction Methods In this work, we evaluate the models with the following prediction methods:
1. label-based prediction: We provide the prompt “{question} {options} The correct answer is” to LLMs and then calculate the prob-
ability of the next token for each option letter (e.g., “A”, “B”, “C”, “D” for four options). The option with the highest probability is selected as the predicted answer. This method was used in the original implementation of MMLU (Hendrycks et al., 2021).
2. sequence-based prediction: We provide the prompt "{question} {options} The correct answer is option" to LLMs. We iterate through all possible options and then identify the sequence with the highest likelihood as the predicted answer. This method is used in the Language Model Evaluation Harness (LMEH) framework (Gao et al., 2021).
3. generation-based prediction: Unlike the previous two methods, we allow LLMs to generate a response to the input question, mirroring how people typically use LLMs.
3.2 Results and Analysis
Inconsistent Predictions between ProbabilityBased Methods and Generation Experimental results on MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022), and Belebele (Bandarkar et al., 2023) are shown in Table 2 and Figure 2.
Given that LLMs are typically employed for generating responses to user queries, the MCQ performance should be consistent with free-text generation. Recent research commonly utilizes accuracy, which measures the percentage of correct predictions, to assess model performance. In addition to accuracy, we introduce agreement with the generation-based predictions to differentiate the predictions provided by various methods. Agree-
t
ment is defined as the percentage of consistent predictions between two prediction methods. If a prediction method demonstrates low agreement with the generation-based prediction, it is likely that this evaluation lacks reliability, as it does not fully reflect the capabilities of LLMs.
Based on our MMLU results presented in Table 2, it is evident that smaller base language models such as Mistral-7B, LLaMA-1-7B, and LLaMA2-7B face difficulties in achieving consensus with generation-based predictions when utilizing both
label-based and sequence-based methods. Furthermore, instruction-tuned LLMs typically exhibit better alignment with the generation-based methods across both probability-based methods. Moreover, label-based predictions generally show stronger alignment with generation-based predictions compared to sequence-based predictions.
Furthermore, we also evaluate LLMs on TruthfulQA, as shown in Table 2. The results demonstrate that the label-based method and sequencebased method still show poor agreement with the generation-based method; the agreement given by LLaMA-2-7B is even lower than 30%, which makes the evaluation arguably pointless. Moreover, as shown in Figure 2, the gap between different accuracies (∆) is even larger compared to the ∆ on MMLU - the smallest ∆ is close to 5, and the largest ∆ is more than 20. Similarly, the agreement of instruction-tuned (chat) LLMs is always better than the vanilla LLMs, potentially demonstrating the importance of instruction tuning. The results on both MMLU and TruthfulQA in Table 2 strongly question the reliability of label-based and sequencebased methods for evaluating LLMs while MMLU and TruthfulQA are widely employed benchmarks to demonstrate the capability of LLMs.
Additionally, we evaluate LLMs on a recently built benchmark MRC dataset, Belebele (Bandarkar et al., 2023), which can reduce the risk of data contamination for LLMs. Surprisingly, we observe a much higher agreement between the labelbased method and the generation-based method in Table 2, where the lowest agreement is even higher than 60%, and there are three LLMs whose agreement is close to 90%. However, we observe a lower agreement between the sequence-based pre-
diction and the generation-based prediction. We also observe that the ∆ between the accuracy of the sequence-based prediction and the generationbased prediction is much smaller, suggesting that the label-based method is more accurate.
Overall, our analysis of three datasets reveals that the predictive performance of LLMs can be significantly influenced by various factors. Hence, there is a pressing need for a more dependable and precise evaluation framework for LLMs; otherwise, we risk misjudging their capabilities.
Inconsistent Correct Predictions In Table 2 and Figure 2, we highlight the low consistency among prediction methods. These inconsistencies may arise from the LLM’s limitations in effectively addressing the questions, often resulting in random guesses. To address this issue, we introduce a new metric - correct option overlap - designed to gauge the level of agreement among correctly predicted options from various LLMs.
We analyze the overlap of accurately predicted
options across different LLMs and present the findings in Table 3. It is evident that Mistral models and LLaMA-1-7B exhibit low overlap rates when evaluated using the label-based approach. Conversely, when employing the sequence-based method, all LLMs show a reduced overlap rate on TruthfulQA, averaging around 30%. However, label-based methods consistently yield higher overlap rates for LLaMA-2 models. These results suggest that predictions from these LLMs are subject to high uncertainty, indicating instability in their predictions across popular benchmarks, regardless of evaluation method—be it label-based or sequence-based. Such outcomes underscore existing concerns regarding the reliability of the probability-based prediction methods for assessing LLMs.
Correlation to Human Preferences We extend our investigation to determine if probability-based prediction methods exhibit discrepancies with human preferences. Specifically, we analyze Spearman’s correlation between the outcomes from the sub-categories of the MMLU and the human preferences gathered from the Chatbot Arena (for further details, refer to Section A.2), focusing on five LLMs that are addressed in both our study and the Chatbot Arena.
We present the categories showing the top-5 and bottom-5 correlations with Elo scores in Figure 3. Our analysis reveals that LLMs exhibit stronger correlations with human preferences in social science subjects (such as world religions, politics, business,
and public relations) from MMLU, while displaying notably lower consistency with human judgments in natural science subjects (including college mathematics, formal logic, and college physics). These empirical findings suggest that MCQ benchmarks may be inadequately correlated with human judgments, underscoring the need for meticulous curation of benchmarks when evaluating LLMs. Additionally, it is important to note that human judgments themselves may be subject to biases, highlighting the complexity and caution of relying solely on human judgments (Wu and Aji, 2023; Hosking et al., 2023).
More Disagreement under Few-shot Learning LLMs typically demonstrate superior performance in few-shot in-context learning compared to zeroshot generation (Dong et al., 2022). Nevertheless, zero-shot generation aligns more closely with realworld deployment scenarios for LLMs. Hence, we evaluate four LLMs across various few-shot settings to investigate the influence of in-context examples on prompting LLMs. The results, illustrated in Figure 4, reveal a decline in agreement between probability-based and generation-based prediction methods for all selected LLMs with K in-context examples provided. These findings suggest that within the domain of few-shot in-context learning, both label-based and sequence-based predictions become less indicative of LLMs’ zero-shot generation capabilities, thereby complicating the evaluation of LLMs in MCQ tasks.
Effect of Multilingual Evaluation We conducted additional experiments on multilingual Belebele to evaluate the performance of two large language models (LLMs), Mistral-7B and LLaMA2-7B, in languages beyond English. Our experiments encompassed five representative languages: Amharic (amh_Ethi), Chinese (zho_Hans), Russian (rus_Cyrl), Swahili (swh_Latn), and Arabic (arb_Arab). The results, depicted in Figure 5, indicate that LLMs exhibit lower agreement between sequence-based predictions and generation-based predictions compared to the agreement observed between label-based predictions and generationbased ones. Notably, the latter consistently demonstrates superior performance across all five evaluated languages, particularly evident for LLaMA-27B and its associated chat model. Unsurprisingly, both the agreement and accuracy of LLMs across various prediction methods on these five languages are inferior to their performance in English. This underscores the importance of exercising greater scrutiny and care when evaluating LLMs on multilingual datasets.
4 Moving Forward
To make sure the future research in LLMs more reliable, it is crucial to reevaluate our current benchmarks and evaluation methodologies. Our analysis indicates a misalignment between these traditional evaluation mechanisms, primarily MCQbased benchmarks and output probability metrics, and the practical usage of generative text appli-
cations in LLMs. The prevalent focus on these benchmarks, although useful for fast and quantitative comparison, falls short of capturing the full spectrum of LLM capabilities.
In response to these challenges, we propose several forward-looking recommendations for the LLM research community:
Do Not Take Leaderboard Scores at Face Value: The emphasis on leaderboard rankings, while serving as a proxy for LLM performance, often overlooks the complexity of tasks that LLMs are now being developed to perform. As a community, we should not be easily over-hyped with leaderboard chasing, especially considering the limitations on either MCQ-based, or voting-based leaderboards as discussed in this paper.
Develop Comprehensive Evaluation Protocols: Future research should focus on creating evaluation frameworks that encompass a broader range of LLM capabilities. The discrepancy between evaluation measures and real-world applicability underscores the necessity for a more holistic approach to LLM evaluation. This includes not just traditional benchmarks but also metrics that evaluate free-text generation, contextual understanding, and conversational engagement. Crafting these comprehensive evaluation protocols will be challenging yet essential for a deeper understanding of LLM performance and applicability.
Embrace Slow Research: The field should adopt a more deliberate pace of research, prioritizing understanding over the speed of advancement and leaderboard-chasing. Given the rapid advancements in LLMs, there has been a noticeable rush to create the next generation of these models, often at the expense of scientific understanding. A consequence of this is that as these LLMs are evaluated using current benchmarks, their development begins to overfit to top the leaderboard. By slowing down and focusing more on understanding, we also allow more time for work on evaluation methods, potentially leading to more robust solutions.
Align Benchmarks with Human Preferences: As a short-term measure, identifying benchmark subsets that more closely mirror human preferences can help improve the correlation between traditional evaluation metrics and the generative capabilities of LLMs. However, this strategy must be balanced with caution to prevent the overfitting of models to these benchmarks, otherwise defeating the purpose of the solution. Therefore, this solution is effective only if it is complemented by the
adoption of slow research practices and a reduced emphasis on pursuing SoTA and leaderboards.
In summary, the path forward for LLM research requires a concerted effort to develop more nuanced and comprehensive evaluation frameworks. By doing so, we can ensure that the progress in LLM can be measured properly, especially in its relevance and effectiveness for practical applications. Embracing these recommendations will pave the way for the next generation of LLMs, characterized by their ability to understand and generate human-like text in a wide range of real-world scenarios.
5 Related Work
Large Language Models LLMs have demonstrated remarkable proficiency across a wide range of NLP tasks (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2022; Touvron et al., 2023a). Furthermore, recent research has shown that supervised fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) can significantly enhance their performance when following general language instructions (Weller et al., 2020; Mishra et al., 2022; Wang et al., 2022b; Chung et al., 2022; Muennighoff et al., 2022; Wu et al., 2023; Li et al., 2023a; Wang et al., 2023c; Wu et al., 2024). Zhao et al. (2023) present a comprehensive overview of the development of LLMs. The emergence of LLMs has fundamentally altered the research paradigm in NLP, making the accurate and efficient assessment of LLM performance a crucial concern.
Human Evaluation of LLMs Human evaluation plays a pivotal role in assessing the performance of LLMs and is often regarded as the “gold standard” for evaluating natural language generation (van der Lee et al., 2019; Howcroft et al., 2020). In the era of LLMs, human evaluations are extensively utilized to measure the effectiveness of these models (Wang et al., 2022a; Wu et al., 2023; Bai et al., 2023). A recent study by Zheng et al. (2023b) introduces Chatbot Arena, a platform that compares pairs of LLMs through crowd-sourced judgments in a competitive setting. Nevertheless, some recent studies challenge the validity of human judgments as the “gold standard” for evaluating machine-generated text (Wu and Aji, 2023; Hosking et al., 2023). Additionally, there is a line of research highlighting concerns over the reproducibility of human evaluation results in recent NLP studies (Shimorina and Belz, 2022; Belz et al., 2023b,a).
Automatic Evaluation of LLMs Given the limitations of human evaluation in terms of scalability and reproducibility, automatic evaluation acts as a proxy for human evaluation. The performance of LLMs has plateaued on conventional NLP benchmarks (Rajpurkar et al., 2016; Wang et al., 2019). Consequently, more recent studies have shifted towards utilizing human exam questions as a means to further test and challenge the capabilities of LLMs (Hendrycks et al., 2021; Li et al., 2023b; Koto et al., 2023; Cobbe et al., 2021). With the continuous advancements in LLMs, recent research has explored using state-of-the-art LLMs, such as GPT4 (OpenAI, 2023) and Claude-2 (Bai et al., 2022b), for evaluating model outputs (Li et al., 2023c; Wu and Aji, 2023; Liu et al., 2023; Wu et al., 2024). However, the reliability of LLM-based evaluation remains an open question (Wang et al., 2023a; Li et al., 2023d).
Ours Considering the limitations of human evaluation in terms of scalability and reproducibility, leveraging automatic evaluation to assess Large Language Models (LLMs) becomes essential. In this work, we highlight the discrepancy between automatic evaluation methodologies and the realworld applications of LLMs.
6 Conclusion
This work critically examines the alignment between probability-based evaluation methods for LLMs and their actual performance in generating text, particularly on benchmarks such as MMLU, TruthfulQA, and Belebele. Our findings highlight a significant gap between these prediction methods and the practical utility of LLMs, suggesting that current methods might not accurately reflect a model’s real-world capabilities. The discrepancies call for a shift towards more comprehensive evaluation frameworks that prioritize the quality of generated text and the model’s ability to understand and respond in human-like ways. Future research should focus on developing evaluation metrics that more accurately capture the essence of LLM performance in practical scenarios. In summary, our study underscores the need for revising LLM evaluation practices to ensure they accurately estimate the models’ effectiveness in real-world applications. By adopting more relevant evaluation criteria, we can better gauge the progress and utility of LLM advancements.
Limitations
In this paper, we selected three representative benchmarks to evaluate various LLMs, but these benchmarks might not be comprehensive enough to reflect the evaluation issue of LLMs since they only cover examination questions (MMLU), factoid questions (TruthfulQA) and general reading comprehension (Belebele). Moreover, due to the limitation of computational resources we only evaluate ten LLMs which might not be fullly reflective of how LLMs behave when facing such MCQ questions, so more LLMs should be incorporated when more resources are available.
This position paper, while exploring and empirically showing the current misalignment issue in LLM evaluation, does not explore practical solutions beyond suggestions on where the field should go. Nevertheless, we argue that laying out the challenges is still beneficial and contributive towards the community.
References Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed
Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023. Palm 2 technical report. CoRR, abs/2305.10403.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A
general language assistant as a laboratory for alignment. CoRR, abs/2112.00861.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. CoRR, abs/2309.16609.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional AI: harmlessness from AI feedback. CoRR, abs/2212.08073.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.
Anya Belz, Craig Thomson, and Ehud Reiter. 2023a. Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP. In The Fourth Workshop on Insights from Negative Results in NLP, pages 1–10, Dubrovnik, Croatia. Association for Computational Linguistics.
Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023b. Non-repeatable experiments and nonreproducible results: The reproducibility crisis in human evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt quality.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. CoRR, abs/2210.11416.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Tom Hosking, Phil Blunsom, and Max Bartolo. 2023. Human feedback is not gold standard. CoRR, abs/2309.16349.
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2305.08322.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.
Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore. Association for Computational Linguistics.
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023a. Bactrian-x : A multilingual replicable instruction-following model with low-rank adaptation. CoRR, abs/2305.15011.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023b. CMMLU: measuring massive multitask language understanding in chinese. CoRR, abs/2306.09212.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023c. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.
Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2023d. Split and merge: Aligning position biases in large language model based evaluators. CoRR, abs/2310.01432.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. CoRR, abs/2211.09110.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. CoRR, abs/2301.13688.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions.

heading: In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
text: 1: Long Papers), pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. CoRR, abs/2211.01786.
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, and Adhiguna Kuncoro. 2023. On “scientific debt” in NLP: A case for more rigour in language model pre-training research. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8554–8572, Toronto, Canada. Association for Computational Linguistics.
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal
Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of
options in multiple-choice questions. arXiv preprint arXiv:2308.11483.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.
Anastasia Shimorina and Anya Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75, Dublin, Ireland. Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023b. Llama: Open and efficient foundation language models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023c. Llama 2: Open foundation and finetuned chat models.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation,
pages 355–368, Tokyo, Japan. Association for Computational Linguistics.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. CoRR, abs/2306.07899.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261–3275.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023a. Large language models are not fair evaluators. CoRR, abs/2305.17926.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a. Self-instruct: Aligning language model with self generated instructions. CoRR, abs/2212.10560.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. 2023c. Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation. arXiv preprint arXiv:2311.16511.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361–1375, Online. Association for Computational Linguistics.
Minghao Wu and Alham Fikri Aji. 2023. Style over substance: Evaluation biases for large language models. CoRR, abs/2307.03025.
Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024. Adapting large language models for document-level machine translation. arXiv preprint arXiv:2401.06468.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. 2023. Lamini-lm: A diverse herd of distilled models from large-scale instructions. CoRR, abs/2304.14402.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023a. On large language models’ selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. CoRR, abs/2306.05685.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: less is more for alignment. CoRR, abs/2305.11206.
A Appendix
A.1 Experimental Setup A.1.1 Datasets MMLU The Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021) benchmark is a comprehensive test designed to assess knowledge acquired during pretraining of language models, especially in zero-shot and fewshot settings. Introduced by (Hendrycks et al., 2021)., MMLU encompasses 57 subjects across diverse fields including STEM, humanities, social sciences, and others, making it a broad measure of both world knowledge and problem-solving ability (Hendrycks et al., 2021). The dataset contains 17,803 examples with a range of difficulties, from elementary to advanced professional levels. Its comprehensive nature allows for a detailed examination of a model’s strengths and weaknesses across various disciplines.
Truthful-QA The Truthful-QA dataset (Lin et al., 2022) is a benchmark to assess the truthfulness of language model responses to questions. This dataset contains 817 questions spanning 38 diverse categories, including health, law, finance, and politics. The key characteristic of Truthful-QA is its design to elicit imitative falsehoods, wherein some questions are crafted to provoke false answers based on common misconceptions or false beliefs. The dataset aims to test language models’ ability to avoid generating false answers that may have been learned through imitating human texts. Importantly, the Truthful-QA questions are adversarial in nature, designed to pinpoint weaknesses in the truthfulness of language models. Additionally, it features a set of true and false reference answers for each question, backed by reliable sources.
Belebele The Belebele Benchmark (Bandarkar et al., 2023) is a massively multilingual reading comprehension dataset designed to evaluate machine reading comprehension (MRC) capabilities across various languages. Developed by Facebook Research, it features 900 multiple-choice questions per language, spanning 122 language variants, totaling 109,800 questions linked to 488 distinct passages. Each question has four answer options, with only one correct answer. This benchmark encompasses a wide range of languages, from highresource to low-resource, making it ideal for assessing the performance of language models in diverse linguistic contexts.
A.1.2 Models LLaMA LLaMA-1 (Touvron et al., 2023b), Vicuna (Chiang et al., 2023) and LLaMA-2 (Touvron et al., 2023c) is a family of large language models (LLMs), encompassing a range of pretrained and fine-tuned generative text models with parameters varying from 7 billion to 70 billion. The model was trained on a new mix of publicly available online data, with a considerable size of 2 trillion tokens, and includes over one million humanannotated examples for fine-tuning. Its training and evaluation emphasize both performance and safety. These fine-tuned models have shown superior performance in human evaluations for helpfulness and safety, matching or even surpassing other well-known models like ChatGPT and PaLM in certain aspects.
Mistral The Mistral model (Jiang et al., 2023) equipped with 7.3 billion parameters, is designed to outperform its counterparts in terms of efficiency and effectiveness. Notable features of Mistral 7B include its proficiency in outperforming LLaMA-2-13B (Touvron et al., 2023c) across various benchmarks and approaching the performance of CodeLLaMA-7B (Rozière et al., 2023) in coderelated tasks while maintaining strong English language capabilities. Additionally, Mistral 7B incorporates Grouped-query attention (GQA) for faster inference and Sliding Window Attention (SWA) to manage longer sequences more economically.
lm-harness The lm-harness (Gao et al., 2021) †, developed by EleutherAI, is a comprehensive framework designed for the few-shot evaluation of autoregressive language models. This library is pivotal in the field of natural language processing for assessing the performance of language models in few-shot settings. It stands out due to its versatility and ability to handle a variety of language models, making it a valuable tool for researchers in the field. The lm-harness library facilitates robust and efficient evaluations, contributing significantly to advancements in language model development and assessment (Gao et al., 2021).
A.2 Elo-based Chatbot Arena Leaderboard
In the Elo-based Chatbot Arena Leaderboard, crowds are given an interface to ask questions to LLMs. The users are then given 2 options from 2
†https://github.com/EleutherAI/ lm-evaluation-harness
anonymous LLMs, in which the user has to vote for the better one, which will be the winner LLM. Based on several win-lose interactions, we can then calculate the Elo score.
Elo scores have been previously designed in rank multiple players that involve multiple matches across different people, such as chess. It is good for determining a unified ranking across every player (in this case, LLMs). From the Elo score of 2 players, we can predict the winning chance of both players. For example, an LLM with an Elo of 1200 will win against an LLM with an Elo of 900 85% of the time.
Chatbot Arena is one of the popular Elo-based leaderboards. It supports a variety of LLMs, both proprietary and open-sourced, and has accumulated hundreds of thousands of votes.

heading: Future Work
text: 4 Moving Forward
To make sure the future research in LLMs more reliable, it is crucial to reevaluate our current benchmarks and evaluation methodologies. Our analysis indicates a misalignment between these traditional evaluation mechanisms, primarily MCQbased benchmarks and output probability metrics, and the practical usage of generative text appli-
cations in LLMs. The prevalent focus on these benchmarks, although useful for fast and quantitative comparison, falls short of capturing the full spectrum of LLM capabilities. In response to these challenges, we propose several forward-looking recommendations for the LLM research community:
Do Not Take Leaderboard Scores at Face Value: The emphasis on leaderboard rankings, while serving as a proxy for LLM performance, often overlooks the complexity of tasks that LLMs are now being developed to perform. As a community, we should not be easily over-hyped with leaderboard chasing, especially considering the limitations on either MCQ-based, or voting-based leaderboards as discussed in this paper. Develop Comprehensive Evaluation Protocols: Future research should focus on creating evaluation frameworks that encompass a broader range of LLM capabilities. The discrepancy between evaluation measures and real-world applicability underscores the necessity for a more holistic approach to LLM evaluation. This includes not just traditional benchmarks but also metrics that evaluate free-text generation, contextual understanding, and conversational engagement. Crafting these comprehensive evaluation protocols will be challenging yet essential for a deeper understanding of LLM performance and applicability. Embrace Slow Research: The field should adopt a more deliberate pace of research, prioritizing understanding over the speed of advancement and leaderboard-chasing. Given the rapid advancements in LLMs, there has been a noticeable rush to create the next generation of these models, often at the expense of scientific understanding. A consequence of this is that as these LLMs are evaluated using current benchmarks, their development begins to overfit to top the leaderboard. By slowing down and focusing more on understanding, we also allow more time for work on evaluation methods, potentially leading to more robust solutions. Align Benchmarks with Human Preferences: As a short-term measure, identifying benchmark subsets that more closely mirror human preferences can help improve the correlation between traditional evaluation metrics and the generative capabilities of LLMs. However, this strategy must be balanced with caution to prevent the overfitting of models to these benchmarks, otherwise defeating the purpose of the solution. Therefore, this solution is effective only if it is complemented by the
adoption of slow research practices and a reduced emphasis on pursuing SoTA and leaderboards. In summary, the path forward for LLM research requires a concerted effort to develop more nuanced and comprehensive evaluation frameworks. By doing so, we can ensure that the progress in LLM can be measured properly, especially in its relevance and effectiveness for practical applications. Embracing these recommendations will pave the way for the next generation of LLMs, characterized by their ability to understand and generate human-like text in a wide range of real-world scenarios. 5 Related Work
Large Language Models LLMs have demonstrated remarkable proficiency across a wide range of NLP tasks (Brown et al. , 2020; Chowdhery et al. , 2022; Scao et al. , 2022; Touvron et al. , 2023a). Furthermore, recent research has shown that supervised fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) can significantly enhance their performance when following general language instructions (Weller et al. , 2020; Mishra et al. , 2022; Wang et al. , 2022b; Chung et al. , 2022; Muennighoff et al. , 2022; Wu et al. , 2023; Li et al. , 2023a; Wang et al. , 2023c; Wu et al. , 2024). Zhao et al. (2023) present a comprehensive overview of the development of LLMs. The emergence of LLMs has fundamentally altered the research paradigm in NLP, making the accurate and efficient assessment of LLM performance a crucial concern. Human Evaluation of LLMs Human evaluation plays a pivotal role in assessing the performance of LLMs and is often regarded as the “gold standard” for evaluating natural language generation (van der Lee et al. , 2019; Howcroft et al. , 2020). In the era of LLMs, human evaluations are extensively utilized to measure the effectiveness of these models (Wang et al. , 2022a; Wu et al. , 2023; Bai et al. , 2023). A recent study by Zheng et al. (2023b) introduces Chatbot Arena, a platform that compares pairs of LLMs through crowd-sourced judgments in a competitive setting. Nevertheless, some recent studies challenge the validity of human judgments as the “gold standard” for evaluating machine-generated text (Wu and Aji, 2023; Hosking et al. , 2023). Additionally, there is a line of research highlighting concerns over the reproducibility of human evaluation results in recent NLP studies (Shimorina and Belz, 2022; Belz et al. , 2023b,a). Automatic Evaluation of LLMs Given the limitations of human evaluation in terms of scalability and reproducibility, automatic evaluation acts as a proxy for human evaluation. The performance of LLMs has plateaued on conventional NLP benchmarks (Rajpurkar et al. , 2016; Wang et al. , 2019). Consequently, more recent studies have shifted towards utilizing human exam questions as a means to further test and challenge the capabilities of LLMs (Hendrycks et al. , 2021; Li et al. , 2023b; Koto et al. , 2023; Cobbe et al. , 2021). With the continuous advancements in LLMs, recent research has explored using state-of-the-art LLMs, such as GPT4 (OpenAI, 2023) and Claude-2 (Bai et al. , 2022b), for evaluating model outputs (Li et al. , 2023c; Wu and Aji, 2023; Liu et al. , 2023; Wu et al. , 2024). However, the reliability of LLM-based evaluation remains an open question (Wang et al. , 2023a; Li et al. , 2023d). Ours Considering the limitations of human evaluation in terms of scalability and reproducibility, leveraging automatic evaluation to assess Large Language Models (LLMs) becomes essential. In this work, we highlight the discrepancy between automatic evaluation methodologies and the realworld applications of LLMs