Beyond accuracy: understanding the performance of LLMs on exams designed for humans

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:26:17

Abstract: Many recent studies of LLM performance have focused on the ability of LLMs to achieve outcomes comparable to humans on academic and professional exams. However, it is not clear whether such studies shed light on the extent to which models show reasoning ability, and there is controversy about the significance and implications of such results. We seek to look more deeply into the question of how and whether the performance of LLMs on  exams designed for humans reflects true aptitude inherent in LLMs. We do so by making use of the tools of psychometrics which are designed to perform meaningful measurement in test taking.  We leverage a unique dataset that captures the detailed performance of over 5M students across 8 college-entrance exams given over a span of two years in Brazil. With respect to the evaluation of LLM abilities, we show that the tools of Item Response Theory (IRT) provide a more informative evaluation of model performance than the usual accuracy metrics employed in previous studies.  Digging deeper, we show that the modeling framework of IRT, by explicitly modeling the difficulty levels of questions, allows us to quantitatively distinguish between LLMs that answer questions in “human-like” patterns versus LLMs that do not. We also show how to quantitatively identify cases in which exam results are not reliable  measurements of an LLM's ability.  Using the tools of IRT we can also identify specific questions that appear to be either much easier, or much harder, for machines than for humans, and we give some reasons for those differences. Overall, our study shows that the conventional focus on accuracy as the primary performance metric for LLM studies  does not allow us to deeply understand the true capabilities of LLMs and compare them to that of humans. Thus, we claim that psychometric modeling should play a larger role in the evaluation of LLM capabilities on exams designed for humans.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have demonstrated an impressive ability in performing well on examinations designed for humans [25, 30] , such as the US bar exam [27] , the US Medical Licensing Exam [21] , and many others [45, 53] . This yields controversy in how researchers should interpret such results, raising two kinds of criticisms of those apparent successes. The first is the potential for publicly-given exams (and answers) to leak into models' training data. The second, and more fundamental, issue is the notion of construct validity [44] . Most exams given to humans are intended to measure a construct, e.g., legal analysis ability, medical analysis ability, etc. However, the reliability of these exams in measuring the relevant construct for non-humans is usually ignored, and exams that are valid in one context may not generalize across different groups, settings or tasks [24] .Formalizing the notion of construct validity in general is challenging. Since the 1950s, the field of psychometrics has been grappling with how to design examinations that validly measure human abilities along specific dimensions. The primary tool developed has been Item Response Theory (IRT) [10] , which has been employed in psychology, medicine, and especially in educational test-Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.ing. IRT formalizes the unobserved construct as a continuous latent variable, and models stochastic responses of humans to questions as a logistic regression conditional on that latent variable.In this paper, we demonstrate how IRT can help shed light on whether LLMs are in fact showing human-like performance on exams intended for humans. As a case study, we use one of the largest university-entrance exams in the world, a dataset comprising the performance of over 5 million Brazilian students on eight multiple-choice exams administered over two years. Each exam was prepared and fitted to an IRT model by educational testing experts, giving us an unparalleled opportunity to examine the performance of LLMs in detail.Our results show that the LLMs we study reveal performance patterns that are consistent with expected human behavior in many cases. Nonetheless, we also frequently observe significant deviation from human-like behavior. We demonstrate how to use the tools of IRT to quantitatively distinguish between human-like and non-human-like behavior. We then explore the differences between models and exam types that correlate with differences in response patterns. Lastly, we use the tools of IRT and psychometrics to identify cases where exams are not producing reliable estimates of LLM ability and understand why this happens. This occurs because exams are in some cases too difficult for the models, and in other cases too easy for them and as such they cannot properly measure the ability of certain LLMs.Moving beyond conclusions about current models, the broader contribution of our study is to demonstrate the power of IRT as a framework for evaluating LLMs. For example, in Classical Test Theory (CTT), no attempt is made to assess the difficulty of individual questions, in-line with majority of in standard LLM benchmarks that pursues accuracy [8, 43, 16, 4] . In contrast, as we will show below, IRT simultaneously measures both test takers and exam questions (on the same scale). In doing so, IRT allows one to distinguish between test takers with similar CTT (accuracy) scores, but differing levels of true ability, by inspecting the pattern of correct or incorrect answers given. Moreover, we deploy a broader set of tools (e.g., goodness-of-fit, Fisher information, discrimination index) which enable us to evaluate which are the cases in which fitting the IRT model to the LLMs response patterns gives us reliable estimates of the models' ability. Thus, we believe that the methods of our study represent a valuable step beyond the use of simple accuracy for assessing whether both current and future LLMs show human-like response patterns. 
related work: 
 Our study connects a number of research areas, spanning benchmarking LLMs, the applications of item response theory, and the evaluation of LLMs using exams designed for humans.Benchmarking LLMs. The most common strategy to evaluate LLMs is through traditional largescale NLP benchmarks [46, 40, 11, 18, 16, 19, 4] . Conventionally, benchmark evaluation relies on some notion of accuracy -the number of correct answers -as a proxy for ability [8, 43] . A key distinction of our study is to draw attention to the limitations of the use of accuracy alone [34] for evaluating the performance of LLMs on benchmarks in understanding the similarity between the performance of models versus humans.LLMs and Exams Designed for Humans. Many attempts to evaluate LLMs use exams designed for humans, e.g., at college-entrance [1, 26] or college-level [14, 37, 47, 13, 41, 53] . These exams also generally use accuracy as a metric of ability; one focus of our work is on how to use IRT analysis to determine when such exams in fact perform meaningful measurement.The Brazilian nationwide college-entrance exams we use in this work (ENEM), detailed in Section 4.1, were used in previous efforts to evaluate NLP models [38, 39, 26] . However, those studies only used accuracy and did not make use of the IRT models associated with the exam, which is a central aspect our work.IRT in Machine Learning. Work in psychometrics (i.e., the measurement of human cognitive abilities), detailed in Section 3, has shown that using accuracy as a exam score may not reflect the true underlying abilities of individuals [15] . As a result, IRT has been advocated for use in machine learning (ML) as an improved tool for benchmarking. The authors in [33] show that it is possible to produce rankings of NLP models which are more reliable and stable using IRT than accuracy. Item response theory has also been shown to help in spotting noisy questions, identifying overfitting, selecting features, and designing better benchmarks for ML [29, 35, 20, 54, 22] . However, there is a critical difference between the previous uses of IRT in ML and our work. Previous work uses IRT by training an IRT model on the results of ML models solving question-answering or classification questions. Our method is different: we leverage the fact that we have access to an IRT model trained on human responses, and we do not retrain on model responses. We take this approach because a central goal of our study is to explore whether LLMs are in fact following response patterns as exhibited by human test takers.Finally, we note that [42] shares some goals with our work. The investigation seeks to understand whether LLMs show human-like response biases in surveys. We also look at the question of whether LLMs show human-like response patterns, but we study the question along different dimensions:(a) patterns of correct and incorrect answers in exams; and (b) the ways in which LLMs choose incorrect answers. Additionally, Xia et al. [51] recognize that accuracy as a single metric does not capture errors LLMs can make in intermediate steps when solving mathematical tasks, and they systematically study those errors. 
background: 
 In this section, we give some background of the tools we use from psychometrics.Classical Test Theory (CTT): CTT [2] evaluates test takers based on the fraction of questions they answer correctly. We call this score accuracy or CTT score of the test taker and we use these two terms interchangeably. Inadequately, CTT does not differentiate between difficult and easy questions, nor does it take into consideration the patterns of correct answers. For example, the CTT score does not penalize a test taker who answers correctly difficult questions, but answers wrongly easy ones -despite the fact that such a pattern might be indicative of randomness or cheating.Item Response Theory (IRT): IRT [12, 5] is a model used extensively in psychometrics to measure the ability level of the test takers and evaluate the difficulty of the test questions (which are referred to as items in psychometrics). IRT takes into consideration the difficulty of the questions when evaluating test-taker's performance and also makes use of the pattern of correct and incorrect responses on the exam. The model associates with every test taker j a parameter θ j , which corresponds to the ability of j. The two-parameter IRT model (2PL) associates every question i with two parametersϕ i = (α i , β i ).The model assumes that a test taker with ability θ j answers question i associated with ϕ i correctly with probability given by the logistic function: -βi) .p ij = e αi(θj -βi) 1 + e αi(θj(EQUATIONParameter α i is the discrimination parameter and β i is the difficulty of question i. Note that the ability θ j and the difficulty level β i are in the same scale; after all, the difference (θ j -β i ) directly affects p ij . For fixed α i , the difficulty parameter β i is the value (on the ability scale) for which p ij = 0.5. Parameter α i characterizes how well question i can differentiate among test takers located at different points of the ability continuum; α i is proportional to the slope of p ij = p i (θ j )at the point where p ij = 0.5 -the steeper the slope, the higher the discriminatory power of i. All the parameters of this model take values in (-∞, +∞). Note that any set of questions comprising an exam spans a certain range of β i values; such a set is not appropriate to assess test takers with abilities outside this range.The 3-Parameter IRT model (3PL for short) is an extension of the above model that also incorporates a pseudo-guessing parameter γ i . Thus, in 3PL every question i is associated with three parameters Φ i = (α i , β i , γ i ); α i and β i are the same as before. Intuitively, γ i is the probability of answering correctly based on random guess with γ i ∈ [0, 1] . Thus, the probability of a test taker with ability θ j to answer question i correctly is:P ij = γ i + (1 -γ i )p ij .Given test-taker responses, the parameters of the model can be estimated using Bayesian methods [5] . In our case, the ENEM dataset came with a set of questions for which the parameters (α i , β i , γ i ) had already been fitted by education experts [17] . Therefore, for each one of the LLMs we considered, we only need to compute their ability parameters -given their response patterns.Intuitively, large values of θ correspond to test takers with high ability levels and vice versa. High ability value θ of an LLM implies better performance.Although the ability levels of test takers can be used as a measure of their performance, one should also know if the test takers are consistent with the model, e.g., they should answer easy questions correctly if they answer difficult questions correctly. One index that enables us to evaluate the consistency of the test takers with the model is the l z index [12] . Intuitively, the l z index is based on the standardization of a test-taker's log-likelihood function given their theta values. Assume a set of I questions and test taker j with ability θ j and response vector r j such that r j (i) = 1 (resp.r j (i) = 0) if j answered question i correctly (resp. wrongly). Then, the log-likelihood of j is simply:L j = i∈I [r j (i) ln P ij + (1 -r j (i)) ln(1 -P ij )] .To standardize L j we need both its mean (E[L j ]) and variance (Var(L j )). Then, the l z score is computed as:EQUATIONIn a well-designed test, the l z scores are expected to have a unit normal distribution -this is the case for humans taking the ENEM test (see for example Figure 3 ). In general, l z values close to 0 are considered good: it means the test takers' response patterns are consistent with what is expected from them by the model. Negative l z (j) scores reflect an unlikely response vector. A positive l z (j) score indicates that j has a more likely response vector than indicated by their ability.We can access the amount of information that an item i provides to estimate θ under the 3PL model by the Fisher information, which is given by:EQUATIONThe total information of a test is simply the sum of item information, i.e., I(θ)= i∈I I i (θ).The Fisher information is connected with the standard error of the estimation, given by SE(θ) = 1/ I(θ). When a test has high Fisher information in a certain θ range, the test has more discriminative power in that range, producing scores with less measurement errors. 
the enem exam: 
 The Exame Nacional do Ensino Médio (ENEM), world's second largest university entrance exam behind Chinese's Gaokao exam, is taken by millions of Brazilian students each year [39] . ENEM comprises questions requiring different levels of domain-specific knowledge and reasoning [3] .The exam is in Brazilian Portuguese and consists of four sections, each of which has 45 multiplechoice questions with five options [17] . Each section is treated as a separate exam for the purposes of modeling via IRT. The four sections consist of the Humanities, the Languages and Codes, the Natural Sciences, and the Math exams. The description of these exams is given in Appendix A.11.Since 2009, the grades assigned to ENEM test-takers have been determined using IRT. Using IRT helps to penalize guessing, differentiate among students that otherwise would get the same (CTT) grade, and compare among students that took exams in different years. The ENEM organizers release not only the exam content and questions, but also the student (anonymized) responses and their CTT and IRT scores, which enables downstream studies.From our standpoint, there are a number of relevant aspects of the process used by the ENEM developers [17] . First, questions are given to a sample of students, whose answers are used to find inconsistencies and errors. Next, an important test of construct validity is to verify the unidimensionality of the latent trait, for which the ENEM team uses Full Information Factor Analysis [7] .Finally, the IRT model itself is fit using the Marginal Maximum Likelihood Estimator [6] . Using the results, the developers may exclude questions having poor model fit.The exams, their solutions, and all the fitted parameters of the 3PL IRT model (θ j , α i , β i , γ i ) are publicly available at the Brazilian government website [17] . To the best of our knowledge, these data are the largest and most comprehensive public dataset based on item-response theory available. The datasets contain questions and complete response patterns of all students taking the exams in 2022 and 2023. Questions for the 2023 exam were released in November 2023, minimizing the chance they are in training data for most of the LLMs we considered. However, we expect fragments of the exam being in the training data (e.g. poems, and any other widely available material used as part of a question) 1 . The number of test takers per year ranged from 2.2M to 3.7M.The ENEM exams are initially made available as PDF files; we used the Python library PyPDF2, followed by regular expressions and some manual adjustments to extract each question from its exam file. In order to account for possible effects of Language, as diagnosed in previous work [31] ,we translated all questions to English and run all experiments in Portuguese and English. For those exam questions that incorporated images, we used the version of the exam designed for blind people containing textual descriptions of the images. We manually audited all questions in 2022 and 2023 exams to ensure their quality (Appendix A.1).We evaluate the following family of models: the open source models Mistral-7B, Gemma-7B, Llama2-7B, Llama2-13B, Llama3-8B, and GPT 3.5. For the open source models, we evaluate on both instructed and non-instructed tuned versions. Our choice of models enables the study of models of similar size (the majority of our models are of size 7B), but also introduces diversity of architectures (GPT, Gemma, Mistral, Llama), size (7B vs. 13B), training data (Llama2 vs. Llama3),and training strategies (with and without instruction tuning).We prompt models with {0, 1, 4}-shots, following conventional question-answer benchmark prompting strategies [32] (example prompts in Appendix A.4). We measure model's next token probability across five option letters, and average predictions across 30 shuffles of the order of the answer choices to correct for the well-known effect of position bias [28] (Details in Appendix A.4). 
results: 
 In this section, we present our main findings. All the results we show here are for the 2023 ENEM exams, with four-shot prompting. Results for the 2022 ENEM exam and for zero-shot and one-shotprompting and for open source instructed tuned models are shown in Appendices A.6 -A.9. The results we show in this section are strongly consistent with the results we get for the 2022 ENEM exam and for one-shot prompting.We first investigate how humans compare to LLMs when IRT parameter θ is used instead of accuracy (the metric that is employed in most LLM benchmarking, e.g., [8, 43] ). In Figure 1 we plot the CTT score (accuracy) vs IRT score (θ) for 30 shuffles of answer options for each model. The light blue background points correspond to the humans who took the exam. Each of the closed curves in the figure corresponds to one LLM, and shows the central 90% of the LMM's distribution.First, we observe that there are many cases where identical accuracy scores result in different θ scores. This reflects the fact that IRT takes into account not just the number, but also the pattern of correct answers. Second, for many LLMs, particularly in the Humanities and Languages exams, there is overall greater variability in the accuracy score than in the IRT score. This suggests that IRT is less sensitive to the variations in LLM output that are due to the LLM's inherent randomness.To compare the performance between LLMs and humans, we compare their IRT scores (θ). Recall that IRT score of 0 corresponds to the average ability of a human test taker. Across all four subjects, the majority of models have CTT and IRT scores overlapping with humans. LLMs in general achieve θ scores above that of the human average in Humanities, Languages, and Natural Sciences, but below human average in Mathematics. Looking at specific models, we find the Llama2 models at the lower end of θ scores, Mistral and Llama3 in the middle range, and GPT-3.5 and Gemma-7B at the higher end of θ scores.The language of the exam affects some models' performance. In Languages and Natural Sciences, GPT-3.5 tends to perform better in Portuguese compared to English, while in Humanities and Natural Sciences, the Llama models tend to perform worse in Portuguese than in English. This suggests that there are differences regarding the reasoning ability and the amount of knowledge accessible to the models in each language.Gemma-7B (EN) Gemma-7B (PT-BR) LLaMA2-13B (EN) LLaMA2-13B (PT-BR) LLaMA2-7B (EN) LLaMA2-7B (PT-BR) LLaMA3-8B (EN) LLaMA3-8B (PT-BR) Mistral-7B (EN) Mistral-7B (PT-BR)Importantly, outlier models all tend to have higher accuracy and/or lower IRT scores than humans.These models answer more questions correctly than humans do, but show error patterns that are not entirely human-like. We dig into this phenomenon next.One of our goals is to assess whether the LLMs we examine show good fit to the ENEM IRT model, as crafted by the educational expert team described in Section 4.1. Intuitively, a test taker showing good fit to an IRT model is an individual j that tends to make less frequent mistakes on "easy" questions (question i with β i < θ j ) while making more frequent mistakes on "hard" questions (question i with β i > θ j ). Thus, to assess fit we need to inspect the response patterns of the LLMs.Figure 2 shows the response patterns of LLMs for the 2023 exam. Every cell (i, j) corresponds to the probability that LLM i answered question j correctly, where probabilities are computed over the 30 shuffles. We use gray scale with a black (resp. white) cell representing 1 (resp. 0). Questions are ordered in increasing order of their β values. Generally, rows with darker overall patterns (higher correctness) are indicative of higher θ scores.The figure demonstrates a number of points. For example, on the Math exam, the figure exhibits a response pattern that appears to show low θ values for all models, which confirms results in Figure 1 .In addition, the figure shows that for some questions, the 30 shuffles of answer choices of a given model are often either all correct or all incorrect. However, there are some grey areas in the figure for all the exams, indicating that shuffling the options can affect the LLM's answers on certain items. Furthermore, the patterns show that many questions appear to be either "easy" (black) or "hard" (white) for all models at the same time. Likewise, in many cases models show similar performance on the English and Portuguese versions of a given question.Overall, the response patterns we observe suggest that the Math exam is "too difficult," with models often resorting to guessing. On the other hand, most LLMs consistently answer correctly the questions in the Humanities exam, implying that this is an easy exam for them. The performance of LLMs in the Natural Science exam is the most interesting as there are blocks of questions that most LLMs answer consistently correctly, interleaved with blocks of questions that most LLMs answer incorrectly. This suggests that there are questions that are easy for humans but difficult for LLMs and vice versa. In the next subsection we analyze this phenomenon more closely.In this section, we investigate whether the ENEM exam is a valid test for LLMs' ability, in the same way it is for humans. Intuitively, we want to define measures that allow us to quantify to what extent we trust the IRT scores we obtained for LLMs. We propose three different ways of doing this. The first is goodness-of-fit that quantifies whether the response of LLMs fit the IRT model. The second is based on Fisher information, measuring how much information the exam provides for estimating the θs in a certain range. Finally, we use the discrimination index which evaluates the capacity of questions to accurately distinguish between high and low performing test takers.We use the l z score (see Section 3) assess whether the test taker is behaving in a manner consistent with the model. Alternatively, we ask what is the appropriateness of a testtaker's estimated θ as a measure of the test taker's true θ? For example, imagine that an LLM has a response pattern of missing easy questions and correctly answering more difficult ones. Such a pattern may arise because the LLM was lucky on the hard questions, or it may arise because the LLM had access to memorized patterns that assisted in answering the hard questions. Generally, low l z scores suggest that the θ estimate of the model is less reliable [12] .In Figure 3 we show l z scores plotted against θ scores of LLMs across the four exams in 2023 (2022 is shown in Appendix A.9). As in previous plots, the light blue points in the background show the distribution of the same two scores for the human test takers. Starting again with the Math exam, we note that l z values are low, but now we can see that the response patterns of the LLMs are indeed quite human-like; LLMs behave like humans with similarly low l z values. One possible reason for this behavior is that the Mathematics exam tends to be the harder exam of ENEM, leading to more guessing, which may make the human l z values for Mathematics smaller.For the Languages exam, models perform better in general (higher θ values) and the most l z scores being close to 0 (and with a similar spread as the human distribution of l z 's) suggest that these θ estimates are reliable -the models are showing human-like response patterns. The results become more nuanced as we look at the Natural Sciences exam. For this exam, most models, including the high performing ones (i.e., GPT-3.5 and Gemma-7B), show values well outside the human distribution, with a long tail in the negative values of l z . Comparing the GPT-3.5 and Gemma-7B results in Figures 1 and 4 , we can infer that the high accuracy (CTT scores) achieved by these models on the Natural Sciences exam are quite misleading; although GPT-3.5 and Gemma-7B answer many questions correctly, their response pattern is very unlikely, with very low l z values.This corroborates with Figure 2 , which shows an interchange of blocks of correct and incorrect answers from the models, creating an unlikely response pattern.In Humanities, almost all LLMs perform reasonably well, achieving θ scores above zero (the average human level). However, Llama2-7B, while obtaining above average accuracy scores (Figure 1 ) and good θ scores, has low average l z scores. This suggests that the IRT scores Llama2-7B may be not reliable. Examination of the corresponding rows in Figure 2 shows that this is the only model that does not have a consistent response pattern across shuffles, leading to the observed low l z score. Fisher Information: We investigate further whether the ENEM exams are giving us accurate estimates of the LLMs ability levels from another standpoint -that of Fisher Information (see Section 3, Equation ( 3)). Intuitively, Fisher Information quantifies whether there was enough information in the test to infer the ability level of a test taker at a certain ability level. Figure 4 shows, for every ENEM exam, the total Fisher Information I(θ) on the top plot, and the θ scores for the models (95% Confidence Interval (CI) computed using the shuffles) on the bottom plot. This plot reinforces the observation that for some models in Natural Sciences and for all models in Mathematics, the models' θ are not in the range of the exam with highest information -the models ability levels fall in the tail of the Fisher Information histogram. Hence, the Math exam is not useful for making meaningful measurements of these LLMs, casting doubt on the informativeness of the models' θ scores on this exam. The lack of discrimination ability of this exam is reflected by the responses for many models showing apparently random response patterns in the corresponding heatmap (see Figure 2 ). Discrimination Index: To further assess the reliability of the IRT scores, we also turn into psychometrics and use the notion of the item discrimination index (DI), which measures how well an item on a test distinguishes between high and low scorers on the entire test [9] . Let P h (resp. P l ) be the proportion of the top 25% (resp. low 25%) LLMs (in terms of θ, including the shuffles) that correctly answer the item; then DI = P h -P l , the difference of the two proportions. DI ranges from -1 to 1, and questions with DI higher than 0.2 are considered good, while lower DI indicates flaws [50] .Figure 5 shows the distribution of the discrimination indices computed for humans and LLMs for the 2023 exam. Overall, we notice that discrimination indices computed for LLMs are more negative compared to those of humans. We also observe that a significant fraction of Math questions have low discriminative power, reinforcing the hypothesis that this exam is not well designed to measure Math abilities for LLMs. Nonetheless, the Humanities and Languages have several questions with very good discriminative power. Interestingly, the Natural Sciences exam appears to follow a bimodal distribution, containing both informative and poorly-designed questions. This may be a reflection of the fact that the Natural Sciences exam is a hybrid test, containing a mix of knowledge-based items and items that demand more complex reasoning over numbers and images, which can be less useful for evaluating the current state-of-the-art LLMs.Attributes affecting reliability of IRT scores: In a further investigation, shown in Appendix A.2, we explore potential causes of low discrimination. We investigate item attributes such as the existence of images or numbers in the questions as we believe that these attributes impede LLMs from understanding the question properly. Our preliminary results suggest that LLMs' ability to understand math questions and parse images is sub-par compared to their capacity in answering pure text-based questions. In Appendix A.10 we show examples of non-discriminating and highly discriminating items for the 2023 Natural Sciences exam. In Appendix A.3, we reach a similar conclusion by looking at model accuracy against model perplexity, a model intrinsic metric. 
conclusions: 
 The ongoing debate in LLM evaluation centers around whether exams designed for humans are appropriate tools for measuring the performance of LLMs. In this paper, we provide a case study that illustrates methods that can be used to address this question, as well as specific results for a range of current LLMs. We leverage the largest known human exam for which a public IRT model is available, and show that IRT can be leveraged to distinguish between human-like and non-humanlike responses under the model. We show cases where LLMs respond in non-human-like ways and show how to identify those cases using a model-fit metric. Further, we show that using IRT we can determine when an exam is capable of making meaningful measurement of an LLM's ability in a given subject area. Using our evaluation framework, we find that the ENEM Math exam is not appropriate to make meaningful measurements of the models' ability, for the LLMs we study. At the same time, Humanities and Language exams are better suited for evaluating the LLMs' abilities on those subjects. We conclude that IRT modeling, drawing on a long history of psychometric theory, provides a set of crucial tools for assessing whether exams designed for humans are actually meaningful measures of LLM ability. Our results suggest that they should be used in future studies when questions are raised regarding the performance of LLMs on human exams.  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We disclose the compute related information in Appendix A.5Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We have reviewed the code of ethics and our work conforms with it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the impact of our work in Section 5 and 6.Guidelines: 
 
 Appendix: A.1 Manual auditing of exam questionsAssuming the original questions written by the ENEM authorities are good test instruments for testing student capability, we focus on ensuring the quality of adapted dataset for LLM evaluation.We manually correct the artifacts for each question in 2022 and 2023. In the next sections, we describe the artifacts from those easier to address (sec A.1.2 A.1.3), to deeper-rooted problems (i.e., harder to correct, sec A.1.4), as well as how we addressed them manually (sec A.1.5).We assume answers are correct as translation and parsing of single characters can be quite reliable, and that the original ENEM test is tested across millions of human test takers and will be discarded if it had a wrong answer. When we look at the label distribution for 2022, options "ABCDE" each occur 39/39/37/36/33 times, making it fairly balanced. We also ran random baselines on the same option shuffles as the model (Table 1 ).We found several issues pertaining to initial round of translation in this dataset. Mainly, independent translation of question context and answer option leads to incoherence. Details are sometimes mistranslated ("p.d.d" translated to "d.d.p") . There are many non-standardized translations pertaining to chemical formulas, proper nouns, and mathematical formulas. In general, there are significant amount of awkward phrasing, incomplete translation, and linguistic idiosyncrasies lost in translation.In a few cases, the answer options are expected to complete the last sentence of the question. After translation, options do not all fit as completions of the sentence (Q11). Translation without context also leads to improper translation of polysemantic terms. "Coagulation" maybe translated correctly in the question, but becomes "coagulating" as a stand-alone word (Q96). "Good" and "fair" (when used as survey options) gets translated to "regular" and "I will" as stand-alone options (Q171)Inconsistent translation details Within the same questions, there are cases where the same concept is translated differently. In one question, the context introduces the concept "potential difference (p.d.d)", and later referred to it as "d.d.p" and "d.p.d". Within different options, the same unit can sometimes be plural and sometimes be singular (when it should be consistently plural)Non-standard translation 1) Chemical formula translation is non-standard. "N2O3" becomes "N 2O3", and "NH4+" becomes "NH4 positively charged". Linguistic idiosyncrasies lost in translation In one question, the problem arises when English translation does not match with literal tokens of expressions in Portuguese ("Next to the man is the message: "Men don't cry", with a large X drawn over the word "no"). The word "no" does not appear in the English phrase "Men don't cry" but the statement as a whole makes sense in the Portuguese version of the instruction. In a separate question, the topic is on testing for a Portuguese specific pronoun inflection. However, when it was translated into one single word in English, the question no longer makes sense ("They told me. Audio description of images, tables, and figures are not always sufficient, or the most intuitive. For instance, a question asks test taker to note why a particular painting stands out, and the answer is due to the painting's "distortion when representing human figure", which is difficult to qualitatively describe, no matter how complete the description of an image is. Similarly, textual description of geometric figures can be impossibly complicated ("...Figure of a grid with 7 horizontal and 7 vertical lines, on which a polygonal path is drawn by means of a continuous line on the grid lines, joining the starting point P , located on the second vertical line, from left to right, and between the sixth and seventh horizontal lines, from top to bottom, to the end point Q , which is located between the sixth and seventh vertical lines, from left to right, and on the second horizontal line, from top to bottom...")The majority of the artifacts begin with incorrect parsing of the PDF documents related to structural components. To address this, we manually audited each question, and added correct spacing and newlines to each question. These improvements result in better translations from DeepL API qualitatively. After translation, we make minimal edits to improve syntactic and semantic issues through Grammarly to obtain a score of at least 95 2 3 . For each answer option, we ensure consistent part-of-speech, especially if they are sentence completions of the questions. For math and science sections, we follow consistent markdown-like format the same way as other mathematical reasoning datasets [16, 11, 52] . Here we list the full set of modification rules for 2022 (question numbers are referenced in parenthesis):• Separate description of the image by '\n' before and after.• "Por cento" becomes %.• Number in the form 7 000 becomes 7000.• From "abre aspas" "fecha aspas" to "".• Remove "Descric ¸ão da estrutura química", "Descric ¸ão do esquema", "Descric ¸ão da associac ¸ão de baterias", "Descric ¸ão da imagem" from the options".• "De carga positiva" to +, "De carga negativa" to -, "de carga dois menos" to (2-).• For a subset of the questions, we follow the non-blind version of the question (157, 158, 163, 166, 168, 171, 174, 177, 178, 179) • Remove period at the end options or questions of math questions (to avoid confusion).Here are the list of rules we use for English version of the exam (2022):• Change number decimal from "3,1415" to "3.1415".• Manual translation fix (49, 162) .There are a few limitations of the dataset:1. Even though the English version of the exam is modified manually, there are still issues with the presentation of the questions. We rely mostly on Grammarly feedback, but it is not perfect. Our judgement of how fluently a question is written is also subjective. The ideal method would be to recruit professional human translators, which is costly and time consuming.2. The content of many of the questions are focused on knowledge common to Brazilian culture, or problems in Brazilian society. The English translations may not cover the full extent of cultural, language specific phenomenons or connotations.3. We assume the transcription of images and tables to be sufficient for the models to understand and solve the question.Given that questions have wide range of discrimination indices for LLMs, we investigate a potential cause described in the psychometrics literature for aberrant response patterns: lack of subabilities [23] , i.e., specific skills required to answer a question correctly. We hypothesize that some item attributes, such as whether the question contains images or numbers in its statement or among the options, may be disproportionately harder for LLMs and hence represent subabilities that explain the aberrant response patterns quantified in Figure 3 .We built a contingency table relating non-discriminative/discriminative items (i.e., items with discriminative index lower/higher than 0.2) and the aforementioned attributes, and run a χ 2 independence test. The results for the Natural Sciences exam are shown in Table 2 . For this exam, we observe high χ 2 values which indicate that the abilities of the LLM models with respect to math reasoning and interpreting images are sub-par compared to their capacity in solving pure text questions. While Language and Humans exams are most purely text and the Math exam mostly demands reasoning with images and numbers, the nature of the Natural Sciences exam is hybrid, containing both types of questions. This may well explain the bimodal distribution of discrimination indices in Figure 5 and the aberrant response patterns identified by the very low l z scores in Figure 3 , and highlights how psychometrics can aid the design of better and more valid benchmarks for LLMs. One reason that models may error differently than humans is due to their training corpus. If models have encountered similar question or topics, if not identical, to those in our dataset during training, they may perform unexpectedly well, even if the questions are difficult. Recent work in data contamination proposed a few model intrinsic metrics that can be used to detect contamination [36] .Mainly, the Min-k% Prob score takes the average probability of the top-k percentile tokens with minimum probabilities4 :EQUATIONwhere x = x 1 , x 2 , . . . , x N denotes the input sequence of N tokens, Min-K% Prob(x) represents the set containing tokens with minimum k percentile probabilities, and E represents the size of such set.Note here that Min-k% Prob is intrinsic to each model, and if a model has been exposed to more similar training data as the questions, its Min-k% Prob would be low for that question.We do not expect any model to have unexpectedly low Min-K% Prob(x) on any of our questions, considering it is highly unlikely that the ENEM questions were parsed and translated to English, and somehow ended up in the training corpus. What we are more interested here, is whether such score is correlated to model's accuracy on the answer predictions. If they are negatively correlated (i.e.high Min-K% Prob corresponds to low accuracy), this is evidence for the hypothesis that training on related data leads to higher accuracy.To investigate this hypothesis, we plot 4-shot model accuracy (averaged across 31 option shuffles)against Min-20% Prob for four subjects in exam 2022 in English along with the Pearson correlations5 in Figure 6 . In all except 1 model-subject pair (Llama2 chat in humanities, we investigate this further) do we see a significant negative correlation (p < 0.05) between accuracy and Min-k 20%Prob, indicate that model doesn't necessarily do better if they have encountered similar data during training. Another way to interpret this, is that it is not likely that these models have seen our data during training.The few negative correlation cases As seen before, we observe a significant negative correlation for Llama-2 7B Chat in humanities. To get a full understanding of whether this is a stand-alone phenomenon, we examine Portuguese version of the exam, as well as exam in 2023, and show our findings below in Table 3 . We do not see the same correlation in the Portuguese version of the exam. However, we additionally see Gemma-it negatively correlated with humanities section in both English and Portuguese version of the exam in 2023, as well as Gemma with languages section in 2023. The later two correlations are robust across a few other metrics we investigated from [36] as well, we think this may suggest data contamination, but we cannot test such hypothesis because Gemma training data is not public.Positive correlations in 2022 science In 2022 Science, both English and Portuguese, we see significant positive correlation across all models (Table 3 ).Through qualitative analysis, we find that the questions with highest perplexities were formatted more in a sentence completion-like structure similar to Question 1. Whereas less perplexity ques- This is similar to what we discover with discriminative index in Section ?? in the main text.1 Question : Technique modifies rattlesnake venom protein to create a drug that modulates blood clotting Listing 1: high perplexity question with high model accuracy.1 Question : On a hot day , two colleagues are playing with the water from the hose . One of them wants to know how high the water jet reaches from the outlet when the hose is positioned vertically .The other colleague then proposes the following experiment : they position the water outlet of the hose in a horizontal direction , 1 meter above the ground , and then measure the horizontal distance between the hose and the place where the water hits the ground .The measurement of this distance was 3 meters , and from this , they calculated the vertical reach of the water jet . Consider the acceleration of gravity to be 10 meters per second squared . Listing 2: low perplexity question with low model accuracy.We also tried filtering for top N percent most difficult questions per subject and recalculate all the correlations. We did not find any significant difference to results above.To administering the test to LLMs, we measure the next token logits across the 5 letter options directly (i.e. letter "A", "B", "C", "D", "E"), and take the argmax as the model's choice (invariant to sampling temperature). We shuffle the option orders (30 runs) and take the average to calibrate model's prior on generating each letter options. For API-based model (GPT3.5), we query for 1 token generation, and obtain top-20 logits, and use that for our prediction. In the sections below we include 0-shot (Listing 3), 5, 6, 7) , and 4-shot prompts (Listing 8) we use in main experiments. For 1-shot, we choose the 1-shot example for each of the four subjects by selecting the easiest question (i.e., with lowest β) from the same subject in the 2021 exam. For 4-shot, we concatenate the 1-shots from four subjects and shuffle the options to evenly distribute the answer among five option letters. year lang subj L2-7b L2-7b-it L2-13b L2-13b-it L3-8b-it L3-8b M-7b M-7b-it G-7b-it G-We ran exploratory experiments with Chain-of-Thought (CoT) like prompting [49] , but and did not see significant changes. We did not include the results because CoT prompting requires generating reasoning strings and parsing answers, making 30-shuffles extremely slow to run for all models. Future directions could explore how much effect more complex prompting techniques have in assimilating model behaviors. Regarding the best prompting strategy, we do acknowledge recent criticisms on first letter evaluation [48] . At the time of our writing, it is still the best evaluation strategy for multiple choice question-answering data. We also acknowledge that there are more capable models than GPT3.5 that is available through API services but as our work is not trying to identify the SOTA model we did not feel the need to evaluate latest and largest models.Lastly, we assume Portuguese and Brazilian culture is present in the training data for the language models we test. Future work could evaluate the amount of multilingual training's affect on some of these IRT metric we propose.1 Here are some questions from a college entrance exam . Choose the correct answer to the best of your ability , and output in the following format :2 Answer : ( Option ) 1 Here are some questions from a college entrance exam . Choose the correct answer to the best of your ability , and output in the following format :2 Answer : ( Option ) Separating the animals properly by sex would minimize the occurrence of these problems .In the text , the author presents ethical outlines that correspond to Options :( A ) the foundation of the idea of surplus value .( B ) concept of class struggle .( C ) fundamentals of the scientific method .( D ) paradigms of the inquiry process . 1 Here are some questions from a college entrance exam . Choose the correct answer to the best of your ability , and output in the following format :2 Answer : ( Option ) In the text , the author presents ethical outlines that correspond to Options :( A ) the foundation of the idea of surplus value .( B ) concept of class struggle .( C ) fundamentals of the scientific method .( D ) paradigms of the inquiry process .( E ) domains of commodity fetishism .Answer : ( B ) concept of class struggle .Question :We used GPUs (V100 or A100) provided by a university cluster 6 . For the main experiments, we used around 200 hours of GPU time (roughly 20 hours per model). Moreover, we used the OpenAI API to run the experiments with GPT3.5. E. it increases breeding sites.Gemma-7B Instruct (EN) Gemma-7B Instruct (PT-BR) LLaMA2-13B Instruct (EN) LLaMA2-13B Instruct (PT-BR) LLaMA2-7B Instruct (EN) LLaMA2-7B Instruct (PT-BR) LLaMA3-8B Instruct (EN) LLaMA3-8B Instruct (PT-BR) Mistral-7B Instruct (EN) Mistral-7B Instruct (PT-BR)The Humanities exam assesses understanding of geographical, cultural, and socioeconomic transformations, as well as comprehension of social and political institutions, technological changes, and the use of historical knowledge to promote conscious engagement in society. It requires recognizing the interactions between society and nature in various historical and geographical contexts.The Languages and Codes exam assesses the use of communication in various contexts. This includes some knowledge and use of foreign languages, understanding of body language, analysis and interpretation of expressive resources in different languages, comprehension of opinions in specific languages, and understanding the impact of communication on personal and social life.The Natural Sciences exam assesses understanding of natural sciences and recognizing their roles in production, economic and social development. It involves associating environmental degradation or conservation with productive and social processes, understanding the interactions between organisms and the environment, and applying specific knowledge of physics, chemistry, and biology.The Math exam assesses the usage of geometric knowledge to represent reality, understanding notions of magnitudes, measurements, and their variations for solving everyday problems, interpreting information of scientific and social nature obtained from reading graphs and tables, and making trend predictions, extrapolations, interpolations, and interpretations.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Gemma models are released inand we suspect contamination issues from analysis in Appendix A.3.grammarly.com/We chose not to use a large model such as GPT3.5 to rephrase the translations because it may artificially lower the perplexity and change the meaning of the questions.We follow the equation in https://github.com/swj0419/detect-pretrain-code/blob/main/src/run.pyhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.htmlWhat prior biotechnological procedure is recommended in this situation ?We will disclose it after the reviewing phase due to the double-blind process. 


Decision: The paper argues that using accuracy for evaluating LLMs does not allow deep understanding on the true capabilities of LLMs. It proposes to use item response theory (IRT) as a tool to better understand LLM performance on human exams. The results show that LLMs exhibit some significant deviation from human-like behaviors in many cases. 

The paper is well-written and the experiments are well-designed as noted and agreed by most reviewers. 

Some reviewers consider using IRT to evaluate LLMs is a significant and novel contribution. However, using IRT to evaluate LLMs has already been explored in several previous works like [22, 33, 54] and the following two papers: 
1. Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., and Bowman, S. R. Comparing test sets with item response theory. ACL, 2021.
2. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, Mikhail Yurochkin, tinyBenchmarks: evaluating LLMs with fewer examples, arXiv, 2024

Therefore, using IRT to evaluate LLMs can hardly be considered as a novel and significant contribution of this paper.

The second concern about the paper is on its findings: a key finding of the paper is that LLMs behave differently from humans on certain questions, but this is kind of already known; some findings are subtle and require more thoughtful interpretations (Wmva), e.g. math exam not meaningful for evaluating LLMs; and there is also a lack of clearer/deeper analysis to gain deeper insight into why certain LLM behaviors occur (KhP9). 

Given the above concerns and after discussion with SAC, we think that the paper is more like a case study of using IRT to understand how LLMs perform on "human exams" using a large educational dataset, and the technical contributions of the paper is quite limited. The paper might be better suited to a conference other than NeurIPS, e.g., conferences at the intersection of AI and Education, or an NLP conference, where the community would be more interested in the hypotheses and findings of the paper. NeurIPS, with its emphasis on AI/machine learning foundations and methodology, does not seem to be an appropriate venue for a "case study" paper.

Reviews: review: paper_summary: paper_summary: This paper provides a fresh perspective to evaluating LLMs by arguing for a stronger emphasis on psychometric methods particularly Item Response Theory (IRT) when evaluating them on exams designed for humans, rather than the reliance on traditional metrics such as accuracy. The authors postulate that IRT provides a more comprehensive evaluation by considering not just the number of correct answers but also the difficulty of the questions and the patterns of responses. The authors utilize the Brazilian college entrance exam ENEM for their case study and compare how various LLMs fare against human test-takers. They show how psychometric methods can be leveraged to distinguish between human like and non-human like responses. Furthermore, they demonstrate how IRT can be used to assess the suitability of an exam for making meaningful measurements of an LLM's abilities in the given area.
strength_weakness: strength_weakness: The error analysis can be more detailed especially in areas where the results are surprising. This would better help support the conclusions.       For instance for the questions in Math and Natural Sciences wherein the models show fluctuating performance it would be useful to know what those questions aim to test. Are LLMs not able to solve the problems due to calculation errors or do these problems involve more complex multi-step reasoning or is it just linked to knowledge cutoff (e.g questions involving current events)?
questions: questions: The evaluations have been performed with relatively smaller models (in comparison to SOTA frontier models e.g  GPT-4 / Gemini 1.5 Pro / Claude 3 ). It would be interesting to benchmark frontier models and see whether the results highlight any obvious potential biases in smaller models.
limitations: limitations: The pre-requisite for this type of evaluation seems to be the existence of a strong IRT model which in turn requires the existence of large amount of carefully annotated human data.
paper_summary: paper_summary: This paper initiates the empirical study of the performance of LLMs using Item Response Theory (IRT) models from a large college-entrance exam.
strength_weakness: strength_weakness: 1. Some of the conclusions drawn by the paper appear unscientific/not well-substantiated. To me, the empirical results are subtle and require more thoughtful interpretations. Most of the interpretations of the experiments are confusing to me (i.e. I'm skeptical the conclusions follow), given the actual plots shown. For example, 

(a) What are "outlier models" (line 237)? We cannot see from Figure 1 that "outlier models ... have higher accuracy and/or lower IRT scores..." - how is this statement supported?

(b) line 223-224. The scale of IRT scores and CTT scores is not comparable. How can you conclude there is "greater variability" in the latter than in the IRT score? This is not scientific.

(c) line 264-265. The statement "...questions that are easy for humans but difficult for LLMs" is again inaccurate. The questions are relatively easier for humans but may not be "easier" than the other questions for humans, if easier means for humans anyway.

(d) Why is the math exam not meaningful for evaluating LLMs? Doesn't it suggest that the models are randomly guessing and therefore bad? I don't agree with this interpretation.

2. A clarity issue with the math writing. Line 154-155: This sentence "...j has a more likely response vector than indicated by their ability" is mathematically wrong. It is not possible to have a random draw from a Multinomial distribution that is "more likely" (i.e. higher probability) than the expectation vector (which is not even in the space of possible draws).

3. Experiment section writing missing some details and figures are somewhat difficult to interpret (esp Figure 1). I have several unanswered questions. How was the closed curve generated from the 30 points (of random shuffles)? The caption for Figure 1 could be more informative, e.g. was the exam answered in English or Portuguese by the LLM. If English, are the IRT models fit still valid? - I don't think so. 

4. Typo in lines 232-233, "Natural sciences" appears twice. and the sentence contradicts the graph.
questions: questions: 1. Is the dataset of the ENEM exams translated to English made publicly available?
2. How was the closed curve generated from the 30 points (of random shuffles)? 
3. The caption for Figure 1 could be more informative, e.g. was the exam answered in English or Portuguese by the LLM. If English, are the IRT models fit still valid? - I don't think so. 
4. What are "outlier models" (line 237)?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper focuses on evaluating LLM abilities on a dataset of 8 college-entrance exams in Brazil (translated to English) measuring Item Response Theory instead of Accuracy. It highlights how such metric is useful to better understand models' performance.
strength_weakness: strength_weakness: While the paper is well structured, I felt it was missing a "what now?" message. The authors wrote a convincing argument in favour of using IRT, how do we convince now the field of ML / AI to use it more extensively? What are its limitations in comparison with accuracy-based metrics (given there are many, for instance you need information on overall human performance) and how do we overcome them?
questions: questions: Could you add more to the text about the way questions were translated? Was it machine translation? Did someone double-check consistency of translated questions? Would you advice a similar approach if someone would like to further translate those questions to other languages?
limitations: limitations: I think the work should have discussed more about the specificities of ENEM - I agree with the authors that this is a relevant test-bed for this sort of evaluation, but in which ways are they specific / tailored to Brazil? Is there anything researchers should know about ENEM, which would make future testing / applications more challenging? For instance which topics are covered in Humanities or Languages, how specific are they about the country cultural context?
paper_summary: paper_summary: This paper shows the use of psychometric modeling techniques to measure the reasoning ability of LLMs on human exams. Specifically, the author(s) use Item Response Theory (IRT) to evaluate a Brazilian college-entrance exam, and demonstrate that IRT can provide a more informative evaluation of LLMs , including: the ability to distinguish human-like vs non-human-like response patterns, and to determine whether an exam can reliably measure an LLM's abilities. The empirical results suggest that traditional accuracy metrics are insufficient to assess the abilities of LLMs, and advocate for using IRT/psychometric theory to evaluate them.
strength_weakness: strength_weakness: 1. The results analysis would benefit from a more detailed and clearer/deeper analysis, some statements made (eg. L293-298) are high level observations based on the results, but lack further insight into why certain LLM behaviors occur. Performing more detailed analyses into the specific subset of questions that contribute to scores could help to further understand the limitations of the LLM (L328-331 alludes to this, but very briefly).

2.  All the evaluations were done on variations of the ENEM exam dataset, showing that these psychometric method would also work on other datasets would make this approach more convincing that it will work for wider applications - I understand that there is limited time to run more experiments, so this is more so just a comment.
questions: questions: 1. L212 mentions "The results we show in this section are strongly consistent with the results we get for the 2022 ENEM exam and for one-shot prompting", there are no descriptions/analysis about the zero-shot prompting models (only graphs in the appendix). Were the findings similar? If no, in what ways did they differ?
2. L181 mentions that "developers may exclude questions having poor model fit" for the IRT model, can we have more details about how "poor model fit" is determined?
limitations: limitations: As mentioned above, as experiments are done on variations of one dataset, there are doubts about the generalizability of these methods on other datasets.

Comments: title: Acknowledgement of rebuttal
Improving rating
comment: Thank you for your reply and the further details. It is a really interesting piece of work and I'm sure the community will find it very useful
Thank you for addressing my questions.
Thank you for addressing the comments and questions. This paper offers a thorough and sound evaluation of the ENEM dataset. My main concern remains the generalizability of the technique beyond this specific context. However, the fact that IRT is a widely accepted and versatile framework does help to alleviate some of this concern, more literature review on this could further strengthen the argument for the generalizability of your approach. Given the additional validation of the method, I am increasing my rating for the paper.