Title: Beyond accuracy: understanding the performance of LLMs on exams designed for humans
Venue: NeurIPS 2024
Accepted: no
Keywords: large language models

model evaluation

psychometrics
Abstract: Many recent studies of LLM performance have focused on the ability of LLMs to achieve outcomes comparable to humans on academic and professional exams. However, it is not clear whether such studies shed light on the extent to which models show reasoning ability, and there is controversy about the significance and implications of such results. We seek to look more deeply into the question of how and whether the performance of LLMs on  exams designed for humans reflects true aptitude inherent in LLMs. We do so by making use of the tools of psychometrics which are designed to perform meaningful measurement in test taking.  We leverage a unique dataset that captures the detailed performance of over 5M students across 8 college-entrance exams given over a span of two years in Brazil. With respect to the evaluation of LLM abilities, we show that the tools of Item Response Theory (IRT) provide a more informative evaluation of model performance than the usual accuracy metrics employed in previous studies.  Digging deeper, we show that the modeling framework of IRT, by explicitly modeling the difficulty levels of questions, allows us to quantitatively distinguish between LLMs that answer questions in “human-like” patterns versus LLMs that do not. We also show how to quantitatively identify cases in which exam results are not reliable  measurements of an LLM's ability.  Using the tools of IRT we can also identify specific questions that appear to be either much easier, or much harder, for machines than for humans, and we give some reasons for those differences. Overall, our study shows that the conventional focus on accuracy as the primary performance metric for LLM studies  does not allow us to deeply understand the true capabilities of LLMs and compare them to that of humans. Thus, we claim that psychometric modeling should play a larger role in the evaluation of LLM capabilities on exams designed for humans.

Abstract

Many recent studies of LLM performance have focused on the ability of LLMs to achieve outcomes comparable to humans on academic and professional exams. However, it is not clear whether such studies shed light on the extent to which models show reasoning ability, and there is controversy about the significance and implications of such results. We seek to look more deeply into the question of how and whether the performance of LLMs on exams designed for humans reflects true aptitude inherent in LLMs. We do so by making use of the tools of psychometrics which are designed to perform meaningful measurement in test taking. We leverage a unique dataset that captures the detailed performance of over 5M students across 8 college-entrance exams given over a span of two years in Brazil. With respect to the evaluation of LLM abilities, we show that the tools of Item Response Theory (IRT) provide a more informative evaluation of model performance than the usual accuracy metrics employed in previous studies. Digging deeper, we show that the modeling framework of IRT, by explicitly modeling the difficulty levels of questions, allows us to quantitatively distinguish between LLMs that answer questions in 'human-like' patterns versus LLMs that do not. We also show how to quantitatively identify cases in which exam results are not reliable measurements of an LLM's ability. Using the tools of IRT we can also identify specific questions that appear to be either much easier, or much harder, for machines than for humans, and we give some reasons for those differences. Overall, our study shows that the conventional focus on accuracy as the primary performance metric for LLM studies does not allow us to deeply understand the true capabilities of LLMs and compare them to that of humans. Thus, we claim that psychometric modeling should play a larger role in the evaluation of LLM capabilities on exams designed for humans.

1 Introduction

Large Language Models (LLMs) have demonstrated an impressive ability in performing well on examinations designed for humans [25, 30], such as the US bar exam [27], the US Medical Licensing Exam [21], and many others [45, 53]. This yields controversy in how researchers should interpret such results, raising two kinds of criticisms of those apparent successes. The first is the potential for publicly-given exams (and answers) to leak into models' training data. The second, and more fundamental, issue is the notion of construct validity [44]. Most exams given to humans are intended to measure a construct, e.g., legal analysis ability, medical analysis ability, etc. However, the reliability of these exams in measuring the relevant construct for non-humans is usually ignored, and exams that are valid in one context may not generalize across different groups, settings or tasks [24].

Formalizing the notion of construct validity in general is challenging. Since the 1950s, the field 36

of psychometrics has been grappling with how to design examinations that validly measure human 37

38

abilities along specific dimensions.

The primary tool developed has been Item Response Theory

(IRT) [10], which has been employed in psychology, medicine, and especially in educational test39

ing. IRT formalizes the unobserved construct as a continuous latent variable, and models stochastic 40 responses of humans to questions as a logistic regression conditional on that latent variable. 41

In this paper, we demonstrate how IRT can help shed light on whether LLMs are in fact show42 ing human-like performance on exams intended for humans. As a case study, we use one of the 43 largest university-entrance exams in the world, a dataset comprising the performance of over 5 mil44 lion Brazilian students on eight multiple-choice exams administered over two years. Each exam 45 was prepared and fitted to an IRT model by educational testing experts, giving us an unparalleled 46 opportunity to examine the performance of LLMs in detail. 47

48

49

50

51

52

53

54

55

56

Our results show that the LLMs we study reveal performance patterns that are consistent with expected human behavior in many cases. Nonetheless, we also frequently observe significant deviation from human-like behavior. We demonstrate how to use the tools of IRT to quantitatively distinguish between human-like and non-human-like behavior. We then explore the differences between models and exam types that correlate with differences in response patterns. Lastly, we use the tools of IRT and psychometrics to identify cases where exams are not producing reliable estimates of LLM ability and understand why this happens. This occurs because exams are in some cases too difficult for the models, and in other cases too easy for them and as such they cannot properly measure the ability of certain LLMs.

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

Moving beyond conclusions about current models, the broader contribution of our study is to demonstrate the power of IRT as a framework for evaluating LLMs. For example, in Classical Test Theory (CTT), no attempt is made to assess the difficulty of individual questions, in-line with majority of in standard LLM benchmarks that pursues accuracy [8, 43, 16, 4]. In contrast, as we will show below, IRT simultaneously measures both test takers and exam questions (on the same scale). In doing so, IRT allows one to distinguish between test takers with similar CTT (accuracy) scores, but differing levels of true ability, by inspecting the pattern of correct or incorrect answers given. Moreover, we deploy a broader set of tools (e.g., goodness-of-fit, Fisher information, discrimination index) which enable us to evaluate which are the cases in which fitting the IRT model to the LLMs response patterns gives us reliable estimates of the models' ability. Thus, we believe that the methods of our study represent a valuable step beyond the use of simple accuracy for assessing whether both current and future LLMs show human-like response patterns.

2 Related Work

Our study connects a number of research areas, spanning benchmarking LLMs, the applications of item response theory, and the evaluation of LLMs using exams designed for humans.

Benchmarking LLMs. The most common strategy to evaluate LLMs is through traditional largescale NLP benchmarks [46, 40, 11, 18, 16, 19, 4]. Conventionally, benchmark evaluation relies on some notion of accuracy - the number of correct answers - as a proxy for ability [8, 43]. A key distinction of our study is to draw attention to the limitations of the use of accuracy alone [34] for evaluating the performance of LLMs on benchmarks in understanding the similarity between the performance of models versus humans.

LLMs and Exams Designed for Humans. Many attempts to evaluate LLMs use exams designed for humans, e.g., at college-entrance [1, 26] or college-level [14, 37, 47, 13, 41, 53]. These exams also generally use accuracy as a metric of ability; one focus of our work is on how to use IRT analysis to determine when such exams in fact perform meaningful measurement.

The Brazilian nationwide college-entrance exams we use in this work (ENEM), detailed in Section 4.1, were used in previous efforts to evaluate NLP models [38, 39, 26]. However, those studies only used accuracy and did not make use of the IRT models associated with the exam, which is a central aspect our work.

IRT in Machine Learning. Work in psychometrics (i.e., the measurement of human cognitive abilities), detailed in Section 3, has shown that using accuracy as a exam score may not reflect the true underlying abilities of individuals [15]. As a result, IRT has been advocated for use in machine learning (ML) as an improved tool for benchmarking. The authors in [33] show that it is possible to produce rankings of NLP models which are more reliable and stable using IRT than accuracy. Item response theory has also been shown to help in spotting noisy questions, identifying overfitting,

selecting features, and designing better benchmarks for ML [29, 35, 20, 54, 22]. However, there is 92 a critical difference between the previous uses of IRT in ML and our work. Previous work uses IRT 93 by training an IRT model on the results of ML models solving question-answering or classification 94 questions. Our method is different: we leverage the fact that we have access to an IRT model trained 95 on human responses, and we do not retrain on model responses. We take this approach because 96 a central goal of our study is to explore whether LLMs are in fact following response patterns as 97 exhibited by human test takers. 98

99

100

101

102

103

104

105

Finally, we note that [42] shares some goals with our work. The investigation seeks to understand whether LLMs show human-like response biases in surveys. We also look at the question of whether LLMs show human-like response patterns, but we study the question along different dimensions: (a) patterns of correct and incorrect answers in exams; and (b) the ways in which LLMs choose incorrect answers. Additionally, Xia et al. [51] recognize that accuracy as a single metric does not capture errors LLMs can make in intermediate steps when solving mathematical tasks, and they systematically study those errors.

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

3 Background

In this section, we give some background of the tools we use from psychometrics.

Classical Test Theory (CTT): CTT [2] evaluates test takers based on the fraction of questions they answer correctly. We call this score accuracy or CTT score of the test taker and we use these two terms interchangeably. Inadequately, CTT does not differentiate between difficult and easy questions, nor does it take into consideration the patterns of correct answers . For example, the CTT score does not penalize a test taker who answers correctly difficult questions, but answers wrongly easy ones - despite the fact that such a pattern might be indicative of randomness or cheating.

Item Response Theory (IRT): IRT [12, 5] is a model used extensively in psychometrics to measure the ability level of the test takers and evaluate the difficulty of the test questions (which are referred to as items in psychometrics). IRT takes into consideration the difficulty of the questions when evaluating test-taker's performance and also makes use of the pattern of correct and incorrect responses on the exam. The model associates with every test taker j a parameter θ j , which corresponds to the ability of j . The two-parameter IRT model (2PL) associates every question i with two parameters ϕ i = ( α i , β i ) . The model assumes that a test taker with ability θ j answers question i associated with ϕ i correctly with probability given by the logistic function:

Parameter α i is the discrimination parameter and β i is the difficulty of question i . Note that the ability θ j and the difficulty level β i are in the same scale; after all, the difference ( θ j -β i ) directly affects p ij . For fixed α i , the difficulty parameter β i is the value (on the ability scale) for which p ij = 0 . 5 . Parameter α i characterizes how well question i can differentiate among test takers located at different points of the ability continuum; α i is proportional to the slope of p ij = p i ( θ j ) at the point where p ij = 0 . 5 - the steeper the slope, the higher the discriminatory power of i . All the parameters of this model take values in ( -∞ , + ∞ ) . Note that any set of questions comprising an exam spans a certain range of β i values; such a set is not appropriate to assess test takers with abilities outside this range.

The 3-Parameter IRT model (3PL for short) is an extension of the above model that also incorporates a pseudo-guessing parameter γ i . Thus, in 3PL every question i is associated with three parameters Φ i = ( α i , β i , γ i ) ; α i and β i are the same as before. Intuitively, γ i is the probability of answering correctly based on random guess with γ i ∈ [0 , 1] . Thus, the probability of a test taker with ability θ j to answer question i correctly is: P ij = γ i +(1 -γ i ) p ij .

Given test-taker responses, the parameters of the model can be estimated using Bayesian methods [5]. In our case, the ENEM dataset came with a set of questions for which the parameters ( α i , β i , γ i ) had already been fitted by education experts [17]. Therefore, for each one of the LLMs we considered, we only need to compute their ability parameters - given their response patterns. Intuitively, large values of θ correspond to test takers with high ability levels and vice versa. High ability value θ of an LLM implies better performance.

Although the ability levels of test takers can be used as a measure of their performance, one should 142 also know if the test takers are consistent with the model, e.g., they should answer easy questions 143 correctly if they answer difficult questions correctly. One index that enables us to evaluate the 144 consistency of the test takers with the model is the l z index [12]. Intuitively, the l z index is based 145 on the standardization of a test-taker's log-likelihood function given their theta values. Assume a 146 set of I questions and test taker j with ability θ j and response vector r j such that r j ( i ) = 1 (resp. 147 r j ( i ) = 0 ) if j answered question i correctly (resp. wrongly). Then, the log-likelihood of j is 148 simply: L j = ∑ i ∈ I [ r j ( i ) ln P ij +(1 -r j ( i )) ln(1 -P ij )] . To standardize L j we need both its 149 mean ( E [ L j ] ) and variance (Var ( L j ) ). Then, the l z score is computed as: 150

In a well-designed test, the l z scores are expected to have a unit normal distribution - this is the case for humans taking the ENEM test (see for example Figure 3). In general, l z values close to 0 are considered good: it means the test takers' response patterns are consistent with what is expected from them by the model. Negative l z ( j ) scores reflect an unlikely response vector. A positive l z ( j ) score indicates that j has a more likely response vector than indicated by their ability.

151

152

153

154

155

We can access the amount of information that an item i provides to estimate θ under the 3PL model 156 by the Fisher information, which is given by: 157

The total information of a test is simply the sum of item information, i.e., I ( θ ) = ∑ i ∈ I I i ( θ ) . The Fisher information is connected with the standard error of the estimation, given by SE ( θ ) = 1 / √ I ( θ ) . When a test has high Fisher information in a certain θ range, the test has more discriminative power in that range, producing scores with less measurement errors.

4 Methods

4.1 The ENEM Exam

The Exame Nacional do Ensino M´ edio (ENEM), world's second largest university entrance exam behind Chinese's Gaokao exam, is taken by millions of Brazilian students each year [39]. ENEM comprises questions requiring different levels of domain-specific knowledge and reasoning [3].

The exam is in Brazilian Portuguese and consists of four sections, each of which has 45 multiplechoice questions with five options [17]. Each section is treated as a separate exam for the purposes of modeling via IRT. The four sections consist of the Humanities , the Languages and Codes , the Natural Sciences , and the Math exams. The description of these exams is given in Appendix A.11.

Since 2009, the grades assigned to ENEM test-takers have been determined using IRT. Using IRT helps to penalize guessing, differentiate among students that otherwise would get the same (CTT) grade, and compare among students that took exams in different years. The ENEM organizers release not only the exam content and questions, but also the student (anonymized) responses and their CTT and IRT scores, which enables downstream studies.

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

From our standpoint, there are a number of relevant aspects of the process used by the ENEM developers [17]. First, questions are given to a sample of students, whose answers are used to find inconsistencies and errors. Next, an important test of construct validity is to verify the unidimensionality of the latent trait, for which the ENEM team uses Full Information Factor Analysis [7]. Finally, the IRT model itself is fit using the Marginal Maximum Likelihood Estimator [6]. Using the results, the developers may exclude questions having poor model fit.

The exams, their solutions, and all the fitted parameters of the 3PL IRT model ( θ j , α i , β i , γ i ) are 182 publicly available at the Brazilian government website [17]. To the best of our knowledge, these data 183 are the largest and most comprehensive public dataset based on item-response theory available. The 184 datasets contain questions and complete response patterns of all students taking the exams in 2022 185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

and 2023. Questions for the 2023 exam were released in November 2023, minimizing the chance they are in training data for most of the LLMs we considered. However, we expect fragments of the exam being in the training data (e.g. poems, and any other widely available material used as part of a question) 1 . The number of test takers per year ranged from 2.2M to 3.7M.

The ENEM exams are initially made available as PDF files; we used the Python library PyPDF2 , followed by regular expressions and some manual adjustments to extract each question from its exam file. In order to account for possible effects of Language, as diagnosed in previous work [31], we translated all questions to English and run all experiments in Portuguese and English. For those exam questions that incorporated images, we used the version of the exam designed for blind people containing textual descriptions of the images. We manually audited all questions in 2022 and 2023 exams to ensure their quality (Appendix A.1).

4.2 Models

We evaluate the following family of models: the open source models Mistral-7B, Gemma-7B, Llama2-7B, Llama2-13B, Llama3-8B, and GPT 3.5. For the open source models, we evaluate on both instructed and non-instructed tuned versions. Our choice of models enables the study of models of similar size (the majority of our models are of size 7B), but also introduces diversity of architectures (GPT, Gemma, Mistral, Llama), size (7B vs. 13B), training data (Llama2 vs. Llama3), and training strategies (with and without instruction tuning).

We prompt models with { 0 , 1 , 4 } -shots, following conventional question-answer benchmark prompting strategies [32] (example prompts in Appendix A.4). We measure model's next token probability across five option letters, and average predictions across 30 shuffles of the order of the answer choices to correct for the well-known effect of position bias [28] (Details in Appendix A.4).

5 Results

In this section, we present our main findings. All the results we show here are for the 2023 ENEM exams, with four-shot prompting. Results for the 2022 ENEM exam and for zero-shot and one-shot prompting and for open source instructed tuned models are shown in Appendices A.6 - A.9. The results we show in this section are strongly consistent with the results we get for the 2022 ENEM exam and for one-shot prompting.

5.1 Accuracy vs. Ability Level

Wefirst investigate how humans compare to LLMs when IRT parameter θ is used instead of accuracy (the metric that is employed in most LLM benchmarking, e.g., [8, 43]). In Figure 1 we plot the CTT score (accuracy) vs IRT score ( θ ) for 30 shuffles of answer options for each model. The light blue background points correspond to the humans who took the exam. Each of the closed curves in the figure corresponds to one LLM, and shows the central 90% of the LMM's distribution.

First, we observe that there are many cases where identical accuracy scores result in different θ scores. This reflects the fact that IRT takes into account not just the number, but also the pattern of correct answers. Second, for many LLMs, particularly in the Humanities and Languages exams, there is overall greater variability in the accuracy score than in the IRT score. This suggests that IRT is less sensitive to the variations in LLM output that are due to the LLM's inherent randomness.

To compare the performance between LLMs and humans, we compare their IRT scores ( θ ). Recall that IRT score of 0 corresponds to the average ability of a human test taker. Across all four subjects, the majority of models have CTT and IRT scores overlapping with humans. LLMs in general achieve θ scores above that of the human average in Humanities, Languages, and Natural Sciences, but below human average in Mathematics. Looking at specific models, we find the Llama2 models at the lower end of θ scores, Mistral and Llama3 in the middle range, and GPT-3.5 and Gemma-7B at the higher end of θ scores.

The language of the exam affects some models' performance. In Languages and Natural Sciences, GPT-3.5 tends to perform better in Portuguese compared to English, while in Humanities and Natural

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

-2

-1

0

1

2

3

IRT score

2023 Languages and Codes

2023 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2023 Natural Sciences

0

15

30

45

CTT score

2023 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

Sciences, the Llama models tend to perform worse in Portuguese than in English. This suggests that there are differences regarding the reasoning ability and the amount of knowledge accessible to the models in each language.

Importantly, outlier models all tend to have higher accuracy and/or lower IRT scores than humans. These models answer more questions correctly than humans do, but show error patterns that are not entirely human-like. We dig into this phenomenon next.

5.2 Response Patterns

One of our goals is to assess whether the LLMs we examine show good fit to the ENEM IRT model, as crafted by the educational expert team described in Section 4.1. Intuitively, a test taker showing good fit to an IRT model is an individual j that tends to make less frequent mistakes on 'easy' questions (question i with β i < θ j ) while making more frequent mistakes on 'hard' questions (question i with β i > θ j ). Thus, to assess fit we need to inspect the response patterns of the LLMs.

Figure 2 shows the response patterns of LLMs for the 2023 exam. Every cell ( i, j ) corresponds to the probability that LLM i answered question j correctly, where probabilities are computed over the 30 shuffles. We use gray scale with a black (resp. white) cell representing 1 (resp. 0 ). Questions are ordered in increasing order of their β values. Generally, rows with darker overall patterns (higher correctness) are indicative of higher θ scores.

The figure demonstrates a number of points. For example, on the Math exam, the figure exhibits a response pattern that appears to show low θ values for all models, which confirms results in Figure 1. In addition, the figure shows that for some questions, the 30 shuffles of answer choices of a given model are often either all correct or all incorrect. However, there are some grey areas in the figure for all the exams, indicating that shuffling the options can affect the LLM's answers on certain items. Furthermore, the patterns show that many questions appear to be either 'easy' (black) or 'hard' (white) for all models at the same time. Likewise, in many cases models show similar performance on the English and Portuguese versions of a given question.

Overall, the response patterns we observe suggest that the Math exam is 'too difficult,' with models often resorting to guessing. On the other hand, most LLMs consistently answer correctly the

2023 Languages and Codes

Question

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

1

45

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2023 Humanities

1

43

2023 Mathematics

2023 Natural Sciences

Question

questions in the Humanities exam, implying that this is an easy exam for them. The performance of 261 LLMs in the Natural Science exam is the most interesting as there are blocks of questions that most 262 LLMs answer consistently correctly, interleaved with blocks of questions that most LLMs answer 263 incorrectly. This suggests that there are questions that are easy for humans but difficult for LLMs 264 and vice versa. In the next subsection we analyze this phenomenon more closely. 265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

5.3 Reliability of IRT scores for LLMs

In this section, we investigate whether the ENEM exam is a valid test for LLMs' ability, in the same way it is for humans. Intuitively, we want to define measures that allow us to quantify to what extent we trust the IRT scores we obtained for LLMs. We propose three different ways of doing this. The first is goodness-of-fit that quantifies whether the response of LLMs fit the IRT model. The second is based on Fisher information , measuring how much information the exam provides for estimating the θ s in a certain range. Finally, we use the discrimination index which evaluates the capacity of questions to accurately distinguish between high and low performing test takers.

Goodness-of-fit: We use the l z score (see Section 3) assess whether the test taker is behaving in a manner consistent with the model. Alternatively, we ask what is the appropriateness of a testtaker's estimated ˆ θ as a measure of the test taker's true θ ? For example, imagine that an LLM has a response pattern of missing easy questions and correctly answering more difficult ones. Such a pattern may arise because the LLM was lucky on the hard questions, or it may arise because the LLM had access to memorized patterns that assisted in answering the hard questions. Generally, low l z scores suggest that the θ estimate of the model is less reliable [12].

In Figure 3 we show l z scores plotted against θ scores of LLMs across the four exams in 2023 (2022 is shown in Appendix A.9). As in previous plots, the light blue points in the background show the distribution of the same two scores for the human test takers. Starting again with the Math exam, we note that l z values are low, but now we can see that the response patterns of the LLMs are indeed quite human-like; LLMs behave like humans with similarly low l z values. One possible reason for this behavior is that the Mathematics exam tends to be the harder exam of ENEM, leading to more guessing, which may make the human l z values for Mathematics smaller.

For the Languages exam, models perform better in general (higher θ values) and the most l z scores being close to 0 (and with a similar spread as the human distribution of l z 's) suggest that these θ estimates are reliable - the models are showing human-like response patterns.

291

292

293

294

295

296

297

298

299

300

301

302

303

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2023 Languages and Codes

6

4

2

0

2

4

l

z

score

2023 Humanities

6

4

2

0

2

4

l

z

score

2023 Natural Sciences

6

4

2

0

2

4

l

z

score

2023 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

The results become more nuanced as we look at the Natural Sciences exam. For this exam, most models, including the high performing ones (i.e., GPT-3.5 and Gemma-7B), show values well outside the human distribution, with a long tail in the negative values of l z . Comparing the GPT-3.5 and Gemma-7B results in Figures 1 and 4, we can infer that the high accuracy (CTT scores) achieved by these models on the Natural Sciences exam are quite misleading; although GPT-3.5 and Gemma-7B answer many questions correctly, their response pattern is very unlikely, with very low l z values. This corroborates with Figure 2, which shows an interchange of blocks of correct and incorrect answers from the models, creating an unlikely response pattern.

In Humanities, almost all LLMs perform reasonably well, achieving θ scores above zero (the average human level). However, Llama2-7B, while obtaining above average accuracy scores (Figure 1) and good θ scores, has low average l z scores. This suggests that the IRT scores Llama2-7B may be not reliable. Examination of the corresponding rows in Figure 2 shows that this is the only model that does not have a consistent response pattern across shuffles, leading to the observed low l z score.

0

10

20

30

40

(

)

2023 Languages and Codes

2023 Humanities

2023 Natural Sciences

2023 Mathematics

-2

-1

0

1

2

3

4

5

IRT score (

)

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

-2

-1

0

1

2

3

4

5

IRT score (

)

-2

-1

0

1

2

3

4

5

IRT score (

)

-2

-1

0

1

2

3

4

5

IRT score (

)

EN

PT-BR

Fisher Information: We investigate further whether the ENEM exams are giving us accurate esti304 mates of the LLMs ability levels from another standpoint - that of Fisher Information (see Section 3, 305 Equation (3)). Intuitively, Fisher Information quantifies whether there was enough information in 306 the test to infer the ability level of a test taker at a certain ability level. Figure 4 shows, for every 307 ENEMexam, the total Fisher Information I ( θ ) on the top plot, and the θ scores for the models (95% 308 Confidence Interval (CI) computed using the shuffles) on the bottom plot. This plot reinforces the 309 observation that for some models in Natural Sciences and for all models in Mathematics, the mod310 els' θ are not in the range of the exam with highest information - the models ability levels fall in the 311 tail of the Fisher Information histogram. Hence, the Math exam is not useful for making meaningful 312 measurements of these LLMs, casting doubt on the informativeness of the models' θ scores on this 313 exam. The lack of discrimination ability of this exam is reflected by the responses for many models 314 showing apparently random response patterns in the corresponding heatmap (see Figure 2). 315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Discrimination Index

0

2

4

6

Density

Humans (2023)

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Discrimination Index

LLMs (2023)

Humanities

Languages and Codes

Mathematics

Natural Sciences

Discrimination Index: To further assess the reliability of the IRT scores, we also turn into psychometrics and use the notion of the item discrimination index (DI) , which measures how well an item on a test distinguishes between high and low scorers on the entire test [9]. Let P h (resp. P l ) be the proportion of the top 25% (resp. low 25%) LLMs (in terms of θ , including the shuffles) that correctly answer the item; then DI = P h -P l , the difference of the two proportions. DI ranges from -1 to 1, and questions with DI higher than 0.2 are considered good, while lower DI indicates flaws [50].

Figure 5 shows the distribution of the discrimination indices computed for humans and LLMs for the 2023 exam. Overall, we notice that discrimination indices computed for LLMs are more negative compared to those of humans. We also observe that a significant fraction of Math questions have low discriminative power, reinforcing the hypothesis that this exam is not well designed to measure Math abilities for LLMs. Nonetheless, the Humanities and Languages have several questions with very good discriminative power. Interestingly, the Natural Sciences exam appears to follow a bimodal distribution, containing both informative and poorly-designed questions. This may be a reflection of the fact that the Natural Sciences exam is a hybrid test, containing a mix of knowledge-based items and items that demand more complex reasoning over numbers and images, which can be less useful for evaluating the current state-of-the-art LLMs.

Attributes affecting reliability of IRT scores: In a further investigation, shown in Appendix A.2, we explore potential causes of low discrimination. We investigate item attributes such as the existence of images or numbers in the questions as we believe that these attributes impede LLMs from understanding the question properly. Our preliminary results suggest that LLMs' ability to understand math questions and parse images is sub-par compared to their capacity in answering pure text-based questions. In Appendix A.10 we show examples of non-discriminating and highly discriminating items for the 2023 Natural Sciences exam. In Appendix A.3, we reach a similar conclusion by looking at model accuracy against model perplexity, a model intrinsic metric.

6 Conclusions

The ongoing debate in LLM evaluation centers around whether exams designed for humans are appropriate tools for measuring the performance of LLMs. In this paper, we provide a case study that illustrates methods that can be used to address this question, as well as specific results for a range of current LLMs. We leverage the largest known human exam for which a public IRT model is available, and show that IRT can be leveraged to distinguish between human-like and non-humanlike responses under the model. We show cases where LLMs respond in non-human-like ways and show how to identify those cases using a model-fit metric. Further, we show that using IRT we can determine when an exam is capable of making meaningful measurement of an LLM's ability in a given subject area. Using our evaluation framework, we find that the ENEM Math exam is not appropriate to make meaningful measurements of the models' ability, for the LLMs we study. At the same time, Humanities and Language exams are better suited for evaluating the LLMs' abilities on those subjects. We conclude that IRT modeling, drawing on a long history of psychometric theory, provides a set of crucial tools for assessing whether exams designed for humans are actually meaningful measures of LLM ability. Our results suggest that they should be used in future studies when questions are raised regarding the performance of LLMs on human exams.

356

357

358

References

Josh Achiam et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 .

M.J. Allen and W. M. Yen. 2002. Introduction to Measurement Theory . Waveland Press.

Thales Sales Almeida, Thiago Laitz, Giovana K Bon´ as, and Rodrigo Nogueira. 2023. Bluex: a bench359 mark based on brazilian leading universities entrance exams. In Brazilian Conference on Intelligent 360 Systems . Springer, 337-347. 361

BIG-bench authors. 2023. Beyond the imitation game: quantifying and extrapolating the capabilities of 362 language models. Transactions on Machine Learning Research . https://openreview.net/forum?i 363 d=uyTL5Bvosj . 364

F. B. Baker and S.-H. Kim. 2004. Item Response Theory: Parameter Estimation Techniques . Marcel 365 Dekkerm, Inc. 366

R Darrell Bock and Murray Aitkin. 1981. Marginal maximum likelihood estimation of item parameters: 367 application of an em algorithm. Psychometrika , 46, 4, 443-459. 368

R Darrell Bock, Robert Gibbons, and Eiji Muraki. 1988. Full-information item factor analysis. Applied 369 psychological measurement , 12, 3, 261-280. 370

S´ ebastien Bubeck et al. 2023. Sparks of artificial general intelligence: early experiments with gpt-4. 371 arXiv preprint arXiv:2303.12712 . 372

Man Ching Esther Chan. 2015. Young learners: an examination of the psychometric properties of the 373 early literacy knowledge and skills instrument. Journal of Psychoeducational Assessment , 33, 7, 607374 621. 375

Yunxiao Chen, Xiaoou Li, Jingchen Liu, and Zhiliang Ying. 2021. Item response theory - a statistical 376 framework for educational and psychological measurement. (2021). arXiv: 2108.08604 [stat.ME] . 377

Karl Cobbe et al. 2021. Training verifiers to solve math word problems. arXiv preprint 378 arXiv:2110.14168 . 379

Rafael Jaime De Ayala. 2013. The theory and practice of item response theory . Guilford Publications. 380

Iddo Drori et al. 2022. A neural network solves, explains, and generates university math problems by pro381 gram synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences , 382 119, 32, e2123433119. eprint: https://www.pnas.org/doi/pdf/10.1073/pnas.2123433119 . 383 DOI: 10.1073/pnas.2123433119 . 384

385

386

387

388

Iddo Drori et al. 2023. From human days to machine seconds: automatically answering and generating machine learning final exams. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '23). Association for Computing Machinery, New York, NY, USA, 3947-3955. ISBN: 9798400701030. DOI: 10.1145/3580305.3599827 .

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

Mehtap Erguven. 2013. Two approaches to psychometric process: classical test theory and item response theory. Journal of Education , 2, 2, 23-30.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. In International Conference on Learning Representations .

Instituto Nacional de Estudos e Pesquisas Educacionais An´ ısio Teixeira. 2024. Microdados do Enem 2023. INEP. https://www.gov.br/inep/pt-br/acesso-a-informacao/dados-abertos/micr odados/enem .

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Vancouver, Canada, (July 2017).

Douwe Kiela et al. 2021. Dynabench: rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 4110-4124.

Adrienne Kline, Theresa Kline, and Joon Lee. 2021. Item response theory as a feature selection and interpretation tool in the context of machine learning. Medical & Biological Engineering & Computing , 59, (Jan. 2021). DOI: 10.1007/s11517-020-02301-x .

TH Kung et al. 2023. Performance of chatgpt on usmle: potential for ai-assisted medical education using 407 large language models. (2023). 408

John P Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In 409 Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on 410 Empirical Methods in Natural Language Processing . Vol. 2016. NIH Public Access, 648. 411

Rob R Meijer. 1996. Person-fit research: an introduction. Applied Measurement in Education , 9, 1, 3-8. 412

Samuel Messick. 1995. Standards of validity and the validity of standards in performance asessment. 413 Educational measurement: Issues and practice , 14, 4, 5-8. 414

Arvind Narayanan and Sayash Kapoor. 2023. Gpt-4 and professional benchmarks: the wrong answer to 415 the wrong question. (2023). https://www.aisnakeoil.com/p/gpt-4-and-professional-benc 416 hmarks . 417

Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, and Rodrigo Nogueira. 2023. Evaluat418 ing gpt-3.5 and gpt-4 models on brazilian university admission exams. (2023). arXiv: 2303.17003 419 [cs.CL] . 420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

OpenAI. 2023. Gpt-4 technical report. (2023). arXiv: 2303.08774 [cs.CL] .

Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. (2023). arXiv: 2308.11483 [cs.CL] .

Fernando Plumed, Ricardo Prudˆ encio, Adolfo Mart´ ınez-Us´ o, and Jose Hernandez-Orallo. 2016. Making sense of item response theory in machine learning. In (Sept. 2016). DOI: 10.3233/978-1-61499-672 -9-1140 .

Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. Ai and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks . J. Vanschoren and S. Yeung, (Eds.) Vol. 1. Curran. https ://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/084b 6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf .

Leonardo Ranaldi and Giulia Pucci. 2023. Does the english matter? elicit cross-lingual abilities of large language models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL) , 173-183.

Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations . https://ope nreview.net/forum?id=yKbprarjc5B .

Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan BoydGraber. 2021. Evaluation examples are not equally informative: how should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Chengqing Zong, Fei xia, Wenjie Li, and Roberto Navigli, (Eds.) Association for Computational Linguistics, Online, (Aug. 2021), 4486-4503. DOI: 10.18653/v1/2021.acl-long.346 .

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems . A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, (Eds.) Vol. 36. Curran Associates, Inc., 55565-55581. https://proceedings.neurips.cc/paper_files/paper/2023/file/adc98a266f45005c403 b8311ca7e8bd7-Paper-Conference.pdf .

Jo˜ ao Sedoc and Lyle Ungar. 2020. Item response theory for efficient human evaluation of chatbots. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems . Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, and Eduard Hovy, (Eds.) Association for Computational Linguistics, Online, (Nov. 2020), 21-33. DOI: 10.18653/v1/2020.eval4nlp-1.3 .

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations .

Bruno Silva, Leonardo Nunes, Roberto Estev˜ ao, and Ranveer Chandra. 2023. Gpt-4 as an agronomist assistant? answering agriculture exams using large language models. arXiv preprint arXiv:2310.06225 .

Igor Cataneo Silveira and Denis Deratani Mau´ a. 2017. University entrance exam as a guiding test for artificial intelligence. In 2017 Brazilian Conference on Intelligent Systems (BRACIS) , 426-431. DOI: 10.1109/BRACIS.2017.44 .

Igor Cataneo Silveira and Denis Deratani Mau´ a. 2018. Advances in automatically solving the enem. In 2018 7th Brazilian Conference on Intelligent Systems (BRACIS) . IEEE, 43-48.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: a question answering challenge targeting commonsense knowledge. ArXiv , abs/1811.00937. https://api.s emanticscholar.org/CorpusID:53296520 .

Christian Terwiesch. 2023. Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course. Tech. rep. Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania.

Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2024. Do llms exhibit human-like response biases? a case study in survey design. (2024). arXiv: 2311.04076 [cs.CL] .

Hugo Touvron et al. 2023. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Sean Trott. 2024. Can large language models help augment english psycholinguistic datasets? Behavior 474 Research Methods , 1-19. 475

Lakshmi Varanasi. 2023. Gpt-4 can ace the bar, but it only has a decent chance of passing the cfa exams. 476 here's a list of difficult exams the chatgpt and gpt-4 have passed. (2023). https://www.businessins 477 ider.com/list-here-are-the-exams-chatgpt-has-passed-so-far-2023-1 . 478

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: 479 a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 480 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , 353-355. 481

Xiaoxuan Wang et al. 2023. Scibench: evaluating college-level scientific problem-solving abilities of 482 large language models. arXiv preprint arXiv:2307.10635 . 483

Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul R¨ ottger, Frauke Kreuter, Dirk Hovy, 484 and Barbara Plank. 2024. ' my answer is c': first-token probabilities do not match text answers in 485 instruction-tuned language models. arXiv preprint arXiv:2402.14499 . 486

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei xia, Ed Chi, Quoc V Le, Denny Zhou, 487 et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural 488 information processing systems , 35, 24824-24837. 489

Margaret Wu and Ray Adams. 2007. Applying the Rasch model to psycho-social measurement: A prac490 tical approach . Educational Measurement Solutions Melbourne. 491

Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024. Evaluating mathematical 492 reasoning beyond accuracy. arXiv preprint arXiv:2404.05692 . 493

Longhui Yu et al. 2023. Metamath: bootstrap your own mathematical questions for large language mod494 els. arXiv preprint arXiv:2309.12284 . 495

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu 496 Chen, and Nan Duan. 2023. Agieval: a human-centric benchmark for evaluating foundation models. 497 arXiv preprint arXiv:2304.06364 . 498

Yan Zhuang et al. 2023. Efficiently measuring the cognitive ability of llms: an adaptive testing perspec499 tive. arXiv preprint arXiv:2306.10512 . 500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

A Supplemental Material

A.1 Manual auditing of exam questions

Assuming the original questions written by the ENEM authorities are good test instruments for testing student capability, we focus on ensuring the quality of adapted dataset for LLM evaluation. We manually correct the artifacts for each question in 2022 and 2023. In the next sections, we describe the artifacts from those easier to address (sec A.1.2 A.1.3), to deeper-rooted problems (i.e., harder to correct, sec A.1.4), as well as how we addressed them manually (sec A.1.5).

A.1.1 Label accuracy

We assume answers are correct as translation and parsing of single characters can be quite reliable, and that the original ENEM test is tested across millions of human test takers and will be discarded if it had a wrong answer. When we look at the label distribution for 2022, options 'ABCDE' each occur 39/39/37/36/33 times, making it fairly balanced. We also ran random baselines on the same option shuffles as the model (Table 1).

A.1.2 Translation artifacts

We found several issues pertaining to initial round of translation in this dataset. Mainly, independent translation of question context and answer option leads to incoherence. Details are sometimes mistranslated ('p.d.d' translated to 'd.d.p'). There are many non-standardized translations pertaining to chemical formulas, proper nouns, and mathematical formulas. In general, there are significant amount of awkward phrasing, incomplete translation, and linguistic idiosyncrasies lost in translation.

Independent translation of context and question In a few cases, the answer options are expected to complete the last sentence of the question. After translation, options do not all fit as completions of the sentence (Q11). Translation without context also leads to improper translation of polysemantic terms. 'Coagulation' maybe translated correctly in the question, but becomes 'coagulating' as a stand-alone word (Q96). 'Good' and 'fair' (when used as survey options) gets translated to 'regular' and 'I will' as stand-alone options (Q171)

Inconsistent translation details Within the same questions, there are cases where the same concept is translated differently. In one question, the context introduces the concept 'potential difference (p.d.d)', and later referred to it as 'd.d.p' and 'd.p.d'. Within different options, the same unit can sometimes be plural and sometimes be singular (when it should be consistently plural)

Non-standard translation 1) Chemical formula translation is non-standard. 'N2O3' becomes 'N 2O3', and 'NH4+' becomes 'NH4 positively charged'. 2) (Proper) nouns are sometimes capitalized when they shouldn't. For instance, one question begins with the sentence ' On the Gravitational Field of a Mass Point According to Einstein's Theory A 'Black Hole is a... ' 3) Mathematical equations are overly verbatim. This we suspect is partially due to an issue with using audio version of the

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

test. For example, if an option is the formula 9( 8! (8 -2)!2! -1) , its Portuguese representation would be ' 9 vezes ( (8 fatorial dividido por ( (8 menos 2) fatorial vezes 2 fatorial)) menos 1) ' and the English translation exacerbates the situation by translating parenthesis literally as well: ' 9 times open parenthesis, open parenthesis, 8 factorial divided by, open parenthesis, open parenthesis, 8 minus 2, close parenthesis, factorial times 2 factorial, close parenthesis, close parenthesis, minus 1, close parenthesis. '. Sometimes, delimiters are omitted after translation: '9,300' becomes '9 300'.

Awkward phrasings There exist awkward phrasings throughout translation. They range from causing minor difficulty in understanding (i.e., ' Life: the science of biology Bears, because they are not truly hibernating, wake up due to the presence of thermogenin, a mitochondrial protein that prevents protons from reaching ATP synthase, generating heat. ') to sometime completely non-sense (i.e., ' articulation of several narrative nuclei ')

Incomplete translation There is no fine line between proper code switching (where proper nouns should remain in Portuguese script) to in-complete translation. The amount of Portuguese left over range from single words, to phrases in options (not consistently across options), to entire sentences within the question.

Linguistic idiosyncrasies lost in translation In one question, the problem arises when English translation does not match with literal tokens of expressions in Portuguese (' Next to the man is the message: 'Men don't cry', with a large X drawn over the word 'no' ). The word 'no' does not appear in the English phrase 'Men don't cry' but the statement as a whole makes sense in the Portuguese version of the instruction. In a separate question, the topic is on testing for a Portuguese specific pronoun inflection. However, when it was translated into one single word in English, the question no longer makes sense (' They told me... - They told me. - Huh? - The correct word is 'they told me'. Not 'they told me'. - I speak the way I want to. And I'll tell you more... Or is it 'tell you'? - What's that? - I'm telling you that you... -'You' and 'you' don't go together. . . ')

A.1.3 Document parsing artifacts

Each section consistently contains an error of this kind, where the last part of the question got wrongfully parsed into part of the first option (option (A)). In a separate instance, a figure was wrongfully parsed into one of the options of the previous question. In the Portuguese version of the exam, structural components of the question (e.g., title, subtitle, caption) are consistently concatenated together without proper separation. This often leads to incoherent English translations.

A.1.4 Audio-version artifacts

Audio description of images, tables, and figures are not always sufficient, or the most intuitive. For instance, a question asks test taker to note why a particular painting stands out, and the answer is due to the painting's 'distortion when representing human figure', which is difficult to qualitatively describe, no matter how complete the description of an image is. Similarly, textual description of geometric figures can be impossibly complicated (' ...Figure of a grid with 7 horizontal and 7 vertical lines, on which a polygonal path is drawn by means of a continuous line on the grid lines, joining the starting point P , located on the second vertical line, from left to right, and between the sixth and seventh horizontal lines, from top to bottom, to the end point Q , which is located between the sixth and seventh vertical lines, from left to right, and on the second horizontal line, from top to bottom... ')

A.1.5 Manual Correction

The majority of the artifacts begin with incorrect parsing of the PDF documents related to structural components. To address this, we manually audited each question, and added correct spacing and newlines to each question. These improvements result in better translations from DeepL API qualitatively. After translation, we make minimal edits to improve syntactic and semantic issues through Grammarly to obtain a score of at least 95 2 3 . For each answer option, we ensure consistent

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

part-of-speech, especially if they are sentence completions of the questions. For math and science sections, we follow consistent markdown-like format the same way as other mathematical reasoning datasets [16, 11, 52]. Here we list the full set of modification rules for 2022 (question numbers are referenced in parenthesis):

Separate description of the image by ' \ n' before and after.

'Por cento' becomes %.

Number in the form 7 000 becomes 7000.

From 'abre aspas' 'fecha aspas' to ''.

Remove 'Descric ¸˜ ao da estrutura qu´ ımica', 'Descric ¸˜ ao do esquema', 'Descric ¸˜ ao da associac ¸ ˜ ao de baterias', 'Descric ¸˜ ao da imagem' from the options'.

'De carga positiva' to +, 'De carga negativa' to -, 'de carga dois menos' to (2-).

For a subset of the questions, we follow the non-blind version of the question (157, 158, 163, 166, 168, 171, 174, 177, 178, 179)

Remove period at the end options or questions of math questions (to avoid confusion).

Here are the list of rules we use for English version of the exam (2022):

Change number decimal from '3,1415' to '3.1415'.

Manual translation fix (49, 162).

A.1.6 Limitations of the dataset

There are a few limitations of the dataset:

Even though the English version of the exam is modified manually, there are still issues with the presentation of the questions. We rely mostly on Grammarly feedback, but it is not perfect. Our judgement of how fluently a question is written is also subjective. The ideal method would be to recruit professional human translators, which is costly and time consuming.

The content of many of the questions are focused on knowledge common to Brazilian culture, or problems in Brazilian society. The English translations may not cover the full extent of cultural, language specific phenomenons or connotations.

We assume the transcription of images and tables to be sufficient for the models to understand and solve the question.

A.2 Attributes that affect goodness-of-fit

Given that questions have wide range of discrimination indices for LLMs, we investigate a potential cause described in the psychometrics literature for aberrant response patterns: lack of subabilities [23], i.e., specific skills required to answer a question correctly. We hypothesize that some item attributes, such as whether the question contains images or numbers in its statement or among the options, may be disproportionately harder for LLMs and hence represent subabilities that explain the aberrant response patterns quantified in Figure 3.

We built a contingency table relating non-discriminative/discriminative items (i.e., items with discriminative index lower/higher than 0.2) and the aforementioned attributes, and run a χ 2 independence test. The results for the Natural Sciences exam are shown in Table 2. For this exam, we observe high χ 2 values which indicate that the abilities of the LLM models with respect to math reasoning and interpreting images are sub-par compared to their capacity in solving pure text questions. While Language and Humans exams are most purely text and the Math exam mostly demands reasoning with images and numbers, the nature of the Natural Sciences exam is hybrid, containing both types of questions. This may well explain the bimodal distribution of discrimination indices in Figure 5 and the aberrant response patterns identified by the very low l z scores in Figure 3, and highlights how psychometrics can aid the design of better and more valid benchmarks for LLMs.

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

A.3 Model accuracy relation to model perplexity

One reason that models may error differently than humans is due to their training corpus. If models have encountered similar question or topics, if not identical, to those in our dataset during training, they may perform unexpectedly well, even if the questions are difficult. Recent work in data contamination proposed a few model intrinsic metrics that can be used to detect contamination [36]. Mainly, the Min-k% Prob score takes the average probability of the top-k percentile tokens with minimum probabilities 4 :

where x = x 1 , x 2 , . . . , x N denotes the input sequence of N tokens, Min-K% Prob(x) represents the set containing tokens with minimum k percentile probabilities, and E represents the size of such set. Note here that Min-k% Prob is intrinsic to each model, and if a model has been exposed to more similar training data as the questions, its Min-k% Prob would be low for that question.

We do not expect any model to have unexpectedly low Min-K% Prob(x) on any of our questions, considering it is highly unlikely that the ENEM questions were parsed and translated to English, and somehow ended up in the training corpus. What we are more interested here, is whether such score is correlated to model's accuracy on the answer predictions. If they are negatively correlated (i.e. high Min-K% Prob corresponds to low accuracy), this is evidence for the hypothesis that training on related data leads to higher accuracy.

To investigate this hypothesis, we plot 4-shot model accuracy (averaged across 31 option shuffles) against Min-20% Prob for four subjects in exam 2022 in English along with the Pearson correlations 5 in Figure 6. In all except 1 model-subject pair (Llama2 chat in humanities, we investigate this further) do we see a significant negative correlation (p < 0 . 05 ) between accuracy and Min-k 20% Prob, indicate that model doesn't necessarily do better if they have encountered similar data during training. Another way to interpret this, is that it is not likely that these models have seen our data during training.

The few negative correlation cases As seen before, we observe a significant negative correlation for Llama-2 7B Chat in humanities. To get a full understanding of whether this is a stand-alone phenomenon, we examine Portuguese version of the exam, as well as exam in 2023, and show our findings below in Table 3. We do not see the same correlation in the Portuguese version of the exam. However, we additionally see Gemma-it negatively correlated with humanities section in both English and Portuguese version of the exam in 2023, as well as Gemma with languages section in 2023. The later two correlations are robust across a few other metrics we investigated from [36] as well, we think this may suggest data contamination, but we cannot test such hypothesis because Gemma training data is not public.

Positive correlations in 2022 science In 2022 Science, both English and Portuguese, we see significant positive correlation across all models (Table 3).

Through qualitative analysis, we find that the questions with highest perplexities were formatted more in a sentence completion-like structure similar to Question 1. Whereas less perplexity ques-

0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

MIN-20% Prob

2022 Humanities (EN)

Model: Pearson r (p-value)

Gemma-7B Instruct: -0.06 (0.69)

Gemma-7B: 0.06 (0.70)

LLaMA2-13B Instruct: -0.16 (0.30)

LLaMA2-13B: -0.06 (0.68)

LLaMA2-7B Instruct: -0.36 (0.02)

LLaMA2-7B: -0.14 (0.36)

LLaMA3-8B Instruct: -0.15 (0.32)

LLaMA3-8B: -0.08 (0.62)

Mistral-7B Instruct: 0.04 (0.78)

Mistral-7B: 0.02 (0.87)

0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

MIN-20% Prob

2022 Natural Sciences (EN)

Model: Pearson r (p-value)

Gemma-7B Instruct: 0.44 (0.00)

Gemma-7B: 0.65 (0.00)

LLaMA2-13B Instruct: 0.56 (0.00)

LLaMA2-13B: 0.58 (0.00)

LLaMA2-7B Instruct: 0.52 (0.00)

LLaMA2-7B: 0.45 (0.00)

LLaMA3-8B Instruct: 0.57 (0.00)

LLaMA3-8B: 0.58 (0.00)

Mistral-7B Instruct: 0.59 (0.00)

Mistral-7B: 0.62 (0.00)

0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

MIN-20% Prob

2022 Languages and Codes (EN)

Model: Pearson r (p-value)

Gemma-7B Instruct: 0.24 (0.11)

Gemma-7B: -0.22 (0.15)

LLaMA2-13B Instruct: -0.27 (0.07)

LLaMA2-13B: -0.34 (0.02)

LLaMA2-7B Instruct: 0.02 (0.89)

LLaMA2-7B: 0.14 (0.35)

LLaMA3-8B Instruct: -0.07 (0.65)

LLaMA3-8B: -0.36 (0.02)

Mistral-7B Instruct: -0.00 (1.00)

Mistral-7B: -0.05 (0.76)

0

0.2

0.4

0.6

0.8

1.0

0

10

20

30

40

50

MIN-20% Prob

2022 Mathematics (EN)

Model: Pearson r (p-value)

Gemma-7B Instruct: -0.27 (0.07)

Gemma-7B: -0.03 (0.86)

LLaMA2-13B Instruct: 0.11 (0.48)

LLaMA2-13B: -0.04 (0.81)

LLaMA2-7B Instruct: 0.27 (0.07)

LLaMA2-7B: 0.23 (0.13)

LLaMA3-8B Instruct: 0.14 (0.36)

LLaMA3-8B: -0.05 (0.75)

Mistral-7B Instruct: -0.13 (0.42)

Mistral-7B: -0.04 (0.80)

We also tried filtering for top N percent most difficult questions per subject and recalculate all the 709 correlations. We did not find any significant difference to results above. 710

711

A.4 Prompting Details

To administering the test to LLMs, we measure the next token logits across the 5 letter options 712 directly (i.e. letter 'A', 'B', 'C', 'D', 'E'), and take the argmax as the model's choice (invariant 713 to sampling temperature). We shuffle the option orders (30 runs) and take the average to calibrate 714 model's prior on generating each letter options. For API-based model (GPT3.5), we query for 1 715 token generation, and obtain top-20 logits, and use that for our prediction. In the sections below we 716 include 0-shot (Listing 3), 1-shot (Listing 4, 5, 6, 7), and 4-shot prompts (Listing 8) we use in main 717 experiments. For 1-shot, we choose the 1-shot example for each of the four subjects by selecting 718 the easiest question (i.e., with lowest β ) from the same subject in the 2021 exam. For 4-shot, we 719 concatenate the 1-shots from four subjects and shuffle the options to evenly distribute the answer 720 among five option letters. 721

722

723

724

725

726

727

728

729

730

731

732

733

year

L2-7b

L2-7b-it

L2-13b

L2-13b-it

L3-8b-it

L3-8b

M-7b

M-7b-it

G-7b-it

G-7b

2022

en

CH

-0.14/0.36

-0.36/0.02

-0.06/0.68

-0.16/0.30

-0.15/0.32

-0.08/0.62

0.02/0.87

0.04/0.78

-0.06/0.69

0.06/0.70

LC

0.14/0.35

0.02/0.89

-0.34/0.02

-0.27/0.07

-0.07/0.65

-0.36/0.02

-0.05/0.76

-0.00/1.00

0.24/0.11

-0.22/0.15

CN

0.45/0.00

0.52/0.00

0.58/0.00

0.56/0.00

0.57/0.00

0.58/0.00

0.62/0.00

0.59/0.00

0.44/0.00

0.65/0.00

MT

0.23/0.13

0.27/0.07

-0.04/0.81

0.11/0.48

0.14/0.36

-0.05/0.75

-0.04/0.80

-0.13/0.42

-0.27/0.07

-0.03/0.86

pt

CH

-0.09/0.56

-0.12/0.43

-0.06/0.70

-0.05/0.73

-0.07/0.65

-0.05/0.74

-0.09/0.56

-0.06/0.69

-0.20/0.18

0.18/0.24

LC

0.10/0.53

-0.02/0.88

-0.06/0.67

-0.05/0.73

0.08/0.61

-0.20/0.20

0.14/0.35

-0.09/0.56

0.14/0.37

-0.21/0.16

CN

0.41/0.01

0.42/0.00

0.49/0.00

0.48/0.00

0.57/0.00

0.52/0.00

0.53/0.00

0.52/0.00

0.46/0.00

0.58/0.00

MT

-0.17/0.26

-0.15/0.34

0.12/0.44

-0.02/0.91

0.07/0.66

-0.08/0.59

-0.18/0.23

-0.14/0.35

-0.05/0.76

0.12/0.42

2023

en

CH

-0.06/0.72

-0.07/0.66

-0.09/0.56

0.06/0.69

-0.20/0.20

-0.18/0.23

-0.20/0.18

-0.07/0.65

-0.32/0.03

-0.16/0.30

LC

-0.06/0.67

-0.22/0.15

-0.24/0.12

-0.21/0.17

-0.30/0.04

-0.18/0.23

-0.08/0.61

-0.05/0.76

-0.32/0.03

CN

0.21/0.17

0.21/0.17

0.31/0.04

0.16/0.31

0.30/0.05

0.28/0.06

0.14/0.35

0.15/0.34

0.20/0.19

0.24/0.11

MT

0.17/0.28

-0.07/0.66

-0.04/0.82

-0.02/0.87

-0.05/0.75

0.15/0.35

0.03/0.85

0.19/0.21

0.06/0.68

0.16/0.32

pt

CH

-0.00/1.00

-0.02/0.92

0.09/0.58

0.18/0.25

-0.02/0.90

-0.11/0.46

-0.04/0.77

-0.30/0.05

-0.09/0.55

LC

-0.21/0.16

-0.23/0.13

-0.27/0.07

-0.17/0.26

-0.20/0.18

-0.24/0.11

-0.18/0.23

-0.10/0.53

-0.13/0.40

-0.36/0.02

CN

0.11/0.49

0.17/0.26

0.25/0.10

0.04/0.82

0.14/0.37

0.36/0.01

0.15/0.32

0.14/0.35

0.08/0.61

0.13/0.41

MT

-0.01/0.96

0.02/0.87

-0.02/0.90

-0.04/0.79

-0.07/0.67

0.06/0.71

-0.08/0.60

0.09/0.56

0.18/0.24

0.28/0.06

.

Potential limitations We ran exploratory experiments with Chain-of-Thought (CoT) like prompting [49], but and did not see significant changes. We did not include the results because CoT prompting requires generating reasoning strings and parsing answers, making 30-shuffles extremely slow to run for all models. Future directions could explore how much effect more complex prompting techniques have in assimilating model behaviors. Regarding the best prompting strategy, we do acknowledge recent criticisms on first letter evaluation[48]. At the time of our writing, it is still the best evaluation strategy for multiple choice question-answering data. We also acknowledge that there are more capable models than GPT3.5 that is available through API services but as our work is not trying to identify the SOTA model we did not feel the need to evaluate latest and largest models. Lastly, we assume Portuguese and Brazilian culture is present in the training data for the language models we test. Future work could evaluate the amount of multilingual training's affect on some of these IRT metric we propose.

A.5 Compute Resources 1048

We used GPUs (V100 or A100) provided by a university cluster 6 . For the main experiments, we 1049 used around 200 hours of GPU time (roughly 20 hours per model). Moreover, we used the OpenAI 1050 API to run the experiments with GPT3.5. 1051

1052

1053

A.6 Zero and One Shot prompting Results for 2023

A.6.1 CTT and IRT θ

-2

-1

0

1

2

3

IRT score

2023 Languages and Codes

2023 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2023 Natural Sciences

0

15

30

45

CTT score

2023 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

-2

-1

0

1

2

3

IRT score

2023 Languages and Codes

2023 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2023 Natural Sciences

0

15

30

45

CTT score

2023 Mathematics

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

-2

-1

0

1

2

3

IRT score

2023 Languages and Codes

2023 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2023 Natural Sciences

0

15

30

45

CTT score

2023 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

-2

-1

0

1

2

3

IRT score

2023 Languages and Codes

2023 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2023 Natural Sciences

0

15

30

45

CTT score

2023 Mathematics

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

A.6.2 Response Patterns 1054

We show 43 items for the 2023 Math exam, instead of 45, because 2 items failed to converge and 1055 produce item parameters when the ENEM organizers fitted the human model. 1056

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2023 Languages and Codes

1

45

2023 Natural Sciences

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2023 Humanities

1

43

2023 Mathematics

Question

Question

1

45

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

2023 Languages and Codes

1

45

2023 Natural Sciences

1

45

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

2023 Humanities

1

43

2023 Mathematics

Question

Question

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2023 Languages and Codes

1

45

2023 Natural Sciences

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2023 Humanities

1

43

2023 Mathematics

Question

Question

2023 Languages and Codes

1

45

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

1

45

1

45

Question

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

2023 Humanities

1

43

Question

2023 Mathematics

2023 Natural Sciences

A.6.3 Comparing IRT θ and l z 1057

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2023 Languages and Codes

6

4

2

0

2

4

l

z

score

2023 Humanities

6

4

2

0

2

4

l

z

score

2023 Natural Sciences

6

4

2

0

2

4

l

z

score

2023 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2023 Languages and Codes

6

4

2

0

2

4

l

z

score

2023 Humanities

6

4

2

0

2

4

l

z

score

2023 Natural Sciences

6

4

2

0

2

4

l

z

score

2023 Mathematics

MODEL

Gemma-7B Instruct

LLaMA2-13B Instruct

LLaMA2-7B Instruct

LLaMA3-8B Instruct

Mistral-7B Instruct

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2023 Languages and Codes

6

4

2

0

2

4

l

z

score

2023 Humanities

6

4

2

0

2

4

l

z

score

2023 Natural Sciences

6

4

2

0

2

4

l

z

score

2023 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2023 Languages and Codes

6

4

2

0

2

4

l

z

score

2023 Humanities

6

4

2

0

2

4

l

z

score

2023 Natural Sciences

6

4

2

0

2

4

l

z

score

2023 Mathematics

MODEL

Gemma-7B Instruct

LLaMA2-13B Instruct

LLaMA2-7B Instruct

LLaMA3-8B Instruct

Mistral-7B Instruct

LANGUAGE

EN

PT-BR

A.7 CTT and IRT θ for 2022 1058

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

-2

-1

0

1

2

3

IRT score

2022 Languages and Codes

2022 Humanities

0

15

30

45

CTT score

-2

-1

0

1

2

3

IRT score

2022 Natural Sciences

0

15

30

45

CTT score

2022 Mathematics

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

A.8 Response Patterns for 2022 1059

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2022 Languages and Codes

1

41

2022 Natural Sciences

1

44

Question

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2022 Humanities

1

42

Question

2022 Mathematics

1

45

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

2022 Languages and Codes

1

41

2022 Natural Sciences

1

44

Question

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

2022 Humanities

1

42

Question

2022 Mathematics

2022 Languages and Codes

Question

1

45

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

1

41

1

44

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2022 Humanities

1

42

2022 Mathematics

2022 Natural Sciences

Question

2022 Languages and Codes

1

45

Gemma-7B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (EN)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA2-7B Instruct (PT-BR)

LLaMA3-8B Instruct (EN)

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (EN)

Mistral-7B Instruct (PT-BR)

1

41

1

44

Question

Gemma-7B Instruct (EN) Gemma-7B Instruct (PT-BR) LLaMA2-13B Instruct (EN) LLaMA2-13B Instruct (PT-BR) LLaMA2-7B Instruct (EN) LLaMA2-7B Instruct (PT-BR) LLaMA3-8B Instruct (EN) LLaMA3-8B Instruct (PT-BR) Mistral-7B Instruct (EN) Mistral-7B Instruct (PT-BR)

2022 Humanities

1

42

Question

2022 Mathematics

2022 Natural Sciences

Gemma-7B Instruct (EN) Gemma-7B Instruct (PT-BR) LLaMA2-13B Instruct (EN) LLaMA2-13B Instruct (PT-BR) LLaMA2-7B Instruct (EN) LLaMA2-7B Instruct (PT-BR) LLaMA3-8B Instruct (EN) LLaMA3-8B Instruct (PT-BR) Mistral-7B Instruct (EN) Mistral-7B Instruct (PT-BR)

Gemma-7B Instruct (EN)

LLaMA2-13B Instruct (EN)

Gemma-7B Instruct (PT-BR)

LLaMA2-13B Instruct (PT-BR)

LLaMA2-7B Instruct (PT-BR)

LLaMA2-7B Instruct (EN)

LLaMA3-8B Instruct (EN)

Mistral-7B Instruct (EN)

1

45

ChatGPT-3.5 (EN) ChatGPT-3.5 (PT-BR) Gemma-7B (EN) Gemma-7B (PT-BR) LLaMA2-13B (EN) LLaMA2-13B (PT-BR) LLaMA2-7B (EN) LLaMA2-7B (PT-BR) LLaMA3-8B (EN) LLaMA3-8B (PT-BR) Mistral-7B (EN) Mistral-7B (PT-BR)

1

41

1

44

ChatGPT-3.5 (EN)

ChatGPT-3.5 (PT-BR)

Gemma-7B (EN)

Gemma-7B (PT-BR)

LLaMA2-13B (EN)

LLaMA2-13B (PT-BR)

LLaMA2-7B (EN)

LLaMA2-7B (PT-BR)

LLaMA3-8B (EN)

LLaMA3-8B (PT-BR)

2022 Humanities

1

42

2022 Mathematics

2022 Languages and Codes

Question

2022 Natural Sciences

Question

2022 Languages and Codes

1

45

2022 Humanities

1

44

Question

LLaMA3-8B Instruct (PT-BR)

Mistral-7B Instruct (PT-BR)

2022 Natural Sciences

1

41

2022 Mathematics

1

42

Question

A.9 Comparing IRT θ and l z for 2022 1060

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

Gemma-7B Instruct

LLaMA2-13B Instruct

LLaMA2-7B Instruct

LLaMA3-8B Instruct

Mistral-7B Instruct

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

Gemma-7B Instruct

LLaMA2-13B Instruct

LLaMA2-7B Instruct

LLaMA3-8B Instruct

Mistral-7B Instruct

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

ChatGPT-3.5

Gemma-7B

LLaMA2-13B

LLaMA2-7B

LLaMA3-8B

Mistral-7B

LANGUAGE

EN

PT-BR

6

4

2

0

2

4

l

z

score

2

1

0

1

2

3

4

IRT score

2022 Languages and Codes

6

4

2

0

2

4

l

z

score

2022 Humanities

6

4

2

0

2

4

l

z

score

2022 Natural Sciences

6

4

2

0

2

4

l

z

score

2022 Mathematics

MODEL

Gemma-7B Instruct

LLaMA2-13B Instruct

LLaMA2-7B Instruct

LLaMA3-8B Instruct

Mistral-7B Instruct

LANGUAGE

EN

PT-BR

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

A.10 Examples of non-discriminating and highly discriminating items for the 2023 Natural Sciences exam.

A.10.1 Poorly discriminative questions

Question 107 (discrimination index -0.013)

Municipalities are responsible for managing their urban waste (garbage) cleaning and collection according to the Federal Constitution. However, there are reports that part of this waste winds up incinerated, releasing toxic substances into the environment and causing explosions-related accidents when incinerating aerosol bottles (e.g., deodorants, insecticides, and repellents). The high temperature causes all the contents inside these bottles to vaporize, increasing the internal pressure until it explodes.

Suppose there is a metal aerosol bottle with a capacity of 100 milliliters containing 0.1 mol of gaseous products at a temperature of 650 degrees Celsius at the moment of explosion.

The pressure, in atmospheres, inside the flask at the moment of the explosion is closest to

A. 756

B. 533

C. 76

D. 53

E. 13

Question 108 (discrimination index -0.076)

The circuit with three identical incandescent light bulbs, shown in the figure, consists of a mixed association of resistors. Each bulb (L1, L2, and L3) is associated in parallel with a resistor of resistance R, forming a set. These sets are connected in series, with all the bulbs having the same brightness when connected to the power supply. After several days in use, only lamp L2 burns out, while the others remain lit.

Figure description: a power supply connected to three sets, arranged in series clockwise, in the 1086 following sequence: the parallel set of L1 and R, the parallel set of L2 and R, and the parallel set of 1087

L3 and R. 1088

R

R

Fonte de

energia

R

In the case where all the bulbs work, after L2 burns out, the brightness of the bulbs will be 1089

1090

1091

1092

1093

1094

A. the same.

B. more intense.

C. less intense.

D. less intense for L1 and the same for L3.

E. more intense for L1 and less intense for L3.

Question 109 (discrimination index 0.013) 1095

A company's transport safety team is evaluating the behavior of the tensions that appear in two 1096 horizontal ropes, 1 and 2, used to secure a load of mass M equal to 200 kilograms to the truck, 1097 as shown in the illustration. When the truck starts from rest, its acceleration is constant and equal 1098 to 3 meters per second squared, while when it arbitrarily brakes, its braking is constant and equal 1099 to 5 meters per second squared. In both situations, the load is about to move, and the direction of 1100 the truck's movement is shown in the figure. The coefficient of static friction between the box and 1101 the bottom surface of the body is 0.2. Consider the acceleration due to gravity to be 10 meters per 1102 second squared, the initial tension in the ropes is zero, and the two ropes are ideal. 1103

Figure description: a truck traveling horizontally to the right (represented by the vector V). A box M 1104 is resting on the central surface of its body. The box is attached to the rear of the body by horizontal 1105 rope 1 and to the front by horizontal rope 2. 1106

When the truck is accelerating and braking, the tensions in ropes 1 and 2 in Newton will be 1107

1108

acceleration: T1=0 and T2=200; braking: T1=600 and T2=0.

1109

1110

acceleration: T1=0 and T2=200; braking: T1=1400 and T2=0.

acceleration: T1=0 and T2=600; braking: T1=600 and T2=0.

acceleration: T1=560 and T2=0; braking: T1=0 and T2=960. 1111

acceleration: T1=640 and T2=0; braking: T1=0 and T2=1040. 1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

A.10.2 Highly discriminative questions

Question 124 (discrimination index 0.650)

Update of the Portuguese Society of Neonatology's recommendation

Glass containing aluminum is an excellent material for packaging medicines and supplements because heating can sterilize it. However, when the drug or supplement contains substances that bind strongly to this metal's ion, the aluminum's dissolution is promoted by the displacement of the chemical equilibrium established between the species immobilized in the glass and the species in solution. For this reason, it is recommended that newborn nutrition supplements containing calcium gluconate be packaged in plastic containers rather than in this type of glass.

If this supplement is packaged in this type of glass, the risk of contamination by aluminum will be greater if the

glass of the bottle is translucent.

concentration of calcium gluconate is high.

glass bottle is thicker.

glass is previously sterilized at high temperatures.

reaction of aluminum with calcium gluconate is endothermic.

Question 91 (discrimination index 0.624)

It is a common requirement to turn off devices, such as cell phones, whose operation involves emitting or receiving electromagnetic waves when traveling by plane. The justification for this procedure is, among other things, the need to eliminate sources of electromagnetic signals that could interfere with the pilots' radio communications with the control tower.

This interference can only occur if the waves emitted by the cell phone and those received by the plane's radio

are both audible.

have the same power.

have the same frequency.

have the same intensity.

propagate at different speeds.

Question 130 (discrimination index 0.621)

The number of bees is in decline in various regions of the world, including Brazil, and multiple factors are contributing to the collapse of their hives. In the United States, seed bombs of native plant species have been used to combat the disappearance of these insects. They are small balls filled with seeds, compost, and clay. When they are thrown and exposed to sun and rain, they germinate even in poorly fertile soil.

This method contributes to the preservation of bees because

it reduces predation.

it reduces the use of pesticides.

it reduces competition for shelter.

it increases the food supply.

it increases breeding sites. 1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

A.11 Description of Exams

The Humanities exam assesses understanding of geographical, cultural, and socioeconomic transformations, as well as comprehension of social and political institutions, technological changes, and the use of historical knowledge to promote conscious engagement in society. It requires recognizing the interactions between society and nature in various historical and geographical contexts.

The Languages and Codes exam assesses the use of communication in various contexts. This includes some knowledge and use of foreign languages, understanding of body language, analysis and interpretation of expressive resources in different languages, comprehension of opinions in specific languages, and understanding the impact of communication on personal and social life.

The Natural Sciences exam assesses understanding of natural sciences and recognizing their roles in production, economic and social development. It involves associating environmental degradation or conservation with productive and social processes, understanding the interactions between organisms and the environment, and applying specific knowledge of physics, chemistry, and biology.

The Math exam assesses the usage of geometric knowledge to represent reality, understanding no1166 tions of magnitudes, measurements, and their variations for solving everyday problems, interpreting 1167 information of scientific and social nature obtained from reading graphs and tables, and making 1168 trend predictions, extrapolations, interpolations, and interpretations. 1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

1214

1215

1216

1217

1218

1219

1220

1221

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We show how IRT can be used to study LLM in comparison to humans through multiple-metric propositions (Section 3) and their results and discussions (Section 5).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations of prompts and models in Appendix A.4, limitations of contamination correlation study in Appendix A.3, limitation of the dataset curation in Appendix A.1

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

1222

1223

1224

1225

1226

1227

1228

1229

1230

1231

1232

1233

1234

1235

1236

1237

1238

1239

1240

1241

1242

1243

1244

1245

1246

1247

1248

1249

1250

1251

1252

1253

1254

1255

1256

1257

1258

1259

1260

1261

1262

1263

1264

1265

1266

1267

1268

1269

1270

1271

1272

1273

1274

1275

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our dataset creation in Section 4.1, data manual auditing process in Appendix A.1, prompting and evaluation details in Section 4.2 and Appendix A.4. We will release our code and data.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

1276

1277

1278

1279

1280

1281

1282

1283

1284

1285

1286

1287

1288

1289

1290

1291

1292

1293

1294

1295

1296

1297

1298

1299

1300

1301

1302

1303

1304

1305

1306

1307

1308

1309

1310

1311

1312

1313

1314

1315

1316

1317

1318

1319

1320

1321

1322

1323

1324

1325

1326

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code with reproducibility instructions. We will also provide all the data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Prompting and evaluation details given in Section 4.2 and Appendix A.4. Evaluation scripts can be seen in our code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We describe our option shuffling procedure in Section 4.2 and Appendix A.4 to account for model bias for generating option letters. For all plots, we explain what were the confidence intervals/means are (Figures 1, 2, 4, and 3)

Guidelines:

The answer NA means that the paper does not include experiments.

1327

1328

1329

1330

1331

1332

1333

1334

1335

1336

1337

1338

1339

1340

1341

1342

1343

1344

1345

1346

1347

1348

1349

1350

1351

1352

1353

1354

1355

1356

1357

1358

1359

1360

1361

1362

1363

1364

1365

1366

1367

1368

1369

1370

1371

1372

1373

1374

1375

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We disclose the compute related information in Appendix A.5

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics and our work conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

1376

Justification: We discuss the impact of our work in Section 5 and 6.

Guidelines: 1377

1378

1379

1380

1381

1382

1383

1384

1385

1386

1387

1388

1389

1390

1391

1392

1393

1394

1395

1396

1397

1398

1399

1400

1401

1402

1403

1404

1405

1406

1407

1408

1409

1410

1411

1412

1413

1414

1415

1416

1417

1418

1419

1420

1421

1422

1423

1424

1425

1426

1427

1428

1429

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not forsee any negative impact with our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and and describe in detail the process through which we obtained ENEM dataset in section 4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

1430

1431

1432

1433

1434

1435

1436

1437

1438

1439

1440

1441

1442

1443

1444

1445

1446

1447

1448

1449

1450

1451

1452

1453

1454

1455

1456

1457

1458

1459

1460

1461

1462

1463

1464

1465

1466

1467

1468

1469

1470

1471

1472

1473

1474

1475

1476

1477

1478

1479

1480

1481

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit our dataset and include details of how we obtained the text in detail in Section 4.1 and Appendix A.1. Human results and related parameters are released by ENEM officials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not crowdsource. The human results are published by ENEM officials.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our study does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

1482

1483

1484

1485

1486

1487

1488

1489

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows the use of psychometric modeling techniques to measure the reasoning ability of LLMs on human exams. Specifically, the author(s) use Item Response Theory (IRT) to evaluate a Brazilian college-entrance exam, and demonstrate that IRT can provide a more informative evaluation of LLMs , including: the ability to distinguish human-like vs non-human-like response patterns, and to determine whether an exam can reliably measure an LLM's abilities. The empirical results suggest that traditional accuracy metrics are insufficient to assess the abilities of LLMs, and advocate for using IRT/psychometric theory to evaluate them.\n\nSTRENGTHS:\n1. Provides more comprehensive evaluation methods for LLM performance, I personally agree that accuracy metrics often do not give a complete picture of LLM ability.\n2. The results section is methodological, it evaluates not only the IRT scores but how reliable they are based on several metrics (increases reliability of the evaluations)\n\nWEAKNESSES:\n1. The results analysis would benefit from a more detailed and clearer/deeper analysis, some statements made (eg. L293-298) are high level observations based on the results, but lack further insight into why certain LLM behaviors occur. Performing more detailed analyses into the specific subset of questions that contribute to scores could help to further understand the limitations of the LLM (L328-331 alludes to this, but very briefly).\n\n2.  All the evaluations were done on variations of the ENEM exam dataset, showing that these psychometric method would also work on other datasets would make this approach more convincing that it will work for wider applications - I understand that there is limited time to run more experiments, so this is more so just a comment.\n\nQUESTIONS:\n1. L212 mentions \"The results we show in this section are strongly consistent with the results we get for the 2022 ENEM exam and for one-shot prompting\", there are no descriptions/analysis about the zero-shot prompting models (only graphs in the appendix). Were the findings similar? If no, in what ways did they differ?\n2. L181 mentions that \"developers may exclude questions having poor model fit\" for the IRT model, can we have more details about how \"poor model fit\" is determined?\n\nLIMITATIONS:\nAs mentioned above, as experiments are done on variations of one dataset, there are doubts about the generalizability of these methods on other datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on evaluating LLM abilities on a dataset of 8 college-entrance exams in Brazil (translated to English) measuring Item Response Theory instead of Accuracy. It highlights how such metric is useful to better understand models' performance.\n\nSTRENGTHS:\nI have found the work very well structured and appreciated the amount of care the authors have been given to the preparation of the dataset for the experiments (PDF processing, translation to English, use of exams designed for blind people in order to address questions based on images, etc). The experiments and results are discussed in details, with clear comparisons with human performance, discussing clear differences (e.g. in Mathematics).\n\nWEAKNESSES:\nWhile the paper is well structured, I felt it was missing a \"what now?\" message. The authors wrote a convincing argument in favour of using IRT, how do we convince now the field of ML / AI to use it more extensively? What are its limitations in comparison with accuracy-based metrics (given there are many, for instance you need information on overall human performance) and how do we overcome them?\n\nQUESTIONS:\nCould you add more to the text about the way questions were translated? Was it machine translation? Did someone double-check consistency of translated questions? Would you advice a similar approach if someone would like to further translate those questions to other languages?\n\nLIMITATIONS:\nI think the work should have discussed more about the specificities of ENEM - I agree with the authors that this is a relevant test-bed for this sort of evaluation, but in which ways are they specific / tailored to Brazil? Is there anything researchers should know about ENEM, which would make future testing / applications more challenging? For instance which topics are covered in Humanities or Languages, how specific are they about the country cultural context?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper initiates the empirical study of the performance of LLMs using Item Response Theory (IRT) models from a large college-entrance exam.\n\nSTRENGTHS:\n- The question of construct validity of LLM evaluations (based on scores in exams designed for humans) is very important. This paper addresses this question in the earnest, by leveraging the primary tool developed in the education and psychometrics field, IRT. As far as I know, this is an original contribution as no previous work has used IRT in the same way to tackle the construct validity issue of LLM evaluations.\n\n- The paper is largely well-written and the concepts (e.g. section 3) are explained clearly.\n\n- Relative strength of empirical work. The experiments are well-designed and there is some breadth in the range of hypotheses explored, e.g. English vs. Portuguese effect on performance, Different topics of exams, response patterns in LLMs vs. humans with questions sorted by IRT difficulty. Seven different LLMs were evaluated.\n\n- Significance. The method of this paper (i.e. using IRT in LLM evals) is an important first step to understanding what LLM evals are trying to measure. The paper already observes interesting phenomena, e.g. \n(1) the Fisher information of the math exam for the LLM test response distributions is low compared to other exams (although this is a somewhat obvious corollary of the p_i's being close to random for the LLMs performance on the math exam, the FI is a metric that points in the right direction).\n(2) the joint distribution of IRT scores and CTT scores for LLMs is meaningfully different from that of the human test takers.\n\nWEAKNESSES:\n1. Some of the conclusions drawn by the paper appear unscientific/not well-substantiated. To me, the empirical results are subtle and require more thoughtful interpretations. Most of the interpretations of the experiments are confusing to me (i.e. I'm skeptical the conclusions follow), given the actual plots shown. For example, \n\n(a) What are \"outlier models\" (line 237)? We cannot see from Figure 1 that \"outlier models ... have higher accuracy and/or lower IRT scores...\" - how is this statement supported?\n\n(b) line 223-224. The scale of IRT scores and CTT scores is not comparable. How can you conclude there is \"greater variability\" in the latter than in the IRT score? This is not scientific.\n\n(c) line 264-265. The statement \"...questions that are easy for humans but difficult for LLMs\" is again inaccurate. The questions are relatively easier for humans but may not be \"easier\" than the other questions for humans, if easier means for humans anyway.\n\n(d) Why is the math exam not meaningful for evaluating LLMs? Doesn't it suggest that the models are randomly guessing and therefore bad? I don't agree with this interpretation.\n\n2. A clarity issue with the math writing. Line 154-155: This sentence \"...j has a more likely response vector than indicated by their ability\" is mathematically wrong. It is not possible to have a random draw from a Multinomial distribution that is \"more likely\" (i.e. higher probability) than the expectation vector (which is not even in the space of possible draws).\n\n3. Experiment section writing missing some details and figures are somewhat difficult to interpret (esp Figure 1). I have several unanswered questions. How was the closed curve generated from the 30 points (of random shuffles)? The caption for Figure 1 could be more informative, e.g. was the exam answered in English or Portuguese by the LLM. If English, are the IRT models fit still valid? - I don't think so. \n\n4. Typo in lines 232-233, \"Natural sciences\" appears twice. and the sentence contradicts the graph.\n\nQUESTIONS:\n1. Is the dataset of the ENEM exams translated to English made publicly available?\n2. How was the closed curve generated from the 30 points (of random shuffles)? \n3. The caption for Figure 1 could be more informative, e.g. was the exam answered in English or Portuguese by the LLM. If English, are the IRT models fit still valid? - I don't think so. \n4. What are \"outlier models\" (line 237)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a fresh perspective to evaluating LLMs by arguing for a stronger emphasis on psychometric methods particularly Item Response Theory (IRT) when evaluating them on exams designed for humans, rather than the reliance on traditional metrics such as accuracy. The authors postulate that IRT provides a more comprehensive evaluation by considering not just the number of correct answers but also the difficulty of the questions and the patterns of responses. The authors utilize the Brazilian college entrance exam ENEM for their case study and compare how various LLMs fare against human test-takers. They show how psychometric methods can be leveraged to distinguish between human like and non-human like responses. Furthermore, they demonstrate how IRT can be used to assess the suitability of an exam for making meaningful measurements of an LLM's abilities in the given area.\n\nSTRENGTHS:\nThe paper is very well written. It provides a comprehensive literature review and does a good job at covering the background information. The experimental analysis is sound with sufficient supporting materials. The paper makes significant novel contributions to evaluation of LLMs. The application of psychometric methods and the insights that can be mined through them when used to compare LLMs can be of significant interest to the research community. The experimental results on assessing whether an exam is a good indicator of an LLM's ability are particularly interesting and open up significant opportunities for future research.\n\nWEAKNESSES:\nThe error analysis can be more detailed especially in areas where the results are surprising. This would better help support the conclusions.       For instance for the questions in Math and Natural Sciences wherein the models show fluctuating performance it would be useful to know what those questions aim to test. Are LLMs not able to solve the problems due to calculation errors or do these problems involve more complex multi-step reasoning or is it just linked to knowledge cutoff (e.g questions involving current events)?\n\nQUESTIONS:\nThe evaluations have been performed with relatively smaller models (in comparison to SOTA frontier models e.g  GPT-4 / Gemini 1.5 Pro / Claude 3 ). It would be interesting to benchmark frontier models and see whether the results highlight any obvious potential biases in smaller models.\n\nLIMITATIONS:\nThe pre-requisite for this type of evaluation seems to be the existence of a strong IRT model which in turn requires the existence of large amount of carefully annotated human data.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}