Correlated Errors in Large Language Models
Authors: Elliot Myunghoon Kim
Avi Garg
Kenny Peng
Nikhil Garg
Venue: ICML 2025 poster (2025)
Keywords: algorithmic monoculture
evaluations
LLMs
Primary area: social_aspects
TL;DR: We evaluate the extent to which different language models are correlated in how they err.

Abstract: Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ \textit{meaningfully}. We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and a resume-screening task. We find substantial correlation in model errors---on one leaderboard dataset, models agree 60\% of the time when both models err. We identify factors driving model correlation, including shared architectures and providers. Crucially, however, larger and more accurate models have highly correlated errors, even with distinct architectures and providers. Finally, we show the effects of correlation in two downstream tasks: LLM-as-judge evaluation and hiring---the latter reflecting theoretical predictions regarding  algorithmic monoculture.

Correlated Errors in Large Language Models
Elliot Kim 1
Avi Garg 2
Kenny Peng
Nikhil Garg
Abstract
different available models differ in meaningful way. How correlated are LLM emrors--.e -i... how often do models con- verge on the same wrong answer? What features of models predict high correlation? Are newer models more or less homogeneous? What are the downstream effects of evolv- ing levels of model correlation in high-stakes settings where LLMs are used? Answering these questions is necessary both understand the current limits of ecosystem diversity, and more effectively engineer multi-agent systems.
Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully. We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and resume-screening task. We find substantial correlation in model errors-on one leaderboard dataset, models agree 60% of the time when both models err. We identify fac- tors driving model correlation, including shared architectures and providers. Crucially, however, larger and more accurate models have highly cor- related errors, even with distinct architectures and providers. Finally, we show the effects of correla- tion in two downstream tasks: L.L--as-judge eval- uation and hiring-the the latter reflecting theoretical predictions regarding algorithmic monoculture.
Here, we answer these questions, using three large datasets of LLM responses: Responses of 349 LLMs on 14,402 multiple choice questions on a HuggingFace leaderboard, 71 LLMs on 12,032 multiple choice questions on the Helm leaderboard (Liang et al., 2023), and 20 LLMs on 1,800 resume-job description pairs.
(1) How correlated are LLM errors? To assess corre- lation on multiple choice questions, we evaluate the agree- ment rate of model wrong. On the resume evaluations, we evaluate the corre- lation in residuals in comparison to human labels. In all three datasets, we find that errors correlate highly across LLMs. For example, on Helm, pairs of models agree av- erage about 60% of the time when both models are incorrect (choosing between incorrect answers uniformly at random would lead to an agreement rate of 1/3).
on both models heing
1. Introduction
Large language models (LLMs) are increasingly involved in high-stakes multi-agent settings. A key characteristic of the emerging ecosystem is the large number of models made available by various providers. One HuggingFace leaderboard currently hosts over 2000 models. Diversity potentially supports healthy ecosystem. In labor markets, different firms using different models could reduce systemic exclusion compared to universal adoption of single model (Bommasani et al., 2022; Toups et al., 2023; Creel & Hell- man, 2022). Systems with multiple models may generally be more robust, allowing "wisdom of crowds" effects and avoiding correlated failure (Verga et al., 2024; Kleinberg & Raghavan, 2021; Peng & Garg, 2024a;b; Jain et al., 2024a;b; Tekin al., 2024; Chen et al., 2025; Yang et al., 2025).
(2) What explains model correlation? Through a regres- sion analysis on the datasets above, we find that models with the same provider (company), with the same base ar- chitecture, with similar sizes have more correlated errors. Importantly, even after conditioning on these factors, pairs of models that are more accurate individually also have more correlated errors. This means that newer models that differ on the surface may be converging in their outputs.
(3) What are the downstream effects of model corre- lation? We then study how these patterns in correlation translate to downstream outcomes in two settings: LLM-as- judge evaluation (Zheng et al., 2023) and hiring applications. In the LLM-as-judge setting, we show that model corre- lations substantially affect LLM model evaluations when another model is used to proxy ground truth labels: judges overinflate the accuracy of models that are less accurate
However, we lack large-scale empirical evidence on whether
Equual contribution 'Cornell University 'Independent. Corre- spondence to: Kenny Peng (atp998@cooeel.ddd..... Nikhil Garg <ngarg@ @comeelledu>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Correlated Errors in Large Language Models
Correlated Errors FHugeingFace)
Correlated Errrs (HELM)
5.03
Plodec Dostted Acconace!
Mooe Sonted by becunay
Eis Aprammenn ha
0.30
Vodal (aared Acouray)
Model by Accuresy
(a) HuggingFace
(b) Helm
Figure Agreement when both models are wrong. Models are sorted by accuracy. More accurate models have more correlated errors. and almost all model pairs have higher agreement rates than random disagreement on errors would imply.
than it especciillyy for models of the same provider chitecture. Furthermore, motivated by concerns regarding algorithmic monoculture and systemic exclusion in hiring markets (Kleinberg & Raghavan, 2021; Creel & Hellman, 2022; Peng & Garg, 2024a), we quantify the implications of correlations across firms when making hiring decisions for example, how model correlation affects who hired (com- pared to hand labels of resume-job fit) and worker welfare (do they match with their most desired firms).
individually accurate agent. Our work provides empirical foundation for this literature: how correlated are different models, and what explains correlation? Our results thus aid in choosing uncorrelated models and analyzing whether there exists sufficient diversity to receive its benefits.
Algorithmic monoculture and systemic exclusion A bur- geoning literature is concerned with the effects of algorith- mic monoculture, when many decision- model (Kleinberg & Raghavan, 2021; Bommasani et al., 2022; Jain et al., 2024a;b; Creel & Hellman, 2022; Peng & Garg, 2024a;b; Toups et al., 2023; Baek et al., 2025). For example, in hiring, monoculture can negatively affect correlated hiring firms (Kleinberg & Raghavan, 2021: Peng & Garg. 2024a). There are further concerns about systemic exclusion, when worker rejected from all jobs because they all use the same algorithm (Bommasani et al., 2022; Creel & Hellman, 2022). However, Peng & Garg (2024a) argue that monoculture can benefit applicants on average be- cause those given offers have more power to choose where to work and, equilibrium, a similar number of workers will be hired. Motivated by this literature, Section 5 we simulate labor market in which firms either the same different large language models to screen applicant resumes.
the same
Section 2 discusses related work. Section analyzes and explains model correlations. Sections 4 and 5 consider downstream tasks. Section concludes. Our code and data available at https://gitte wwwccoooiiiiiieeeeeeeeec homm/iikhgarg///
2. Related Work
Multi-agent and multi-model use large literature, ei- ther implicitly explicitly, assumes that using different LLMs provides benefits using the same model. For ample, to avoid seff-preferencing (Panickssery et al., 2024; Wataoka et al., 2024), many papers use separate LLM model to evaluate outputs from one or more models (Zhou et al., 2024; Verga et al., 2024). Raghavan (2024) models a game where is beneficial to both be accurate and differ- ent than other players: he finds that increasing noise (c.g., via increasing temperature) and choosing different LLM than competitors may be preferable over using the most
Ecosystem evaluation A closely related literature studies the ecosystem of large language models and foundation models more broadly. This literature focuses on the degree to which models use shared components. This motivated,
2

---

Correlated Errors in Large Language Models
for example, by the component sharing hypothesis studied by Bommasani et al. (2022); Toups et al. (2023)i..e. that models with the same components are more likely to have correlated outputs. Further work documents how different models share components (Bommasani et al., 2023). Our paper can be viewed partially as studying the component sharing hypothesis at a large scale.
across multiple networks (Wolfram & Schein, 2025). These findings suggest mechanistic explanation for results.
3. Correlated Errors
We start by establishing that LLMs broadly have correlated errors, and that this correlation is both substantial and higher for more individually accurate models.
Generative diversity and monoculture in large language models Generative diversity and mode collapse is com- mon worry regarding large language models (Zhang al., 2025; Senthilkumar et al., 2024; Xu et al., 2024; Alvero et al., 2024; Wang et al., 2025). Wu et al. (2024) find se- vere monoculture within an LLM in instances of a task; for example, they find that the distribution of code written by an LLM has less variance than the distribution of code in its training set. They further find that changing inference parameters (such as temperature, top-p, and prompts) does not mitigate this monoculture, and that it is worse for mod- els with RLHF fine-tuning. Our work complements this analysis by measuring monoculture across L.Mss are dif- ferent LLMs more correlated with each other than they are with ground truth? We find that they are, and that this is worse among models within the same model company and larger siidelsswhhle within-model generative diversity is a concern, using multiple different models not a panacea.
3.1. Data and methods
We use three datasets of LLM responses: HELM, HUG- GINGFACE. and RESUMES. HELM and HUGGINGFACE are large, existing datasets of LLM answers to multiple-choice questions. RESUMES is a dataset of LLM ratings of resumes (given a job description) that we generate. Details, including prompts and full lists of models, are in the appendix.
Multiple choice questions. We started from two LLM leaderboards: (1) HuggingFace's ' Open LLM Leaderboard 2 and (2) Stanford's Holistic Evaluation of Language Mod- els (Helm) (Liang et al., 2023). Both use questions from the Massive Multitask Language Understanding (MMLU) question -nnwwering dataset (Hendrycks et al., 2020), with multiple choice questions and labeled correct answers. Both leaderboards provide model answers on each question, from 349 and 71 LLMs, respectively.
In complementary and concurrent work, Wenger & Kenett (2025) and Goel et al. (2025) also study model correla- tion. Wenger & Kenett (2025) show that, on creative tasks, "LLM responses are much more similar to other LLM re- sponses than human responses are to each other." TM Goel al. (2025) build on the observation that error consistency metrics (Geirhos et al., 2020) do not capture differing predic- tions on errors. They develop novel metric for measuring similarity that adjusts for model accuracy, considers differ- ent wrong predictions as a disagreement, and uses the prob- ability distribution over answer choices; we likewise use a metric (do models agree on an answer when both err) that satisfies the first two criteria but does not incorporate model probabilities. Goel et al. (2025) then study implications of model correlation, such as affinity bias in LLM-as-jude judge and weak-to-strong training, and find that more capable models make more similar mistakes. We similarly find-across three datasets-ther that more accurate models make similar mistakes, and that this affects LLM-as-judge setups. How- ever, we also broadly quantify other correlates of similarity (shared developer, base architecture, model size). We then focus on different downstream outcomes, especially how correlation affects multi-agent systems like in hiring.
For each model, we further extract the model company and performance features (such as accuracy across the leader- board questions). HuggingFace provides model details, in- cluding the number of parameters and the base architecture (known due to the open-soures nature of the models). These features are not consistently available for the Helm and Resume models, which are often proprictary.
Resume-Job Description Evaluations. Starting from large datasets of job postings (Asaniczka, 2024) and re- sumes (Bhawal, 2022; Jiechieu & Tsopze, 2021), we select subset of 30 job descriptions and 60 resumes chosen via cluster analysis (to find related job descriptions and re- sumes). This provides 1,800 resume-job description pairs, which we evaluate using 20 LLMS (from Meta, Mistral AI, Amazon, Anthropic, and OpenAI). Again, we extract avail- able features describing these models. We hand-label 450 resume-job pairs (30 unique resumes and 15 job descrip- tions) using the same criteria
prompts.
Measuring correlated error. On HUGGINGFACE and HELM, where responses are multiple choice and where we have ground truth, we measure pairwise error correlation
Finally, recent work explores underlying homogeneity in language model representations, e.g., demonstrating that embeddings across models can be translated (Jha et al., 2025) and that activations are consistent at similar depths
col lecti i 1mm 1eadeeeborrddrrr open- 1een-11m-ledden 1n-deeeraaddddrrrd-d- (5.00d77601esss 6854431 fffe
3

---

Correlated Errors in Large Language Models
through agreement rate when both models are wrong: when two models wrong, how often do their wrong answers coincide? On RESUMES, we compare the correlation be- tween residuals: the difference between a model's 's rating and the human rating for each job-resume pair. Here, we treat the human rating as "ground truth," but note that job- resume evaluations are subjective. These measures of error correlation aim reduce confounding based on model ac- curacy: two independently accurate models will agree large fraction of questions in which they both give the correct answer. We, in contrast, are primarily interested in correlated errors. Appendices B and C contain results for alternate correlation measures (such as overall agreement).
Table Agreement Errors
HUGGINGFACE
HELM
RESUMES
Intercept
0.398
0.602
0.653
(0.001)
(0.001)
(0.007)
Same Company
0.066
0.022
0.021
(0.003)
(0.005)
(0.012)
Same Architecture
0.076 TM
(0.001)
Acc. 1
0.014
0.055
0.015
(0.000)
(0.001)
(0.006)
Acc. 2
0.013
0.054"*
0.028
(0.000)
(0.001)
(0.006)
Acc. 1: Acc. 2
0.023
0.026"*
0.043
(0.000)
(0.001)
(0.005)
models
349
71
20
3.2. Results
# responses/model
14,042
12,032
1,800
R²
0.340
0.618
0.415
Figure la shows the agreement rate between each pair of models (when both are wrong) on HUGGINGFACE, and Fig- ure lb shows the same on the HELM data. See Figure 9b for the analog on RESUMES. Two conclusions are immediately apparent from the agreement rate matrices.
Notes: Standard errors in parentheses. "'p < 0.001. Dependent variable: Agreement rate when both wrong (HUGGINGFACE, HELM), correlation in residual = predicted true (RESUMES), Numeric features (e.g., accuracy) are standardized, For HUG- GINGFACE, we omit subset of covariates in this table for concision; see Appendix C for the full regression.
(1) First, models substantially agree, even when both are wrong. Conditional on having the wrong answer, uniformly random model responses would yield an agreement rate of on the HELM, since each question has 3 incorrect answer choices. On HUGGINGFACE, this agreement rate random would be O. 0.12 On both datasets, almost all pairs (100% of pairs HuggingFace; 97.5% on Helm) of models have a higher agreement rate than the respective baselines. The mean agreement rate across pairs is 0.423 on HuggingFace and 0.6 on Helm, about double or higher than the baselines.
rates. Notably, more accurate models (and especially if both models are accurate) are more correlated. The included features explain between 34% and 62% of the variation in error agreement across the three datasets. In Appendix C we report full regressions, including for overall agreement rate and agreement rate when one model is wrong. Together, these results establish (a) there is substantial correlated er- rors in LLMss, and (b) this correlation more severe for more accurate models and those with shared characteristics.
(2) Second, some models agree with each other more than others. This can be expected: for example, on Helm, and meta/llama-
meta/llamaa332299056 2--90b-visiin-ninstttttt--eeee have an agreement rate (when both wrong) of .97, consistent with the vision-language model being based on the related language-only model. However, some models are unexpectedly correlated. For example, on Helm, google/text-unice untittrnninnnneen @ 001 and writer/palmyra-xx agree on 0.9987 fraction of the questions on which both are incorrect (about 22% of all questions), and a higher overall agreement rate; to our knowledge, there is publicly stated direct relationship between the models.
4. Case Study: LLM as judge
3. 1.1--0b-instruutt-eee
In this section, we examine the effect of LLM correlation in L.LM-as-judge setups, in which a judge model is used to evaluate the accuracy of other models. Then, we evaluate the accuracy of other models according to the judge's 's answers. We consider judges chosen be the most accurate model within shared model provider (on HELM) or architecture (on HUGGINGFACE), Figure 2 plots the results for HELM, giving models' true accuracy on the x-axis and the accuracy inflation (judged accuracy true accuracy) on the y-axis. Inflation above 0 indicates that the judged accuracy is higher than a model's true accuracy. We plot models from the same provider/architecture in red and other models in blue.
Sources of' model correlation. Table shows the results of a regression of the error agreement rate for each pair of models with the model characteristics. We find that models by the same developer, using the same base architecture, and having similar sizes are all associated with higher agreement
A stark pattern emerges: each judge systematically inflates the accuracy of models that are less accurate than itself, due to correlated errors (the judge marks incorrect answers as correct if both models agree on the incorrect answer). On the other hand, each judge underinflates the accuracy of models that are more accurate than itself (the judge cannot
"The questions have different numbers of answer choices, be- tween and 10. Let pk be the fraction of questions with k answer choices. Then, the agreement rate at random would be Ei Ph - Over 80% of the questions have à 10 choices.
4

---

Correlated Errors in Large Language Models
antrropk:
google:
openal:
meta:
quett:
anthropiciclsute--555 3-5-sonnet- 20241022
google/gemimi-11 S-pr-002
opensiggt--o-22244000
mete/llamm--3
1-00b--nnttttt-uuroo gwervqwen2.5> Sttp:ee22---2--nnininaae 1-2b--s-ut----tttttt
1.06
0.06
inftaton
0.06-
eccuraty inflatinn
accuracy inftst
accuracy lifflat
0.08
0.06
0.04
eccuraty
0.04
0.07
0.02
0.00
0.00
0.00
0.00
-0.07
066
0.8
0.6
true accuracy
true accuracy
true accuracy
accuracy
true accuracy
mistrala::
01-al:
microsoft:
writer/paliyyra. 004
mistralaimiitttal large 2407
aizi/jama 1.5 large
microsoftjphi medium 4k inst
0.125
0.050
Intiction
0.100
accurccy Infocton
intoctio
accurscy infatiio
0,050
accuracy Infitttion
0.075
0.060
0.00
0.000
0.000
9.025
-0.02
-0.425
-0 5.04
-0.050
-0.050
-8.025
..06
-0.075
0.6
0.8
1.0
0.5
0.8
1.5
0.5
0.0
1.
0.6
0.0
10
0.6
0.8
1.0
true eccuracy
true accuracy
true acturacy
true accuracy
true accuracy
databricks:
deepseok-af: al:
cohere:
snowflake:
allenal:
detair cksjdbrx-instuct
srowfleke/wwwet wttp///wwwwwwweeeeetiiiiiioe www.--aciiiiiinnttttttt
ellenaitoimo-11 1.7-b
accuracy intlutinn
eccuracy Inflation
0.05
Initation
accuurny intlatioon
accuracy Inftation
0.00
0.00
0.00
-0.05
-0.03
-0.3
-0.05-
-0.05
-8.10
-0.
-0.10
0.30
-0.15
0.8 accuracy
1.0
0.6
0.8
0.6
1.0
0.8 accuracy
0.6
1.0
accuracy
true acturacy
accuracy
Figure Evaluating LLM-as-judke on HELM. each plot, one model is used as the judge. Each dot is another model; the y-axis is the accuracy inflation (compared to ground truth) of using the given model as the judge, and the x-axis is the model's true accuracy. The vertical red line corresponds the true accuracy of the judge. Each judge tends inflate the accuracy of models that are less accurate than itself, especially models from the same provider or family (shown in red dots). Results for HUGGING are shown in Figure 8.
reward model for answering correctly question it itself answers incorrectly). We also see examples in which judges significantly inflate the accuracy of models from the same provider (shown in red). This reflects concerns of self- preferencing (Panickssery et al., 2024; Wataoka et al., 2024), further suggesting that relative self- self-preferencing can occur across different models from the same family; it further con- nects to other limitations of the LLM-as-judge paradigm (Gu et al., 2024; Kamoi al., 2024b;a; Stureborg al., 2024). More generally, the results suggest that, when comparing models or establishing eтror rates using LLM-as-judge, it is important to calibrate error metrics for each model-judge pair using ground-tre -trth data.
outcomes studied in this prior literature. First, we study the systemic exclusion rate (Bommasani et al., 2022), the proportion of applicants who are screened out of all jobs. Second, we study downstream outcomes of applicants in a matching market setting (Peng & Garg, 2024a). primary aim of this analysis is to test theoretical predictions on the impact of homogeneity. In contrast to existing empirical work, our experiments use real resumes and job descriptions. We also use LLMs opposed more classical tabular chine learning models, more closely resembling potential deployments of LLMs as hiring tools.
We study the effect of firms using the same or correlated LLMS by considering five experimental settings where firms rank applicants under the following methods:
5. Case Study: LLMs in labor markets
In this section, study the implications of LLM corre- lation in hiring settings. LLM-use in hiring decisions has been identified as a particularly important setting in which homogeneity-ine i.c.. algorithuic monoculture- may have negative effects (Kleinberg & Raghavan, 2021; Bommasani et al., 2022; Peng & Garg, 2024a), on top of general con- cerns regarding biased decisionmaking (Wilson & Caliskan, 2024; Gaebler et al., 2024). We study several downstream
All firms use the same LLM, simulating a setting in which one model dominates the hiring process, such through shared service provider.
Same LLM:
2. Same Company LLM:
Each firm uses a random LLM from the same company, simulating setting in which one company monopolizes the LLM space.
Latest LLM:: Each firm uses one of the most recent
5

---

Correlated Errors in Large Language Models
models from each company from the given LLMs, simulating a setting in which companies only use state- of-the-art LLMs.
systemic exclusion may be acceptable if some resumes are definitely poor fits for position.)
In Figure 3b, we further consider the systemic exclusion rate when firms form applicant preferences according to the 5 specified methods. We find that if firms all adopt models from the same company or they all adopt the latest LLMs, there is a somewhat higher degree of systemic exclusion in comparison to when they adopt random LLMs. However, these differences are fairly small in contrast to settings in which firms all adopt the same LLM.
4. Random LLMs:
Each firm independently at random selects an LLM from a given set of models, simulating a setting in which companies all use LLMs, but in a maximally diverse way, unrelated to ground truth fit and without correlated errors.
5. Uniformly Random: All firms have uniformly random applicant preferences, providing a baseline in which firms make uncorrelated decisions.
We note that the systemic exclusion rate does not account for true resume-job fits, e.g., as measured using hand la- bels, capacity constraints (each applicant can only work for one firm), and applicant preferences. Next, we analyze the implications of correlation in the presence of these effects.
These settings span the spectrum of complete monoculture (fully correlated decisions) to complete polyculture (fully in- dependent decisions). Settings 2-4 represent an intermediate state. In our previous results, we demonstrated that models tend to be correlated in general (even in how they err). Here, we examine the practical effects of this correlation.
5.2. Results: Matching markets effects
Above, we study systemic exclusion in setting where each firm offers interviews
fixed number of top applicants. However, theoretical work monoculture in matching mar- kets suggests that it is also important to consider market- level effects, since firms may adjust offers to fill capacity (Peng & Garg, 2024a). Under a stable matching framework, Peng & Garg (2024a) make three sets of predictions under algorithmic monoculture, corresponding to firm outcomes (do firms collectively hire the best-fit applicants), applicant outcomes (do they match with their most preferred firms), and outcomes under differential access (some applicants apply to more jobs than do other applicants).
5.1. Results: Systemic Exclusion
A key concern in algorithmic monoculture is systemic ex- clusion, in which an applicant is screened out of all oppor- tunities (Creel& & Hellman, 2022; Bommasani et al., 2022).
More formally, for set of firms F and a set of applicants A, we let sy(a) denote the percentile ranking of applicant E A in the preference list of firm f F. Then applicant a receives an interview from firm f if and only if sa(a) 2 1- - p. Therefore, an applicant is systemically excluded if sy(a) < 1 - for all firms f E F, i.e., they are not interviewed by any firm. The systemic exclusion rate of an economy equal to
In this section, we test these predictions under a stable matching framework and LLM correlation. We consider markets with a set of 60 applicants A and set of 30 firms F (or the hand-labeled subset). Each firm has capacity of 1: each applicant accepts at most one job offer, and each firm can hire at most one applicant. For all experiments in this section, each applicant E A has uniformly random pref- erences over firms. (Appendix B.2 reproduces experiments where applicant preferences are determined by an LLM.) To understand how markets behave when firms use LLMs to rank applicants, consider markets in which firms form their preferences according to the five methods outlined above, spanning full monoculture to full polyculture.
la E (a) < 1 - p for all f fe Fl
r(F)
(1)
[A|
the fraction of applicants interviewed by no firm. In our ex- periments, we set p = 0.25, so the top quarter of applicants receive interviews at each firm. We further assume that each firm uses the same job description.
In Figure 3a, we consider markets with firms, ranging from 1 to 20. We compare settings when firms have uni- formly random preferences over applicants and when firms each use a random LLM to rank applicants. When firms have random preferences, the systemic exclusion rate goes to 0 as n grows large (indeed, we should expect it to be (1- (.2)). When firms each use random LLM, how- ever, see that even with 20 firms, around 20 percent of applicants continue to be systemically excluded conse- quence of general correlation across LLMs. (Some level of
Firm Outcomes: match probability conditional on ap- plicant rating First, we study how an applicant's true rating affects match probability across different firm pref- erence methods. The theoretical literature establishes that monoculture worsens firm outcomes, i.e., applicants with the highest true ratings are less likely to match with firms (Kleinberg & Raghavan, 2021; Peng & Garg, 2024a); in contrast, more diverse models can result in a "wisdom of crowds" effect, with firms collectively matching with the
'Meta's Llama 3.3 70b, Mistral AI's Mistral Large (24.07), Amazon's 's Nova Pro, Anthropic's Claude Sonnet (20241022), and OpenAT's GPT-4o mini.
6

---

Correlated Errors in Large Language Models
Systemic Exclusion Rate
0.8
0.7
Systemic Exclusion Rate
Offer Method
0.7
0.6
LLM
0.5
Random
0.5
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
Same LLM (1)
Same Company LLM (3-7)
Latest LLM (5)
Random LLM (5)
Uniformly Random (5)
0.0
2
10 12 14 Number of LLMS
6 8
16
18
20
(a) The effect of the number of distinct LLM configurations used in a job hiring market systemic exclusion rate actoss different offer methods. For each where represents the number of LLMs use, we sample up to (29) combinations LLMs. If this exceeds 100, we limit the selection to 100 random combinations. LLMs are then evenly distributed across firms for a given joh. The graph shows that offers made applicants hased LLM scores consistently exhibit higher systemic exclusion rate than uniformly random selection applicants. This highlights that regardless heterogeneity in LLMs in job market, considerable amount of systemic exclusion will be present when giving offers based on scores generated by LLMs.
(h) Average systemic exclusion rates across markets under differ- ent firm preference methods. In each market, all firms share randomly sampled job description, and systemic exclusion rates are calculated by each preference method. As market settings shift from full monoculture (Same LLM) to full polyculture (Uni- formly Random), systemic exclusion rates decrease. The plot generated averaging over 1500 random markets.
Figure 3. Systemic exclusion (fraction of resumes with no job offers) when = 0.25: (a) for varying number of distinct LLMs used in job hiring market, (b) for varying firm preference methods. Lower systemic exclusion better.
highest rated applicants (Peng & Garg, 2024b).
We use our human hand labels as a ground truth for appli- cant quality. each market, we track which applicants successfully matched and their corresponding human- labeled scores. To analyze the effect of applicant rating, we group scores into discrete buckets (c.g.. [1, 2). [2, 3), etc.). The match probability for a given bucket then defined as the fraction of applicants in that bucket who are matched.
0.9
Same LUM (3)
Same Company LLM (3-7)
Match Probability
Latest LLM 15)
0.8
Ranoom LUM
0.7
0.6
0.5
Figure shows how match probability changes with true applicant rating, in each market. As expected from the thcoretical literature- among the LLM-based markets due to monoculure- usinu the same (randomly chosen) single LLM leads to the worst firm welfare, with relatively smaller match probabilities for the highest-ranked applicants. On the other hand, using the latest LLMs which are the most accurate individually but also more correlated than random MMs-mmximizes firm welfare. These results suggest that, in Raghavan (2024), there two competing effeetts for firm welfare: individual model accuracy, and diversity. LLM diversity may not yield of effects that outperform choosing the best L.Mss.
0.3
1
3
8
10
True Applicant Rating
Figure The effect of "true applicant rating" match probability across different market environments. We use a subset of 30 applicants and 15 firms, where the fit of applicant evaluated by humans. On the x-axis, we have true applicant rating, for each E [1, 10). represents scores in the range On the y-axis, we have match probability. The plot is generated by averaging over 1500 random markets.
firm is
1).

---

Correlated Errors in Large Language Models
Average Applicant Rank
6
5
4
8
Preference
Same LLM
3-
Relative Match Probability
7
Company L.M (3-7]
Latest LLM (5)
6
Random
Uniformly Random
1
5
4
Same LLM (1)
Same Company LLM (3-7)-
Latest LLM (5)
Random LLM (5)
Unifarmly Random (5)
2
1
0
5
10
15
20
25
30
# of Applications
(a) Average applicant ranks (averaged across applicants, the rank of the firm according g to the applicant's preference list to which they matched) across markets under different firm preference methods. Lower better, indicating higher applicant welfare.
(b) The effect of differential application access across different matching market environments. On the x-axis, we have the number of applications submitted by applicant firm. On the y axis, we have the relative match probability as defined eq. (4).
Figure 5. Market outcomes depending on firm LLM usage: (a) gives average applicant rank, (b) gives the effect of differential application access. Both plots are generated by averaging over 1500 random markets; in each, all firms share a randomly sampled job description.
Applicant Outcomes Another basic metric is applicant welfare: do applicants match to their top choices? Peng Garg (2024a) suggest that monoculture (increased cor- relation in firm decisions) yields higher overall applicant welfare (intuitively, the applicants that receive offers do from more firms, and get to choose which firm to accept an offer from). Correspondingly, we expect applicants in markets with more LLM monoculture to match with more- preferred firms than in markets with less correlation.
systemic exclusion analysis in Section 5.1, as predicted by Peng Garg (2024a). Markets with the highest systemic exclusion rates produce the highest-quality matches for ap- plicants, suggesting trade-off between systemic exclusion (without considering market effects) and applicant welfare when market effects considered. As above, markets with correlated models (either by the same company or the latest model from each company) have effects in between those of markets with firms using the same LLM or one at random.
Define rank(æ, y) as the rank the firm y in the preference list of applicant x-a ranking of indicates an applicant's most preferred firm. The output of the stable matching algorithm results in matching M A + F U (0). For set of applicants A, and firm-ape firm-applicat pairs M, average applicant ranking of match of a labor market is equal to
Differential Application Access Finally, study differ- ential application access, in which applicants may differ in the number of firms to which they apply. Instead of sub- mitting application to every firm, each applicant applies to a random subset of f firms, where f. Uniform(1,) (Fl), where is drawn for each applicant independently. This mimies job searching process which various factors (time, effort, environment, etc.) may affect the number of applications applicant submit. Peng & Garg (2024a) predict theoretically that monoculture (using the same algo- rithm) is more robust to differential access. We explore the effect of the number of applications submitted on relative match probability of applicants.
AvgRank(MM) Ea:(ays rank (a, (a)) la E A: M (a) + 0
Figure 5a shows that markets using single LLM (full mono- culture) yield the best average applicant rank; as market set- tings shift towards using uniformly random firm preferences (full polyculture). applicant welfare worsens.
These market-level results show an opposite trend from the
We run 1500 simulations for each method, in which each
8

---

Correlated Errors in Large Language Models
of the 60 applicants draws a number of applications inde- pendently. Then, across simulations, we calculate the match probability P manchh (f), for each number of applications t:
wrong), and find similar results. Finally, note that one limita- tion of current metrics (including ours and that of Goel et al.
(2025)) is that they treat incorrect answers identically; in practice, some incorrect answers may be "closer" to correct, and so preferred by more accurate models; some questions may also be harder than others. Future work should consider developing a metric that is robust to these characteristics.
P (1) = [{a E At a is matched)
[Ac
where A, is the set of applicants who applied to exactly firms. The relative match probability (t) is then:
Pmmaic( (t)
Multi-agent performance measurement and limitations. How to properly evaluate L.LM output is a highly contested, complex question (Wallach et al., 2024; Guerdan et al., 2025; Perlitz et al., 2024; Weidinger et al., 2025), and this is es- pecially true for multi-agent systems. Here, we leverage leaderboard metries based multiple-choice evaluations, and further have LLMs score resumes numerically, in an of- fline manner. While these measurements provide only a lim- ited view of LLM capabilities, their standardized nature and broad coverage across models enable us to conduct a large- scale empirical analysis of the correlation between model errors. In comparison to work evaluating open-ended model output diversity (Wu et al., 2024), we are able to evaluate many more (over 350) models, across over 20,000 questions with ground truth labels. Richer evaluation of open-ended generation and complex reasoning tasks remains an impor- tant direction for future work. Recent methods may allow analysis over the open-ended types of questions models are correlated on and the ways in open-ended responses cor- relate (Movva al., 2025). Our evaluations reflect many proposed use cases of LLMs in high-stakes settings: for example, natural language parsers of resumes already in use produce linle-dimeesioonl numeric scores of resumes, such as to shortlist resumes for human hiring managers. Our work does not however directly measure the implications of correlations induced by LLMs helping write job descriptions resumes (Wiles et al., 2025; Wiles & Horton, 2025).
P Pretative (t)
(4)
Pacchh (1)
where P manchh (1) represents the match probability when appli- cants submit only one application. This normalized metric measures the advantage of submitting more applications.
We find that overall, the relative probability of applicants matching increases based on the number of applications sub- mitted, but at different rates depending on the marketplace structure. As shown in Figure 5b, similar to trends from Figure 5a, as market settings shift from full monoculture to full polyculture, relative match probability is consistently higher for any t. An applicant is approximately 7 times more likely to match when submitting 30 applications than 1 application in markets under uniformly random prefer- ences, and conversely, an applicant is approximately just 2 times more likely to match when submitting 30 applications versus 1 application when firms use the same LLM. These results align with the hypothesis in Peng & Garg (2024a) markets under full monoculture are most robust to differen- tial application access, since those with more applications receive fewer independent "lottery tickets."
6. Conclusion
We show that LLMs have correlated errors, and that this correlation is substantially higher for individually accurate models and those by the same developer or using the same base architecture. These findings suggest that as model per- formance increases, models are also converging in the errors that they make. This error correlation has implications for the effectiveness of the LLM-as-jun jujuuge paradigm, the use of LLMS in hiring, and of multi-agent systems broadly.
Implications for ecosystem monitoring. While work evaluating models individually is prevalent, analysis of cor- relations across models has been surprisingly rare. This is true even though, as leverage, such analyses can be done using the same data as is used to evaluate models in- dividually, since benchmarks often repeat questions across models. We thus encourage leaderboard developers to con- tinuously track model correlation as we show, doing so reveals surprising patterns between individual pairs of mod- els and provides insights on which models might be most effective use together. Similarly, in our application to study LLMs in labor markets, there exist laws (such as NYC Local Law 144) mandating that companies audit hiring al- gorithms (Groves et al., 2024; Wright et al., 2024; Terzis et al., 2024); however, empirical analyses of correlations across models or of multiple firms sharing algorithms are relatively unexplored.
Measure of correlation For our multiple choice analysis in the main text, we use "agreement rate when both mod- els err" as our measure of correlation. Like in the work of Goel et al. (2025), such a metric is designed to correct for model accuracy: two models will not be deemed to be more correlated simply because they both get question correct. The metric of Goel et al. (2025) also leverages agreements on correct answers and model output probabilities, and so may be preferable when such probabilities are available. In the appendix. we also show results using alternate metrics (overall agreement rate, and agreement rate when either is

---

Correlated Errors in Large Language Models
Impact Statement
Creel, K. and Hellman, D. Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philoso- phy, 52(1):26 43, 2022.
The algorithmic leviathan:
We demonstrated that LLMs exhibit large amounts of cor- relation, even in how they err. This has implications for high-stakes settings such as hiring, where there risk that applicants systematically screened out of opportunities, and in LLM-as-judge settings, where models may inflate the estimated performance of other models. While our analysis is limited to a subset of tasks on which homogeneity may be a concern, it serves as a foundation for further evaluation and monitoring of the LLM ecosystem.
Gaebler, J. D., Goel, S., Huq, A., and Tambe, P. Auditing the use of language models to guide hiring decisions, 2024. URL https::/p/.... werxiivorrr /abb//////// 24404 03086.
Geirhos, R., Meding, K., and Wichmann, F. A. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. Advances in neural information processing systems, 33:13890-13902 2020.
Acknowledgements
We thank Manish Raghavan for helpful discussions. NG is supported by NSF CAREER IIS-2339422 and Cornell Tech Urban Tech Hub, Meta, and Amazon research awards, the latter of which provided API credits for this project. EK was supported by Cornell BURE undergraduate research fellowship.
Goel, S., Struber, J., Auzina, I. A., Chandra, K. K., Ku- maraguru, P., Kiela, D., Prabhu, A., Bethge, M., and Geiping, J. Great models think alike and this undermines ai oversight. arXiv preprint ar S..iv:22502.04311.. 2025.
Groves, L., Metcalf, J., Kennedy, A., Vecchione, B., and Strait, A. Auditing work: Exploring the new york city algorithmic bias audit regime. In The 2024 ACM Confer- ence on Fairness, Accountability, and Transparency, pp. 1107-1120, 2024.
References
Alvero, A., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., and Antonio, A. L. Large language models, social demography, and hegemony: comparing author- ship in human and synthetic text. Journal of Big Data, 11 (1):138, 2024.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on Ilm-as-a- judge. arXiv preprint arXiv:2411.15594 2024.
Guerdan, L., Barocas, S., Holstein, K. Wallach, H., Wu, Z. S., and Chouldechova, A. Validating Ilm-as-a-judge systems in the absence of gold labels. arXiv preprint wxXi:2503.059633 2025.
Asaniczka.
Upwork job postings dataset 2024 (50k records), 2024. //www, kaggle. upwork
URL https:
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint MoXiv:2009.0333 2020.
Back, J., Bastani, H., and Chen, S. Hiring under congestion and algorithmic monoculture: Value of strategic behavior. arXiv preprint 2rX2:2250..2066, 2025.
Bhawal, S. Resume dataset, 2022.
URL
Jain, S., Creel, K., and Wilson, A. Scarce resource alloca- tions that rely on machine learning should be randomized. arXiv preprint arXiv:2404. 08592, 2024a.
https: // /wwww/wwwww//////////wo htt:/w/wkgg/t http:/ll...om//aaaateeeeeee treehaarblaaall/eesm---eeer
Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., and Liang, P. S. Picking on the same person: Does algo- rithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 3663-3678, 2022.
Jain, S., Suriyakumar, V., Creel, K., and Wilson, A. Algorith- mic pluralism: structural approach to equal opportunity. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 197-206. 2024b.
Jha, R., Zhang, C., Shmatikov, V., and Morris, X. Har- nessing the universal geometry of embeddings. arXiv preprini arXiv: /2540, 2025.
Bommasani, R., Soylu, D., Liao, T. I., Creel, K. A., and Liang. P. Ecosystem graphs: The social footprint of foun- dation models. March 2023. URL https: //arxv....vvv org/ org/abs/// abs/ 2303. 15772..
Jiechieu, K. F. F. and Tsopze, N. Skills prediction based on multi-label resume classification using cnn with model predictions explanation. Neural Computing and Applica- tions, (300) 2021. ISSN 1433-3058. doi: 10. 10.1007/00521-0000-00002.20 URL https: / /doi.
Chen, Z., Li. J., Chen, P., Li, Z., Sun, K., Luo, Y., Mao, Q., Yang, D., Sun, H., and Yu, P. S. Harnessing multiple large language models: A survey on Ilm ensemble. arXiv preprint arXiv:2502. 1in.:2002..80666 2025.
org//00.000//800221-0000
10

---

Correlated Errors in Large Language Models
Kamoi, R., Das, S. S. S., Lou, R., Ahn, J., Zhao, Y., Lu, X., Zhang, N., Zhang, Y., Zhang, H. R., Vumman- thala, S. R., Dave, S., Qin, S., Cohan, A., Yin, W., and Zhang, R. Evaluating LLMs at detecting errors in LLM responses. In First Conference on Language Modeling, 2024a. URL wwththts///// /ttp:////ppeeeeiewe/ee net/ forum? (b=dnwRSccjxx:
Raghavan, M. Competition and diversity in generative ai. arXiv preprint arXiv:2412.08610, 2024.
Senthilkumar, N. K., Ahmad, A., Andreetto, M., Prab- hakaran, V., Prabhu, U., Dieng, A. B., Bhattacharyya, P., and Dave, S. Beyond aesthetics: Cultural competence in text-to-image models. Advances in Neural Information Processing Systems, 37:13716-13744 2024.
Kamoi, R., Zhang, Y., Zhang. N., Han, J., and Zhang, R.
When can Ilms actually correct their own mistakes? critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417- 1440, 2024b.
Stureborg, R., Alikaniotis, D., and Suhara, Y. Large lan- guage models are inconsistent and biased evaluators. CoRR, 2024.
Tekin, S. F., Ilhan, F., Huang, T., Hu, S., and Liu, L.. Llm- topla: Efficient IIm ensemble by maximising diversity. arXiv preprint arXiv:2410.03955 2024.
Kleinberg. J. and Raghavan, M. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 11822))=2018340113 2021.
Terzis, P., Veale, M., and Gaumann, N. Law and the emerg- ing political economy of algorithmic audits. In The 2024 ACM Conference Fairness, Accountability, and Trans- parency, Pp. 1255- 1267, 2024.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang. Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C. A., Manning, C. D., Re, C., Acosta-Navas, D., Hud- Son, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao. H., WANG, J., Santhanam, K., OIT, L., Zheng, L.,, Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N. S., Khattab, O., Henderson, P., Huang, Q. Chi, R. A., Xie, S. M., Santurkar, S., Gan- guli, S., Hashimoto. T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models. Transac- tions on Machine Learning Research, 2023. ISSN 2835- forum?
Toups, C., Bommasani, R., Creel, K., Bana, S., Jurafsky, D., and Liang, P. S. Ecosystem-level analysis of deployed machine learning reveals homogeneous outcomes. Ad- vances in Neural Information Processing Systems, 36: 51178-51200 2023.
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. Replacing judges with juries: Evaluating IIm gener- ations with a panel of diverse models. arXiv preprint arXiv:2404. 18796, 2024.
8856. URL https::/ http:////pppeeeeeeeeooome id=104LZibEch
Movva, R., Peng. K., Garg, N., Kleinberg, J., and Pierson, E. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502. 04382, 2025.
Wallach, H., Desai, M., Pangakis, N., Cooper, A. F., Wang, A., Barocas, S., Chouldechova, A., Atalla, C., Blodgett, S. L., Corvi, E., et al. Evaluating generative ai systems is a social science measurement challenge. arXiv preprint arXhv:2411.10933 2024.
Panickssery, A., Bowman, S., and Feng. S. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772- 68802, 2024.
Wang. A., Morgenstem, J., and Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Na- ture Machine Intelligence, pp. 1-12, 2025.
Peng. K. and Garg, N. Monoculture in matching markets. In The Tiirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024a. URL https: / //openreview net met/ffoumm forum? iddPSeezHuSS..
Wataoka, K., Takahashi, T., and Ri, R. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819, 2024.
Peng, and Garg. N. Wisdom and foolishness of noisy matching markets. In Proceedings of the 25th ACM Con- ference on Economics and Computation, pp. 675, 2024b.
Weidinger, L., Raji, I. D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., and Isaac, W. Toward an evaluation science for gener- ative ai systems. arXiv preprint arXiv:2503.053366 2025.
Perlitz, Y., Gera, A., Arviv, O., Yehudai, A., Bandel, E., Shnarch, E., Shmueli-Scheuer, M., and Choshen, L. Do these Ilm benchmarks agree? fixing benchmark evalua- tion with benchbench. arXiv preprint arXiv:4407.13666, 2024.
Wenger, E. and Kenett, Y. We're different, we're we're the same: Creative homogeneity across 1lms. arXiv preprint arXiv:2501.19361, 2025.

---

Correlated Errors in Large Language Models
Wiles, E. and Horton, J. J. Generative ai and labor market matching efficiency. Available at SSRN 5/87344, 2025.
Wiles, E., Munyikwa, Z., and Horton, J. Algorithmic writ- ing assistance on jobseekers' resumes increases hires. Management Science, 2025.
Wilson, K. and Caliskan, A. Gender, race, and intersectional
bias in resume screening via language model retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume pp. 1578-1590, 2024.
Wolfram, C. and Schein, A. Layers at similar depths gen- erate similar activations across Ilm architectures. arXiv preprint arXiv:2504.0885 08775, 2025.
Wright, L., Muenster, R. M., Vecchione, B., Qu, T., Cai, P. Smith, A., Investigators, C. S., Metcalf, J., Matias, N., et al. Null compliance: Nyc local law 144 and the challenges algorithm accountability. The 2024 ACM Conference on Fairness, Accountability, and Trans- parency, pp. 1701-1713, 2024.
Wu, F., Black, E., and Chandrasekaran, V. Generative monoculture large language models. arXiv preprint arXiv:2407.02200 02209, 2024.
Xu, W., Jojic, N., Rao, S., Brockett, C., and Dolan, B. Echoes ai: Quantifying lack of plot diversity in Ilm outputs. arXiv preprint 3xXi:22501.0027: 2024.
Yang, H., Li, M., Zhou, H., Xiao, Y., Fang, Q. Zhou, S., and Zhang, R. One Im is not enough: Harnessing the power of ensemble learning for medical question answering. Journal of Medical Internet Research, pp. 70080, 2025. doi: 10.2196/70080. URL https: : preprints s jmir.. org. http///rrpriit/0000000 forth- coming/in press.
Zhang, S., Xu, J., and Alvero, A. Generative ai meets open- ended survey responses: Research participant use of ai and homogenization. Sociological Methods & Research, 2025.
pp.
Zheng, L.,, Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et Judging llm-as-a with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hypothesis Generation with Large Language Models. (44Xiv:2004.04326), August 2024.
12

---

Correlated Errors in Large Language Models
The appendix is organized follows:
In Appendix A we describe details on how we constructed our three datasets of LLM responses: HUGGINGFACE, HELM, and RESUMES.
In Appendix B, we provide additional figures and analyses that were excluded from the main text.
In Appendix C, we provide exact results from our regression analyses.
In Appendix D, we provide full list of models available in our datasets.
A. Dataset Construction
A.I. HUGGINGFACE, HELM
We started from two prominent LLM leaderboards: (1) HuggingFace's Open LLM Leaderboard 24; and (2) Stanford's Holistic Evaluation of Language Models (Helm) (Liang et al., 2023). Both use questions from the Massive Multitask Language Understanding (MMLU) question-answe -anseernnnn dataset (Hendrycks et al., 2020), with multiple choice questions and labeled correct answers. Both leaderboards provide model answers on each question, along with metadata about the models and questions.
As we are interested in explaining model correlations, we further collect features for each model. For models it evaluates, HuggingFace further provides a range of model features (all known due to the open-source nature of the models), including the number of parameters and the base architecture. Such features are not consistently available for the Helm models, given the proprictary nature of the models. For each model in both data sources, we further extract the model company and performance features (such as accuracy across the leaderboard questions).
The HuggingFace leaderboard focuses open-source models, including both "base" models (often provided by larger developers) and adaptations by others (such as additional fine-tuning). For tractability," we filtered the original 2041 models in the dataset to set of 349 models. 6 The Helm model list includes both open and proprietary LLMs, focusing on larger companies; at the time of our analysis, it provided evaluations for 71 models across 15 companies including OpenAI, Anthropic, Google, DeepSeek, and Qwen. Additionally, if company has released multiple versions of the same model, or has multiple model offerings, the most recent models are present in their respective datasets.
HuggingFace's ' MMLU dataset samples 12,032 multiple choice questions from 91 different MMLU datasets and across 14 different categories such as business, history, economics, and computer science. The raw response in which the model ranks the options is provided and we extract the model's top-ranked predicted answer. Most questions have 10 answer choices. The Helm Dataset is similar and has 14,042 multiple choice questions across 57 categories; questions are limited to questions with 4 options, and the model is prompted to directly select one of the choices. The categories include high school and college computer science, college computer science, jurisprudence, abstract algebra, astronomy, and US foreign policy.
A.2. RESUMES
We started from a dataset of 53,058 job postings on Upwork, spanning various categories and countries to represent our jobs dataset (Asaniczka, 2024). Our resume dataset consists of a combination of 2,484 resumes from LiveCareer (Bhawal, 2022), a database of high-quality resumes, and 29,780 resumes from publicly available resume repository used by Jiechieu & Tsopze (2021).
We removed duplicates as well as resumes or job descriptions that were more than standard deviation below the median length. We calculated sentence embeddings for each resume and job description, via Sentence Transformers (SBERT). We
"sttpa:/ httpe///hgggggngggeeeee facce. an oc llett i focetion///ae aon//oonnnonniioeoeooo 1appnn11mm- 11m- loaderboord/open- 1m- leaderbcard- 2-rrnd22-66000m 660-66600000000 26t.222600000000oooee2222333111- cha6a52431fft
"Since we are analyzing correlations between each pair of models, our analyses scale by the (# of questions) x m(n-1) where v is
the number of models analyzed.
'Hugingface allows users to upload their own models trained from scratch (mostly done by large developers, for example, the Llama family from Meta is available) and to allow individuals to modify base models (such as via additional tine-tuning), Starting with the list of the 500 most accurate models on the leaderboard, we first found 451 models that had well-structured MMLU datasct available through API query. Then, we select all models from any major developer (such as those that upload their own base models), and up to five models each from any individual that fine-tunes other models, picking the most accurate models. This procedure allows analyzing the effect of shared developer and base model, without any individual dominating the dataset, and led to 349 models that we analyze.
13

---

Correlated Errors in Large Language Models
then applied KMeans clustering to each job description and resume dataset; we chose the number of clusters (28) maximize similarity between the most similar pair of resume and job clusters, respectively (see Figure 6a). We selected highly similar pairs of (job description, resume) clusters as shown in Figure 6b, each representing different fields (general IT, web development, and consultin/finance to simulate applicants applying to jobs from across and within various industries. Finally, we sampled 20 applicants and 10 jobs from each cluster to construct dataset of 60 resumes and 30 job descriptions.
Resume:
Resume:
Resume: 23
0.90
Resume:
Jobr Resumer
Jobi Resuree
Highest Cosine Similrity
0.85
0.80
Job: 13, Resume: 10
Job: 13, Resume: 26
Job: 13. Resume:
0.75
0.70
aasser damad Sodd morelto
Linaz ta heraes
Caud 3 Sith Comdomm
Matted aid goll ann hara
3 Cinnt 1
No
Cheers
Number of Clusters
(a) Highest cosine similarity value between a resume cluster and a job description cluster. The axis represents the number of clusters both the resume and job description datasets y axis represents the highest cosine similarity value given pairs of clusters. This graphs shows that as the number clusters we set in the KMeans algorithm increases, the highest cosine similarity also increases, but this peaks when there are 28 clusters.
(b) Average scores outputted by each LLM, categorized by each pair of resume, job clusters. The graphs along the main diagonal of the figure the three pairs of clusters chosen, and observe that the overall scores are higher compared to those of other rows/columns. This shows accurate to simulate real job markets.
clustered to. The
clustering process was
Pigure Cluster analysis for selecting resumes and job descriptions in RESUMES.
We used 20 open-source models available on Amazon Bedrock, specifically models made from Meta, Mistral AI, and Amazon, Anthropic API, OpenAl API (see Appendix 10), and 2 prompts. For each applicant-job pair, we prompt an LLM to score the fit of given resume to given job description through a single score, following structured criterion, as shown in Figure?. 7.
B. Additional Figures and Analyses
B.1. LLM-as-judge
Figure shows the LLM-as-judge analysis for HUGGINGFACE models.
B.2. Matching Markets with LLM-Specified Applicant Preferences
We now replicate several of the main results the market results, when applicant preferences are determined by Llama
3.1 405B instead of random. Figure 10a shows average applicant welfare. Figure 10b shows the effect of differential application access.
14

---

Correlated Errors in Large Language Models
"You are going evaluate the fit of resume job description using single score scale from 1 10. When grading, you consider overall fit, category fit, and skill fit. Your output format a json file must follow this example: "Score":
Resume: {resume}
Job Deseription: (job description}
Json File:
"You are going evaluate the of resume a job description using a single score on scale from When grading, consider the following:
1. Overall fit: This metric evaluates how well the candidate's 's resume aligns with the joh description, taking into account their background, experiences, and qualifications. Consider not only their current suitability but also their potential for growth, adaptability, and ahility thrive in the role.
2. Category fit: This metric assesses the relevance of the candidate's primary field or industry experience to the job's sector. For example, a resume comes from a professional a different field but with some relevant transferable skills, consider their potential to succeed in the new field, rather than focusing solely on mismatched backgrounds.
3. Skill fit: This metric evaluates the match between the candidate's listed skills and those required by the job. Consider both the relevance and proficiency of these skills, but also factor in the candidate's ability to learn and grow in areas where there may be gaps, ensuring the evaluation takes into account future potential. This is the grading scale:
10 Excellent Fit: The resume aligns very closely with the job description in all key areas. The candidate's s background, industry experience, and skills are not only relevant but also demonstrate high proficiency, making them a strong match for the role.
7 Good Fit: The resume shows strong alignment with the job description in most areas. The candidate's hackground and skills are largely relevant, and while there may be few gaps, they have the necessary qualifications and potential to perform the role effectively.
5 Average Fit: The resume meets some of the key requirements the job description. While the candidate may not have perfect alignment. there is still moderate relevance in terms of background, skills, and experience, though some key areas may be lacking.
3 Poor Fit: The candidate shows limited relevance the job description. Although there may be few transferable skills or some related experience, significant gaps exist across background, skills, industry relevance, making the candidate a less likely match.
1 Extremely Poor Fit: The resume shows little to alignment with the job description. The candidate's 's background, experience, and skills are largely unrelated to the job's requirements, making them an unlikely match for the role.
Your output format is a json file must follow this example: { "Score":
Resume: [resume}
Job Description: [job description]
Json File:
Figure The prompts used to score
resume to a job description. The first prompt named "firm rate comb short and the second prompt is named "firm rate comb 2". The resume and job description texts taken directly from the dataset. A similar prompt used applicants, named "app_rate_comil ratt__comh_hhort"" rating job description based on their resume, hut the references to person The prompt that used in the main experiments was "firm rate comb 2" because the prompt laid out specific criteria for the LLM to consider.
("candidate", "applicant", "resume") being replaced with references job, and vice
15

---

Correlated Errors in Large Language Models
Qwen2FarCausalLM:
LasaaooCausallM
MistralFaCCuuallMM
MaziyarPanahjraale http://nnnniiilll----sssstt----e wtt.esstnnnt--hhgiiieiddeeeeeeee lama-3 1-708
Queni))en2-V1-22 -77-innnntt
mistralaa/Misrra. arge-Instruct
theppirt/Raniii-oo 2ww.ne 2-5-14
Intatio
0.20
accuracy inflation
eccuracy intlation
accuracy inflation
0.05
accuracy Infration
22
0.10
0.03
0.3
0.05
-0.05
0.0
-0.55
accuracy
true accuracy
accuracy
accuracy
accuracy
ForCauselLMM
Phi3farCausalUMA
Owen2Moeff Creuneeeeeerrreeeleeeeeee
MrisforCamdtionlooeeettto
claudyu/Mixtrl 348K2 McE
Coppcctto//hoccoettee Instruct
Qwen/Swer2 AA14B Instruct
infrtton
acursay Inflation
acourocy Infitton
accurecy Infration
0.2
0.2
8.0
0.1
eccuracy
0.0
-0.3
-0.3
+0.3
-0.
0.4
0.6
n.8
1.0
0.4
0.6
1.4
2.1
0.6
0a
1.0
0.4
0.6
n.
1.0
0.4
0.6
n.
1.0
true sccurecy
true accuracy
true accuracy
true accuracy
true sccurecy
ExconoFonCausall..
CohereForCaussal..t
InterotW2ForCaucallkk internimminteminn2 5-20b-chat
RalcanMammbaoorCaussll L:
CohereforA/ayyeeeeeee Sta
Sccuracy inflation
0.1
accuraicy inflation
accuracy inflation
0.
-0.3
9.3
-0.2
-0.3
-0.3
-0.3
accuracy
accuracy
accuracy
accuracy
Figure Evaluating LLM-as-judge on HUGGINGFACE. each plot, one model used judge. Each dot another model: the y-axis is the accuracy inflation (compared to ground truth) of using the given model as judge, and the x-axis is the model's true accuracy. Each judge tends to inflate the accuracy models that less accurate than itself, especially models from the same provider or family (shown in red).
16

---

Correlated Errors in Large Language Models
Gut
Uamas sib
49
Uamusa--73
Gpe-ob-rii
Woo-Tro
N Large
Gal rt
Mava Large
Clacde Serret222243122
Combaai
Claudo 2-5 Senvin 602614022
Chade 25.jaala22204102
Wivv
Masrotiagg
(aank
J, 2.
Mocc
Lonss
45
Matral intruct
Cotd S-take
M sa Ab
(a) Heatmap of correlations between LLM configurations used experiments. We define an LLM configuration as combina- tion of an LLM and a prompt that was used. We calculate the correlation between the scores of each pair of LLM configura- tions. After analyzing strong correlation across prompts, we constrain the heatmap for configurations only using the prompt "firm rate comb2" to focus on the effect models. We can serve that there is strong correlation amongst larger models from the same company.
(b) Heatmap of error correlations between LLM configurations used experiments.
calculate correlation between the "residuals" (subtracting off the hand-labelen scores) of each pair of LLM configurations, where hand-labeled scores are available. Similarly to the trends of Figure 9a, larger models from same company similarly.
Figure Correlation heatmaps on RESUMES.
17

---

Correlated Errors in Large Language Models
8
Average Applicant Rank
Fimm Preference Method
Relative Match Probability
7
Same LLM
LLM (3-7)
6
Latest (5)
2
Uniformly Random
Random
5
4
Same LLM (1)
Same Company LLM (3-7)
Latest LLM (5)
Random LLM (5)
Uniformly Random (5)
3
2
1-
25
of Applications
(a) Average applicant ranks markets under different firm preference methods. Analog of Figure Sa, where applicant pref- erences are now determined by Llama 3.1 40SB. While similar patterns hold relatively, average applicant rank is generally worse this market, since preferences between applicants related. case, adoption of multiple LLMs does appear to significantly improve applicant rank in comparison the uni- formly random firm preference setting.
(b) The effect of differential application matching market environments. Analog Figure 5b, where appli- cant preferences are now determined by Llama 3.1 40SB. Similar patterns hold.
different
more cor-
Figure 10. Replication of matching markets experiments given L.MM-detemmneed applicant preferences.
18

---

Correlated Errors in Large Language Models
C. Full Regressions
For all regressions, samples are pairs models. We randomize the order of the models.
C.1. HUGGINGFACE
Table 2. Agreement Errors: HUGGINGFACE
coef
std
P>
[0.025
0.975]
Intercept
0.3984
0.001
656.121
0.000
0.397
0.400
same.company| Truee]
0.0658
0.003
21.428
0.000
0.060
0.072
same architectureeT........ Truue
0.0759
0.001
102.157
0.000
0.074
0.077
i..moe[ moruT.TTuee is 1s_moc_211 2[T.Truc]
0.0006
0.003
0,200
0.842
-0.005
0.007
0.0053
0.003
1.682
0.092
-0.001
0.011
params. billions. params.billions.e log params billions log generation generation_2
0.0012
0.000
2.569
0.010
0.000
0.002
0.0014
0.000
2.989
0.003
0.000
0.002
0.0031
0.000
7.710
0.000
0.002
0.004
0.0032
0.000
8,100
0,000
0.002
0.004
haram-dfff
-0.0215
0.000
-52.082
0.000
-0.022
-0.21
accuracy_ 1 accuracy
0.0138
0.000
31.917
0.000)
0.013
0.015
0.0131
0.000
30.087
0.000
0.012
0.014
accuracy 1:accuracy 2
0.0227
0.000
65.631
0.000
0.022
0.023
Dependent variable: Agreement rate when both models are wrong. No. ohservations: 60726.
R²² = 0.340.
Table 3. Agreement on Errors when either model is wrong: HUGGINGFACE
coef
std err
P>
[0.025
0.975]
Intercept
0.2381
0.001
376.046
0.000
0.237
0.239
Company[T.Trur
0.0793
0.003
24.765
0.000
0.073
0.086
same Architecture(TTTrus
0.0773
0.001
99.690
0.000
0.076
0.079
[s.moe[[
0.0023
0.003
0.724
0.469
-0.004
0.009
A.mooe_[T.TTre''
0.0071
0.003
2.175
0.030
0.001
0.013
params_billin ggrams_billions_lgg
0.0015
0.000
3.122
0.002
0.001
0.002
params billions log
0.0019
0.000
3.793
0.000
0.001
0.003
generation
0.0026
0.000
6,281
0.000
0.002
0.003
generation.2
0.0028
0.000
6,887
0.000
0.002
0.004
param diff
-0.0230
0.000
-53.509
0.000
-0.024
-0.022
accuracy 1
0.0004
0.000
0.880
0.379
-0.000
0.001
accuracy
-0.0006
0.000
-1.340
0.180
-0.001
0.000
Securacy_Iaaccraae
0.0226
0.000
62.639
0.000
0.022
0.023
Dependent variable: Agreement rate when either model is wrong. No. observations: 60726.
R² 0.300.
19

---

Correlated Errors in Large Language Models
Table 4. Agreement all questions:
HUGGINGFACE
coef
std err
P>
[0.025
0.975]
Intercept
0.4834
0.000
979.834
0.000
0.482
0.484
same.com wateeecmommall...e Truuel
0.0530
0.002
21.223
0.000
0.048
0.058
same. www.aaaccceteee...Tee..
0.0602
0.001
99.743
0.000
0.059
0.061
is is_moe[ moeeT.True]
0.0026
0.002
1.062
0.288
-0.002
0.007
0.0064
0.003
2.509
0.012
0.001
0.011
params. billions log params billions log generation generation_2
0.0025
0.000
6.650
0.000
0.002
0.003
0.0028
0.000
7.360
0.000
0.002
0.004
0.0024
0.000
7.297
0,000
0.002
0.003
0.0025
0.000
7.815
0.000
0.002
0.003
param-diff
-0.0182
0.000
-54.147
0.000
-0.019
-0.018
accuracy_1
0.0248
0.000
70.477
0.000
0.024
0.026
accuracy
0.0241
0.000
68.139
0.000
0.023
0.025
accuracy 1:accuracy 2
0.0225
0.000
80.179
0.000
0.022
0.023
Dependent variable: Agreement rate overall. No. observations: 60726. R² 0.451.
C.2. HELM
Table 5. Agreement on Errors: HELM
coef std err
P> |t]
[0.025
0.975]
Intercept same wwwpwoymmmmccccccccccc [T.rue]
0.6018
0.001
437.617
0.000
0.599
0.605
0.0216
0.005
4.753
0.000
0,013
0.031
accuracy_1 accuracy accuracy accuracy_I 1
0.0549
0.001
41.914
0.000
0.052
0.058
0.0544
0.001
41.478
0.000
0.052
0.057
0.0264
0.001
19.460
0.000
0.024
0.029
Dependent variable: Agreement rate when models are wrong. No. observations:
2485. R² = 0.613.
Table 6. Agreement on Errors when either model is wrong:
HELM
coef
std err
P> |t]
[0.025
0.975]
Intercept
0.2404
0.001
195.475
0.000
0.238
0.243
same company(TT.Trus
0.0264
0.004
6.498
0.000
0.018
0.034
accuracy.1
0.0219
0.001
18.708
0.000
0.020
0.024
accuracy accuracy 1:aceuracy 2
0.0224
0.001
19.101
0.000
0.020
0.025
0.0358
0.001
29.504
0.000
0.033
0.038
Dependent variable: Agreement rate when either model is wrong. No. observations:
2485. R² 0.406.
Table 7. Agreement on all questions: HELM
std err coef
P> [t]
[0.025
0.975]
Intercept same www.poomommerrueeeeeoooo
0.6895
0.001
786.886
0.000
0.688
0.691
0.0121
0.003
4.182
0.000
0.006
0.018
accuracy_1
0.0724
0.001
86.742
0.000
0.071
0.074
accuracy 2
0.0703
0.001
84.129
0.000
0.069
0.072
accuracy 1:a.uruccccuurcy2
0.0293
0.001
33.880
0.000
0.028
0.031
Dependent variable: Overall agreement rate. No. observations: 2485. R²
0.865.
C.3. RESUMES
20

---

Correlated Errors in Large Language Models
Table 8. Correlation in residuals of estimating joh-resume fit: RESUMES
coef
std err
P> It]
[0.025
0.975]
Intercept
0.6534
0.007
89.839
0.000
0.639
0.668
same_com gaaeeecommaayy.Iivee latest model sedest aatest_mode 2[T.Tree
0.0210
0.012
1.694
0.092
-0.003
0.046
1[T.True]
0.0155
0.013
1.211
0.228
-0.010
0.041
0.0047
0.014
0.346
0.729
-0.022
0.032
correlation. tiththooowwtttttt hwmahhumaaoooo _conee correlation with human score.2 2coman.ccore22
0.0151
0.006
2.627
0.009
0.004
0.026
0.0283
0.006
4.993
0.000
0.017
0.039
correlation with human score ::correlation. with human score
0.0429
0.005
8.529
0.000
0.033
0.053
Dependent variable: Correlation in residual (predicted human evaluation). No. observations: 190. R² 0.115.
Table Agreement on job-resume ratings: RESUMES
coef
err
P> |t]
[0.025
0.975]
Intercept
0.6377
0.005
117.254
0.000
0.627
0.648
same..omme came__copanyy[TTnug latest_model _oo.el_ latest model 2rT.Truee]
0.0340
0.009
3.660
0.000
0.016
0.052
-0.0069
0.010
-0.719
0.473
-0.026
0.012
-0.0104
0.010
-1.016
0.311
-0.031
0.010
correlation with human score.
0.0857
0.004
19.960
0.000
0.077
0.094
correlation with human score
0.1129
0.004
26.633
0.000
0.105
correlation with human huran.score score l:correlation with human score.2 human.score.2 2
0.0421
0.004
11.204
0.000
0.035
0.050
Dependent variable: Correlation. No. observations: 190. n² = 0.896.
D. Models analyzed
Table 10: Models analyzed for market analysis (correlation scores with prompt used in the main text)
model
company correlation with www.hhhhhmanscoo
latest.model
0
Nova-Pro
Nova
0.73
True
1
Nova-Micro
Nova
0.64
False
Nova-Lite
Nova
0.55
False
3
Mistral-Large
Mistral
0.69
True
4
Histral-Large(244
Mistral
0.66
False
5
Mistral-8x70
Mistral
0.54
False
6
Mistral-7B-IIntruuct
Mistral
0.46
False
Llama3-1-408)
Llama
0.67
8
Llama3-3-70b
I.ama
0.67
True
9
Llama3-2-90b
Llama
0.64
False
10
Llama3-1-77
Llama
0.63
False
Llama3-2-11D
Llama
0.46
False
12
Llama3- 1-8b
Llama
0.46
False
13
Llama3-70b
Llama
0.40
False
14
Gpt-ol-mini
Gpt
0.70
False
15
Gpt-4o-mini
Gpt
0.68
True
G-t-3.5-turbo
Gpt
0.30
17
Claude 3.5. Sommet(20241022
Claude
0.71
True
18
Claude 3.5 Haiku((0241022)
Claude
0.66
False
19
Claude
0.34
False
Table 11: Models analyzed from Helm
model
company
accuracy
writer/palmmra-x-000
writer
0.82
writer/palmmra-xx-33
writer
0.78
2
flake
0.66
Continued on next page
21

---

Correlated Errors in Large Language Models
Table 11: Models analyzed from Helm
model
company
accuracy
3
wwen//wwe225-772--insrtcee
qwen
0.83
4
qwenn/gwen2-72--nistnuc
qwen
0.83
gwen/qwenl..-1100aee 10bbcchh
qwen
0.77
6
qwen/qwenl.5-72b
qwen
0.77
7
gwen/qwenl.5-322
qwen
0.74
8
gwenkkywen2.5-77-nnnnnu-----
qwen
0.71
gwen/qwenl.5-14b
qWen
0.67
10
gwen/qwen1.5-7b
qwen
11
openailgn--4o-20022-88-06
openai
0.85
12
openaiigmi-4o-2024-0-111
openai
0.85
13
openai/gpt-4-0613
openai
0.84
14
0penalggs--turrbo-22440000999999
openai
0.82
15
openal/ggt441100-ppeeieww
openai
0.81
16
openailgnt-oomminn-22220---1111
openai
0.76
17
opcnai/gn-33.55uurbo-00113
openai
0.68
18
openaiggnt33.-tuubo-00333
openai
0.66
19
mistrlli/mistaallagg-22007
mistralai
0.81
20
mistralai/miixrral-82222
mistralai
0.77
21 mistrll//mixxtaa4427---222ss
mistralai
0.71
22 mistrdlai/mistral-smalll224244
mistralai
0.69
23
mistrala/miistallkkag----
mistralai
0.67
24 mistrhaloppenn-miitaaannnme---
mistralai
0.65
25 mistrla//miistral---nnnnnuueee
mistralai
0.59
26
mistrlla/mmistral-b--11..1
mistralai
0.56
27 microsotiphi---mmdiim--------ss-
microsoft
0.77
28 microsofi/phi----mall-------e
microsoft
0.76
29
microsoft/phi-2
microsoft
0.57
30
meta
0.85
31
0.81
32
0.81
33
mcta/llama-3-7001
meta
0.79
34
meta/llama-227001
meta
0.69
35
meta/llama-3-8h
meta
0.65
36
meta/llama-2-133
0.55
37
meta/llama-3.2-1
ib-visioon-instruc--ubboo
meta
0.54
38
meta
0.54
39
meta/llama-2-7b
meta
0.45
40
goggleggemini--.--po---0
google
0.86
41
googlelgemmiii--5-ppr--0011
gongle
0.83
42
google/eemiill5--ppppprvoeovvvve
google
0.81
43
Dooglefiext-umicco@@000
google
0.78
44
google/gemiini-15-1lss--00001111
google
0.78
45
google/eemiii--55llls-prvvvvvev
google
0.77
46
google/germma-22277
google
0.75
47
Bonglddgemini--.--fass----
google
0,74
48
google/gemma-2299
google
0.70
49
googlc/geminni-1O-ppo-000
google
0.70
50
googlefiext-bisom@0001
google
0.68
google/gemma-77h
google
0.65
52
deepseekai
0.73
53
databricks/dbrx-insttee
databricks
0.73
54
cohere/command--rplus.
cohere
0.69
55
cohere/commmad-rr
cohere
0.65
56
anthropic
0.88
57
matropicclaadd------oone------.2
anthropic
0.87
58
antropicclaudd---opu--22200000
anthropic
0.85
59 autinpeckccuule33ssmmee000000000
anthropic
0.76
60
a
anthropic
0.73
61
autinppiccaudde-HHniik--22222220
anthropic
0.73
Continued on next page
22

---

Correlated Errors in Large Language Models
Table 11: Models analyzed from Helm
model
company
accuracy
62
anthropiciclaudd--nssaa--- 1.2
anthropic
0.68
63
allenai/olmo-1.7776
allenai
0.53
64
allenai/olmo-7b
allenai
0.29
65
ai2 11fjamba-1.5-large
ai21
0.78
66
ai21/jamba-l.5 1.ppmmba-1.5-mppe
mini
ai21
0.69
67
ai2 innssaaaa-nntt
ai21
0.66
68
01-ai/yi-large-pree
0lai
0.80
69
01-ai/yi-34b
01ai
0.76
70
01-ai/yi-6b
01ai
0.63
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
-atae//alconn-1OB--BB 10B-Instruct
0.44
10.31
LlamaForCausalLM
tiuua//Flconn-- 1-B-BBase
0.42
10.31
LlamaForCausaiL..
tituacFalcon3-77B Instruct
0.41
7.46
LlamaForCausalLMM
0.34
7.27
FalcomMambaaFoCCausllll
4
Hiiuee/falcoo333BB-nnttuuuutt
0.30
3.23
Ll
theprine/ReWii2-00ee222..-...1
0.51
16.74
?
lenyxi.lamaa--TeeyxChha---00
0.52
70.55
LlamaForCausalLM
0.57
32.76
Qwen2ForCausalll
0,48
14.77
Qwen2ForCausal
Quaypaallo/Rommboo----TTEEE.
0.44
7.62
Foen2FoCCaullL.. L
10
0.44
7.62
Owen2ForCausall .M
11
shhenn--oom/mmicci-448---022
0.52
14.77
Qwen2ForCausall
12
smmiss/Qweon2.55558--nsmaa
0.52
94.65
Qwen2ForCausall .
13
spow spow12/ChatWaifu 22B v2.0 preview
0.40
22.25
Mistral
14
spow
Wawh v2.0 22B
0.38
22.25
MistralForCausall
speakteah//Bieiii-11B-.222.1
0.34
11.17
MistralForCausall
16
sometiincaaottionlLaaaaade.ll 1.B-..0 14B-v0.3
0.54
14.77
17
sometinesanotioo/laaaarkk 148-00.44-4-----------
0.54
14.77
18
sometinnsanntiio//aaaccc.ll wea-brrcsoomoeeel....e
0.54
14.77
ForeaForCa
19
somettinesanoiiom//veeee22.-..- Virmarckeeoo
0.54
14.77
Qwen2 ForCaaaalll
0.54
14.77
Qwen2ForCausall..
21
scchiiyer(Qween2.-777AAnnii
0.42
7.62
FornaForCa
22
squechos//lamaa311-70-P1tum
0.52
70.55
LlamaForCausia CausalI.M .M
23
sam pacch/Delliriummv
0.42
9.24
Gemma2ForCausalll
24
sam paapppecc/Daakest-uuee-l/
0.42
10.16
Gemma2ForCaussil
25
sam sam-paechhQuiill-v1
0.42
9.24
Gemma2ForCausall
26
0.59
72.71
Dwwn2ForCausall
27
0.59
32.76
Twen2ForCausalll M
28
0.54
14.77
Forn2ForCausalll
29
0.53
70.55
LlamaFor CoraFFaaaaaMMMM
30
0.50
14.77
Owen2ForCausie
31
thymes-ai/Aria
0.44
25.31
AriaForConditionall
Conditionallenneraaioo
32
ncoline/recolnne-eemma2222
0.42
10.16
Gemma2ForCausia
33
reoolnee/Gemmm-222Atureee Gemmasutra 9B-seer
0.42
10.16
Gemma2ForCausallM.
34
sooiime/rcoollme-eemmma- 2 9-.1.v0.1
0.42
10.16
Gemma2ForCausallM
35
rcoinne/rcailmme-emmm222--9--::2 2-99--00.2
0.41
10.16
Gemma2ForCauuse
36
0.40
10.16
Gemma2ForCausil
37
ging:220224Q0eeen2511114---mnmm
0.53
14.77
Qwen2ForCausall .M
38
gingyy2024/Qwwenn...MMaaa 14B-Instruett
0.52
14.00
Qwen2ForCausse
39
qingy20244/ussonn2
0.51
14.77
Dowen2FoCCausall
40
gingy20244/Fusionn-4BBBBB In-srrtuutt
0.50
14.00
Owen2ForCausalL
41
0.50
14.77
Qwen2ForCausall.M
42
gingy20199QQwan2.5-Mathh
-4B-ttuuttttt
0.53
14.00
Continued page
23

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
43
gingy20119/Qwen225-Martht 148-nnttruut-tA/phee
0.53
14.00
Qwen2ForCausall
44
177221199Qwen225-Ullimmaaa
0.49
14.77
Qwen2ForCausalLM,
45
paloalma/TW3-JRGLvv22
0.49
72.29
LlamaForCausall
46
pabolnnn/ECC--TW--JRGGGY
0.46
72.29
LlamaForCausalLMM
47
pdoalma/EECE/TW3--RGGLLV
0,42
68.98
LlamaForCausalLM
48
oxyapifoxy-I-small
0.50
14.77
Owen2ForCausalll
49
avddaaLLaama33J--Nemorro
0.49
70.55
JlamaForCausalL.
50
msten/franngwenstein-33555
0.56
34.71
Owen2ForCausall.a
51
0.45
7.62
Owen2ForCausal
52
nhyha/N3N Delirium-v1 1030 0227
0.41
10.16
Demma2ForCausiae MM
53
nhyha/N3N gemma-2-9b-it 20241029 1532
0.41
10.16
Gemma2ForCausalLM
54
0.40
10.16
Gemma2ForCaussil
55
new.bong//lonmee--..0000ee2222
0.45
7.62
Qwen2ForCausale
56
newsbang/Hoomer-7B--0)]]
0.45
7.62
Qwen2ForCausal
57
0.45
7.62
Qwen ForCasaCalll
58
newsbang/HHomer-7B-vv222
0.44
7.62
Owen2ForCausalL
59
new.baug/loommee=0550veeee22-2.
0.44
7.62
Qwen2For Ca
60
fbeetbowerllaamm3333kkkooooooonn
0.53
70.55
Llama Ja
61
0.49
14.77
62
(bertbowwe//Lanma3. D00
0,47
70.55
LlamaForCausalLMM
63
aberbbowee/Gemmm222-nnennnnrr -Dppeel-9B
0.41
9.24
Gemma2ForCausalLa
64
abeerbower/Miittaal--mllllll
Doppel-222
0.41
22.25
MistralForCausall..
65
moeru moeru-ai/L3.
0.39
13.67
MixtralForCausall
66
0.52
70.55
LlamaForCausia .M
67
mlahamne//iggQOeee225555555
0.55
52.27
Fornaallll M
68
0.47
47.39
Qwen2ForCausall
69
mladomme/ieermmss3-lamm-111----1
0.47
70.55
LlamaForCausall
70
0.38
8.03
LlamaForCausse
71
mitrall/Missall-angennssttu---
0.56
122.61
MistralForCausalLM.
72
0.41
22.05
FistralForCausalL.MM
73
microsoft/Phi-3--mciuu----
0.47
13.96
Phi3ForCausalLMM
74
microsot/hhi---mdiiu-----i---i
0,47
13.96
Por3ssCCa
75
micrsoff/PPhh33.5.MEE-iiisnut
0.47
42.00
Phi3ForCausalL.M
76
micosooffPhi--rmin--kkininnuuu
0.40
3.82
Phi3ForCausalL.M
77
micmosof/Phh33.5-miin-nnnmm
0.40
3.82
Phi3ForCausall..
78
microsof//Pii--mmiii-- - -nntsnrrttt
0.37
3.82
Phi3ForCausalLMM
79
meta- aetelllama/MMeta-Laan--33 -----mm---.--0--n---ttu
0.53
70.55
LlamaForCausall
80
meta- metalllana//Lann-3333-------- Instruct
0.53
70.55
LlamaForCausalLM
81
mothllamm//Mccae-laaa3-3------
0.52
70.55
LlamaForCausall/
metallama//cca--JJaa------7
0.47
70.55
LlamaForCausall
83
0.37
8.03
LlamaFoCaausal
84
0.40
3.82
Phi3ForCausalL.M
85
0.39
8.03
LlamaForCausalL.M
0.35
MistralForCausall
87
mattshumer/eet.70..3
0.53
70.55
LlamaForCausalL.M
88
0,44
10.16
2erma2Foo
89
0.44
10.16
Gemma2ForCausalL.
90
0.44
10.16
Gemma2ForCausall
91
0.43
10.16
Gemma2ForCausall..
92
0.43
10.16
Gemma2ForCauslll.n
93
0.37
8.03
Ll
94
|pacifico/Choocolatiee-III 14B-Instruut-44-DDOO
0.48
13.96
PhiSForCausalLM
95
ppcificoo/Choccolatneellte --B-Innstrctt---O--
0.47
13.96
Phi3ForCausalLM
96
0.39
3.82
Phi3ForCausalL.M
97
jeffnodoyyQpccn22.--7-------
0.46
7.62
Owen2ForCausall
98
jefinedoy(Q@een225--7--nnne
0.44
7.62
Owen2ForCausaee M
99
0.44
7.62
Owen2ForCausalll
100
jelimellyyfeffmellyy
0.44
7.62
Owen2ForCausall
101
0.44
7.62
Owen2ForCausalll
Continued on next page
24

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
102
0.36
22.25
MistralForCausalL..
103
inlemnnn/intermmmm2.. 5-20b-chat
0,40
19.86
MieemLM2ForCausii
104
nneriln/mmtemnnn2.5-7-hhhhhhh
0.37
7.74
105
informatiker/Owwenn--00
0.39
7.62
Owen2FooCausaaLMM
106
bului-ai/QQwQ-32B-CCmm Cower--Fsioen--Peeee-e- 9010
0,56
32.76
107
buihui-ai/Qwen2.5-
0.50
14.77
Dwen2ForCausalLMM
108
nuibui-ai/Qwen2.5-77B- -
0.42
7.62
PownsFooCausalL..
109
0.42
7.62
Owen2ForCausallA
110
hommalluser/Geemmaaaalaaa--2
0.47
27.23
Femma2ForCauslll
111
0.46
27.23
Gemma2ForCausil M
112
Dotnalluse/Cemmms22rrnoo07777
0.46
27.23
Gemma2ForCausllLMM
113
hounailuserGeeumma2nggnnn--77
0.46
27.23
Gemma2ForCausall
114 bounadissee/Qeeee...HHoeeeettrp
0.45
7.62
Qwen2ForCausile
115 google/gemmaa-2227-it
0.45
27.23
Ferma22orCausall
116 gmousoongeemmaa//bbbbbbbtttliild
TTES
0.43
9.24
Gemma2ForCausalll
117 glavec/ReeiicctoonJJaaa-3-----
0.63
69.50
LlamaForCausall
118 @huemos//Driicbbbaaa--LLmammm.m.
0.52
70.55
LlamaForCausalLM
119 freevheeiiffieeeeoo--vee//------
0.49
72.29
LlamaForCausalLM
120
Bamurnai/lLamma.111annnneeeess
0.48
70.55
LlamaForCausall
121 faammeni///Maabooo11.llminnaiii
0,47
70.55
LlamaForCausalLM
122 Bamnmenai/Maahooo-11--imsss
0.36
12.25
MistralForCausalll
123 hbggi//hheBccllee22eee-2322
0.59
32.76
124 fbgpr//ybbrrrrooo44444477---A
0,45
7.62
Owen2 ForCausal
125 hbgaldyyherttonn-44gg07-B--CCC
0.45
7.62
Qwen2ForCausmn
126
0.45
70.55
LJamaForCausee
127 experiment-llm/eexp--44-
0.43
7.62
Qwen2ForCausaa
128
chiisoofon/RRQ@ea--0...
0.52
14.77
Fwen2ForCausall
129
chnistofonn/RQQwen--0122
0.52
14.77
PoensForCausall
130
chinsofoon/Gennnna2----ittttttne
0.39
9.24
Gemma2ForCausalla.
131
chrrstofonnGGeemma---------
manae---aalall Ok-mmetaal.hhhhth
0.38
9.24
Femma2ForCausall..
132
christooon/HaapyyLLlaall
0.35
8.03
LaamaFor
133
dwi.kithecducc//gn--tty--nnooeo
0.52
14.77
Fwen2ForCausall
134
dwikiitheduck/gee--ttll
0.51
14.77
Qwen FreenaCCCuua
135
dwikkitheduck//gen-nnns--
0.51
14.77
Owen2ForCausse
136
0.54
77.97
Qwen2Foo
137
dankkn/RRYS-XL..n
0.54
77.97
Qwen2For
138
dnbkkn/RYSS-XLarge
0.54
77.97
Owen2ForCausalL..
139
0.52
81.68
LlamaForCausall
140
0.51
73.98
LlamaForCausall,
141
djuna(G2-BigGGSHT-2Z7
0.45
27.23
Genma2ForCausall..
142
djuna/Q2.5-Partron-77
0.43
7.61
Qwen2ForCausaa L
143
djuna/L3.1-Promissum
ManessuumMMM--81111 Mane-8B-Der Della--l.--alll
0.39
8.03
JlamaForCausall
144
Mane-8B-Della-ccall
0.38
8.03
LlamaForCausall
145
Jjuna/L331-FooSutIS
0.37
8.03
LlamaForCausall
146
dfurnan/Callneyyys-88-00op
0.70
77.97
Owen2ForCausalll
147
0.55
72.70
Owen2ForCausail
148
deepseek
ceepsock-aiddeeppseclllle
aiddeepsscc-llln-66--hhh
0.39
67.00
LlamaForCausalLMM
149
cstr/llama3.1-8---peeete
0.37
8.03
LlamaForCausall
1.50
0.55
72.00
Owen2ForCausall
1.51
0.45
13.96
MistralForCausall
152
cloudyu/Mixtral 34Bx2 MoE 60B
0.48
60.81
MixtralForCausall
153
byronnevrrso//MMisrraallamaalll AnnSmmmll-nsttuut-240---lllll
0.39
22.25
MistralForCausalL.M
154
byroneversso//i--...9g... abitttttrrtee
0.38
8.83
LlamaFor ComaaloCausalL.MM
155
bunnycoorsQQmmdorrr225555775
0.45
7.62
Qwen2ForCausall
156
bumnycooc(Qveen2.5---B-nimmeee
0.45
7.62
157
bumnycone/Cyherrcore-qqeee-1
0.44
7.62
158
Dumnycore/QanndorrEEpp-111
0.44
7.62
Owee2ForCausalll
159
0.44
7.62
Forn2ForCausall
160
0.45
3.00
Owen2ForCausalll
Continued on next page
25

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
161
0.45
7.62
Qwen2ForCausal
162
arcee-ai/Arcee-Nova
0.55
72.71
Foen2ForCausaLLMM
163
arcee-ai/Virtuoo-SSmie
0.52
14.77
Qwen2ForCausall
164
arce-ai/SuperNovv-Medius
0.50
14.77
OweenFFoCausalll
165
arcce-ai/Llama-3//
0.39
8.03
LlamaFor CaarrForCauaaMM
166
arcee aice-aai/llama-Spatt
0.37
8.03
LlamaForCausalLM
167
authaaciee-orgmmggnu---------
0.55
72.71
Qwen2ForCausall
168
anthaaclee-orrmmagauu---222211
0.55
72.71
Qwen2ForCausalla.
169
unthracile-rgg/magnnum---33---
0.48
34.39
LlamaForCausall.a
170
anthasclee-orrgmmggnuu---+2-7
0.44
27.23
Demma2ForCausse M
171
0.42
27.23
Gemma2ForCausalLM
172
appncale/WizzdddM222222222
0.46
140.62
MixtralForCausall
173
alluz-ory/M6-MMeadovllla222222
0.38
22.25
LM
174
0.54
14.77
Qwen2 ForCausal
175
0.54
14.77
Qwen2 ForCausall M
176
alkenoviinrogggo//vvvnenoo----g
0.54
14.77
FornnFooCa
177
0.54
14.77
Owen2ForCausaal
178
0.54
14.77
Owen2ForCausalll
179
allemi//Jlmmaa3111100--3-7788
0.47
70.55
CamaaooCCausalla
180
allmaa//lamma3311-uut---3------
0,46
70.00
LlamaForCausalL.
181
allemi/LLammaa33--1Uu:33777-6
0.46
70.55
182
allemi//lammm3311-Tut---------
0.29
8.00
183
ajjidaa532441Laaaa333SSoooo
0.38
8.03
JamaaooCausalla
184
0.48
70.55
JlamaForCausall
185
0.55
72.71
DwwnaForCl M
186
ahacnsaiSmmaung--Onnn2--------
0.52
72.71
Qwen2ForCausall
187
abccusi/Smmang--Lnnn3--7000 Instru-t-ttttt322K
0.48
70.55
LlamaForCausall
188
abacusai/Smnugg-2B--111
0.46
72.29
Llama JamaaorCausalLMM
189
aadiyy//Lamma3OQqelBBi1LLMMM
0.49
70.00
LlamaForCausalL.M
190
0.45
7.62
Qwen
191
ZemoXClemm/wwenn25--7B HomerCreative-Ne
0.44
7.62
Owen2ForCausall
192
Z HomerAnvita-Noor
0,44
7.62
Owen2ForCausall
193
Zero PeeoXCCllmmQveee222-Aeettteeeee
0.43
7.62
Qwen2ForCausee
194
Weyax//Baggl--eeeess34BB----
0.47
34.39
LlamaForCausall..
195
Vaiiamt.abb//llma3. hhr.:aammn..--------i--ignnnn Valiant2 Vallinn
0.52
70.55
LlamaForCausall.
196
VAGOOsolatioons/Laaaa33.11 SrOOoouttoosLLaaaaa31-uuuucuuut M 70b-Instuuct
0.53
70.55
LlamaForCausall
197
0.52
14.77
Owen2ForCausalll
198
(DDDsuuttoo//SSucccccuuLL1111110 14b-DPO
0.51
14.77
ForCaaaCC
199
0.47
13.96
HistralForCausail M
200
VAGOoolationn/LLama33115
S-aaaerkrutL.M-----naamta MM88--Instruet
0.39
8.03
LlamaForCaure
201
Undi95/MG-FinalMiix2222
0.54
72.71
Owen2ForCaussee L
202
Tsunami-th/Tsunan -tt.:////TTaaam---11-44....
1.0-14B-Instruue
0.52
14.77
Owen2ForCausalll
203
0.45
7.62
Free2ForCausall
204
Tsunami-th/Tsuaanee 1.0-7B-Instrur
0.44
7.62
Qwen2ForCausall
205
moimami-th/Tsunai---0
0.44
7.62
Qwen2ForCaussi
206
The wwwp///enn-capppmsssl..ooooooo
0.52
14.77
Qwen2 ForCausa
207
Taar 10wpa///a/aaaaaaacsscsoooo00000
0.51
14.77
FornsFFMCa
208
The 129%/qeen-cappmmssseeee0.
0.51
14.77
Qwen2 ForCaaaClll
209
Tse heeeaaar///oooveeeeaaaaasseeeeee
0.51
14.77
Qwen2ForCausall
210
Tae 12
0.51
14.77
Qwen2ForCausall
211
The-DrumnerrCyddoia-222-11111
0.41
22.25
MistralForCausall
212
T145ILLama-331-SB-IInnnnuuuulee
0.39
8.03
ClauaForCausalLMM
213
1145/ZEUS-BB-Y V2L2
0.39
8.03
LlamaForCausian
214
T145/ZEUS-8B-V
0.38
8.03
LlanaForCausall
215
T145/ZEUS-8B-
0.38
8.03
LlamaForCausal
216
T145/ZEUS -U55/EUS--B-V4
0.38
8.03
LlamaForCausall
217
0.39
3.82
218
0.37
12.25
HistralForCausalll
219
SicamusSiaaniisutiiiweees....
0.53
14.00
OvvenForCausalli
Continued on next page
26

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
220
Uncensored Sinyyah22100Umm-444---nnntttttt Uncensored
0.53
14.00
Qwen2ForCausallMM
221
StariusSiarriSSul/QQeea2----- Unceesooed In...
0.51
14.77
Qwen2ForCausalLMM
222
0.39
3.82
223
0.46
27.23
Gemma2ForCaualLMM
224
0.37
8.03
LlamaForCausalLM
225
Qwcn/Qwen2.5-72B
0.60
72.71
Qwen2ForCausalLM.
226
Qwen/Qwen2.55422B
0.58
32.76
Owen2ForCausalLL
227
Qwen/Qwen2-72BB
0.57
72.71
Qwen2ForCausall.M
228
QwenlQwwe2-VV--222Bmmeee
0.57
73.41
229
Wven/QQwQ:32B-Preevie
0.57
32.76
230
Qwen/Qwen2.5-322B Instruct
0.57
32.76
Qwen2ForCausalLM.
231
QwennQwenn.55-22BBBm
0.56
72.71
Owen2ForCausaan
232
Qwenl(Qwen-772B-nnsttee
0.54
72.71
More2ForCausalll
233
Qwen/Qwen1.5-11000
0.54
111.21
Fore2ForCausalll
234
Qwenl(ween.55-Coder:222
0.53
32.76
M
235
Qwen/Owen2.5-144B
0.52
14.77
Owen2ForCausall..
236
QwenIQwen2.5-144--Inse
0.49
14.77
Owen2ForCaussll
237
Dwen/Qwenl.5--110--54
0.48
111.21
Qwen2ForCausalL..
238
QvenlQween.5-MMMth222B----
0.48
72.71
Qwen2ForCaussnn
239
Qwen/Qwen2-557B-A1
Insttcntttuct
0,46
57.41
Qwen2MoeFFoCausall..
240
Qwen/Qwenl.5-322
0.45
32.51
Owen2ForCausalL
241
Qwen/Qweel.5-332B-
0.45
32.51
Down2ForCausallll
242
0,44
32.76
243
Dwemm/Qwen255-77--nnrrrurt
0.43
7.62
Owen .M
244
Qwan(Qwen2-Mathh-22--imns
0.43
72.71
DwwnaForCa
245
Quem/OOwen2-VI--7BBnnnnne
0.41
8.29
Qwen2V1 Owwn2VL.ForCondlee
246
Qwen//Qwen2.5-Codeertee
0.39
14.77
Forn2 Fo
247
Qwenn(wwen-77B-Instruct
0.38
7.62
Qwen2For Fere2Forra
248
Qwenn/Owenn25--Caaeer-------
0.34
7.62
249
QwwnmQQwen2.5-33--niitre
0.33
3.00
Owen2ForCausalll LM
250
Orion Drionn2hen/Qwwc22555--8
Uncensored
0.44
7.62
Fwen2ForCausalLM
251
0.38
8.03
LlamaForCausalLMM
252
0.38
8.03
LlamaForCausall.
253
0.53
70.55
LlamaForCausalLMM
254
0.52
70.55
LlamaForCausalL
255
0.51
70.55
LlamaForCausalL.M
256
0.48
70.55
LlamaForCausalL.,
257
0.48
14.77
LlamaForCausalLM
258
0.47
70.55
LlamaForCausall
259
Nchobby\MS-Scchssndnr----
0.41
22.25
MistralForCausall
260
NhhobyyAS--Sciisddna-2---00
0.41
22.25
Mistral MistralForCausall M
261
WLLPatk/hiiCC--Roinnneets.A-
0.51
70.55
LlamaForCausall
262
0.47
14.77
LlamaForCausalLMM
263
NL
T..aak/AnnFeng.333--vvee
0.44
34.39
Llama
264
NAPS AAPS-ai/napx-llama-... hit.:lammmm---------stttt-----
0.35
8.03
LlamaForCausaa
265
M 788
0.73
77.97
Owen2ForCauseee
266
M
0.72
77.97
Qwen Fwen2FoCCa
267 MariyarPammhhiccllme222444422
0.70
77.97
Qwen2 ForCausal.MM
268
MariyyrPemmnicccmmm2222-eeeeee.
0.56
72.70
Foen2ForCausall
269
MariyaaPaaahiccllee22---vveeee/
0.56
72.70
ForCaaara
0.45
7.62
Owen2ForCaussin
271
Lil
0.45
7.62
FornsFFrCa
272
0.54
14.77
Owen2ForCaussall
273
0.41
7.82
ExaoneFor
274
0.33
2.40
FxaoncForCausalL
275
0.44
7.62
Qwen2
276
0.53
70.55
LlamaForCausalLMM
277
KSL-INW.SSCLLLL333-OOSSAAALPLLL
0.52
70.55
LlamaForCausall
278
0.52
70.55
JlamaForCausall
Continued on next page
27

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
279
0.52
70.55
LlamaForCausaIL.M
280
Jumboee/Qwen-Meegnnin
0.42
15.23
?
0.39
8.03
LlamaForCausalLM
282
0.43
7.62
Qwen2ForCausallM..
283
Isaak-Carter/ Cutterr//ossccccioo./2......
Instrruct--ab....
0,41
7.62
Qwen2ForCausal
284
0,41
19.86
LlamaForCausalLMM
285
HunnanLLLyyHHHuaaiiiiQQeeeee222- Instrrcttt
0.44
7.62
Qwen2ForCausall
286
0.46
140.62
MixtralForCauuall...
287
IPAABBSCTJlmma33AAleeeBeeea
0.36
8.03
LlamaForCausall
288
GuntlhonuCGemmms----MMerre
0.39
10.16
Gemma2ForCausal M
289
Gryphe/Panthhen--RPPPee---22222
0.39
22.25
MistralForCausall
290
0.39
9.24
Gemma2ForCausall
291
Goekdeniz Goekdeniz- Quweiane///seeeiicieeeeee222..5
0.50
14.77
Owen2ForCausiee
292
0.41
7.62
Qwen2ForCaussi L
293
Goekdeniz- Gookkeeniz-Gnnelmezjjjaii
0.40
7.62
Qwen2ForCausal
294
-iiistrrutt
0.43
9.24
Gemma2ForCausalLM
295
Etherll/Qwenn.5-7BB--ee
0.44
7.62
Qwen2ForCausan
296
Etherl//Superllermes
0.39
8.03
LlamaForCausall
297
Cheel//ferppee--LMMM4.laaaa-
-ie-----
0.38
8.03
LlamaForCausian
298
Shhertl/Oween255-Cooeer- --n-ooerrt7--nntuutt---iiii-
0.35
7.62
Qwen2ForCausalLM
299
EpistemeAlD/FirrballlPPhii m-edim------i-----------
0.46
13.96
MistralForCausall
300
EVA
UNIT-OL/EVA-QWeen2..
0.58
72.71
Swen2FooCausalll
301
DreadPoor/Emu.BB
Model_Stock
0.42
9.24
Gemma2ForCausall
302
0.39
8.03
LlamaForCausan Causall. .M
303
0.39
8.03
LlamaForCausall.
304
DreadPoo//BBe2eetBBLLLNNNNA
0.39
8.03
LamaaFoCauusall
305
DreadPourProomissum
0.38
8.03
LlamaForCausalLMM
306
DeepMoun(W/LLaana333135 -
0.39
8.03
LlamaForCausal
307
DegpMouuntO/LLanne-3-----tt
0.39
8.03
LlamaForCausalLMM
308
0.38
8.03
JamasaCCa
309
DecpAutoAV/kdmmssuu http://mmmmmmupppmeeeee
-
0.39
8.03
LlamaForaausall
310
DecpAutoAl//dm.soup Llama-3.1-88
0.39
8.03
LlamaForCausalla
311
Aal.auuoAAA/dddmmsuuulaaaaaaaaaa 1-8B-Inst
0.39
8.03
LlamaForCausalla
312
0.39
8.03
LlamaForCausalL.M
313
DeepAutoA//Explore.... 1atpa//////Aoloeeaaaaaaa-----1-- L.ama-3.1-8--mm
0.38
8.03
LlamaFoCCausalL.M
314
Danielbruz/Barreens--11 1eb--hi----PP------
0.47
13.96
MistralForCausall
315
Dauppficchen/Llamaa.//
0.38
8.03
Llama L.
316
Culri\/QQeen...114BB Wernicke
0.54
14.77
Qwen ForCausalL
317
Culmn\/Owestton-1148
0.54
14.77
Qwen ForCasall .M
318
0.54
14.77
Owwn2ForCausse
319
Culmn\/SeQQwecee-144B
14B-BvolMerge
0.54
14.77
2 ForCassal L
320
Culti\//SeQweeee-14B
0.54
14.77
Owen2ForCausee
321
Cran-May/T.E-8.1
0.44
7.62
Fwen2ForCausall
322
0.57
32.76
Owen2For
323
0.50
14.77
Qwen2ForCausall
324
CombirHHotzzoohhhiiiiaaaaiicccce 2-2222222w-..... --.....
0.49
14.77
FornnFor
325
Combiitforiien///SSSMbbooooos --44-SLLRPPP SLERP
0.47
34.39
LlamaForCausaILM
326
0.43
7.62
Owen2ForCausal
327
CohereForrAllayeexxpanss-:3b
0.41
32.30
CohereForCausalll LM
328
ColtreforrAiccchcccmmmnnd----p
0.40
103.81
CohereFor
329
Clantiolalliintliiecnee-oo-oo--d
0.42
7.62
ForenFooCaauaaLL.
330
BoltMonkey//DreaMMixx
0.38
8.03
LlamaForCausall
331
0.37
8.03
332
0.35
8.03
LlamalorCausall
333
Blac-Becmice/cooo-Phhiii11-----
0.46
13.96
Phi3ForCausalL..
334
0.56
72.70
ForCaaFora
335
0.46
70.55
LlamaForCausalLMM
336
0.46
70.55
LlamaForCausall
337
0.41
8.83
JlamaForCausalll
Continued on next page
28

---

Correlated Errors in Large Language Models
Table 12: Models analyzed from HuggingFace
accuracy
params (B)
architecture
338
Azure999loossom-55333)
0.42
32.51
Qwen2ForCausallM
339
Auralnddustries/Auuz-888
0.39
8.03
LlamaForCausalL.M
340
Aashrra99SSCrraaiiee---neeed
0.45
7.62
Qwen2ForCausalLM
341
Austrra1995/QQwee--EEvo--7
0.45
7.62
Owee2FooCausaall
342
BELLMMifommm-2-ccriiinnnintt
0.39
9.24
Formma2ForCusaall LM
343
AELLLWigomma-22yyy-nnnitt--
0.38
10.16
Gemma2ForCausall
344
AALFFFusseChttLLaaa--11888
priv
0.37
8.03
LlamaForCausalLMM
345
AAL.F/FusSCha--Jaaaa-----3
0.37
8.03
LlamaForCausall.a
346
01-ai/YY-1.5-34B-22K
0.47
34.39
alasaForCausall
347
01-ai/Yi115-33BB-hha
0.45
34.39
LlamaForCausal M
348
01-ai/Yi-1.5-9B-CII
0.40
8.83
LlamaForCausalL.
29


Reviews:
[{"review_id": "hFngMKT65U", "reviewer": "Reviewer_5BGN", "summary": "The paper studies agreement of LLMs on samples they make mistakes on, showing model pairs have well above chance error correlation. It shows that models from the same developer, and of higher accuracies, make more correlated errors. It then measures effects on candidates when LLMs are used to score CVs, simulating various scenarios such as using the same LLM, latest LLM, same company etc, showing how systematic exclusion rates can differ in trends from average applicant ranks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We’re glad that you found our claims well substantiated. We appreciated the questions you raised, which we aim to address below:\n\n\n\n\n**“How would figure 2 look if we replaced qwen 2.5 72b with other llms?”** Thanks for this question. We extended this analysis to other models (the top performing model of each architecture/company in HuggingFace/Helm), and find that same high-level result holds for the high-performing models (which are most likely to be adopted as a judge). An interesting subtlety is that low-performing models tend to underinflate model performance of better-performing models. We will describe these additional experiments in the main text and add the experiments to the appendix. \n\nWe plot the amount of accuracy inflation across different judges in the following anonymous doc (“Extension of Figure 2: LLM as judge”: https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub)\n\n**“what architectures are considered in your data, and how many models of each type is available?”** We considered 13 architectures. We will make note of this in the main text.\n\nQwen2ForCausalLM (156), LlamaForCausalLM (108), Gemma2ForCausalLM (35), MistralForCausalLM (20), Phi3ForCausalLM (12), MixtralForCausalLM (4), ExaoneForCausalLM (2), CohereForCausalLM (2), InternLM2ForCausalLM (2), Qwen2VLForConditionalGeneration (2), FalconMambaForCausalLM (1), AriaForConditionalGeneration (1), Qwen2MoeForCausalLM (1) \n\n**Alternate metric of correlation on errors proposed in parallel work.** Thank you for sharing the reference to the parallel work. We enjoyed reading the paper (first made public after the ICML deadline). The paper develops a novel metric for measuring similarity that adjusts for model accuracy, considers different wrong predictions as a disagreement, and uses the probability distribution over answer choices. So while our primary similarity metric (do models agree on an answer when both err) satisfies the first two criteria, it does not incorporate model probabilities or information about which questions the models erred on. Both metrics account for the fact that more accurate models are more correlated simply because they agree on correct answers. (We also note that the papers focus on distinct downstream outcomes, with our focus being on the algorithmic monoculture and hiring literature.) We will also add a discussion of the parallel work in our related work, as well as of the limitations of our metric. \n\n**Incorporating measures of job candidate quality.** You noted that homogeneity may be desirable in the hiring setting (e.g., if they all accurately identify the best candidates). This is a great point. While there is significant literature in algorithmic monoculture that focuses on measures like systemic exclusion that do not consider candidate quality, there are other works that do. To this end, we hand labeled 450 resume-job pairs in the dataset as “ground truth” for the resume’s quality. These labels allow us to assess the quality of matched applicants in stable matching. The experiment demonstrates that adopting the latest LLMs (that are more homogeneous and more accurate) improves the overall quality of selected candidates (higher quality candidates are more likely to be matched), illustrating tensions between different desiderata in the hiring process. This plot is given in the bottom of the anonymous doc (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub))\n\n**Improvements to Figures** Thank you for the comments on figure clarity. In particular, we agree that Figure 1 (the heatmap of error correlation) is not current very parsable. We’ve created a new version, in which models are ordered by their accuracy. This makes a primary finding clear: that more accurate models are more correlated when they err. We’ve also made some edits to Figure 2 and Table 1. For Figure 2, we directly plot the **accuracy inflation** to convey the main point: that judges can inflate the accuracy due to correlated errors. (We do this for the set of models described above.) For Table 1, we incorporated the analogs for Helm and Resumes to increase the informativeness. These updated versions are given in the anonymous doc (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub)\n\nThank you also for pointing us to the LLM evaluation paper, which we will add to our related work.", "claims_and_evidence": "Most claims are well substantiated. I have only two issues:\n\n1. How would figure 2 look if we replaced qwen 2.5 72b with other llms? The claim of models over-rating other model outputs currently seems cherry picked. It just shows that qwen is maybe a liberal judge (perhaps also of human responses).\n\n2. For the result showing models with the same architecture are more likely to agree with each other than chance, what architectures are considered in your data, and how many models of each type is available?", "methods_and_evaluation_criteria": "Yes, more or less. I appreciate the use of HELM for error correlation (wide task coverage) and also resume datasets. However, one main concern is that measuring error correlation by counting agreement when both models make an error leaves out information about whether models also make errors on similar samples. Such information is accounted for in the model similarity metric proposed in parallel work [1], so it may be useful to use that instead. \n\n[1] Great Models Think Alike and this Undermines AI Oversight, Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, Jonas Geiping, ArXiv 2025.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, I carefully checked experiment design and have two main concerns:\n\n1. Since scores are used to measure LLM correlation in the hiring setting instead of errors, it's unclear whether the observed results are undesirable or not. In particular, one might hope that different human reviewers too would arrive at similar scores for candidates, and the LLMs would provide similar scores to humans (leading to capable LLMs having similar scores). Overall, it's unclear what to take away from the resume/market analysis section. \n\n2. Figure 1 is a bit hard to process. Are there any alternative ways of visualisation that can convey the conclusions?", "relation_to_broader_scientific_literature": "Prior work has shown that models prefer their own outputs (which explains some of the within-family results shown in this paper). Parallel work [1] shows very similar results: models are getting similar with increasing capabilities, and LLM judges favour more similar models. It may be useful to use their similarity metric, as it takes into account what samples models make errors on, likelihoods provided by the models etc.", "essential_references_not_discussed": "LLM Evaluators Recognize and Favor Their Own Generations, Arjun Panickssery, Samuel R. Bowman, Shi Feng, (NeurIPS 2024) -- paper showing models favour their own generations should probably be cited in Section 2.", "other_strengths_and_weaknesses": "**Strengths**: \n\n1. The paper shows that different models have correlated errors on popular leaderboards.\n\n2. The paper demonstrates effects on a downstream application of LLMs in hiring under realistic modelling assumptions.\n\n**Weaknesses**:\n\nHighlighted above. Particularly, it's unclear why the metric used is optimal (there is very little discussion around this), and what to take away from the downstream study where score correlation is measured instead of error correlation. Further, Figure 1, 2 and Table 1 can be improved for both clarity, presentation, and informativeness.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7aDWbyYmN", "reviewer": "Reviewer_eRHz", "summary": "This paper studies how correlated the mistakes of large language models (LLMs) are across two public leaderboards. This study considers a large range of LLMs and a large set of questions. The findings are that there is substantial correlation in model errors. A regression analysis suggests that high agreement is correlated with two models being from the same company, having the same base architecture, and having a similar number of parameter sizes. They show an implication of these results: under the LLM-as-judge paradigm, LLM accuracy is inflated by about 7%. The paper concludes with a case study where a labor market is simulated wherein firms use LLMs to screen job applications. The authors find similar error correlation here, and the implications are larger-than-random levels of systemic exclusion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and detailed review. We are happy to see that you found the paper to be a significant contribution to the LLM behavior and monoculture literature, and that you found the analyses to be interesting and thorough.\n\nWe now discuss several of the points you raise, which we sought to address:\n\n**Generalizability beyond MMLU / limited # of models on hiring task.** We extended the resumes analysis to consider 20 models, adding: Meta models (Llama 3 70B Instruct, Llama 3.3 70B Instruct); Mistral AI models (Mistral Large 24.02, Mistral 7B Instruct); Amazon models (Nova Pro, Nova Macro, Nova Lite); Anthropic models (Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 Haiku); and OpenAI models (GPT-4o-mini, GPT-3.5-turbo, o1-mini) to our experiments.\n\nWe also extended the regression analysis on this updated resumes dataset. We hand labeled 450 resume-job description pairs which serve as ground truth. We can then measure the correlation among models in their residuals (subtracting out ground truth label from model evaluations). We find that the same results hold (more accurate models and models from the same company are more correlated in their errors). This is shown in Table 1 in the following anonymized link (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub). \n\n**External validity of systemic exclusion.** Systematic exclusion in our hiring experiments may not be an issue if some candidates are unqualified. Using the hand labels, we computed an additional metric in our stable matching experiments: the probability that an applicant with a given human label is matched. Adopting the latest (most accurate/homogeneous) models results in improved quality of matched applicants, illustrating tensions between different desiderata. (See the bottom of the doc linked above for this plot.)\n\n**More sophisticated models of the relationship between accuracy and homogeneity.** We agree that it would be interesting to consider nonlinear models, and will explore that in the future. \n\n**Extension to overall agreement.** Thanks for this suggestion! We’ll include this analysis in the appendix – overall agreement as well as agreement when either model errs. The high level findings remain the same with these metrics. (We do also find that google/text-unicorn and writer/palmyra-x-v3 in fact agree overall as well—they basically have the same answers everywhere.) Table 2 in this anonymous link (under \"Extensions of Table 1\") gives the main analyses for overall agreement (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub)\n\nThanks also for the formatting notes and suggestions (on the regression tables and footnote 3). We’ll make those changes.", "claims_and_evidence": "The papers claims are generally well-supported. The authors provide extensive analysis across a large number of models to demonstrate the main point: a high degree of error correlation in LLMs. The result in Figure 2 is particularly clever and compelling. The finding that pairs of models with higher individual accuracy have more correlated errors is interesting, surprising, and could benefit from further exploration (see \"Other Strengths and Weaknesses\").", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally sound. The labor market analysis seems especially careful. Moreover focusing on conditioning on incorrect answers makes sense and is intuitive. The main weakness is that the main analysis is limited to MMLU questions, which represent a narrow slice of possible LLM applications (see \"Other Strengths and Weaknesses\" for more discussion).", "theoretical_claims": "The paper doesn't make substantial theoretical claims requiring proofs.", "experimental_designs_or_analyses": "The main analysis and the labor market case study are both sound and carefully executed. The only comment I had was regarding the regression analysis (see \"Other Strengths and Weaknesses\").", "relation_to_broader_scientific_literature": "The paper makes a substantial contribution to our understanding of LLM behavior and the effects of monoculture. I would say its main contribution is taking analysis that is generally theoretical and providing substantial empirical evidence.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- The result in Figure 2 is a very nice and clever analysis. It is convincing of the real-world implication of the findings of the paper -- that correlated errors in LLMs means they can't be used in the LLM-as-judge paradigm, to rate the performance of other LLMs.\n- It's a really interesting result that the two seemingly unrelated models -- google/text-unicorn and writer/palmyra-x-v3 are almost perfectly correlated when they make errors. \n- The result that pairs of models that are more accurate individually also have more correlated errors is interesting and surprising. It would've been nice to explore this more. For example, one possibility this brings up is that we're not controlling sufficiently well enough for variables. Have you considered running more sophisticated methods for measuring these effects, e.g. using nonlinear outcome models (like in the double ML literature) to estimate these effects?\n- The model evaluation is extensive across models. However, all the questions in the main analysis are limited to MMLU questions (either on HuggingFace or Helm). These multiple choice exam-like questions are a very narrow slice of the possible questions and benchmarks we may ask of LLMs, and so correlated errors on this dataset may not extend to other uses of LLMs. It's interesting that in the second half of the paper the resume setting is considered -- which is indeed quite different from MMLU -- but this analysis is more limited, considering only 8 models.\n- The reason for conditioning on both model answers to be incorrect makes sense and is clear to me. Still it would be nice to have overall agreement statistics (perhaps in the appendix) for comparison. For example, it would be interesting to see in the google/text-unicorn and writer/palmyra-x-v3 whether the overall agreement rate extends to whether each question is answered correctly or incorrectly. \n- I liked the case study of LLMs in labor markets. One minor comment I had was the external validity of studying systemic exclusion here was unclear. I fully understand that systemic exclusion is problematic when qualified candidates are being screened out due to monoculture, but the way that resumes/jobs are sampled in this simulation seem to allow the possibility that some candidates aren't well-suited for the chosen job in which case having higher-than-random systemic exclusion rates is not a problem.", "other_comments_or_suggestions": "- The OLS tables in the appendix should be formatted more nicely (e.g. we don't need to know the date they were executed)\n- Footnote 3 (line 218, left column) cuts off early.", "questions_for_authors": "No questions outside of the \"Other Strengths And Weaknesses\" section\n\n=====================\n\nPOST-REBUTTAL UPDATE\n\n=====================\nAfter reading the rebuttal I will main my original (positive) review of the paper", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "com24417D0", "reviewer": "Reviewer_kVi8", "summary": "This paper investigates the correlated errors of different LLMs, which I find interesting and novel. The authors conduct extensive experiments analyzing the performance of different LLMs, revealing a high correlation in model errors. The experimental results largely support the core claims of the paper, and the study concludes with a discussion on the potential implications of this correlation for multi-model collaboration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and comments. We’re glad that you found the experiments to be extensive, and supportive of our primary claims, and that you found the paper “interesting” and “highly relevant to the existing LLM research.”\n\nWe appreciate the comments you give, which we discuss below.\n\n**Regarding the causes of error correlation:** One way that we tried to understand this is through the regression analysis, in which we determined that shared model architecture and developer were predictive of higher error correlation. We also found that more accurate models tend to be correlated. (Section 2.2) We agree, however, that underlying theoretical explanation is an interesting direction for future work.\n\n**Regarding generalizability of downstream tasks:** We focus on the hiring setting since it has been a focus of literature in the algorithmic monoculture literature (e.g., [1-4]). One other downstream task we consider in the paper is in the LLM as a judge setting. We have extended our experiments on this task, finding that the homogeneity of models can inflate scores (especially of models with shared components) for models that are less accurate than the judge. See the anonymized link below for plots showing the amount by which model accuracies are inflated (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub) \n\n**Regarding the theoretical insights of our work:** While we agree that our analysis is empirical, it is motivated directly by theoretical work:\nFirst, it tests the “component-sharing hypothesis” (i.e., that models that share components are more correlated) proposed by [1]. Our experiments support this hypothesis. However, it adds nuance to the hypothesis by showing that having independent components does not ensure independence: in fact, more accurate models (regardless of shared of components) tend to be more homogeneous.\nSecond, it test hypotheses of theoretical work in algorithmic monoculture, which has identified potential downstream outcomes of more homogeneous models (especially in the labor market settings) (e.g., [4]). Our results test these predictions empirically.\n\nWe very much appreciate your comments, and will make sure to highlight the connections of our work to existing theory.\n\n[1] Bommasani et al. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? NeurIPS (2022).\n[2] Kleinberg and Raghavan. Algorithmic Monoculture and Social Welfare. PNAS (2021).\n[3] Creel and Hellman. The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision-Making Systems. Canadian Journal of Philosophy (2022).\n[4] Peng and Garg. Monoculture in Matching Markets. NeurIPS (2024).", "claims_and_evidence": "This paper empirically reveals a high correlation between model errors. The experiments cover a variety of tasks, and the experimental setup is reasonable, supporting the main claims of the paper. However, a deeper analysis of the specific causes of error correlation is somewhat lacking.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "In my opinion, this paper is primarily an experimental analysis, with relatively insufficient in-depth theoretical analysis.", "experimental_designs_or_analyses": "The experimental setup and analysis in this paper are reasonable. The experiments cover multiple tasks, the evaluation metrics are appropriate, and the complete experimental results are provided in the appendix. However, the downstream tasks are only evaluated on the labor markets task, which may limit its generalizability.", "relation_to_broader_scientific_literature": "This paper investigates the correlated errors among multiple LLMs, providing new insights into multi-LLM collaboration and evaluation. I find the paper interesting and believe it is highly relevant to the existing LLM research.", "essential_references_not_discussed": "In my understanding, there are none.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Is there any deeper theoretical insight regarding the correlated errors between LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7yagx5RbT2", "reviewer": "Reviewer_jJmX", "summary": "This paper investigates the extent of correlation across large language models (LLMs) and its implications for systemic bias and multi-model collaboration. Key factors influencing this correlation include shared base model architecture and development organization. The study highlights that larger, more accurate models are highly correlated, even when both are incorrect. The findings emphasize the risks of \"algorithmic monoculture\" and its impact on decision-making.\n\n## update after rebuttal\n\nThanks for your repsonse about my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive review. Given that you didn’t have any questions, we will be very brief in our response. \n\nOne thing you point out is that leaderboard metrics do not fully represent real-world performance. We appreciate this comment, and will make note of it more clearly in the main text. This was also one of our motivations for including the resume / hiring analysis in the paper. To more directly show that our main findings generalize beyond MMLU tasks, we’ve extended the regression analysis on the Resumes dataset. We hand labeled 450 resume-job description pairs which serve as ground truth. We can then measure the correlation among models in their residuals (subtracting out ground truth label from model evaluations). We find that the same results hold (more accurate models and models from the same company are more correlated in their errors). This is shown in Table 1 in the following anonymized link (https://docs.google.com/document/d/e/2PACX-1vSlxmq6qMRG55uCdpwSuNqt9PbrEvi2hi2MEIQU-oZY8bw17lBo2W7Z8NFhKeDv4s23ZT42zJrStEeP/pub).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not contain explicit formal mathematical proofs for the theoretical claims but relies on empirical analyses to substantiate its conclusions.", "experimental_designs_or_analyses": "The authors analyze model correlation by comparing the responses of 170 LLMs on 12,032 questions from the HuggingFace leaderboard and 71 models on 14,042 questions from the Stanford Helm leaderboard, aiming to calculate question-level agreement between pairs of models, particularly focusing on cases where both models are wrong. However, leaderboard metrics might not fully represent real-world performance, as they may be models finetuned for specific tasks, and the question sets may not reflect the distribution of tasks models will face in other applications.", "relation_to_broader_scientific_literature": "This paper highlights that using correlated models as \"judges\" leads to inflated performance estimates and potentially incorrect rankings. The authors show that accuracy inflation occurs when one model is used to proxy ground truth for other models, which can skew rankings and lead to overestimation of model quality.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies the correlated errors of different LLMs across a large range of LLMs and a large set of questions. The studied problem is interesting and novel. Also, some findings are also insightful and interesting, such as google/text-unicorn and writer/palmyra-x-v3 are almost perfectly correlated when they make errors. The evaluation is also extensive across different models. Therefore, I recommend to accept this paper. I also recommend the authors to include the additional results in the final revision.
Decision: Accept (poster)