Great Models Think Alike and this Undermines AI Oversight
Authors: Shashwat Goel
Joschka Strüber
Ilze Amanda Auzina
Karuna K Chandra
Ponnurangam Kumaraguru
Douwe Kiela
Ameya Prabhu
Matthias Bethge
Jonas Geiping
Venue: ICML 2025 spotlightposter (2025)
Keywords: Model Similarity
AI Oversight
LLM as a Judge
Weak to Strong Generalization
Model Differences
Primary area: deep_learning->large_language_models
TL;DR: We propose a measure for model similarity, finding increasingly correlated failures as model capabilities improve, and show the negative effects of similarity on AI oversight paradigms like LLM-as-a-Judge and Weak-to-Strong Generalization.

Abstract: As Language Model (LM) capabilities advance, evaluating and supervising them at scale is getting harder for humans. There is hope that other language models can automate both these tasks, which we refer to as *AI Oversight*. We study how model similarity affects both aspects of AI oversight by proposing *Chance Adjusted Probabilistic Agreement (CAPA)*--a metric for LM similarity based on overlap in model mistakes. Using CAPA, we first show that *LLM-as-a-judge* scores favor models similar to the judge, generalizing recent self-preference results. Then, we study training on LM annotations, and find complementary knowledge between the weak supervisor and strong student model plays a crucial role in gains from *weak-to-strong generalization*. As model capabilities increase, it becomes harder to find their mistakes, and we might defer more to AI oversight. However, we observe a concerning trend--model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures. Our work underscores the importance of reporting and correcting for model similarity, especially in the emerging paradigm of AI oversight.

Great Models Think Alike and this Undermines AI Oversight
Shashwat Goel 12 Joschka Strüber 34
Tize Amanda Auzina 34 Karuna K Chandra
Ponnurangam Kumaraguru
Douwe Kiela 67
Ameya Prabhu 34 Matthias Bethge 34 Jonas Geiping 123
Hample-wise Predictions ons
model-similaree setrritttggttt..ii... ic
Im-similarita
Abstract
Great Models Think Alike
As Language Model (LM) capabilities advance, evaluating and supervising them at scale is get- ting harder for humans. There is hope that other language models can automate both these tasks, which we refer to as "AI Oversight". We study how model similarity affects both aspects of Al oversight by proposing Chance Adjusted Prob- abilistic Agreement (CAPA): a metric for LM similarity based on overlap in model mistakes. Using CAPA, we first show that LLM--a-a--udde scores favor models similar to the judge, gener- alizing recent self- eeffpreferrece results. Then, we study training LM annotations, and find com- plementary knowledge between the weak supervi- sor and strong student model plays crucial role in gains from "weak-to-strong generalization". As model capabilities increase, becomes harder to find their mistakes, and might defer more Al oversight. However, observe concern- ing trend model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures. Our work under- scores the importance of reporting and correcting for model similarity, especially in the emerging paradigm of Al oversight.
Similaity, by
Modill
Capebality
this Undermines Al Oversight
LM-as--Jddge
LM-as-Annotetor
Julgemont Soome
Judga
Annonator
Chii
Arboay
Model
Similurity Ag
Student
Sinni Sinnlaator, C.
Evaluation
Training
Figure 1. Our Main Contributions. We develop novel proba- bilistic metric for model similarity, CAPA (Kp)). which adjusts for chance agreement due to accuracy. Using this, we find (1) LLM-as- a-judge scores are biased towards more similar models controlling for the model's capability (2) Gain from training strong models on annotations of weak supervisors (weak-to-strong generalization) higher when the two models are more different, (3) Concerningly, model errors are getting more correlated as capabilities increase.
is increasing interest in post-training, either with human preferences (Ouyang et al., 2022), or task-specific expert annotations (Lightman et al., 2024). Collecting human pref- erences or annotations is slow and expensive. Therefore, with increasing model capabilities, an attractive alternative is to use LMs to annotate training data (Gilardi et al., 2023) and score model outputs (Zheng et al., 2023), to boost both training (Stiennon et al., 2020) and evaluation (Li et al., 2024b). In this paper, we refer to both these techniques together Al oversight!
1. Introduction
Machine Learning model capabilities have improved im- mensely over the last few years. Scaling up the amount of data used for training has played crucial role in these improvements (Kaplan al., 2020). Initially, most of the gains in Language Model (LM) capabilities came from scal- ing pretraining data (Llama Team, 2024a). Recently, there
Can we rely on Al oversight going forward? This remains topic of much debate. In this work, we study oversight from the perspective of model similarity. When assessing or teaching humans, it is well recognized that individuals have different strengths and weaknesses. Similarly, two models with 50% accuracy may misclassify completely dif- ferent samples and thus be highly dissimilar (having differ-
'ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems TTübinee AI Center Uniiversity of Tübingen 'ITT Hy- derabad 'Contextual AI Stanfood University "Core contributors. Correspondence to: Shashwat Goel <shashwatnow @ gmail.com. com.>.
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 202.5 by the author(s).
'The term inspired by "scalable oversight" (Bowman al., 2022), which studies -aman-in-the-loop mechanisms for Al Safety.

---

Great Models Think Alike and this Undermines AI Oversight
ent 'strengths"). To measure model similarity, we build on error consistency (Geirhos et al., 2020), which measures overlap in the samples where two models err beyond what is expected by chance due to the two models' accuracies. In Section 2. we extend the error consistency metric in two crucial ways - 1) by counting differences in predictions rather than correctness for each sample, and 2) incorporat- ing output probabilities. In this way, novel similarity metric, Chance Adjusted Probabilistic AlliablmssticAliimment (CAPA), allows us to quantify functional similarity between models. We use this to analyze both evaluation and training using Al oversight as depicted in Figure 1:
wise model predictions alongside benchmark scores (Bur- nell et al., 2023; Ghosh al., 2024), as they enable richer analysis like measuring similarity.
2. Methodology: Measuring LM Similarity
We begin by deseribing how we quantify model similarity,
2.1. Background
Functional similarity: Prior work on model similarity has focused two classes of similarity measures: representa- tional and functional similarity (see Klabunde et al. (2025) for a recent survey). Representation similarity metrics focus on the weights and activations of the networks (Kornblith et al., 2019), comparing how features are represented in- temally. In contrast, functional similarity metrics focus on the input-output behavior of the model. Functional similar- ity metries are more broadly applicable than representation metrics as (1) they allow comparisons across model families and architectures and (2) are applicable for models behind an API (where weights are not released). Functional similar- ity metries are more interpretable because they operate on data samples rather than noisy, complex internal representa- tions (Golechha & Dao, 2024). Despite large architectural differences across models and model families, their outputs can still be fairly similar. Moreover, Geirhos et al. (2020) argue models with similar internal mechanisms make simi- lar mistakes, and thus mistakes can proxy whether models use similar internal mechanisms. Therefore, in the present work we focus on functional similarity metrics.
1. LLM-as-a-Judge. Prior work has shown that LM judges are biased towards their own generations (Liu et al., 2024; Panickssery et al., 2024). It might seem possible to avoid this concern by simply using a different model as the judge. However, just like human evaluators prefer candidates with similar traits (Bagues & VilaadVill,,,,,a, 2012), could LM judges also exhibit this affinity bias? In Section 3, we study this using CAPA, finding LM judges indeed assign higher scores to models that are more similar to themselves.
2. Training LMs on annotations of other LMs. Next, we study the effect of similarity on inter-LM training set- ups, where one model annotates data used to train another model. We hypothesize that performance gained through such training leverages complementary knowledge among models, and is thus inversely proportional to CAPA. We investigate this hypothesis in Section 4, following the weak- to-strong generalization setup (Burns al., 2024), where strong (larger) student model is shown to outperform the weaker (smaller) supervisor whose annotations it is fine- tuned on. Indeed, we find performance gains are higher when the weak supervisor and the strong student model are more different. Moreover, our findings indicate a higher performance ceiling for weak-to-strong generalization than previously estimated, if the weak supervisor's 's complemen- tary knowledge is leveraged effectively.
Error Consistency: A popular similarity metric designed in the context of comparing mistakes of image-classffiers to humans is error consistency (Geirhos et al., 2020). It quantifies the overlap on samples where two models make mistakes while normalizing for chance overlap due to accu- racy. First, they define Cabs as the "observed error overlap" i.c., the fraction of samples on which both models are cor- rect or both models are wrong. This itself is used metric in recent work on LM similarity, Dutta et al. (2024). How- ever, Geirhos et al. (2020) point out, Cabs has crucial shortcoming: two independent models with high accuracy will have a higher Cab. by chance than two models with low accuracy ). An independent model here is one that is correct on uniform random subset (size corresponding to accuracy) of samples, and wrong on the others. For instance, two independent models with 90% accuracy will agree on at least 81% % of the samples by chance, whereas for two models with 50% accuracy, the lower-bound on chance agreement drops to 25%. Consequently, to account for this, Geirhos et al. (2020) calculate the "expected error overlap" (Ceup) as Coxp acc1 aacc2 +(1 (111100 acc1) (1 acc2) where acc, is the accuracy of model i. Similar to Cohen's 's k (Cohen, 1960), error consistency (Bq. 1) is then defined as the fraction of ex-
3. With increasing LM capability errors are becoming more correlated. AI oversight gaining popularity as ca- pabilities increase. The above results show the benefits of diverse models for Al oversight- - less similarity between models reduces bias in ALM-ass---jdde and also leads to greater gains when training on LM annotations. Unfortu- nately, in Section we find that as popular frontier LMs become more capable, their mistakes become more similar as captured by CAPA. This trend indicates risk of com- mon blind-spots and failure modes when using Al oversight, which is concerning for safety (Kenton et al., 2024).
Overall, our work proposes a novel probabilistic metric for model similarity, and demonstrates the risks of corre- lated mistakes in the emerging paradigm of AI oversight. We hope the community shifts towards releasing sample-
2

---

Great Models Think Alike and this Undermines AI Oversight
Table Comparison Functional Model Similarity Metries. Only our metrie, CAPA, satisfies three desiderata:
Adjusts for accuracy- The metric should not inflate scores for high accuracy model pairs due to lesser scope to disagree. Distinguishes different mistakes - The metric should consider different wrong predictions as a disagreement.
Incorporates probabilities- - The metric should use the probability distribution over predictions provided by the models.
Metric
Adjusts for Accuracy
Distinguishes different mistakes Probabilities
Incorporates
%Flips = - Cobs (Dutta et al., 2024)
X
X
Cohen's k, Scott's π, Fleiss k
x
X
AAgreement (Zheng al., 2023)
X
X
Error Consistency (Geirhos et al., 2020)
X
X
Pearson / Matthew's Correlation of Errors
X
X
Divergence metrics like KL, JSD
X
CAPA (Ours)
cess agreement observed (Cobs Gexp) from what is possible beyond chance Cexp):
benchmarks for LMs. We provide a detailed derivation in Appendix A. with extensions to classification and exact match evaluations in Appendix A.3.
Cabs
Gexp
Observed Agreement (cite): It represents the probability of agreement if the model's 's predictions were sampled based on the observed likelihoods assigned over options. Formally,
Coxp
2.2. Our Contribution
We identify two key limitations of error consistency (k)):
Σ Σ pi(os) P2(0:), D
1
obs
DED (EE()))(a)
Does not distinguish differing mistakes If two mod- els make wrong but different predictions, error consistency still counts that an agreement. For example, two models that are always wrong, even in different ways, have perfect error consistency (k= = 1). It thus overestimates similarity.
where pi (o:) and p2(0:) are the output probabilities for model 1 and 2, respectively, on a data sample for option og, O(z) are the possible options: O(z) = [o.,..., on]. and [D is the total number of data points.
Does not capture probability information ): For com- parison to humans, error consistency assumes single top prediction, whereas models inherently output probability distribution. Ignoring these probabilities can lead to incor- rect conclusions about model similarity. Consider two mod- els whose outputs are [0.49, 0.51] and [0.51, 0.49]. Despite their small differences, binary labels would classify them as making entirely different predictions (0 and 1). Conversely, models with predictions [0.99, 0.01] and [0.51, 0.49) may share the same binary output (0 and 0) but differ significantly in confidence distribution.
Chance Agreement (cEsp): To account for higher accuracies inflating Eeh.. we normalize by the agreement expected from two independent models. First, we define Pj as the average probability model assigns to the correct option across all samples. For a perfectly calibrated model Pj approaches accuracy, thus aligning with the motivations in error consis- teney. Then, we define independent models as assigning Pj probability to the correct option, and uniformly distributing the remaining 1 - Pj probability over the incorrect options. The latter is necessary, as there is no coherent concept of "classes" for MCQ data, i.c. the options can be permuted. This prevents us from computing class marginals for the remaining options, such as in inter-annotator agreement metrics like Cohen's 's Kappa, Scott's Pi (Scott, 1955), Fleiss' Kappa (Fleiss et al., 1981). Formally,
Novel Metric. We redefine Cobs and Cesp to address the above limitations. For clarity we adjust the notation of our agreement metrics to e and cásp- To compute ca we directly use the model output probabilities (Eq.2), thus accounting for disagreement on incorrect options and better capturing model similarity. This approach lets us calculate d without sample-wise ground-trus -trtt annotations. For Cosp we take into account that the model output predictions can span over multiple options rather than only looking at sample-wise accuracy.
CP
chance agreement on correct option
(1 (1-P1) -1-
1
(3)
0(x))
ZED
chance agreement on incorrect optoneecttptiionn
Definition. We define Kp in the context of Multiple Choice Questions (MCQs), which is the format of many popular
where 10(z)) is the number of options in question x.
3

---

Great Models Think Alike and this Undermines AI Oversight
08
metries do not adjust for chance agreement due to higher accuracy, and thus lead to higher similarity scores being reported as the second model's accuracy increases, even though the two models continue to make independent and uncorrelated predictions. In Appendix A.I we prove that CAPA is a generalization of error consistency and reduces to it for binary classification tasks if discretize predic- tions to 1 on the correct option and 0 on others. As such, in the simulation presented here, eπтоr consistency would be quite similar to CAPA, as both metrics adjust for accuracy. In Appendix A.7 we show that CAPA demonstrates the ex- pected trend most clearly also in settings when models have correlated predictions (Appendix Figure 8-9). Lastly, CAPA can be extended beyond pairwise comparisons multiple models (Appendix A.2), and for completeness, we present probabilistic extensions for Cohen's K, Scott's x, Fleiss' KF in Appendix A.5.
Similarity Reported by Metric
CAPA (ours)
0.7
Coben's
0.0
JS- Disttisance
% agreement
04
03
04
0.0
0.70
Accuracy for Second Model
Figure 2. Metric comparison for independent models with un- correlated predictions. In this simulation, for each model we select an independent random subset of samples as correct, with the first having fixed 90% accuracy, while for the second curacy is varied from 50% to 90%. CAPA correctly reports 0 similarity when models have uncorrelated errors.
3. Affinity Bias in LLM-as-a-Judge
Evaluating free-respons, model generations automatically at scale is tricky (Biderman et al., 2024). This has led to popular leaderboards like Arena-hard-auu (Li et al., 2024b), AlpacaEval 2.0 (Dubois et al., 2024), and Aidan- Bench (McLaughlin al., 2024) adopting LLM-as-a-Judee for scoring. Recent work cautions that language models are biased towards their own outputs (Liu et al., 2024; Pan- ickssery et al., 2024), and these leaderboards assume that ex- cluding the judge model from the rankings circumvents this problem. However, human interviewers have been shown to also be biased towards candidates with similar knowl- edge and traits, a phenomenon called affinity bias (Bagues & Perez- Vilez--illadoiia, 2012). We study whether LMs also exhibit a bias toward similar models. This would indicate that it is not sufficient to just use a held-out LM as the judge; one must account for the confounder of similarity.
Finally, the equation for CAPA is:
P obs
Cesp
Kp Cexp
(4)
Interpretation. We prove Kip is bounded between -1 and 1 in Appendix A.6. A value of means the models have the same agreement independent models given their ac- curacies. A negative value means the models disagree, and a positive value indicates they agree beyond independent models with their accuracy. As Kip increases, it means mod- els make more similar mistakes, their errors become more correlated, and they are functionally less different. We use these interpretations interchangably.
Alternatives and Justification. We summarize compar- isons to existing functional similarity measures based on key desiderata
in Table 1. To demonstrate the em- pirical effects, we perform simulation, where we take two binary classification models, that are correct on independent random subset of samples (more details in Appendix A.4). Since these models are independent, any agreement is by chance, and their predictions are entirely uncorrelated. In such case, we want a metric that reports O similarity. In Figure 2 report metric behavior when one model has fixed 90% accuracy, while varying the ac- curacy of the other model from 50% to 90%. Our proposed metric, CAPA, indeed reports O similarity, indicating that the models are independent and their predictions uncorre- lated. This is in contrast popular metrics like Cohen's (Cohen, 1960), used to measure agreement between two annotators, Jensen Shannon (JS) Distance, used to measure a normalized divergence between two probability distri- butions, and %Agreement, which is the number of same predictions made by the two models. All three alternative
3.1. Experimental Setup
To study whether LM judges prefer more similar models, we evaluate a large set of judges and models on MMLU- Pro (Wang et al., 2024), a benchmark for hard problem solving questions across 14 domains. We filter 8,707 ques- tions that can also be answered in a free-text response style, without options, following Myrzakhan al. (2024). Each question is posed to every evaluated model as MCQ and as an open-style question. The per-sample results of the former are used to compute the similarities of judge- pairs, whereas responses to the latter are given to an L.M-- as-a-judge. The judge has to make a binary decision on whether a free-text response correct or wrong. This is done based its own internal knowledge without access to ground-truth solution. We call the average of binary judg- ments across the benchmark the model's 's Judgment Score for a given judge. Using a parallel MCQ evaluation with
4

---

Great Models Think Alike and this Undermines AI Oversight
1.-
Table 2. Partial Correlation and Multiple Regression Results. The table reports partial correlation results between judgment scores and model similarity when controlling for accuracy (r- Pearson correlation), as well as multiple regression results be- tween judgment scores (DV) ~ similarity (IV) and accuracy (IV). and ** indicate significance level p<0..0 and p<00.01 respec- tively. Across all judges we find significant partial correlation, which implies that after controlling for accuracy there remains a relationship between judge score and model similarity. With respect to Multiple regression, across all judges we find signifi- cant effect of both IV on judgment scores while holding the other constant, suggesting strong positive relationship (for details, see Appendix B.6).
0.8
Judgment Score
3.6
Fudge
9.6
Queert3 3- 12 to00
Quent 2-0322---213. 1- spen
Mittad Jamnn-3 1-11-11 rataree
Mleta-llurn-1 1-2054t e-du02e
04
Llams-8 S-10B-it
granned 27twit in EL50++
Minitral- 48---3010 c-0las
0.05
0.10
0.35
0.40
Similarity
Judge
Partial Cor.
Multiple Reg.
Figure 3. Judgment Score Relation with Model Similarity on only across family pairs. Each line is regression model fit between judgment and similarity scores. The circle shape indicates that only across-family judge-model pairs are plotted. We report for each fit the corresponding Pearson correlation values, r. We found significant positive correlation between judgment scores and CAPA across all judges, ** indicates < 0.01.
sim
acc
Qwen2.5-7B-It
0.60*
0.59
0.51
Qwen2.5-32B-It
0.43**
0.41*
0.86
Qwen2.5-72B-It
0.42**
0,47**
1.04
Meta-Llama-3.1-8BB
0.65**
1.15
0.53
Mcta-Llama--31-7000
0.45**
0.61**
0.92 *
Llama-3.3-70B-II
0.35*
0.50*
1.02 TM
gemma-2-9b-It
0.65**
0.76
0.69
ground-truth answers allows us to compare the judgment scores with verifiable accuracy measurements (details and comparisons with alternatives are in Appendix B.3), consis- tent with prior scalable oversight research (Bowman et al., 2022). We compute pairwise similarity with CAPA, Kg- across 9 judge and 39 model pairs. For a complete overview of models investigated, question filtering, inference setup, and prompts used for judges, see Appendix B.
gemma-2-27b-It
0.65**
0.71**
0.68**
0.60**
0.82**
0.43
shows that both, accuracy and similarity, have a significant positive effect on the judge score. The coefficient of accu- racy increases for more capable judge models, consistent with prior work showing improved alignment with human judges (Thakur et al., 2024). We find that especially for small models (<32B) the effect of similarity is greater. The partial correlation results control for accuracy and confirm that there is still a significant effect of similarity on judg- ment scores even for the best judge models. Altogether, the statistical analysis confirms that judgment scores are con- founded by affinity bias. For extended results also including model size as a confounder please see Appendix B.
3.2. Results & Discussion
Q1: Do LM Judges favor more similar models? As motivating example, Qwen2. judge scores Qwen2. 5-7B-Innstrut as being 71% cor- rect, while the more capable (41% vs 51% MCQ accuracy) a.lama-3. 1-100--Inntttuu
as
is deemed less accurate at 67%. In Figure we show that model favoritism extends beyond self- or family- preference all models that are func- tionally similar. We find significant (p- < 0.01) positive correlation (average Pearson r=0.84) between LLM-as-a- judge scores and model similarity (xip) for all judges. To ensure this trend is not only an artifact caused by aggregat- ing across all MMLU-Pro categories, per-category results for the same experiment are shown in the Appendix B.5. These exhibit the same affinity bias observed here for all different topics.
Q3: Does this transfer LLM-assa--udge for binary preference in free-form generation? In most cases, LLM- as-a-judge is not used to check the technical correctness of responses, but rather to match human preference by select- ing the better one from a pair of responses. To extend our findings to the additional domain of chat responses, we pro- vide an additional experiment on the AlpacaEval benchmark in Appendix B.4 (Dubois et al., 2023). We show that the Elo obtained by using the same judges as before to obtain binary preferences still strongly correlates with model similarity on MMLU-Pro. This indicates the existence of an affinity bias in settings other than QA.
Q2: Is this merely due to better accuracy? Note that while Kip adjusts for inflation in agreement of highly accu- rate models, we do expect models with lower accuracy to be less similar with highly capable judge models, and thus have lower Kp. To control for the accuracy of the evaluated model we perform multiple regression and partial correla- tion analysis (Table 2). The multiple regression analysis
5

---

Great Models Think Alike and this Undermines AT Oversight
Weals Strong Markers Lines
Elaus 82 3B Lamno-11 51-50 (z=00.00)
Liama 3.m 32-31
Que82:70 (e-- 0.91)
Weake Weaak-t--Strong Strong Trained Moddl Acmmarc
0.25
Queedd 5..5
2lama-32300
1.ama-3.1-0B (mm-2000)
24h (r0-000)
Weak Supervisor Prediction
Correct
1.3.3
Both
Complementary
Qpenn 5-1 58
Quwni 5-7 (:99
gemma-2-4t
Gain from W2S Training
Nemuns--222 Liarr S.1-BB (cm-0,50)
2.2.
6-7B (c- 0.87)
proma 2:2h peomas th (+-- 0.80]
pi--2 Llmma-1).400 (+o-0193)
2lb-2 Quent
(cm0..0)
0.15
piá 2-
peuan 2.3b (Lm0
74277
Wrong
0.10
Elicitable
Neither
4435-166
0.05
Q-25--
G22
0.00
Correct
Wrong
Strong Elicited Model Prediction
Figure 5. Role of Complementary Knowledge and Elicitation in Weak-to-Strong Generalization. We decompose the accuracy of the weak-to-strong to-strnng trained model on four parts the test data distribution, based on the correctness of the weak supervisor and an oracle strong elicited model which uses ground-truth annotations. Sub-rectangles represent weak, strong model pairs. Results are averaged across tasks. Complementary knowledge transfer explains weak-to-strone model accuracy beyond elicitation.
0.00
0.05
0.10
0.16
0.20
0.25
0.30
0.35
Similarity
Figure Similarity vs Gain from Statong Training. Across 12 model pairs, the strong student gains more from weak- to-strong training on tasks where it more different from the weak supervisor 0.01).
4. Learning from Complementary Knowledge of LM Annotators
curacy student on a 70% accuracy task-finetuned teacher lead 75% accuracy? We adopt a lens of complementary knowledge to understand
generalization.
We now study the role of similarity in AI supervising train- ing. This can allow scaling up training data by reducing reliance on costly and time-intensive expert human inputs. There is hope that models can learn from other models to improve further even on tasks where they surpass human capabilities (Hughes et al., 2024). Where could this im- provement come from? We hypothesize that the existence of complementary knowledge or complementary capabili- ties between two LMs can be one mechanism for driving improvements from LM annotations, if effectively lever- aged. This complement can exhibit itself in the form of differing predictions training data, and can thus be quan- tified using functional similarity between the supervisor and student model. Lower Kp is indicative of more complemen- tary knowledge, and as an implication of our hypothesis, should inversely correlate with the performance gain student model when trained on the supervisor's annotations.
Following Burns et al. (2024), we define "weak" and "strong" based on model size. We study weak models the 3B parameter range, and 3 strong models in the 9B parameter range, for a total of 12 model pairs, and 15 of the binary classification NLP tasks studied in Burns al. (2024), specified in Table 9. The smaller models do have lower accuracy (before finetuning) than the bigger ones across tasks, as shown in Appendix Figure 17, justifying calling them "weak" and "strong". We measure similarity between the weak supervisor and base student model on the validation set. We then perform weak-to- to-sttong- training on the student model, using the confidence confidence-weighted finetuning objective proposed in Burns et al. (2024). We investigate if similarity is an apriori predictor of performance gained on the test set. The full setup is consistent with the open- weight model reproduction by (Scherlis al., 2024), and is described in Appendix C.I.
4.1. Experimental Setup
4.2. Results & Discussion
Burns et al. (2024) study training larger student model annotations from small task-finetuned "expert" teacher. They find the student can outperform the supervisor, phe- nomenon they call "weak to strong generalization". We study this setup as can seem counter intuutiv when viewed from the lens of accuracies. How can training a 60% ac-
Q1: Does Complementary Knowledge Influence Perfor- mance Gain? Figure 4 shows that for all model combi- nations, similarity between the weak supervisor and initial strong student inversely correlates with the improvement obtained from weak-to- -oak-oo- training (r -0.55). Even
6

---

Great Models Think Alike and this Undermines AI Oversight
0.30
Yordd
training. We average accuracies across 15 datasets and 12 model pairs (180 training runs) and report gaps to the student model's initial accu- racy. Complementary knowledge transfer can enable higher gains than the previously considered ceiling estimate from elicitation.
Table Accuracy gains possible from
0:35
Shord
Lone
0.30
Avg Similarity
OrandEa
0.19
Quea
Model
Accuracy Gap
0.10
Initial Strong Student
75.1%
0:59
Weak Supervisor
+4.1
000
Weak Strong Trained Student
+7.4
Datasets
800
Ceiling Estimate
Accuracy of Models
Ground-truth Elicitation (previous)
+11.2
Elicitation I Complementary (ours)
+14.1
Figure 6. Average Similarity (Kp) vs Model Capability. We split 130 LMs into buckets based on their accuracy percentile. For each LM we compute its mean similarity within the bucket (across models from different developers), and plot against model ac- curacy. The size of the scatter points indicates model size. As Kpp measures overlap in mistakes, the positive correlation indicates LM mistakes getting more correlated with increasing capabilities.
after using partial correlation analysis to control for the ac- curacy gap between the weak supervisor and strong student, similarity is inversely correlated with weak-to-strong gain (r -0.35, p < 0.01). Thus, tasks where the supervisor and student make less correlated emrors tend to yield greater improvements. This contributes towards understanding why gains from weak to strong training vary across tasks, an open question posed by Burns et al. (2024). The observation is not specific to our similarity metric, Appendix Figure 16 shows the same trend holds with alternate metries, albeit with lesser variance explained.
Q3. Where can weak-to-strong training improve? The strong elicited model is considered represent upper-bound performance. However, recent work has shown that the per- formance of the strong elicited model can be surpassed (Shi et al., 2025), indicating it is not really an "upper-bound". Indeed, the strong elicited model's performance does not account for potential gains from leveraging the complemen- tary knowledge of the weak supervisor. We compute a new ceiling, taking the union of correct predictions between the weak supervisor and strong elicited model, which is signif- icantly higher as shown in Table 3. Interestingly, on the training set, the weak-to- st-tong trained model shows simi- lar accuracy on the top-left and bottom-right quadrants as shown in Figure 15. Yet, when generalizing to unseen sam- ples, it falls back more often to its initial priors. We hope this analysis guides future work on improving weak-to veak-to-strong training methodology, by highlighting leveraging comple- mentary knowledge as a concrete avenue for improvement.
Q2. Does Complementary Knowledge Add Beyond Elic- itation? The original explanation for performance gains from weak-to-strong generalization is that the weak super- visor "elicits" the latent knowledge in the superior repre- sentations of the stronger student (Burns et al., 2024). To investigate whether complementary knowledge adds to this explanation or is subsumed within it, we first obtain the strong model with "upper-bound" elicitation, by finetun- ing on ground-truth annotations. We refer to this as the strong elicited model. We can then separate the test data into four parts based on whether the strong elicited and weak supervisor model were correct or wrong, measuring average accuracy of the weak-to-strong model on each part to disentangle gains from different factors. The experiment setup is discussed further in Appendix C.2.
5. Models are making more similar mistakes as capabilities increase
Figure reports aggregate values across 15 tasks for 12 model pairs. Accuracy on the bottom-left quadrant (avg. 71.9%) can only be due to successful elicitation, as here the weak supervisor was wrong. Accuracy on the top-right quadrant (avg. 42.2%) can only be due to complementary knowledge transfer as here the upper-bound elicitation model was wrong. This confirms that elicitation plays an important role in weak-to-strone generalization, with complementary knowledge transfer from the weak supervisor also contributing to significant gains.
The previous two sections highlighted two major advantages of having access to more diverse LMs: a) it leads to less biased judges. b) it can drive more performance gains from training on I.M annotations. This points to the importance of diversity, or lower model similarity, for Al oversight. As AI oversight becomes increasingly relevant with advancing capabilities, we now study similarity trends in existing LMs across different levels of capability. It has been shown model representations across modalities are converging with increasing capabilities (Huh et al., 2024). Does this also lead to more similar mistakes?
7

---

Great Models Think Alike and this Undermines AI Oversight
5.1. Experimental Setup
et al., 2022; 2023; Dunlap et al., 2024; 2025). Our work proposes metrics to quantify LM differences (or similar- ity). Huh et al. (2024) used representation similarity met- rics (Kornblith et al., 2019; Bansal et al., 2021) to show con- vergence in visual representations and their alignment with language representations. In contrast, we show model mis- takes are becoming more correlated as capabilities improve, using sample level evaluations (Burnell et al., 2023) such as those available on OpenLLMLeaderbboar (Myrzakhan et al., 2024) and HELM (Bommasani et al., 2023). Geirhos et al. (2020) proposed measuring "error consistency" be- tween image classifiers and humans, with Geirhos et al. (2021) showing early trend of data-rich models making more similar mistakes to humans. We enrich this metric, distinguishing between different mistakes and incorporating probabilistic information. Bommasani et al. (2023) used a similar metric to demonstrate homogenization in outcomes when using LMs for decision making. Our work shows that such risks from algorithmic monoculture (Kleinberg & Raghavan, 2021) are increasingly relevant as LM capabili- ties improve.
We collect sample-wise evaluation files for 130 official mod- els from the OpenLLM Leaderboard 2 released by Hugging- Face, listed in Appendix D.5. We use MMLU-Pro (Wang et al., 2024) and Big Bench Hard (BBH) (Suzgun et al., 2023) as they measure a broad range of capabilities using MCQ, and frontier models have reasonable accuracies while not saturating these datasets. We first bucket these mod- els into five performance percentile ranges. Then, for each model, we compute its mean similarity (Rp) with models in the same bucket from different developers, to prevent con- founding from distillation or continual training. More setup details are provided in Appendix D. In Appendix D.3 we also report pairwise results, and using the extension of Kip for sets of M >> 2 models.
5.2. Results & Discussion
Q1. Are model errors becoming more correlated with improving capabilities? Figure 6 shows strong positive correlation between model capabilities and Kp. which mea- sures similarity beyond chance agreement due to accuracy. In Appendix D.4 we find this also holds across individual categories in both datasets, not just in aggregate.
Our results on Al judges fall in a broader line highlighting their pittalls (Zheng et al., 2025). These include biases such as favoring verbose texts or options at certain positions (Koo et al., 2024; Ye et al., 2025). Interestingly, these biases are also sometimes found in human annotators (Chen et al., 2024). In fact, there is rich literature documenting biases in human judgements of other humans. One such bias affinity bias, where recruiters prefer candidates with similar knowledge and skills as them (Bagues & Perez-Villadonge 2012). We show LM judges also systematically favor other models that make similar mistakes, generalizing previous results that showed LMs favor their own outputs (Liu et al., 2024; Panickssery et al., 2024). Overall, we believe AI evaluators should be accompanied with formal checks like consistency (Fluri et al., 2024).
Potential Implications. If this trend continues, it could mean greater affinity bias when using LM judges, and lower potential for gains from inter-LM training in the context of our earlier results. It could undermine benefits from using LM juries by compromising independence and amplifying collective biases. Most concerningly, our results indicate that as model blind-spots get harder to detect, making us defer more to AI oversight, models also make more similar mistakes, posing safety risks from correlated failures.
Q2. Why are model errors becoming more correlated? This is an interesting research direction in itself. We perform a preliminary analysis in Appendix D.2, summarizing key conclusions here. First, we observe only slight increase in similarity for harder questions in our datasets, indicat- ing difficulty is not a significant confounder for this trend. We find this trend is stronger in instruction-uue tuned models, and using alternative architectures like Mamba (Gu& & Dao, 2024) may not be enough to increase diversity.
A second aspect of AI oversight is using another model's supervision to train better models. This is similar to train- ing on text generated by an LM (Chiang et al., 2023) with ongoing debates about its benefits (Kazdan et al., 2025), and an emerging paradigm of exploiting gap in difficulty between solution generation and evaluation (Song et al., 2025). In this paper, we study the more established setup of training LMs on LM annotations, where Burns et al. (2024) demonstrated the phenomenon of weak to strong general- ization, and it has been leveraged for other applications like image classification (Guo et al., 2024) and aligning models (Zhu et al., 2025). Prior work has attempted to understand weak to strong generalization, notably using "misfit error" (Charikar et al., 2024), which shows that the student's disagreement with the weak supervisor after weak to strong training correlates with its accuracy gap from the weak supervisor. Instead, we show similarity between the
6. Related Work
There is increasing interest in finding differences between models for applications like visual tools for comparative analytics (Strobelt et al., 2021; Kahng et al., 2024), efficient human evaluation (Boubdir et al., 2023), comparing learn- ing algorithms (Shah et al., 2023), identifying side-effects of API updates (Eyuboglu et al., 2024) or quantization (Dutta et al., 2024). Prior work has also looked at qualitatively describing differences between data distributions (Zhong
8

---

Great Models Think Alike and this Undermines AI Oversight
weak supervisor and strong student can apriori predict gains from weak-to-strong training. The benefit of model diversity has previously been discussed in related settings like knowl- edge distillation for image classifiers (Roth et al., 2024) and training chess models that outperform the humans they are trained on (Zhang et al., 2024).
can benefit from composing interventions (Kolbeinsson et al., 2025). If the benefits are decorrelated, composing would lead to greater compound safety. If the side effects are correlated, composing would lead to lower accumulation. More broadly, measuring LM similarity post-intervention can help characterize decorrelation in research bets, for granters (Canton, 2025). For example, LM unlearning was recently found to be functionally similar to refusal train- ing (Lucki et al., 2025), even though it was proposed as a complementary safeguard (Li et al., 2024a). Finally, as we transition towards language agents, similarity can help understand collective "blind spots" (He et al., 2023), and could lead to better cooperation (Lowe et al., 2017) but also scheming (Balesni et al., 2024) between multiple agents.
7. Conclusion, Limitations, Future Work
Our paper shows the importance of measuring functional similarity for language models. We derive a novel, proba- bilistic metric for model similarity, CAPA (Kp)). We then use it to study the implications of similarity for AI oversight showing affinity bias in AI judges, and the role of comple- mentary knowledge when training on LM annotations, such as in weak-to-strone generalization. AI oversight will be- come more relevant as capabilities improve, so our finding that increasing capabilities could lead to more correlated er- rors is particularly concerning. Thus, we believe measuring and accounting for model similarity is going to be increas- ingly important. We now list some limitations of our work and avenues for future work, that can help develop a better understanding of model similarity and its implications.
Qualitative analysis of model differences: We developed quantitative methods for measuring LM similarity on a given data distribution. One exciting direction is to use these met- ries to provide qualitative difference descriptors (Dunlap et al., 2025) between models, by describing data distribu- tions where models are least similar.
Acknowledgments
We established similarity corre- lates with both aspects of AI oversight evaluation and train- ing supervision. To establish causality, we need methods to make a model less similar without harming capabilities, which is itself a challenging open problem.
Establishing Causation:
The authors would like to thank (in alphabetical order) Arvindh Arun, Nikhil Chandak, Thomas Klein, Ankit Son- thalia, Guinan Su, Mark Tygert, Vishaal Udandarao for help- ful feedback. We thank HuggingFace for the public sample- wise predictions provided in OpenLLLMLaaderbaard, which enabled our work. This work was supported by the Tühhingen AI Center. JS thanks the International Max Planck Re- search School for Intelligent Systems (IMPRS-IS) for sup- port. AP and MB acknowledge financial support by the Federal Ministry of Education and Research (BMBF), FKZ: 01 011528B and Open Philanthropy Foundation funded by the Good Ventures Foundation.
Extending similarity metries for free-teet outputs: Ev- eryday use of generative models is based on their free-text responses. Like much work on benchmarking, we had to limit to MCQ tasks as the science of precisely evaluating free-text is still evolving (Biderman et al., 2024). For exam- ple, both model-free (Papineni et al., 2002) and model-based metrics (Pillutla et al., 2021) suffer from a wide range of syntax and style sensitivity issues (Kocmi et al., 2021; He et al., 2023). We hope the community takes up the challenge of designing similarity metrics for free-response text and reasoning. This would allow studying the role of similarity using more promising oversight setups like debate (Kenton et al., 2024) and process supervision (Lightman et al., 2024).
Author Contributions
Shashwat conceived the project, proposed the CAPA metric, and led the LM Annotators experiments (Section 4). Joschka led the LLM-as-a-Judde experiments (Section 3). Ilze led statistical analysis of all results, and characterized properties of CAPA (Section 2, Appendix A). Karuna analyzed trends for similiiittsssimiliitttyyy (Section 5). Ameya helped across sections. The manuscript was written by Shashwat, 1lze, Ameya and Joschka. Douwe, Matthias and PK provided feedback and advice throughout the project. Jonas advised the design of all experiments.
Generator- Verifier gap: Al oversight has recently shown promise for tasks where it is easier to validate a solution than generate it (Song et al., 2025). Similarity may continue to play a role here. (1) In evaluations, similarity in stylistic preferences of the generated solution may influence judge scores. (2) In training, the generator-verifier gap may be larger if models are more different.
Safety implications:
Researchers separately develop many post-training interventions to reduce harmfulness, dual-use knowledge, dishonesty etc. In real world model deploy- ments, all these problems have to be tackled at once, which
Impact Statement
Our paper can be considered as foundational research ana- lyzing mechanisms of AI oversight and identifying an im- portant shortcoming, and not tied to particular applications.

---

Great Models Think Alike and this Undermines AI Oversight
References
Hernandez-Orallo, J. Rethink reporting of evaluation results in ai. Science, 2023.
Allal, L. B., Lozhkov, A., Bakouch, E., Blüzquez, G. M., Tunstall, L.,, Piqueres, A., Marafioti, A., Zakka, C., von Werra, L., and Wolf, T. Smollm2 - with great data, comes great performance. attps://githhn com/ huggingface/ stte//mmlllli 1m, 2024.
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L... Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 4971- 5012, 2024.
Bagues, M. and Perez-Villadonige M. J. Do recruiters prefer applicants with similar skills? evidence from a randomized natural experiment. Journal of Economic Behavior Organization, 82(1):12-20, 2012.
Canton, E. A portfolio approach to research funding. Re- search Policy, 54(1):10512) 2025.
Balesni, M., Hobbhahn, M., Lindner, D., Meinke, A., Ko- rbak, T., Clymer, J., Shlegeris, B., Scheurer, J., Stix, C., Shah, R., Goldowsky-Dill, N., Braun, D., Chughtai, B., Evans, O., Kokotajlo, D., and Bushnaq, L. Towards evaluations-based safety cases for ai scheming, 2024.
Charikar, M., Pabbaraju, C.,, and Shiragur, K. Quantifying the gain in weak-to-stron generalization. In Conference on Neural Information Processing Systems, volume 37, pp. 126474-126499, 2024.
Bansal, Y.. Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. In Confer- ence Neural Information Processing Systems, vol- ume 34, pp. 226-236, 2021.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang. B. Humans or LLMs as the judge? study on judgement bias. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 8301-8327, 2024.
Biderman, S., Schoelkopf. H., Sutawika, L., Gao, L., Tow, J. Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Hsu, J., Jaiswal, M., Lee, W. Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J., Skowron, A., Tan, S., Tang, X., Wang, K. A., Winata, G. I., Yvn,, F., and Zou, A. Lessons from the trenches on reproducible evaluation of language models, 2024.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, F. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: /wslmmsssss.....s..........a...a org/blog/ 2023 1023-03-30-vieumm
Chicco, D., Warrens, M. J., and Jurman, G. The matthews correlation coefficient (mec) is more informative than cohen's 's kappa and brier score in binary classification assessment. IEEE Access, 9:78368-78381, 2021.
Bommasani, R., Liang, P., and Lee, T. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 2023.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 2924-2936. 2019.
Boubdir, M., Kim, E.,, Ermis, B., Fadaee, M., and Hooker, S. Which prompts make the difference? data prioritization for efficient human I1m evaluation, 2023.
Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūté, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Kernion, J. Kerr, J., Mueller, J., Ladish J., Landau, J., Ndousse, K., Lovitt, L.,, Elhage, N., Schiefer, N., Joseph, N., Mercado, N. DasSarma, N., Larson, R., McCandlish, S., Kundu, S. Johnston, S., Kravec, S., Showk, S. E., Fort, S., Telleen-Lawton T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hltfield-Dodds Z., Mann, B., and Kaplan, J. Measuring progress on scalable oversight for large language models, 2022.
Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37- 46, 1960.
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from human feedback. In Conference on Neural Information Processing Systems, volume 36, pp. 30039- 30069, 2023.
Burnell, R. Schellaert, W., Burden, J., Ullman, T. D., Martinez-Plumed, F., Tenenbaum, J. B., Rutar, D., Cheke, L. G., Sohk-Dickstein, J., Mitchell, M., Kiela, D., Shana- han, M., Voorhees, E. M., Cohn, A. G., Leibo, J. Z., and
Dubois, Y., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
10

---

Great Models Think Alike and this Undermines AI Oversight
Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., and Yeung-Levy. S. De- scribing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 24199- 24208, 2024.
Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outper- forms crowd workers for text-annotation tasks. Pro- ceedings of the National Academy of Sciences, 120(30):
e2305016120, 2023.
Golechha, S. and Dao, Challenges in mechanistically interpreting model representations. In ICML 2024 Work- shop on Mechanistic Interpretability, 2024.
Dunlap, L., Mandal, K., Darrell, T., Steinhardt, J., and Gon- zalez, J. E. Vibecheck: Discover and quantify qualitative differences in large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Gu, A. and Dao. T. Mamba: Linear-time sequence model- ing with selective state spaces. In First Conference on Language Modeling, 2024.
Guo, J., Chen, H., Wang. C., Han, K., Xu, C., and Wang. Y. Vision superalignment: Weak-to-strong generalization for vision foundation models, 2024.
Dutta, A., Krishnan, S., Kwatra, N., and Ramjee, R. Ac- curacy is not all you need. In Conference on Neural Information Processing Systems, 2024.
Eyuboglu, S., Goel, K., Desai, A., Chen, L., Monfort, M., Ré, C., and Zou, J. Model changelists: Characterizing updates to ml models. In ACM Conference on Fairness, Accountability, and Transparency, pp. 2432 2453, 2024.
He, T., Zhang, J., Wang, T., Kumar, S., Cho. K., Glass, J., and Tsvetkov, Y. On the blind spots of model-based evaluation metries for text generation. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Voltmme 1: Long Papers), pp. 12067- 12097, 2023.
Fleiss, J. L., Levin, B., Paik, M. C., et al. The measurement of interrater agreement. 1981.
Hendrycks, D., Burns, C.,, Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning Al with shared human values. In 9th International Conference on Learning Representations, 2021a
Fluri, L., Paleka, D., and Tramèr, F. Evaluating superhuman models with consistency checks. In 2024 IEEE Con- ference on Secure and Trustworthy Machine Learning (SaTML), pp. 194. IEEE, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, 2021b.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.. Schoelkopf. H., Skowron, A., Sutawika, L.,, Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation, 2023. URL https: //enodo.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S.. Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022.
Huang, L., Le Bras, R., Bhagavatula, C., and Choi, Y. Cos- mos QA: Machine reading comprehension with contex- tual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing 2391-2400 1201, 2019.
Geirhos, R., Meding, K., and Wichmann, F. A. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In Conference on Neural Information Processing Systems, volume 33, pp. 13890-13902, 2020.
pp.
Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., and Brendel, W. Partial success closing the gap between human and machine vision. In Conference on Neural Information Processing Systems, volume 34, pp. 23885-23892 2021.
Hughes, E., Dennis, M. D., Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktäschel, T. Position: Open-endedness is essential for artificial superhuman intelligence. In Proceedings of the 41st In- ternational Conference on Machine Learning, volume 235, pp. 20597-20616, 2024.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024.
Ghosh, A., Dziadzio, S., Prabhu, A., Udandarao, V., Al- banie, S., and Bethge, M. Onebench to test them all: Sample-level benchmarking over open-ended capabilities, 2024.
Hub, M., Cheung, B., Wang, T., and Isola, P. Position: the platonic representation hypothesis. In Proceedings of the 41st International Conference on Machine Learning, 2024.

---

Great Models Think Alike and this Undermines AI Oversight
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J.. Reif, E., Kallarackal, K., Chang, M., Temry, M., and Dixon, L. Lim comparator: Visual analytics for side-by- side evaluation of large language models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2024.
Korbblihh, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In Proceedings of the 36th International Conference on Ma- chine Learning, volume 97, pp. 3519-3529, 2019.
Krippendorff, K. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 2004.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020.
K Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023,
Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., and Koyejo, S. Collapse or thrive? perils and promises of synthetic data self- generating world, 2025.
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Mukobi, G., Helm- Burger, N., Lababidi, R., Justen, L., Liu, A. B., Chen, M., Barrass, I., Zhang, O., Zhu, X., Tamirisa, R., Bharathi, B., Herbert- Voss, A., Breuer, C. B., Zou, A., Mazeika, M., Wang, Z., Oswal, P., Lin, W., Hunt, A. A., Tienken- Harder, J., Shih, K. Y., Talley, K., Guan, J., Steneker, I., Campbell, D., Jokubaitis, B., Basart, S., Fitz, S., Ku- maraguru, P., Karmakar, K. K., Tupakula, U., Varadhara- jan, V., Shoshitaishvili, Y., Ba, J., Esvelt, K. M., Wang, A., and Hendrycks, suring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference Ma- chine Learning, volume 235, pp.
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie. S., Bulian, J., Agarwal, R.. Lindner. D., Tang, Y., Goodman, N. D., and Shah, R. On scalable oversight with weak Ilms judging strong Ilms. Conference on Neural Information Processing Systems, volume 37, pp- 75229-75276, 2024.
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 252-262, 2018.
The WMDP benchmark: Mea-
2024a.
Klabunde, M., Schumacher, T., Strohmaier, M., and Lem- merich, F. Similarity of neural network models: A survey of functional and representational measures. ACM Com- put. Surv. 57(9). 2025. ISSN 0360-0300.
Li, T., Chiang, W.-L.., Frick, E., Dunlap, L.,, Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-haue and benchbuilder pipeline, 2024b.
Kleinberg. J. and Raghavan, M. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, (18(22)xe01834000
Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report, 2023.
2021.
Koemi, T., Federmann, C., Grundkiewicz, R., Junczys- Dowmunt, M., Matsushita, H., and Menezes, A. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 478 494, 2021.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2024.
Kolbeinsson, A., O'Brien, K., Huang, T., Gao, S., Liu, S., Schwarz, J. R., Vaidya, A., Mahmood, F,, Zitnik, M., Chen, T., and Hartvigsen, T. Composable interventions for language models. In The Thirteenth International Conference on Learning Representations, 2025.
Liu, Y., Moosavi, N., and Lin, C. LLMs as narcissistic eval- uators: When ego inflates evaluation scores. In Findings of the Association for Computational Linguistics: ACL 2024.
2024, pp.
Llama Team. The llama herd of models, 2024a.
Koo, R., Lee. M., Raheja, V., Park, J. I., Kim, Z. M., and Kang, D. Benchmarking cognitive biases in large lan- guage models as evaluators. In Findings of the Asso- ciation for Computational Linguistics: ACL 2024, pp. 517-545, 2024.
Llama Team. Llama 3.2 model card. https: /ttp:/ggttttt com com/meta -emetealllmmee www.llaaaaaamm www.ammmdeelllee mwwemm main/models/llame 2/MODEL CA___AARD.m. 2024b.
12

---

Great Models Think Alike and this Undermines AT Oversight
Llama Team. Llama 3.3 model card. ht httpp://githut com/meta-llama modeeaaaaammmemaaeeeee main/modelss 1twooodessllaaaaa_11 3/M.///LMO_ARRR md, 2024c.
Association for Computational Linguisties, pp. 311-318, 2002.
Pilehvar, M. T. and Camacho-Collados, J. WiC: the word-in- context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 1267-1273, 2019.
Lowe, R., WU, Y., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed environments. In Conference on Neural Information Processing Systems, volume 30, 2017.
Lucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, An adversarial perspective on machine unlearn- ing for AI safety. Transactions Machine Learning Research, 2025.
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Measur- ing the gap between neural text and human text using di- vergence frontiers. In Conference on Neural Information Processing Systems, volume 34, pp. 4816-4828, 2021.
McI.aughlin, A., Campbell, J., Uppuluri, A., and Yang, Y. Aidanbench: Stress-testing language model creativity open-ended questions. In NeurlPS 2024 Workshop on Language Gamification, 2024.
Qwen Team. Qwen2.5 technical report, 2025.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I Language models are unsupervised multitask learners, 2019.
Microsoft Research. Phi-4 technical report. Tech-
nical report, Microsoft, 2024. URL https:
com/ //www http///ccoot.ccoo/eeea////eeeeee hoooliitttoonppl---teeeeeeit
Rogers, A., Kovaleva, O., Downey, M., and Rumshisky, A. Getting closer to ai complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligenee 34(05):8722-8731, 2020.
Mistral AL Ministral 8b instruct model card.
https: httpp////hgniggaccccooooooo
Ministral-Bn 8B Instruct 2410, 2024.
Roth, K., Thede, L.,, Koepke, A. S., Vinyals, O., Hénaff, O. J., and Akata, Z. Fantastic gains and where to find them: On the existence and prospect of general knowl- edge transfer between any pretrained model. In The Twelfth International Conference on Learning Represen- tations, 2024.
Myrzakhan, A., Bsharat, S. M., and Shen, Open-llm- leaderboard: From multi-choice open-style questions for ilms evaluation, benchmark, and arena, 2024.
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. Adversarial NLI: A new benchmark for natu- ral language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885 4901, 2020.
Safak, V. Min-mid-man scaling, limits of agreement, and agreement score, 2020.
OpenAl. Gpt-4 technical report, 2024.
Sap, M., Rashkin, H., Chen, D., Le Bras, R. and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 2019.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, Training language models to follow instructions with human feedback. In Conference on Neurul Information Processing Systems, volume 35, pp. 27730-27744, 2022.
Scherlis, A., Mallen, A., Quirke, L., and Bel- rose, N. Experiments in weak-to-strong generaliza- tion, 2024. URI https: http///ll....////.. eleuther. weak to-ttttssoooo strong/.
Panickssery, A., Bowman, S. R., and Feng. S. LLM eval- uators recognize and favor their own generations. In Conference on Neural Information Processing Systems, volume 37, pp. pp. 18772-68802. 2024.
Scott, W. Reliability of content analysis: The case of nominal scale coding. The Public Opinion Quarterly, 19 (3):321-325, 1955.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th Annual Meeting of the
Shah, H., Park. S. M., Ilyas, A., and Madry, A. ModelD- iff: A framework for comparing learning algorithms. In
13

---

Great Models Think Alike and this Undermines AI Oversight
Proceedings of the 40th International Conference on Ma- chine Learning, volume 202, pp. 30646-30688, 2023.
Thakur, A. S., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., and Hupkes, D. Judging the judges: Evaluating alignment and vulnerabilities in Ilms-as- judges, 2024.
Shi, J., Cheng, Q. Fei, Z., Zheng, Y., Guo, Q. and Qiu, X. How to mitigate overfitting in weak-to-strone general- ization? In Scaling Self-Improving Foundation Models (SSI-FM) Workshop, The Thirteenth International Con- ference on Learning Representations, 2025.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aruiraj. A., He, X., Jiang, Z., Li, T., Ku, M. Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Conference on Neural Information Processing Systems, volume 37, pp. 92266, 2024.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1631-1642, 2013.
Warstadt, A., Singh, A., and Bowman, S. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625- 641, 2019.
Song. Y., Zhang, H., Eisenach, C., Kakade, S. M., Foster, D. P., and Ghai. U. Mind the gap: Examining the self- improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations. 2025.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 106, 2017.
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P., Chawla, N. V., and Zhang, X. Justice or prejudice? quantifying biases in Lim-as-a-judge In The Thirteenth International Conference on Learning Representations, 2025.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Con- ference Neural Information Processing Systems, vol- ume 33, pp. 3008-3021, 2020.
Zhang, E., Zhu, V., Saphra, N., Kleiman, A., Edelman, B. L., Tambe, M., Kakade, S., and Malach, E. Transcendence: Generative models can outperform the experts that train them. In Conference on Neural Information Processing 2024.
Strobelt, H., Hoover, B., Satyanaryan, A., and Gehrmann, S. LMdiff: A visual diff tool to compare language mod- els. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 96-10, 2021.
Systems, volume 37, pp.
Zhang, Y., Baldridge. J., and He, L. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 1298-1308, 2019.
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., and Cardie, C. DREAM: A challenge data set and models for dialogue- based reading comprehension. Transactions of the Asso- ciation for Computational Linguistics, 7:217-231, 2019.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le. Q., Chi, E., Zhou, D., and Wei, J. Challenging BIG-bench tasks and whether chain-of sauin-offthord offthought can solve them. In Findings of the As- sociation for Computational Linguisties: ACL. 2023. pp. 13003-13051, 2023.
Zheng, L., Chiang, W.-L, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a- judge with me-bench and chatbot arena. In Conference on Neural Information Processing Systems, volume 36, pp. 46595-46623, 2023.
Tafjord, O., Gardner, M., Lin, K., and Clark, P. QuaRTz: An open-domain dataset of qualitative relationship ques- tions. In Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
Zheng, X., Pang. T., Du, C., Liu, Q., Jiang, J., and Lin, M. Cheating automatic LLM benchmarks: Null models achieve high win rates. In The Thirteenth International Conference on Learning Representations, 2025.
pp. 5941-5946, 2019.
Zhong, R., Snell, C., Klein, D.,, and Steinhardt, J. Describ- ing differences between text distributions with natural language. In Proceedings of the 39th International Con- ference on Machine Learning, volume 162, pp. 27099 27116, 2022.
Technology Innovation Institute. Welcome to the falcon family of open models! https: /hwgggnggagcc....c co co/blog fa 1 con3, 2024.
14

---

Great Models Think Alike and this Undermines AI Oversight
Zhong, R., Zhang, P., Li, S., Ahn, J., Klein, D., and Stein- hardt, J. Goal driven discovery of distributional differ- ences via language descriptions. In Conference on Neural Information Processing Systems, volume 36, pp. 40204- 40237, 2023.
Zhou, B., Khashabi, D., Ning, Q. and Roth, D. "going on vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- 2019.
IJCNLP), pp.
Zhu, W., He, Z., Wang, X., Liu, P., and Wang, R. Weak- to-strong preference optimization: Stealing reward from weak aligned model. In The Thirteenth International Conference on Learning Representations, 2025.
15

---

Great Models Think Alike and this Undermines AT Oversight
Appendix
Contents
Metries
18
A.I Derivation of CAPA
18
A.2 Extending CAPA to more than two models
19
A.3 How to use CAPA beyond multiple choice?
19
A.4 Detailed Discussion on Design Choices
20
A.5 Probabilistic versions of popular agreement metrics
22
A.6 Theoretical bounds for CAPA
22
A.7 CAPA comparison with other inter-rater metrics
24
B
LLM-as-a-Judge
27
B.1 Extended Multiple Regression Analysis with Model Size as a Confounder
27
B.2 Comparison of Judge Scores for Our Similarity vs Error Consistency
27
B.3 Evaluating Judge Scores Against Ground-Trae
27
B.3.1
MCQ Ground-Truth
................................
27
B.3.2
Judge Ensemble with Access to Reference Answers
28
B.3.3
Judge Score Validity Against Siterence-Basee Ensemble
29
B.4
Judge Preference in Chat Generation Tasks Against Similarity
29
B.5
Judge Score Against Similarity per Category
30
B.6
Statistical Testing
30
B.6.1
Quantifying Correlation Strength Using Partial Correlation
30
B.6.2
Multiple Regression
32
B.7 Experimental Setup for Filtering MMLU-Pro
36
B.8
Experimental Setup Perform Free-Form Inference on Filtered MMLU-Pro
36 ................................
B.9
Experimental Setup for LLLM-as---Judge Filtered MMLU-Pro
37
B.10 List of Judges and Evaluated Language Models
37
B.11
Prompts
39
B.11.1 LM-Judge Prompt without Reference Answer
39
B.11.2 LM-Judge Prompt with MCQ Options
39
B.11.3 Original MCQ CoT Prompt
39
B.11.4 Open-style CoT Prompt
42
B.11.5 Coarse Filtering Prompt
43
B.11.6 Fine-grained Filtering Prompt
44
16

---

Great Models Think Alike and this Undermines AT Oversight
Weak-to-Strong Training
45
C.I Setup
45
C.2 Elicitation vs Complementary Knowledge
46
C.3 Effect of Different similarity metrics
47
C.4 Accuracies in Weak-to-Strong training
48
C.5 Weak-to-strong Accuracy Value Details in Elicitation Complementary Knowledge Analysis
48
D Similarity Trends with Increasing Capabilities
52
D.I Setup Details
52
D.2 Why are model mistakes becoming more similar? A preliminary analysis
................................
52
D.2.1
Instruction-tuning exacerbates the trend
................................
52
D.2.2 Is the trend confounded by question difficulties?
................................
52
D.2.3 Can changing architecture reduce model similarity?
53
D.3 Alternative Similarity Metrics
53
D.4 Model capability similarity across domains
54
D.5 List of models
55
17

---

Great Models Think Alike and this Undermines AI Oversight
A. Metrics
The following section covers design details for CAPA Kip. Firstly, we address derivation of CAPA Section A. I and its theoretical bounds in Section A. Secondly, we explain how extend CAPA to multi-model set-up (Section A.2) and how to adapt CAPA to classification and exact match settings (Section A.3). Lastly, we introduce probabilistic versions of popular agreement metrics (Section A.5) and provide comparison between them and CAPA (Section A.4).
A.1. Derivation of CAPA
CAPA is intended to be used as a similarity metric in the context of model accuracies. As such, it extends Error consistency (Geirhos et al., 2020), metric that adjusts chance agreement by taking into account model accuracy. In particular, the same formula used to define Cohen's ' R, Scott's π, Fleiss' Ki Error Consistency and CAPA:
observed agreement - chance agreement maximum possible agreement chance agreement"
(5)
where the excess agreement subtracted both from the numerator and denominator, essentially calculating the proportion of possible excess agreement that is observed between the two models. Across all metries, the maximum possible agreement is 1. Where CAPA differs from the existing metrics is how we calculate the observed and change agreement.
We redefine error consistency (Geirhos et al., 2020) by incorporating probabilistic information. To achieve this we introduce a probabilistic computation of the observed agreement, Cobs as eoss and the chance agreement, Cesp as cesp. The new equation becomes:
obs
Cexp
Kp 1
Cexp
Observed agreement 2olm Given that have the predicted output probabilities, Pı (1.))a )a by LM for all possible options, O2 = [01, oN for data sample e.g. Vo, On. we can compute the relative observed overlap as:
D 0
dbs = D
Pi(o.)= P2(0:)=
==1
where (-) is the predicted probability by model I and (-).) is the predicted probability by model We would like to highlight that the above calculation is performed on sample level to avoid confusion with the common chance agreement Pe calculation in Cohen's kappa 2.
Agreement by chance dapp that a given model is correct p.:
To estimate the model chance agreement Cosp first start by computing the average probability
D
0 T(o; (o:)2 where gt = ground truth
P+
Σ
(8)
D
Performing the above calculation per model accounts for the possibility that each model may have different marginal distributions. An assumption that is fair to assume in the context of LMs. Subsequently, given the p. per model can compute the probability that two models are correct by chance as: Pi P2- Conversely, to account for model chance disagreement we (1) group all the remaining options as incorrect and (2) adjust for the number of options: EP oo (1- pz). These steps are necessary because (1) MCQ options can be permuted, therefore, class marginal probabilities cannot be computed, and (2) the chance disagreement without adjusting for the number of options overestimates the agreement by chance:
D
0
Σ 0
PT)
PT) P5)
99)
D
2Cohen's 's kappa uses the marginal probabilities across categories estimate Pe. However, in MCQ there "class categories" the options can he permuted across data samples. Therefore, marginal prohahilities cannot be estimated.
18

---

Great Models Think Alike and this Undermines AT Oversight
In particular, if the number of options is ignored then the underlying assumption is that both models put their 'incorrect" probability mass on the same option, following a Dirac delta 8(o.) distribution. This is very strong assumption, that overestimates model error agreement. Therefore, we propose to adjust this by assuming that the distribution for the incorrect
options follows uniform distribution Uf0 On-1) as adjusted by normalizing factor [b] Σx=1 jo.i-I+ where [Oz] is the total number of options for sample æ. As such, the overall agreement by chance probability is:
D D mein
exp
P1P2
1 uniformity assumption
(1 (1-P))L-PP)
(10)
chance agreemeent correct
chance agrecment incorrect
Moreover, for perfectly calibrated models the mean correct probability P. would approach model accuracy, P. - pi, and is upper bounded by it 17 < p. as pr is computed based on probabilities (1i. TP-TN
[
Reduction of CAPA Error Consistency for binary classification In binary classification setting when the underlying probabilities are unavailable CAPA reduces to error consistency, as (1) co [D] Σ.= Ularg max p1 = arg max p2] = Cobs
and (2) obs Cobs P. CCC,, and the normalizing factor simplifies to 1.
A.2. Extending CAPA to more than two models
In Section 2.2, computed functional similarity between a pair of models. Here, extend CAPA to multi-model comparisons. In the inter-annotator agreement literature, Fleiss' & (Fleiss et al., 1981) is commonly used for this. However, it is suited to our modeling paradigm as it defines <epp using the assumptions of Scott's x instead of Cohen's & (this is problematic when measuring model similarity as discussed in the previous section). We derive CAPA for more than two models using first principles logic, similar how Fleiss' was derived.
c For -0 Fleiss' к measures the proportion of observed pairwise agreements from the total possible for each question, averaging across questions. This is equivalent to averaging the observed agreements for each pair of models when all models annotate all questions, which is true in our case". This gives us e = M(M-1
Suppose the number of models is M > 2. We still use the
Te Gep formula, but change the definition of co and clsp. 1-cl
2
Eigicigm [o] Σ Ee= P:(o))2 (o)())
Second, for dxsp. Fleiss' k measures the expected pairwise agreements all M models were independent. This can be obtained by averaging the cosp for two models across all possible pairs of M models. This gives us
D Σ
2
Lop
15 (1) (1 P))
+
M(M - 1) ISK4JSM
O
A.3. How to use CAPA beyond multiple choice?
In Section 2.2 we defined CAPA for MCQs as this is used throughout the paper, and more commonly for language models. For completeness, we now define CAPA for classification settings and exact match settings, which are alternate strategies for evaluating models.
Unlike MCQs, in this setting are coherent classes (categorics), representing nominal data. The model output now is probability distribution over C classes. Therefore, we compute dy across categories as follows:
Classification:
Σ Σ pi(c.) p2(cs).
(11)
D c.D()
where pa (cx) is the output probability for class c.. For the computation of &epp we follow the same definition as in the main paper, but now pi is computed for the correct class and the chance agreement on the incorrect class is adjusted by the number
Fleiss i also defined when not all annotators respond to every question, as long as the number respondents per question fixed.
19

---

Great Models Think Alike and this Undermines AT Oversight
Metric
Formula
Description
Cohen's Kappa
K
P.
Measures inter-rater reliability Po while accounting for chance agreement Pe.
Scott's Pi
π P.
Similar to Kappa, but uses marginal probabilities for P..
1 Po
Error Consistency
k Goo
Adjusts for accuracy via Cosp acc1 acc2 (1 acci) ac02)
Coep of
CAPA
Kp Czap
Sap
Accounts for sample level probabilities Cobs
[D] Eæ=1 P1(0)) p2(0))x
1
and accounts for accuracy via coxp = P1 P2 + [D]
1 Ei=1 (1 - p1 p1)(1= - P22
Table 4. Comparison of different inter-rater metrics
of classes instead of number of options:
8 exp
Pi P2
+
(12)
chance agreement class
(I pI-Pi) (1 (2-
Σ
1
(13)
D C(x)
chance agreement incorrect class
In principle, the above implementation could also be adjusted to further take into account the class categories by computing the marginal probabilities per class as:
1 D Ep((c) where & A ground truth,
p(c:).
(14)
and replacing the chance agreement on incorrect class with the product of per class 'incorrect' probabilities.
Exact or Fuzzy Match: Here, models are not provided categories or options to choose between, and instead provide an answer from unconstrained set. The model's output string is matched with reference answer. Here, the probability of independent models agreeing by chance approaches zero due to an unconstrained set of outputs. Further computing probabilistic agreement is difficult over conditional distributions across multiple tokens. We recommend calculating the discrete version of CAPA, where ..M.
2= (z) == m2(x)), and Eexp ace1 acc2, finally computing D Σ
Cobs
Regression: For tasks with numeric predictions like regression, for the observed agreement Cobs, one could measure a distance metric over the two models' predictions, aggregating across samples. Once again, models with lower error would have lower distance in prediction, so the challenge lies in defining chance agreement Cesps for model with given error. We would have make appropriate assumptions about the distribution of errors, such assuming they gaussian, and compute Cexp accordingly.
A.4. Detailed Discussion on Design Choices
In this section, we discuss alternative design choices could have taken. For an overview of the equations for each metric, see table 4.
Why not use inter-annotator agreement metrics like Cohen's x?? Cohen's k, Scott's म, Krippendorf's a measure how people differ when answering survey questions, focusing on the reliability of those questions and the data (Krippendorff, 2004). They assume nominal data and computes marginal probability distributions per category. However, MCQs do not have an inherent category 'a' or 'b', i... options can be permuted, so cannot compute such marginal probability distributions. Moreover, measuring LM similarity requires adjusting for chance agreement due to accuracy to avoid inflating similarity for high-accuracy models (Geirhos et al., 2020), For inter-annotator agreement metrics stemming from human survey studies - where there is no built-in concept of accuracy - and thus they are unsuitable for LM analysis without additional modification.
20

---

Great Models Think Alike and this Undermines AI Oversight
Should Cesp be defined similarly to Cohen's K or Scott's x=? When measuring similarity between LM Judges and human annotators, Thakur et al. (2024) recommend using Scott's 's т over Cohen's K, as it is better metric for inter-annotator agreement studies (Krippendorff, 2004). The two differ in how they compute Geap, Scott's 's л assumes that the two human raters are sampled from a common distribution, estimating it by averaging the marginal probabilities of the two raters. This is in contrast to Cohen's K, which assumes the given different marginal distributions for the two raters. In our case, we wish to account for chance agreement due to accuracies rather than the marginal distribution over classes. To see the relative comparison of how Cohen's x and Scott's π behave in our setting, we consider an example.
Suppose we have a binary classification problem, where both models always agree when they are both wrong as there is only one incorrect option. We now consider two pairs of models. Pair has accuracies 0.2, 0.8, whereas in pair 2, both models have accuracies 0.5. Intuitively, if both pairs were to have the same observed agreement, it would be more surprising if this happened for pair than pair 2. given the vast difference in their accuracy. In other words, models in pair 2 are more similar than expected for independent models with the given accuracies than pair 1. We want this to be reflected in our similarity metric.
For pair 1, Scott's πT, Cexp would be computed assuming joint accuracy of 0.2+0.88 = 0.5, and for pair 2 with the same joint accuracy 0.5+0.5 0.5, giving Cexp = 0.5 + (1 0.5)² = 0.5. Cohen's 's K of pair 2 would be computed as 0.5 0.5 + (1 - 0.5) (1- - 0.5) = 0.5 too. However, for Cohen's 's k of pair 1, Coxp = 0.2 0.8 + 0.8 0.2 = 0.32. This means for a fixed observed agreement Cabs. say 0.5, π = 0.5-0.5 = O for both models, and similarly x = 0 for pair 2. However, for T-0.5
2
1-0.22 0.264. Indeed, Scott's x would lead to think both pairs are equally similar, whereas k indicates pair 2 is more similar, beyond chance agreement arising due to accuracy. Thus k has the more desirable behavior.
pair 1, 0.5-0.32
More broadly, we do not wish to assume both models are drawn from a joint distribution, assigning them a common mean accuracy. However, Scott's ग does this, which makes sense when calculating reliability of surveys or measuring alignment between human and LLM judges. However, this does not make sense in our setting where we wish to adjust for chance agreement expected due to the two model's given accuracies. Hence, we choose to define Ceap similar to Cohen's where we retain the difference in the two model's ' accuracies when computing chance agreement.
Why not use Matthews Correlation Coefficient: or Matthews Correlation Coefficient of the binary vectors of sample-wise correctness for the two models (Chicco et al., 2021). However, it would be difficult to incorporate probabilistic information, and that models can be incorrect and still disagree by predicting different options. In other words, it suffers from the same issues error consistency, and we found it more difficult to extend.
We could take a completely different approach by computing the Pearson
Why not use regression analysis? We could have performed a multinomial regression using the probabilities of the first model to predict probabilities of the second model, using this predictability as a measure of similarity. However, it is unclear whether linear model would be enough. Ideally this prediction should also be contextualized on the input sample, but for this would need a model-based metric to obtain representation of the input sample. We chose to stick to more interpretable, closed-form metric.
Why not use divergence metries like KL or JSD? KL-divergence or Jensen-Shannon Distance (JSD) can measure the divergence between probability distributions assigned by models to the options with lucrative properties. Further, JSD is a valid distance metric, and normalized between 0 and 1. We could use the mean JSD over all questions as model similarity metric. However, higher-accuracy models are expected have lower JSD simply because they have more correct answers, i.e end up assigning more probability mass to correct options across samples. Retaining the coemmatioo-thhoretee properties of ISD while adjusting for chance agreement due to accuracy remains an interesting open problem.
conmattion-thorettie
Why not use JSD of the two distributions instead of overlap in computing CAPA?:
We could have plugged 1 1 JSD into Los in the Kp formula. It is also possible to define &eep by computing JSD between the two independent model distributions defined and subtracting from 1. However, JSD instead of probabilistic overlap is not intuitively interpretable, especially when divided by the possible excess agreement as in Kp- <ie. computes the expected agreement when sampling from both models based on the probability distribution they assign to the options. Intuitively, it gives us the fraction of times the two models would agree if we kept sampling predictions from these distributions infinitely.
Why a uniform prior? In the present work we make the assumption that the prior error distribution follows a uniform distribution. In the context of multiple choice tasks, where by design each answer should have equal distribution in the dataset, this is a valid and data grounded assumption. However, our metric is not limited to uniform prior assumption. In
21

---

Great Models Think Alike and this Undermines AI Oversight
principle, one easily adjust the agreement by chance calculation (eq. 10) by a different prior distribution assumption given the data observed, for example, a categorical distribution if the class distribution in the data is non-uniform.
A.5. Probabilistic versions of popular agreement metries
We now provide probabilistic versions of Cohen's K, Scott'ss n, and Fleiss' Ky, that the interested reader can contrast them with CAPA.
Probabilistic Cohen's One can obtain a probabilistic Cohen's 's x by computing Po as cobs- therefore accounting for the observed agreement based on model output probabilities. While P. = Ei= [ Pi (ci) T Er=1Pa(c))) P2(Ci)o where we compute the product of marginals for each class.
Probabilistic Scott's Similarly to Cohen's x. to compute the observed agreement probabilistically we compute the average product across probabilities for 2 models, meaning Po becomes cabs While adjust P. computation as follows: where we now compute the sum of the marginal probabilities
(i(c)== P. = -i= Pi + DDi
per class as we assume that both models have shared marginal distribution.
Probablistic Fleiss' Kappa (Rp): It extends the Gae-Cog formula to more than two models, where the observed and chance T-p agreement is computed across pairs of two in the set of models. Like ı it assumes chance predictions are sampled from a common combined distribution. While generally Fleiss' Kappa allows a partial random subset of annotators for each question, in our work we assume all models annotate all questions. Let M be the number of models, and |C*] be the number of classes. Let mas be the number of models that put sample in class i. Let Pa M(M-1) Σiec (m 1) be the proportion of observed pairwise agreements for each question. Cobs Gexp = -Eieo( EisIsM p=(()) )2
D EEeD Px. For the chance agreement,
M
Let M be the number of models, and |C|| be the number of classes. Let mas be the number of models that put sample æ in class i. Let Pa M(M-1) Eiec mas(miz: - 1) be the proportion of observed pairwise agreements for each question.
Cobs 1 [D] ExED Pa. For the chance agreement, Geap = Eiec( Eietens (2)
A.6. Theoretical bounds for CAPA
Bounds for 22 Compared to Geirhos et al. (2020) the resulting observed agreement is strictly greater than 0, as all probabilities are positive values, and strictly smaller than 1, as the sum of probability products is strictly smaller than the sum of probabilities:
<
<
(15)
obs
Theorem: If0 < a < 1 and 0 < b < 1, and b = 1, then a² + < <a+b
Proof:
a² +2 < a ab
a² - a + <
a(a 1) + b(b
<0
0 and b 6-1 < 0, therefore b(b - 1) < 0. Since a(a- - 1) < and b(b - 1) < O, their sum will also be negative a(a - 1) + b(b - 1) < O, this implies that indeed b2 +a b.
For < a < 1, > 0 and a - < 0, therefore, a(a- - 1) < 0. For 0 <
Bounds for capp The lower bound for cásp is when the first term approaches zero and the scaling fraction approach 0, thus resulting in deep = 0. The upper bound is maximized when both terms are maximized, but as the second term is the inverse of the first times a scaling factor, the maximum upper bound is (as pi P2 - 1, 1 p1) (1 -22) + 0), resulting in:
Ceep
(16)
22

---

Great Models Think Alike and this Undermines AT Oversight
Bounds for Kip The upper bound for Hy In particular, Kp will always be strictly smaller than but approaching it in the limit.
Theorem:
Given Kip = 1. Then by definition:
Proof:
2
obs
Coxp
Cexp
obs
exp
obs
However, as L < 1, Ki < 1.
Although the above implies that CAPA does not obtain 'perfect agreement' as originally defined by Cohen's k, we show that this is not a concern for our metric as (I) when model probability for the correct class approach Kip - and (2) using probabilities allows us capture observed agreement more precise level:
1. Theorem:
Given probabilities [a.b] and [c.d], where a, c - and conversely b, d + 0, Kp -
Proof:
La a c b.d
- 1 and +
obs +1
which confirms Ky + 1.
2. Geirhos et al. (2020) computes Cohs Cobss,, Cig where ei.j is the number of equal responses. As such, Cabss, is independent of the observed output probabilities. However, for a model pair with output probabilities [0.999.." 0.000..1] versus [0.8. 0.2] (assume the same for both models), we would like the first case to have a higher observed agreement than the second, but Geirhos al. (2020) fails capture this, while e does:
Thcorem: Given two probabilities [a.b] and [c.d] where a, b,b, 1, b c d and a > C, a > d, d, indicates that + b. b c c.c d d Proof:
a.a+b.b> be
a² (0
2a² 22- - 2x+2>>
2(a - () cc 1
(a (a-c c)((++)))>>0
as < (a-c)>0
as > d c 1, thus, = (a+c- 1) > 0,
therefore, a a- b.
d.d.
The lower bound for Kp is -1. particular, K2 will always be strictly greater than -1.
Theorem: Given Kp 2 11 and 0 < c&ep < and 2 1.
23

---

Great Models Think Alike and this Undermines AT Oversight
Proof:
-1
1
Coxp
2 obs
C 2 -(1
Coop
obs
+ 1 exe 2 0
c 2 2reap
minimal possible cº
0 (complete disagreement)
0 2 2Goxp
2 20e
0.5 2 Cxp
therefore, Kip 2 11 Even though, the theoretical lower bound for Kp - to achieve Kip = 11 in practice de must be O (both models perfectly oppose each other), leading that ca 2coxp 1, + coxp 0.5. As cotn is computed based on probabilities its value is Cabs < 1, therefore, the actual lower bound for rip > -.
Altogether, the bounds for CAPA are as follows:
-1 Kpp <1
(17)
A.7. CAPA comparison with other inter-rater metries
Numerical Example For simple mathematical example consider two models with 2 data samples with the following probability distributions, model [[0.9,0.1].[0.3 0.2]] and model 2 [[0.7,0.3].[066 0.4]]. The underlying ground truth index is [0,1]. For Cohen's 's k and Scott's 's π we treat this is example the probabilities to model 1= [A.A], model = [A, A] (these metrics do not take accuracy into account). The accuracy for both models 50%. In table 5 we report the computed similarity for each metric as well specify the exact computation values. As can be noted, all other metries suffer from the following limitations: (1) Cohen's and Scott's r treat the problem as a classification, as such both metries report that the similarity between models is 0.00, indicating no relationship Po Pe. (2) Probabilistic versions of the metrics slightly deviate from 0.00 however still undermine model similarity, (3) Error consistency over estimates model similarity by ignoring model output probabilities in its Cobs calculation. As such, only CAPA is able to accurately account for the observed sample level similarity across the two models.
binary classification with option A and B, converting
Table 5, Numerical Example
Metrie
Similarity
Computation
0.00
P. = 1.0, P. - 2 1.0
Probabilistic
0.01
P, 3 (0.9 -0.9-0..7 0.1 + 0.2.0..6.......... = 0.61
P. 0.9-0.8 2
0.7+0.6 + 0.140.2
0.30.1 = 0.605
π
0.00
Po 30=1. 1.00, P. - + 1.0
(
Probabilistic π
0.04
0.61
P.
0.5.0.8 + 0.7+08) ( (2 0.625
error consistency CAPA
1.00
Cubs 1.00, Gesp (1--05-+1-05511--000
0.5
0.21
20
(0.9 0000110----000000000000000000000
0.61
P1 - (0.7+00 0.55
0.55 0.55 + 22 1 0.5) = 0.51
Coxp
24

---

Great Models Think Alike and this Undermines AI Oversight
Simulation Experiments We design a simulation set-up to compare the 'behavior' of the above listed inter-rater metrics with our novel contribution CAPA. In particular, we limit the simulation to a binary classification prob- lem as standard metrics like Cohen's s k and Scott's ' x are ill-suited for multiple choice question settings. In total we investigate the performance of 5 metrics: Cohen's 's k Probabilistic, Scott's Probabilistic, Error consistency, JS- DS-Daitance and CAPA. We simulate N=10000 observa- tions for 2 models.
Similarity Reported by Metric
1.0
0.6
0.
0.4
0.2
CAPA [ouas)
Cohrn's proh
Scotla a probs
Error consistency
First, we investigate the metric behavior when both mod- els tend towards agreement and their errors are corre- lated. In particular, we set the accuracy of model to 90% and it always favors the 1st option (has high cal- ibration, 0.99), meaning the model is highly confident in it's predictions (e.g. single data point is [0.99, 0.01)). For the other model, model B, we iteratively increase it's accuracy by adjusting it's calibration from 0.01 to 0.99 for the first option, as such, making the models more similar artificially.
02
J8+1 Distance
botaa mndels al 90% sorarany
0.55 Probability of Correct Output for Second Model(p)
6.55
Figure 7. Metric comparison when two models tend towards agreement. For model A we set accuracy to 90% and calibration to 0.99 of the correct answer. For model B increase its calibration from 0.01 to 0.99 of the correct answer, so its answer distribution becomes increasingly similar model A. On y-axis we are plotting metric value on x-axis we are reporting p for model which, as the model becomes more calibrated, approaches accuracy of model A.
As can be noted in Fig. 7 all previous similarity metrics fail to capture the full range of increasing model similarity: (I) EC only captures model similarity up to when their accuracies are matched (both models reach 90% accuracy), yet fails to further distinguish between models that have high agreement [0.99, 0.01] and [0.99, 0.01] versus medium [0.99, 0.01] and [0.75, 0.25]; (2) while -haennon distance (computed as JS-distance) correctly captures the gradual increase in model similarity it overestimates it initially, this is due to the fact that the distance is computed from the average joint distribution; (3) the probabilistic counterparts of standard inter-rater metrics like Cohen's 's x and Scott's 's π (derivation details in App. A.5) are also ill suited. In particular, in the case of Cohen's 's k prob the class marginals (for computing the expected agreement) match the observed agreement, therefore leading to inter-rater agreement of 0 even with increasing model similarity, while in the case of Scott's æ prob we do observe change in similarity score due to it's assumption of joint marginal distribution, but also because of this assumption the metric overestimates the expected agreement leading to negative (opposing) inter-rater agreement, which is incorrect conclusion given the simulation data.
Second, we explore the metric behavior when both model accuracies are increasing yet errors remain random (uncorrelated), We again simulate 10'000 binary data points with uniform class distribution. For model A we set the accuracy to 90%, while for model we incrementally (100 steps) increase it's accuracy from 50% to 90% by randomly selecting data samples which are correct at each step. As such, the errors are uncorrelated between the two models. Results are reported in main paper in Sec. A.7.
Limitations of CAPA. In addition, we perform a simulation experiment when the two models become increasingly dissimilar. In this set up we change that the model B always prefers the second option. Thus, by iteratively increasing its calibration we obtain models that maximally differ in their probability distribution, e.g. [0.99, 0.01] and [0.01, 0.99] respectively. As such, also the accuracy of model B decreases overtime from random chance (50%) to 0.10 %, and we would like to obtain metric of -1. As it can be seen in Fig. 8, CAPA never reaches -1. Importantly, the same issue also can be observed for eттor consistency. This observation comes from the fact that both metrics use the original Cohen's 's k equation. As explained in Section A.6, Kp = - iff 2oep = 0.5. For probabilistic Cohen's 's K we see the same observation as in Fig. 7, the marginal probability computation is not suited for the given problem. Interestingly, probabilistic Scott's 's the only metric that approaches- -1. Whilst desired final outcome, Scott's æ overestimates model disagreement when model probabilities are independent, [0.99, 0.01] and [0.5, 0.5].
Possible solution for lower bound. In the context of the current work, the above limitation not an issue, as models are trained to maximize accuracy, hence, there will always be some level of agreement. However, CAPA would be used in
25

---

Great Models Think Alike and this Undermines AT Oversight
Similarity Reported by Metric
Similarity Reported by Metric
022
0.2
0.
-044
-0.1
CAPA (sris)
CAPA (ouns)
CAPA,b
Cchre's prob
0.6
Cohen's
Sootss peoh
Stotts prob
Errce consistency
monistimey
D8-Disi
18-Distauee
-10
-10
Probability of Correct Output for Second (odel(p)
Probability of Correct Output for Second Model(p)
Figure 8. Metric comparison when models tend towards disagree- ment (Read plot from right to left). We compare different metric values for two models in hinary setting. For model A, we set accuracy to 90% and calibration 0.99 (the model is highly confi- dent in its answers). For model B, incrementally increase its disagreement with model A by pushing its probability mass to the sccond option and increasing its calibration 0.99.
Figure 9, Metric comparison when models tend towards disagree- ment with adjusted CAPA. Replication of fig. with adjusted CAPA Kp, computation following eq. 18.
settings like preference judgments, we would advise to adjust the computation of cosp as described by Safak (2020):
cap
P
2 clep
ohs
Kp
(18)
obs ceep
where 2 mins computed as aa max(0.pp P1 + P2 1). negative domain, see Fig. 9. Now, the models become increasingly dissimilar Kp approaches -1.
This resolves the observed limitation of CAPA over the
26

---

Great Models Think Alike and this Undermines AT Oversight
B. LLM-as-a-Judge
In this section, we extend the LLM-as-a-judge experiments introduced in Section 3. First, we compare CAPA with the related concept of eπrоr consistency (Geirhos al., 2020), demonstrating its advantages in this context. We then present additional experiments analyze the quality and behavior of the judges, well as the performance of the evaluated models on the open-style MMLU-Pro benchmark.
To validate our findings, we provide detailed results from the statistical tests summarized in Table 7. Specifically, we conduct Shapiro- Wilk and Breusch-Pague tests to confirm that the assumptions of normality and homoscedasticity required for partial correlation and multiple regression analyses are satisfied.
Additionally, outline the experimental setup, including: (1) the filtering process for MMLU-Pro to obtain open-style questions only, (2) the methodology for free-form chain-of-thougle inference on this benchmark, and (3) the design of the LLM-as-a-judge evaluation framework. To ensure full reproducibility, we include all prompts and specify the language models used as judges and evaluated models at the end of this section.
B.1. Extended Multiple Regression Analysis with Model Size as Confounder
We extend the main paper regression analysis (Sec. 3) by also including model size as a possible confounder of judgment score. We standardize model size by removing the mean and scaling to unit variance, so the model size as a independent variable is on comparable scale as model similarity and accuracy. We remove outliers based on the fact that the studentized residuals > All outliers where in combination with model extended regression results are reported in table 6. We report that even when controlling for model accuracy and model size, our computed model similarity score remains a significant predictor of judgment score. Furthermore, whilst for some models model size is a significant predictor of judgment score, when controlling for model similarity and accuracy, the effect size is very small, around All in all, the additional results confirm that our metric accounts for model size as a possible confounder.
FacennnnlaaTB---l-M222222 152-135M-Instruct The
Table 6. Extended Multiple Regression Results
Judge
sim
acc
size
Qwen2.5-7B-It
0.49
0.68**
-0.02
Qwen2.5-32B-Ir
0.40
1.02
-0.03**
Qwen2.5-72B-Itt
0,42
1.23
-0.03**
Meta-Llama-331-8BBI1
0.93**
0.61**
-0.02
Meta--Llaaa-311-000
0.74
0.92
-0.03**
Llama-3.3-70B-II
0.61 **
1.03
-0,04**
gemma-2-90
2-9b-It
0.67
0.78
-0.02
2-2-----27b-Itt
0.62
0.77**
-0.02
Ministral-8B-11-24110
0.65
0.50**
-0.02
B.2, Comparison of Judge Scores for Our Similarity vs Error Consistency
In Figure 10 we compare the relationship of judgment scores on the filtered MMLU-Pro dataset using different similarity metrics. On the left, we use CAPA and on the right, we compare against the original error consistency of Geirhos et al. (2020). In both cases, we can see correlation between the judge scores and the similarity of the LLM-as-a-judge and the model being evaluated. However, the relationship for CAPA is stronger, as shown by mean Pearson r of 0.9 which is greater than the one of 0.85 if error consistency is used.
B.3. Evaluating Judge Scores Against Ground- -ruth
B.3.1. MCQ GROUND-TRUTH
Since we source MCQ evaluations from Huggingface OpenLLM Leaderboard throughout the paper, we use their default method to obtain probabilities across MCQ options. For MMLU-Pro and BBH they report the log-likelihood each option. We apply softmax to normalize these We checked this leads to calibrated predictions for base models and overconfident
27

---

Great Models Think Alike and this Undermines AT Oversight
Judgment Score
0.
Judge
Qoca25-322--
QwenZ 5-728-t
Meta Liama 3.1 1--5-
0.2
Mets Neta- 3.1-70B-it
1t
gammn-220044
grmme-3-27-it
0.0
0.1
0.8
0.5
06
0.1
0.2
0.3
0.6
0.7
Ours (Kp)
Error Consistency
Figure Judgment Scores vs CAPA and vs Error Consistency. We compare the relationship of judge scores on the filtered MMI.U-Pro to our improved error consistency and to the original version of Geirhos et al. (2020).
Judge Ensemble OSQ, Accuracy
00
0a
Model Family Type
SuolL322
Genmu-2
Lians-XXX
Pli-d
Falecn-3
Taned
Buse Modd
0.2
0.1
0.4
05
06
MCQ Accuracy
Figure 11. Accuracy of free-form responses compared with multiple-choice accuracy MMLU-Pro. The free-form responses rated using an ensemble of five capable LM judges. Each judge was given access to the original MMLU-Pro reference answers and their aggregated using majority voting.
decisions whether given response is correct
predictions for instruct models, consistent with prior observations about uncertainty of language models (OpenAI, 2024).
B.3.2. JUDGE ENSEMBLE WITH ACCESS TO REFERENCE ANSWERS
Since we are evaluating model responses given in open style on filtered MMLU-Pro questions using LM-as-a-judge, important to investigate whether the responses of the evaluated models are reasonable. ensure that qualitative differences between models of different sizes and families remain, compare their performance using free-form responses the multiple-choice accuracy the same set of questions. This is shown in Figure 11. Using the same question base for free-form and MCQ evaluation draws direct connection between functional similarity and the behavior of LLM-aasjudges. Focusing on a setting where we have access to ground-truth responses is important to accurately analyze the affinity biases of different LMs when used as evaluators.
Experimental Setup Every response is evaluated using an ensemble of five capable LMs used LLM-as-a-judee from range of different model families. The judge is given access to the question, the model's free-form response and all wrong option. Using this information,
MMLU-Pro reference options. For each option we indicate if it is the correct the judge has to decide whether the model's response is correct or wrong. The prompt can be seen in Prompt B. For every per-sample response, we aggregate the five binary decisions using majority voting. Since there are five judges and binary decision task, there are no ties. qualitative analysis has shown the high quality of this process
28

---

Great Models Think Alike and this Undermines AT Oversight
Judge Qucat TB-a
10
Qreadd 383
Qum22.-728-4
Judgment Score
genma 2. 2.16-X
0.6
genmas 3. 371-
Mirieral -ateern----------0
62
64
Judge Ensemble OSQ Accuracy
Figure 12. Judgment Scores compared with the ensemble judgment accuracy given access to reference answers. We compare the judgment scores of each judge using only their own knowledge and capabilities to the rating of a judge ensemble that has access to the ground-truth options. The latter is good proxy of the real correctness of responses.
in determining the correctness of responses. The judges used are gemma gemma-2-27b--a -2-27b-it, 5wen2.5-328-Instruee Qwen2. 5-7------------ Instruct, Llama- -aa Instrrut and Llama 3.3-70B--0 - (Gemma Team, 2024;
Qwen Team, 2025; Llama Team, 2024a;c).
Open-style and Multiple Choice Correlate As we can see in Figure 11, there is high alignment between the performance in MCQ style compared to free-form. For the majority of models, the ordering with MCQ accuracy and open-style accuracy is very similar. There is a consistent trend that performance on the more challenging open evaluation is approximately 5-10% lower. The exception is the instruction-ture models from the Qwen2.5 and Gemma-2 model families that performed particularly well when giving free-form responses. For all other model families, the instruction tuned and base models show similar performance.
B.3.3. JUDGE SCORE VALIDITY AGAINST REFERENCE-BANE ENSEMBLE
To evaluate the quality of different judges and to analyze their similarities and differences. we compare judge scores to the correctness assessments of the previously introduced ensemble of judges. The results are shown in Figure 12. As we can see, most models used as LLM-as-a-judge are able to correctly rank capable and less capable models.
Cappeelity-Dependent Affinity Effects Even if the ordinal ranking of evaluated models is mostly accurate, there is consistent trend that too many wrong responses are judged as being right. The exact behavior varies from fudmeudge to judge. Consider the small Llama-3. 1.133.11111 --BB-Innttinn responses as correct, even for models of low capability. more capable in identifying the wrong responses of low-capability models. However, as the evaluated LMs become stronger, it exhibits the same bias as the smaller Llama judge. This aligns with the findings of Section that LLM-assjudges show an affinity bias, because more capable models are also more similar Owen2. -
for instance: it has a consistent positivity bias and rates too many wrong 5-72B
ruct on the other hand appears to be much
B.4. Judge Preference in Chat Generation Tasks Against Similarity
Experimental Setup To evaluate judge bias in different context, we evaluated each of the 23 instruction-tunee models of the original experiment the benchmark AlpacaEval for free-form generation (Dubois et al., 2023), After generating responses for all models on the full set of AlpacaFarm prompts, we used the same nine LMs as judges as before to determine their preferences on all response pairs between the 253 model combinations. re-implementation of AlpacaEval 2.0 was used to determine preferences (Dubois et al., 2024). The resulting judge preferences were shuffled and used to compute Judgment Elo for each judge. total, there were 203,665 combinations of prompts and pairs of models to be judged.
Free-form Judge Preference Figure 13 shows the relation between model's Judgment Elo on Alpaca Eval and judge-model similarities using our proposed metric CAPA on MMLU-Pro. Since Alpaca Eval cannot be easily turned into an MCQ benchmark, as required by our similarity metric, using the same similarities on MMLU-Pro as before was inevitable. Despite the domain gap between the data used for the LLM-as-are aLudge-a-uudee experiments and CAPA, there a clear positive correlation between the two, indicating the existence of an affiniyy bias. This also provides initial evidence that similarity on a diverse
29

---

Great Models Think Alike and this Undermines AT Oversight
1903
1982
Judgment Eilo
1890
0.02
Qumn 2-82B-i-
Queas Qw..22.-728-4 p-0.3
Liumo 211 m00 -u
Lbura-8 8.5. 8-00-- 5-0.511
r-0.39
-B:1--0l1-1::--40000 o-0
02
Similarity (MMLU-Pro)
Figure 13. Elo of LMs on AlpacaEval 2.0 as judged by different LLM-as-judges in relation with our similarity metric on MMLU-
Pro. We compare the Elo obtained by using different judges for binary preference pairs Alpaca Eval to their similarity the filtered MMLU-Pro dataset.
MCQ benchmark might transfer as predictor across tasks.
B.5. Judge Score Against Similarity per Category
MMLU-Pro is very comprehensive MCQ benchmark evaluate model's common knowledge. By filtering some of the questions ensure that only those that can be answered in free-form remain, may have unintentionally introduced biases such as Simpson's s paradox that affect the preference bias we observed. To ensure that this is not the case, we group the questions for free-form Judgment Scores and MCQ similarity by their original category in Figure 14.
For each category, we only used the judgments and MCQ responses of the questions belonging to that category to compute the Judgment Scores and CAPA metric for model similarity. As in Figure 3, each line corresponds to one judge model. The same positive correlation between CAPA and Judgment Score be seen for every category the original, aggregated Filtered MMLU-Pro benchmark. This indicates the existence of an affinity bias of ALM-as-a-judg- irrespective of the category.
For better readability, we do not report the Pearson in individual legends for each subplot. Apart from one exception, all Pearson correlation coefficients of the LLM-as-judge each category were above 0.75 and statistically significant (p=0.01). The only exception google gemma-2m222 -2 9b-it the category "engineering" with 0.28 and p 0.211.
B.6. Statistical Testing
This section provides detailed statistical validation of the affinity bias observed in Section We confirm that judge- similarity correlates with judgment scores after controlling for MCQ ground-truth accuracy, using partial correlation and multiple regression. Additionally, we verify the statistical assumptions (normality, homoscedasticity) required for these tests across all nine judges.
B.6.1. QUANTIPYING CORRELATION STRENGTH USING PARTIAL CORRELATION
We compute partial correlations between judge scores and judge-model similarity while controlling for ground-truth accuracy. All judges show statistically significant positive correlations (Table B.6.1), with coefficients ranging from 0.35 am.1 3.te-333-7000 3-3--70---M 1--70BIInstrueee Instruct) to 0.65 3 1----- smaller judges from the same gemma-2 family (genma- 2-27b-1- exhibit moderate correlations (r 0.42 and r = 0.43). All p-values remain significant (p < 0.05), with the most robust results for the larger gemma judge (p 0.00001).
The strongest correlations occur for 27b-it it and genma 2-9b 9b it), while larger Qwen2.5 judges
30

---

Great Models Think Alike and this Undermines AT Oversight
Riology
Thasineas
Chemistry
0.00
0.23
Comp ulerr Science
Exuoonie
Engineciug
Judcaust
0.00
6:26
Health
Hlistory
1.0
Judement
0.00
0.20
Math
Other
Philosopky
1.00
Jubazoci
0.00
Physics
Prychology
1.00
Judge
0.76
Quwn2 5 1-522B Tratrut
Iadgment
[ L-1-bbaoooo
Eluns 1.718 Instrett
28-7UB-lasteut
gumand Masi:
0.00
0.0
0.1
0.2
0.3
0.3
05
08
0.1
0.3
0.5
0.
271 SiliiitniBBB-tirrrr....
Suuilarit
Similarity
Figure 14. Judgment Score Relation with Model Similarity per Category. This plot shows the same experiment as Figure except that all questions were grouped by MMLU-Pro category. Each line regression model fit between judgment and similarity scores computed between model and judge pairs. Each point represents a single pair, and indicates that both, the model and the judge, come from same model family.
31

---

Great Models Think Alike and this Undermines AT Oversight
Detailed Partial Correlation Results
Judge
CI 95%
p-val
Qwen2.5-7B-Inss
38
0.60043
[0.34 0.77]
0,00009
38
0.43376
[0.13 0.66]
0.00732
Bwenn.55-2B-Innsruue
38
0.42353
[0.12 0.66]
0.00900
Meta-Llama-3, --1-:aaammaa..-----n----- --8tttrruuettt
38
0.65172
[0.42 0.81]
0.00001
I-sttuuctttt
38
0.44770
[0.14 0.67]
0.00546
Llama-3.3-70B--nsttre
38
0.34882
[0.03 0.6
0.03435
gemma-2-9b-it
38
0.64639
[0,41 0.8
0,00002
gemma-2-27b-it
0.64808
[0.41 0.8
0.00001
Ministral-8B
Soristral-8B-Instuutt---- 18--nstrrut-2410
39
0.59745
[0.34 0.77]
0.00007
B.6.2. MULTIPLE REGRESSION
We perform multiple regression analysis with judgment scores as the dependent variable, using model similarity and ground-truth accuracy from the filtered set of MCQ questions as independent variables. Key results across all judges include:
Coefficient Significance: Both similarity and accuracy show statistically significant effects (p < 0.05) for all judges. The similarity coefficients range from / = 0.35 for the large Llama-3, S-70BInnsruuct to B 1.15 for the smaller
while accuracy coefficients span / = 0.43 for the small Ministral- Bin-aB--Innnttttteeeeee 2410 to B = 1.04 for the large Qwen2. 5-72B- Instruct.
--88-B-----11-------- Inst
Model Fit: All regressions achieve high explanatory power with adjusted R² values between 0.87 (Ministral-9B) and 0.92 (gemma- 2 2-9b-it).
Assumption Verification:
Normality:
Residuals are normally distributed (Shapiro-Wilk > 0.05) for 7 of 9 judges. Exceptions: (p = 0.002) and Ministral 8B (p = 0.012).
-ooooxcedasticit:: All models satisfy constant variance assumptions (Breusch-Pagan > 0.05).
For example, the Qwen2. 7B- InB-Inntu judge model shows:
Significant positive effects for both similarity (3 = 0.59, < 0.001) and accuracy (3 00.51, < 0.001)
Strong model (R² = 0.91, (2,35)) = 182.9, 2.95 x 10-19)
Normally distributed residuals (Shapiro- Wilk p = 0.690)
Full regression outputs for judges are provided in the Tables below. We present detailed regression results for each judge model. Each judge's statistical analysis includes three components: (1) Test summary, (2) Coefficient estimates, and (3) Diagnostic statistics. The consistent significance of similarity coefficients confirms that affinity bias persists even when controlling for actual model capability.
Judge: Qwen2 5---5--BB--- Instruttt (Qwen Team, 2025)
Model:
OLS
Adj. R-squared:
0.908
Dependent Variable:
scores
AIC:
-134.3091
Date:
2025-01-30 11:42
BIC:
-129,3963
No. Observations:
38
Log Lig-Liili:oood:
70.155
Df Model:
2
F-statistic:
182.9
Df Residuals:
35
Prob (F-statistic):
1.95e-19
R R-squared:
0.913
Scale:
0.0015837
Coef.
Std.Err
P:
95% CI
Intercept
0.092
0.019
4.885
0.000
0.054
0.131
similarity
0.586
0.132
4.442
0.000
0.318
0.853
accuracy
0.506
0.098
5.185
0.000
0.308
0.704
32

---

Great Models Think Alike and this Undermines AT Oversight
Omnibus: 2.363
Durbin Watson:
2.097
Ommnibus::
0.307
Jarque-Bera (JB):
1.400
Skew:
-0.437
Proh(JB):
0.496
Kurtosis:
3.348
Condition No.: 27
Normality & Homoscedasticity; Shapiro- Wilk Test for Normality: normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 0.456 (p-value: 0.796), F-value: 0.213 (p-value: 0.809). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
Statistic=0.979, (p-value=0.690), Residuals are likely
Judge: Swoo2.5-32B-Insteae (Qwen Team, 2025)
Model:
OLS
Adj. R-squared:
0,907
Dependent Variable:
scores
AIC:
-114.8801
Date:
2025-01-30 11:42
BIC:
-109.8674
No. Observations:
Log-Likelihond:
60.390
Df Model:
F-statistic:
182.2
Df Residuals:
Prob (F-statistic):
3. 3.14c-19
R-squared:
0.912
Scale:
0.0026477
Coef.
Std.Frr
Pg Pa-t-
95% CT
Intercept
0.045
0.028
1.620
0.114
-0.011
0.101
similarity
0.414
0.145
2.848
0.007
0.119
0.709
accuracy
0.861
0.132
6.513
0.000
0.593
1.129
Omnibus:
0,227
Durbin-Watson: Watson:
2.052
Prob(Omnibus):
0.893
Jarque-Bera (JB):
0.411
Skew:
0.127
Prob(IB):
0.814
Kurtosis:
2.558
Condition No.:
25
Normality & Homoscedastieity; Shapiro- Wilk Test for Normality:
Statistic= (p-valuc=0.965), Residuals are likely
normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 3.097 (p-value: 0.213), F-value: 1.553 (p-value: 0.226). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
Judge:
Qwon2 5-72B-Innsrrue (Qwen Team, 2025)
Model:
OLS
Adj. R-squared:
0.913
Dependent Variable:
scores
AIC:
-103.8097
Date:
2025-01-30 11:42
BIC:
-98.8969
No. Observations:
38
Log-Likelihood:
54.905
Df Model:
2
F-statistic:
195.4
Df Residuals:
35
Prob (F-statistic):
1.03c- 19
R-squared:
0.918
Scale:
0.0035339
Coef.
Std.Err
P
95% CI
Intercept
0.064
0.032
2.038
0.049
0.000
0.128
similarity
0.474
0.171
2.766
0.009
0.126
0.822
accuracy
1.043
0.156
6.702
0.000
0.727
1.359
Omnibus:
0.139
Durhin- Watson:
1.776
Prob :
0.933
Jarque-Bes Bera (JB):
0.286
Skew:
-0.124
Prob(JB):
0.867
Kurtosis: 2.655
Condition No.: 25
Normality & Homoscedasticity; Shapiro-Win Wilk Test for Normality: Statistic= 9899, (p-value=0.968)) Residuals are likely normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 3.562 (p-value: 0.168), F-value: 1.810 (p-value: 079)... No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity
33

---

Great Models Think Alike and this Undermines AT Oversight
met).
Judge: Veta--Llama-331-88--111
(Llama Team, 2024a)
Model:
OLS
Adj. R-squared:
0.881
Dependent Variable:
scores
AIC:
-114.9610
Date:
2025-01-30 11:42
BIC:
-110.0482
No. Observations:
38
Log-Likelihood:
60.481
Df Model:
2
F-statistic:
138.6
Df Residuals:
Prob (F-statistic):
2.34c-17 17
R-squared:
0.888
Scale:
0.0026351
Coef.
Std.Err
Pi-t-
95% CT
Intercept
0.327
0.023
14,309
0.000
0.281
0.374
similarity
1.149
0.226
5.083
0.000
0.690
1.608
accuracy
0.532
0.106
5.035
0.000
0.317
0.746
Omnibus:
23.344
Durbin Watson:
2.611
Prob(Omnibuse
0.000
Jarque-Bec Bera (JB);
52.336
Skew: -1.424
Proh(IB):
0.000
Kurtosis: 7.994
Condition No.:
Normality & Homoscedasticityy Shapiro- Wilk Test for Normality: Statistic=0.892. (p-value=0.002)) Residuals are likely not normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 4.186 (p-value: 0. 0.123), F-value: 2.166 (p-value: 0.130). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
- -nst (Llama Team, 2024a)
Model:
OLS
Adj. R-squared:
0,903
Dependent Variable:
scores
AIC:
-103.3457
Date:
2025-01-30 11:42
BIC:
-98.4329
No. Observations:
38
Log-Likelihod::
54.673
Df Model:
F-statistic:
172.4
Df Residuals:
Prob (F-statistic):
7.60c-19 19
R-squared:
0.908
Scale:
0.0035773
Coef.
Std.Err
Pi-t
95% CI
Intercept
0.140
0.031
4.533
0.000
0.077
0.202
similarity
0.615
0.208
2.962
0.005
0.193
1.036
accuracy
0.917
0.157
5.827
0.000
0.598
1.237
Omnibus:
4.624
Durbin-Watson. Watson:
1.984
Prob(Omnibus):
0.099
Jarque-Bera (JB):
3.237
Skew: -0.616
Prob(JB):
0.198
Kurtosis: 3.724
Condition No.:
28
Normality & Homoscedasticity: Shapiro-Wii Wilk Test for Normality: Statistic=0.974 (p-value=0.502), Residuals are likely normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 2.975 (p-value: 0.226), F-value: 1.487 (p-value: 0.240). No evidence of heteroscedassiccty (the residuals have constant variance, homoscedasticity met).
34

---

Great Models Think Alike and this Undermines AT Oversight
Judge:
1,1 am1ama-3... 3.-ma--..-708--- 33-708- Inst (Llama Team, 2024c)
Model:
OLS
Adj. R-squared:
0.884
Dependent Variable:
scores
AIC:
-94.3204
Date:
2025-01-30 11:42
BIC:
-89,4077
No. Observations:
38
Log- Log-Likelihood:
50.160
Df Model:
2
F-statistic:
142.6
Df Residuals:
35
Prob (F-statistic):
1.49e-17
R-squared:
0.891
Scale:
0.0045363
Coef.
Std.Err
P
95%
Intercept
0.162
0.036
4.544
0.000
0.089
0.234
similarity
0.487
0.221
2.202
0.034
0.038
0.935
accuracy
1,022
0.177
5.770
0.000
0.662
1.381
Omnibus:
4.168
Durbin- Watson: 1.898
Prob (rob(Omnibus):
0.124
Jarque-Bor Bera (JB);
2.830
Skew:
-0.584
Prob(JB):
0.243
Kurtosis: 3.652
Condition No.: 27
Normality Homoscedasticity: Shapiro-Win Wilk Test for Normality: Statistic=0.977 9778, (p-value=0.642), Residuals are likely
normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 2.500 (p-value: 0.287), F-value: 1.232 (p-value: 0.304). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
Judge:
gemma-22 22-9--i-- it (Gemma Team, 2024)
Model:
OLS
Adj. R-squared:
0.917
Dependent Variable:
scores
AIC:
-122.8074
Date:
2025-01-30 11:42
BIC:
-117.8946
No. Observations:
38
Log-Likelihonn
64.404
Df Model:
2
F-statistic:
206.0
Df Residuals:
35
Prob (F-statistic):
4.36e-20
R-squared:
0.922
Scale:
0.0021435
Coef.
Std.Err
P --
95%
Intercept
0.134
0.021
6.356
0.000
0.091
0.177
similarity
0.763
0.152
5,012
0.000
0.454
1.072
accuracy
0.688
0.097
7.129
0.000
0.492
0.884
Omnibus:
6.751
Durbin- Watson:
1.901
Prob(Omnibus)
0.034
Jarque-Bera (JB):
5.969
Skew:
-0.621
Prob(JB):
0.051
Kurtosis: 4.492
Condition No.: 25
Normality Homoscedasticity: Shapiro-Wii Wilk Test for Normality:
Statistic=0.955 (p-value=0.179) Residuals are likely
normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 2.550 (p-value: 0.279). F-value: 1.259 (p-value: 0.297). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
Judge: gemma-2-2777 (Gemma Team, 2024)
Model:
OLS
Adj. R-squared:
0.919
Dependent Variable:
scores
AIC:
-121.3075
Date:
2025-01-30 11:42
BIC:
-116,3947
No. Observations:
38
Log-Likelihood:
63.654
Df Model:
2
F-statistic:
212.0
Df Residuals:
35
Prob (F-statistic):
2.76e-20
R-squared:
0.924
Scale:
0.0022298
35

---

Great Models Think Alike and this Undermines AT Oversight
Coef.
Std.Err
Pg --
95%
Intercept similarity 0.705
0.191
0.022
8.655
0.000
0.146
0.235
0,140
5.034
0.000
0.421
0,989
accuracy
0.677
0.106
6.407
0.000
0.462
0.892
Omnibus:
8.920
Durbin-Watson.
1.882
Prob(Omnibus))
0.012
Carque-Berl (JB);
8.659
Skew: -0.791
Prob(JB):
0.013
Kurtosis: 4.722
Condition No.: 24
Normality & Homoscedasticity; Shapiro-Wil Wilk Test for Normality: Statistic=0.9k (p-value=0.0622, Residuals likely normally distributed. Breusch-Pagan test for homoscedasticity: Lagrange Multiplier statistic: 1.645 (p-value: 0.439), F-value: 0.792 (p-value: 0.461). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
Judge:
(Mistral AI, 2024)
Model:
OLS
Adj. R-squared:
0,868
Dependent Variable:
scores
AIC:
116.80866
Date:
2025-01-30 11:42
BIC:
-141.8179
No. Observations:
39
Log-Likelihood:
76.404
Df Model:
F-statistic:
125.6
Df Residuals:
Prob (F-statistic):
5.80c-15
R R-squared:
0.875
Scale:
0.0012608
Coef.
Std.Err
Pi--
95% CI
Intercept
0.117
0.016
7.187
0.000
0.084
0.150
similarity
0.825
0.185
4,470
0.000
0.451
1.199
accuracy
0.432
0.063
6.803
0.000
0.303
0.560
Omnibus:
9.707
Durhin-
Durbin-Watsom Watson:
1.766
Prob(Omnibus):
0.008
Jarque-Bera (JB):
8.755
Skew:
-0.999
Prob(JB):
0.013
Kurtosis:
4.180
Condition No.: 36
Normality & Homooeedastieity:SShpii. -iilk Test for Normality: Statistic=O. 925, (p-value=0.012). Residuals are likely normally distributed. Breusch- Pagann test for homoscedasticity: Lagrange Multiplier statistic: 1.270 (p-value: 0.530), F-value: 0.606 (p-value: 0.551). No evidence of heteroscedasticity (the residuals have constant variance, homoscedasticity met).
B.7. Experimental Setup for Filtering MMLU-Pro
We evaluate models and judges set of questions that can be answered as MCQ as well as in open-style without access to reference options. This benchmark is obtained by using the filtering process proposed by Myrzakhan et al. (2024) on MMLU-Pro, whereas it was originally used to filter MMLU (Hendrycks et al., 2021b; Wang et al., 2024). Every question evaluated twice using a Owen-2 5- ---BBIInnttuc: LM: first, is judged in a binary way whether is possible to answer the question without access to the MCQ options. the second iteration, the judge gives fine-grainde confidence score. If either the binary decision is positive the confidence is above threshold, the question becomes part of filtered benchmark. After this filtering process, 8707 of the original 12032 questions remain. The detailed prompts are described in Prompts 1.1 and 1.1.
B.8. Experimental Setup to Perform Free-Form Inference on Filtered MMLU-Pro
To obtain the per-sample responses of every model on the filtered MMLU-Pro benchmark, we evaluate them using a custom task on the LM Evaluation Harness (Gao et al., 2023). Whereas the MCQ results from the Open LLM Leaderboard were generated using 5-shot evaluation without chain-of -f-thugght (CoT) prompt, we included CoTs when performing free-form inference (Myrzakhan al., 2024). This was necessary to ensure sufficient instruction following and response quality even
36

---

Great Models Think Alike and this Undermines AT Oversight
for small base models, because free-form generation is more challenging than MCQ evaluation, where access reference answers is given.
We modified every 5-shot CoT prompt by removing the answer options from the end of the question and replacing every reference to them in the CoT with the corresponding answer text. An example of this process is shown in Prompts B. 1.3 and 1....4. Our benchmark is implemented as a task for the LM Eval Harness (Gao et al., 2023). Every CoT response generated until a stop condition met. The final response that is judged is extracted using regex matching. We use vLLM the backend for the LM Eval Hamess (Kwon et al., 2023). Even for instruction-tume turrn were omitted because for the majority of LMs inspected
models, the options
--appl saappyychhat hy-chhhtttemnn a and -- www.wwhot.teeee the quality of responses decreased slightly to severely. However, we did not thoroughly investigate whether this is the case for every single model.
B.9. Experimental Setup for LLM--as-a-Judge on Filtered MMLU-Pro
This section describes the setup of the experiment for Figure 3. On the x-axis we show the similarity between our LLM-as-a- judge and the LM that is being evaluated, whereas the y-axis shows how the given responses of that model were rated by the same judge. The list of judges is shown in Table 7 and the pool of models evaluated can be seen in Table 8.
For the computation of similarities, provided on huggingface.co. The set of responses is filtered to include only those questions that were rated as answerable in open-style without access to the reference options, as previously described in Section B.7. Using the logarithmic probabilities of the models for the answer options of this set of questions, we compute CAPA and other similarities. In addition, for each model-judge pair data samples where the ground truth option differed were excluded from the final analysis, for the final sample count per pair see from table ?? table ??.
the logs of the official evaluation runs of Myrzakhan et al. (2024) that
Next, the judgment scores are obtained by prompting each LL LLM--a----judee to decide whether given response to question is correct or not. To mimic more common, but ungrounded settings for automatic AI evaluation, such as Arena-hard-aae AlpacaEval 2.0, we do not provide the judge with access to ground-trutt responses or MCQ answer options (Li et al., 2024b; Dubois et al., 2024). Since ground-truth responses for each question are available, of different judges and determine if there is any unfair preference. The prompt given to the judge shown Section B. 1.11 Each final decision was given token "0" (incorrect) or "1" (correct). Instruction-following used as LLM-as-a aaa--a----uuge, so the amount of discarded samples due to invalid responses is negligible. Finally, the Judge Score of an evaluated model is computed by averaging the judge decisions across the set of questions.
or
possible to analyze the affinity bias
exceptional for the models
B.10. List of Judges and Evaluated Language Models
Our judge preference experiments were performed using nine high-capability, open-weight models from four different model families. The models that represent the current state-of-the-ar of open-weight language models from very small up to models with 72 billion parameters. Whereas the judges are instruction-tuned the list of evaluated models contains base models as well.
Whenever possible, we evaluated both the base model and the instruction-tuned model for every combination of size and model family. Sometimes this was not possible, because the base model's weights were not available on huggingface, evaluations on the Open LLM Leaderboard v2 were not provided or the LM consistently crashed in vL.LM when performing inference (Myrzakhan et al., 2024; Kwon et al., 2023). The list below shows all models that are part of our experiments.
37

---

Great Models Think Alike and this Undermines AT Oversight
Table LMs used as LLM--a--aJudge
Judge Model Name
google/gemmn--2-33 googlee/gemmaa2222bbi
(Gemma Team, 2024)
(Gemma Team, 2024)
Dwen//Qren225577--nttruuu
(Qwen Team, 2025)
(Qwen Team, 2025)
Qven//phen2255778-1nstttm
(Qwen Team, 2025)
(Llama Team, 2024a)
(Llama Team, 2024a)
nate-1laamm/iiene33333771--11--1
(Llama Team, 2024c)
(Mistral AI, 2024)
Table 8. LMs Evaluated on the Filtered MMLU-Pro Benchmark
Model Name
Base Models
Instruction-tuned Models
Gemma-2 Family (Gemma Team, 2024)
google/gemma
- -2 2-2b
google/gemma -2-2b-it
google/gemma-2
google/gemma- -t----mmmma------------
google/gemma -2-27b
google/gemma -2-21m-------------------
SmolLM2 Family (Allal et 2024))
2024)
Bhrggingga.ce133/mmLL1111111111
Llama 3.1/3.2/3.3 Model Family (amnlyy(llammmaaa22200444333
2024,;1;2
meta--llamm//ettaa-llmm-----1111
3
meta- Beta--1aama/L23aa-32--11
meta-llamas 11amammmmmaaaa..222222
met Leceaa1laaaam//2amm3333------n
Phi-4 Family MMirosooft
microsoft/ppii-4
Qwen2.5 Family (Qwen Team, 2025)
Qwen/Qwen2.5 .-0..
Qwen/Qwen2 http://wen...-..5------tttttt
Qwen/Qwen2.55
Qwen/Qwen2 .5-..55-nnstruu
Twen/Qwen2.5-4
#wen//ouen2.5-77-Inntttm
Qwen BeennQwen2.5-111
Swent/guen22.5514-tnnssm
Qwen/Owen2.5-2 5-
Cwen:/owen22--332-Innnnte Qveat//wen22.5572--nnnttttt
Qwen/Owen2.5- 5-72B
Falcon-3 Model Family
InnhnnalggyImmovatioon Institute,
Insttuut 2024)
tiinaa/Fallcos3----Inssess
ciluee//Falcoo3----BBee
ciluee/rallco33-----ntticec
tiluse/Falcon3 10B Base
silue//aalcon3-11-Instttuu
38

---

Great Models Think Alike and this Undermines AT Oversight
B.11. Prompts
B.11.1. LM-JUDGE PROMPT WITHOUT REFERENCE ANSWER
Prompt for Free-Form Evaluation
Your task is judge whether the given response to question is correct or not. You are only given question and the response you are judging.
Possible judgments:
"O": The response is incorrect.
"1": The response correct.
Question: "[Insert the question here]" Response: "[Insert the response here]"
To the best of your knowledge: Does the provided response answer the question correctly? This is part of an automated evaluation process,
therefore you must only output single word: "0" or "1" Do not justify your decision.
Evaluation (0/1):
1.2. LM-JUDGE PROMPT WITH MCQ OPTIONS
Prompt for Free-Form Evaluation with Access MCQ Reference Options
Your task is judge whether the given response to question is correct or not. You are given question, ground truth response, incorrect options and the response you are judging.
Possible judgments:
"0": The response is incorrect.
does not match the ground-truth answer is more similar to any of the incorrect options than to the ground-truth answer.
"1": The response is correct. It matches the ground-truth.
Question: "[Insert the question here]
Ground truth: "[Insert the ground-truth option here]" Incorrect option (1): "[Insert the 1st wrong option here]
Incorrect option (9): "[Insert the 9th wrong option here] Response: "[Insert the response here]"
the best your knowledge: Does the provided response answer the question correctly, taking the ground-truth and wrong answer options into account? This is part of an automated evaluation process, therefore you must only output single word: "0" or "1", Do not justify your decision.
Evaluation (0/1):
B. 1.3. ORIGINAL. MCQ COT PROMPT
We describe how an original MCQ prompt on MMLU-Pro transformed into an open-style prompt for free-form inference without access to the reference options. The original chain-of-thought (CoT) prompt consists of general information about
39

---

Great Models Think Alike and this Undermines AT Oversight
the task, few-shot list of questions-answer pairs and finally the actual question that is to be solved.
Each question is preceded by the keyword "Question:", followed by the question text and the list of answer options. Every option text is marked with a letter. Next, a reference chain-of- of-thoughtt is given after the key-phrase "Answer: Let's think step by step" to provide an in-context example on how to solve related questions. This CoT can include references to the answer options. The CoT answer ends with the key-phrase "The answer is (X)" where "X" is the letter of the correct option. The phrase nudges the evaluated LM answer in the same way, allowing extract the final response using regex matching. The number of in-context examples depends the --nuum_ fewshot parameter. In our experiment, for reasons of brevity, only a single one is part of the example prompt below. Finally, the phrase that starts a CoT is repeated right before the model's response.
five examples, but
We automatically transform these MCQ into OSQ CoT prompts. The general information slightly adjusted to indicate the type of task. All key-phrases remain the same. We completely omit the MCQ options at the end behind the question. Any reference to option in the chain-of-thought is replaced with the option text itself e.g. "(G)" is replaced with the corresponding "(The second and third pharyngeal arches)". This includes the final response: "The answer is (XYZ).". Our experiments have shown that even the smallest models evaluated are able to follow these instructions and provide free-form responses that can be automatically extracted the vast majority of cases.
40

---

Great Models Think Alike and this Undermines AT Oversight
Few-shot CoT MCQ Prompt
The following are multiple choice questions (with answers) about health.
Think step by step and then finish your answer with the answer is (X) Where is the correct letter choice.
Question: What is the embryological origin of the hyoid bone?
Options:
A. The third and fourth pharyngeal arches
B. The fourth pharyngeal arch
C. The third pharyngeal arch
D. The second pharyngeal arch
E. The second, third and fourth pharyngeal arches
F. The first pharyngeal arch
G. The second and third pharyngeal arches
Н. The first and third pharyngeal arches
I. The first, second and third pharyngeal arches
J. The first and second pharyngeal arches
Answer: Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let's s solve this problem step by step. The hyoid bone, which is also known as the hycoid, is small U-shaped bone located in the anterior neck. In its resting position, it lies between the base of the mandible and the third cervical vertebrae. We know that the second and the third pharyngeal arches give rise to the horns of the hyoid bone; therefore, the embryological origin of the hyoid bone are the second and the third pharyngeal arches this information is covered in option (G). Therefore, we conclude that (G) must be the correct answer. The answer is (G)
Question:
Question: Which disease do polyomaviruses predominantly cause?
Options:
A. Tumours
В. Brain pathology
C. No disease at all
D. Kidney infections
Answer: Let's think step by step.

---

Great Models Think Alike and this Undermines AT Oversight
B. 1.4. OPEN-STYLE COT PROMPT
Few-shot CoT OSQ Prompt
The following are multiple choice questions (with answers) about health. Think step by step and then finish your answer with the answer is (X) where X is the correct letter choice.
Question: What is the embryological origin of the hyoid bone?
Answer: Let's think step by step. We refer to Wikipedia articles on anatomy for help. Let's solve this problem step by step. The hyoid bone, which is also known as the hyeoid, is small U-shaped bone located in the anterior neck. In its resting position, it 1ies between the base of the mandible and the third cervical vertebrae. We know that the second and the third pharyngeal arches give rise the horns the hyoid bone; therefore, the embryological origin of the hyoid bone third pharyngeal arches/this information is covered in option (The second and third pharyngeal arches) Therefore, we conclude that (The second and third pharyngeal arches) must be the correct answer. The answer is (The second and third pharyngeal arches)
the second and the
Question:
Question:
Which disease do polyomaviruses predominantly cause?
Answer: Let's think step by step.
These are the two prompts used for coarse and fine-grained filtering to get the OSQ version of MMLU-Pro. They almost exactly match the original ones provided by Myrzakhan al. (2024), but we performed minimal adjustments to make them more suitable to MMLU-Pro.
42

---

Great Models Think Alike and this Undermines AT Oversight
B.1 1.5. COARSE FILTERING PROMPT
Coarse Prompt
Your task is review series of multiple-choice questions and evaluate their ability to be answered without the provided answer choices.
For questions that begin with an incomplete sentence (e.g., "During swallowing, ..."), use your knowledge to attempt to complete the sentence accurately. For direct questions that ask for specific information or identification (e.g., "Which the following structures part of the small intestine?"), assess whether the question is formulated clearly enough that an informed answer can be given without seeing the multiple-cheice options. For mathematical or analytical questions (e.g., "Find all cosets of the subgroup 4z of 2 determine if the question provides enough context and information for solution to formulated without additional options.
Please follow this format for your evaluation:
QUESTION: [Insert the question here]
VERDICT:
Respond with "YES" if the question is clear and can be directly answered based its content alone, or "NO" if it relies the answer choices to be understood or answered. Your response should include only the verdict without any justification or reasoning.
43

---

Great Models Think Alike and this Undermines AT Oversight
B. 1.6. FINE-GRAINED FILTERING PROMPT
sine-grained Prompt
You will assign a numerical score from based on how confidently it be answered without the choices. The scoring criteria as follows:
1: The question is entirely dependent on its choices for an answer, making it impossible to answer without them. Example: Which of the following statements is correct.'
10: The questien can be easily and confidently answered based solely the question stem, without any need to refer to the provided options. Example: What is the first law of thermodynamics in physics?'
Intermediate Scores:
2-4: The question stem gives very little information and is highly reliant on the choices forcontext. Example: Which of these is prime number?' 'The perspective on sustainability resulted from growth models that analysed the carrying capacity of the planet, overall concluding that the finite capacity of the earth and ---_. generations could reduce quality of life for future generations.'
and ---- by current and past
5: The question provides some context or information, that gives moderate possibility to answer the question. Example: 'Which of the following best describes the structure that collects urine in the body?'
The question provides a good amount context or information, that gives a moderate possibility to answer the question. Example: 'Statement I A factor group of non-Abelian group is non-Abelian. Statement If is a normal subgroup of и and H is normal subgroup of G, then K is normal subgroup of G.'
7: The question provides a good amount of context or information, that gives a high possibility to answer the question. Example: 'The element (4, 2) of 7_12 7.8 has order'
8-9: The question provides good amount of context information, that gives a high possibility to answer the question. Example: 'A "dished face" profile is often associated with'
ONLY GIVE THE VALUE BETWEEN 1-10 AS YOUR ANSWER. DO NOT INCLUDE ANY OTHER INFORMATION IN YOUR RESPONSE.
44

---

Great Models Think Alike and this Undermines AI Oversight
C. Weak-to-Strong Training
C.1. Setup
Table 9. Datasets, Wcak Models and Strong Models Used in the Weak to Strong Experiments.
Models
Datasets
Weak Models
sciq (Welbl et al., 2017)
google/gemme 2-2b (Gemma Team, 2024)
anli-r2 (Nie et al., 2020)
Owen/Qwen2.5 5-1. 111.551150 5B (Qwen Team, 2025)
@oolq(Clark et al., 2019)
Deta-llama/LLaanee 2-1 B (Llama Team, 2024a)
cola (Warstadt et al., 2019)
microsoft/pphi-2 (Li et al., 2023)
ethice-utilitaria anismmmmiinmiiiimmmm (Hendrycks et al., 2021a)
Strong Models
sst2 (Socher al., 2013)
google/genma-22-
seitter-sentimens (Zhang et al., 2019)
/tren/Qwen..-11 B-
dream (Sum et al., 2019)
meta leta-1lama/ aaaaamaLaaaa LLamma 3.1-8B
mc-taco (Zhou et al., 2019)
multirc (Khashabi et al., 2018)
quail (Rogers et al., 2020)
quartz (Tafjord et al., 2019)
social-i-qa (Sap et al., 2019)
wic (Pilehvar & Camacho-Collaan 2019)
cosmos (Huang et 2019)
We follow the weak to strong generalization setup proposed in Burns et al. (2024), focusing on NLP tasks. The original paper reported results with GPT (Radford al., 2019) model versions. Instead, we use larger, more capable and recent open-weight models to make observations at the frontier. For this, used the codebase of Scherlis et al. (2024) that uses open-weight models on Huggingface instead. We now describe the full setup here.
The setup uses a pretrained weak base model W, a pretrained strong base model S and a dataset D, where Dtr, Deat, Drc are the training (10,000 samples), validation (1,000 samples) and test (5,000 samples) datasplits respectively. Dr. is divided into two halves, independently assigning each sample to Dtr1, Der with 50% probability each. All the datasets studied convert standard NLP MCQ datasets into binary classification, by randomly sampling one of the wrong options. Predictions 2 0.5 are considered class 1, and < 0.5 as class O. We highlight the models and datasets used in our study in Table 9.
First, the weak base model W is finetuned on ground-truth labels in Deri to obtain the weak supervisor W,. In the original setup, this meant to simulate a human that is an expert at the given task. Then, Wyt annotates samples in Dir2, and the strong student model finetuned on these annotations to obtain the Weak to Strong trained model Swrss the original setup, the strong base model simulates future model with superhuman intelligence, but not finetuned for specific domain knowledge.
Finetuning Methodology: For the above finetuning steps we use Low Rank Adapters (LoRA) (Hu et al., 2022) due to budget constraints, and train a binary classifier the same way as Scherlis et al. (2024). We use the confidence weighted loss proposed by Burns et al. (2024). This loss encourages the strong model's predictions to align with both a weaker model and its own "hardened" predictions. The hardened predictions are derived by thresholding the strong model's output. The loss function is defined as:
C(f) (1 (1 a) CE(f(z),
CE(f(z), (CC())))
(19)
where f(x) is the strong model's 's output, fw (x) is the weak model's output, f(x) = [zf(z) > t] represents the hardened predictions using an adaptive threshold f, and a is weight that increases over the initial phase of training.
Following Scherlis al. (2024) we use cosine learning rate schedule, with 40 warmup steps, the learning rates for the weak, strong model are 5 x 10-4, 4,8 x 10 respectively, and we train for 3 epochs which is sufficient for the train and validation loss to stabilize.
Weak to Strong Gain Metrie: We wish to study the gain achieved from weak to strong training for the strong student model. To characterize the initial accuracy of the strong student model, we train a binary classifier head obtain Sg. The
45

---

Great Models Think Alike and this Undermines AI Oversight
weak to strong gain is then quantified as:
Acc(Sm.2)) Acc(S.)
(20)
Note that this is different from the PGR metric reported by Burns et al. (2024). Their goal was to show weak to strong training can make the strong student cross the accuracy of the weak supervisor. Thus, they measured accuracy gained over the weak supervisor Ace(Swes) Ace(Wagg gt), normalizing it by an "upper-hound" obtained by training the strong student on In our work, we show that leveraging complementary knowledge effectively might actually allow Sw2s S Sgt. questioning their "upper-bound". Thus we stick to reporting how much the student model improved as described in Equation 20.
Ace(Sw...-AAcc(M
ground-truth labels on Dir2, giving PGR = -
Similarity vs Weak to Strong Gain: In Figure 16 we reported weak-to-strong gain (Equation 20) on the Y-axis, and similarity (Kp) on the X-axis. We group by model pairs, and vary the task within each model pair for the linear fit. Figure 16 shows the same scatter points but colored based the dataset. This shows that weak-to-strong gain is consistently higher for tasks where models are less similar, and how similar two models are depends mostly on the task, i.c. there is not much variance in similarity across the model pairs for fixed task.
Discarded Results:
We had initially run experiments with three more weak models: SmolLM 1.7B, Qwen-2.5-0.5B, Llama- 3.2-1B against the same list of strong models reported above. However, we found that on some tasks, the weak-to-strong gain was negative. The weak supervisor (Waz) models had lower accuracy compared to the strong student S,, leading to decrease in accuracy for the strong student after weak to strong training. We thus removed these weak models from our analysis. Similarly, we had also tried the Hellaswag dataset, but found that both weak and strong models had very low accuracies, often below 60% where chance is 50% for binary classification, consistent with Scherlis et al. (2024), and decided to not include it in our analysis.
C.2. Elicitation vs Complementary Knowledge
Table 10. Models and Sources of Knowledge in Complementary Knowledge vs Elicitation Comparison.
Model Ground-truth labels in Dir Latent Knowledge of W
Latent Knowledge of S
W ge
X
Sgt
X
S Sw2.
Figure 4 points to the fact that similarity or difference between the weak supervisor and the initial strong student are strong predictors of weak-to-strong gain. However, the initially proposed explanation of weak-to-strong generalization is "elicitation", i.e. the strong student has latent capabilities that are brought out by finetuning on weak annotations (Burns al., 2024). To quantify the contribution of these two sources for weak-to-strong gain, elicitation and complementary knowledge, we establish the following setup.
First, our functional similarity metric cannot capture latent knowledge in the strong student's representations. For this, we follow Burns et al. (2024) and finetune the strong student S on ground-trus -tunh labels of D to obtain the elicited strong student model Syt. Note that we use D.r instead of Dtr2 here so that the training set of Swas. Dtr2, remains held-out, and we can analyze the relative effect of eliciation and complementary knowledge on both the train and test set.
Table 10 summarizes sources of knowledge for the weak supervisor W gt. strong elicited Syt, and weak-to-strong trained student Sw2, in our setup. Sw2. benefits from the latent knowledge of S, complementary knowledge transfer of latent knowledge of W, and distillation of knowledge in Dar from Wgt- It learns imperfectly from all three sources of knowledge. Given this, we now discuss how Figure compares elicitation and complementary knowledge transfer:
Bottom-Left Elicitation: W does not benefit from latent knowledge of S, so Sw22 accuracy on samples where wrong but Syt is correct signify knowledge that could only be from elicitation.
Top-Right = Complementary Knowledge Transfer: Syt does not benefit from the latent knowledge of W, so Sw2. accuracy on samples where it is wrong but Wyt is correct signify knowledge that could only be from complementary knowledge transfer.
46

---

Great Models Think Alike and this Undermines AT Oversight
P44.70
Weaks Strong Trained Model Accuracy
Currect
Weak Supervisot Prediction
Both
Complementary
G-3-b
144.7b
Wrong
Elicitable
Neither
Gam
Q-35m
1-1-4b
Gdaa
Q 6m
Correct
Wrong
Strong Elicited Model Prediction
Figure 15. We decompose the accuracy of the weak to strong trained model on four parts of the train data distribution based on whether the weak supervisor and oracle strong elicited model (using ground-truth annotations) correct or wrong. All results averaged over 15 datasets. Sub-res rectaneessgles represent weak, strong model pairs. On the train dataset, complementary knowledge transfer (mean accuracy 0.59) plays an equal role as elicitation (mean accuracy 0.56).
Top-Left = Could be Both: Accuracy of Sw2s samples where both Wgt: Sgt are correct could come from both their latent knowledge, and the ground-trute truth annotations Dr Thus, these could be both elicitation and complementary knowledge transfer, or also learning from the finetuning data.
Bottom-Right = Random flips: We find that 10% predictions can flip even when finetuning W, S on ground-truth labels from Dira instead of Diri which were split into halves random from Drr. Thus, the roughly 10% accuracy samples that both W gts Sgr got wrong could just be random flips to the correct prediction (since its a binary classification setting).
Behavior on the Train Set: Figure 15 reports the same comparison elicitation and complementary knowledge transfer but Drra which the weak-to-strong training occurs. This set unseen for both the weak supervisor Wgt and the strong elicited model Sgt- We find that in fitting the training data complementary knowledge transfer plays an equal or bigger role than elicitation. This is to be expected as Swes is trained by fitting on Wgr 's annotations of Dtr2. The weak-to-strong trained student however still generalizes more similarly to the strong elicited model than the weak supervisor, though complementary knowledge transfer is also visible on test set predictions as seen in Figure
C.3. Effect of Different similarity metrics
We now report similarity vs weak-to-strong gain for various alternate similarity metries. Here, we color the seatter points by dataset instead model pair, and fit single line, for ease of interpretation. We report the following similarity metries:
Error Consistency In this setting of binary classification, this is equivalent to the -on-probabilistic version of CAPA, as there is only one incorrect option so models cannot disagree when both are incorrect on sample.
CAPA (Ksp) - Our metric which incorporates probabilistic information into error consistency.
JSD Since Jensen- ShasennShaanon Distance measures difference between distributions and is normalized between and 1, by subtracting from we can obtain a similarity metric for ease of comparison with the previous metrics.
In Figure 16 we see that all metrics can show the same trend, that is, tasks where models differ more have larger gain from weak-to-strong training. The highest correlation is seen for CAPA, though in the binary classification setup of weak-to-strong generalization the probabilistic information does not add much value compared to error consistency. 1 1 JSD gives more noisy scatter plot, with lower correlation (r).
47

---

Great Models Think Alike and this Undermines AT Oversight
-0.85
-0.7
121
Cata from W2s Tainiig
foom Tooiniii
0.20
016
from
000
0.38
4.00
000
0.0
0.00
-0.03
0.30
0073
0.23
8.50
0.20
022
1.00
6:50
0.50
0073
0.50
Ecror Comistoncy
ISD
anli-r2 boolq
cola
sciq
twitter-sentimee
me-taco multire
quail
social-i-qa
COsmoss-q
ethhice-uiilitariiniim
ast
dream
quartz
wic
Figure 16. Various Similarity Metries Weak-to-Strong gain. The highest correlation is for CAPA Kpp, though in the binary classification setup of weak-to-strong generalization the probabilistic information does not add much value compared to error consistency. JSD gives more noisy scatter plot, with lower correlation (r).
C.4. Accuracies in Weak-to-Strong training
In Figure 17 we report average across the 12 model pairs for all 15 datasets. Consistently, ordering as follows: the initial strong student has lower accuracy than the weak supervisor, but surpasses it after weak-to weak-to-strong training. However, it is not able match the performance ceiling of ground-truth elicitation. Finally, if the take union over the correct predictions of the weak supervisor and strong elicited model, the performance ceiling can be even higher.
C.5. Weak-to-strone Accuracy Value Details in Elicitation vs Complementary Knowledge Analysis
In Table 11 and Table 12 we report the underlying numbers for Figure 5 and Figure 15 respectively. The astute observer may be confused about the around 12% accuracy on the test set when when both the weak supervisor and strong elicited model are wrong (bottom--rghht quadrant). We find that merely finetuning on different random subset of training data leads to around 11 % predictions being flipped. Thus, much of this accuracy could just be due to random chance because the binary classification setup. This also indicates that complementary knowledge transfer explains much of the beyond-chance accuracy not accounted for by elicitation.
48

---

Great Models Think Alike and this Undermines AT Oversight
100
90
Accuracy
80
70
60
50
anli-12
sthhccss-n atitiaaniiit
boolq
cola
twitter-saee writter-sentiment
sciq
ast2
dream
me-taco
multirc
quail
quartz
social-i-qa
cosInos-qe qa
wic
Models
Weak Superviso visor for
Weak-to-Strong
Elicitation + Complementary Knowledge Ceiling
Initial Strong Student
Elicitation Ceiling
Figure 17. Test Accuracies for various models and ceiling estimates in Weak-to-Strong training. The accuracies averaged over 12 model pairs. The initial strong student model has consistently lower accuracy than the weak supervisor consistent with Burns et al. (2024); Scherlis et al. (2024). The wcak-to-strong trained student surpasses the weak-supervisor across datasets. However it has lower accuracy than the elicitation ceiling which trains the strong student on ground-truth annotations. Finally, our new estimated ceiling which incorporates the complementary knowledge weak supervisor has even higher accuracies, showing even more scope for improvements.
49

---

Great Models Think Alike and this Undermines AT Oversight
Table Weak-to-strong trained model's accuracies on four parts of the test distribution, based on relative mistakes veee-ssup,,,soo, strong elicited model. This table reports the underlying numbers for Figure 5, with accuracy averaged across the datasets studied for each model pair. We see the weak-to-strong model is almost always correct when both weak-supervisor. strong elicited model are correct. It is more correct when the strong elicited model is correct and the weak-supervisor is wrong than vice-versa. This indicates weak-to-strong training currently exploits more the possible gains from elicitation, but gains from complementary knowledge transfer.
possible
Common Knowledge
Complementary Knowledge Transfer
Pair
Acc (%)
Pair
Acc (%)
(gemma-2-22) 2--b, gemma-2-9b)
97.4
(gemma-22 2-2b, gemma-2-9b)
45.2
(gemma-2-22) 2--b, Qwen2.5-7B)
97.1
(gemma-2-4) 2-2b, Qwen2.5-7B)
34.9
(gemma-2-2h, Llama-3.1-8B)
97.0
(gemma-2-22 L.lmm-331-8BB
40.1
(Qwen2.5-1.5B, gemma-2-9b)
97.1
(Qwen2.5-1.5B, gemma-2-9b)
47.1
(Qwen2.5-1.5B, Qwen2.5-7B)
97.4
(Qwen2.5-1.5B, Qwen2.5-7B)
36.9
(Qwen2.5-1.5B, 1.
96.5
(Qwen2.5-1.5B, Llama-3.1-8B)
39.6
(Llama-3.2-3B, gemma-2-9b)
97.6
(Llama-3.2-3B. 3.m.mmg..m....m-220001 gemma-2-9b)
46.2
(Llama-3.2-3B, Qwen2.5-7B)
97.5
(Llama-3.2-3B, Qwen2.5-7B)
36.2
(Llama-3.2-3B. Llama-3.1-8B)
97.3
(Llama-3.2-33B, Llama-3.1-8B)
41.7
(phi-2, gemma-2-9b)
97.3
(phi-2, gemma-2-9b)
50.2
(phi-2, Qwen2.5-7B)
97.3
(phi-2, Qwen2.5-7B)
44.2
(phi-2, Llama-3.1-88
97.4
(phi-2, Llama-3.1 1--BB)
44.2
Elicitation
Both Wrong
Pair
Acc (%)
Pair
Acc (%)
(gemma-2-4 2-2b,, gemma-2-9b)
71.0
(gemma-22 2-2b, gemma-2-9b)
12.9
(gemma-244 2-2bb, Qwen2.5-7B)
75.0
(gemma-2-4 2-2b, Qwen2.5-7B)
10.7
(gemma-2-2h, Llama-3.1-88)
72.3
(gemma-2-2b, Llama-3.1-8B)
13.0
(Qwen2.5-1.5B, gemma-2-9b)
69.4
(Qwen2.5-1.5B,
gemma-2-9b)
11.4
(Qwen2.5-1.5BB Qwen2.5-7B)
73.3
(Qwen2.5-115B,
Qwen2.5-7B)
11.6
(Qwen2.5-1.5B, Llama-3.1-88)
72.1
(Qwen2.5-1.5B, Llama-3.1-8B)
13.5
(Llama-3.2-3B, gemma-2---- -2-bb)
71.0
(Llama-3.2-3B. g.mmma. gemma-2-9b)
12.5
(.lama-3.2-3B, Qwen2.5-7B)
77.2
(Llama-3.2-33) Qwen2.5-7B)
11.2
(Llama-3.2-3BB, Llama-3.1-88
73.4
(Llama-3.2-3B. Alama--311888
13.8
(phi-2, gemma-2-9b)
67.9
(phi-2, gemma-2-99)
12.3
(phi-2, Qwen2.5-7B)
71.1
(phi-2, Qwen2.5-7B)
11.6
(phi-2, Llama-3.1-80
69.0
(phi-2, Llama-3.1
11.5
50

---

Great Models Think Alike and this Undermines AT Oversight
Table 12. Weak-to-strong trained model's accuracies four parts of the train data distribution, based relative mistakes of weak-supervvsor, strong elicited model. This table reports the underlying numbers for Figure 15, with accuracy averaged across the 15 datasets studied for each model pair. On the train distribution, the weak-to-strong model almost equally correct the only-elicitable and only learnable from complementary knowledge samples, with a slight lean towards the latter. Yet, Table 11 showed it generalizes more similarly the strong elicited model.
Common Knowledge
Complementary Knowledge Transfer
Pair
Acc (%)
Pair
Acc (%)
(gemma-2-20 22-2b, gemma-2-9b)
98.6
Deemma-2-2b. gemma-2-9b)
66.8
(gemma-2-20) 2--b, Qwen2.5-7B)
98.6
(gemma-2-22) Qwen2.5-7B)
52.5
(gemma-2-2b, Ilama-3. 1-8B)
98.5
(gemma-2-2b, Llama-3.1-8B)
56.8
(Qwen2.5-1.5B, 2et.:--a......ma2222..... gemma-2-9b)
98.5
(Qwen2.5-1.5B,
65.2
(Qwen2.5-1.5B, Qwen2.5-7B)
98.6
(Qwen2.5-1.5B,
Qwen2.5-7B)
56.9
(Qwen2.5-1.5BB Llama-3.1-8B3)
98.5
(Qwen2.5-1.5B,
Llama-3.1-8B)
57.0
(L.ama-3.2-3B, gemma-2-9b)
98.7
(1lama-3.2-33B
67.2
(Llama-3.2-3B, Qwen2.5-7B)
98.7
(Llama-3.2-3B, Qwen2.5-7B)
53.8
(Llama-3.2-3B, Llama-3.1-8B)
98.5
(Llama-3.2-3B, Llama-3.1-8B)
58.8
(phi-2, gemma-2-9b)
98.0
(phi-2, gemma-2-9b)
64.4
(phi-2, Qwen2.5-7B)
98.4
(phi-2, Qwen2.5-7B)
53.7
(phi-2, Llama-3.1-8B)
98.2
(phi-2, Llama-3.1-8B)
54.8
Elicitation
Both Wrong
Pair
Acc (%)
Pair
Ace (%)
(gemma-222)) 2-2b, gemma-2-9b)
50.7
(gemma-22 2-2b, gemma-2-9b)
8.6
(gemma-24 2-22,, Qwen2.5-7B)
63.5
(gemma-2 2-2b, Qwen2.5-7B)
8.3
(gemma-2-2h, Llama-3.1-8B)
57.3
agemma-2-2b, Llama-3.1-8BB
9.5
(Qwen2.5-1.5BB gemma-2-9b)
49.2
(Qwen2.5-1.5B, gemma-2-9b)
9.6
(Qwen2.5-1.5B, Qwen2.5-7B)
62.2
(Qwen2.5-1.5B,
Qwen2.5-7B)
7.4
(Qwen2.5-1.5BB Llama-3.1-8B)
58.2
(Qwen2.5-1.5B, Llamm-3.1-8B)
7.7
(Llama-3.2-3BB gemma-2-9b)
47.9
(Llama-3.2-3BB
20mma--223Bggemmm22290 gemma-2-9b)
8.8
(Llama-3.2-3BB Qwen2.5-7B)
60.3
(Llama-3.2-3BB Qwen2.5-7B)
7.9
(Llama-3.2-3B, Llama-3.1-8B)
52.4
(Llama-3.2-3BB, Llama-3.1-8B)
8.5
(phi-2, gemma-2-9b)
52.6
(phi-2, gemma-2-9b)
10.8
(phi-2, Qwen2.5-7B)
62.3
(phi-2, Qwen2.5-7B)
9,4
(phi-2, Llama-3, 1luma-3.1-88 3.18BB
60.1
(phi-2, Llama-3.1
9.1
51

---

Great Models Think Alike and this Undermines AT Oversight
D. Similarity Trends with Increasing Capabilities
D.I. Setup Details
We utilize two prominent benchmark datasets from the OpenI.MM leaderboard to explore the relationship between model similarity and capability: MMLU Pro and BBH. For the BBH dataset, we aggregate 23 distinct tasks that can be studied multiple-choice questions from the Big-Bende Hard benchmark to ensure that each model is evaluated questions, thereby ensuring statistically significant results. The MMLU Pro dataset consists of MCQs across 14 different subjects, with varying numbers of options per question. Notably, some questions are repeated with shuffled option orders. To maintain consistency, we filter the dataset by retaining only those questions for which both the question text and the correct option index remain consistent across all models. This filtering process yields a refined dataset of 11,828 questions.
sufficient
To analyze trends across model capabilities, we divide 130 models (Table 14) into five bins based their individual accuracy percentiles. This binning strategy is followed for all experimental setups and ensures approximately equal distribution of models per bin, maintaining a consistent sample size across bins. We select model pairs within each bin and compute their similarity and average accuracy. This approach ensures that the average accuracy of the pairs remains representative of the individual model accuracies within the bin. We do not consider model pairs from the same family to avoid confounding effects of model similarity being attributed model family rather than the capability.
D.2. Why are model mistakes becoming more similar? preliminary analysis
D.2.1. INSTRUCTION-TUNING EXACERBATES THE TREND
Instruct
Base
1.0
BBH
BBH
Linoar fit BBH; slope 0.82
Linear fit BRF slope 00.65
0.8
MMLU Pro
MMLU Pro
Linear fi MMLU Pro; slope 0.71
Linear fi MMLU Pro: slape 0.46
0.6
Similarity
0.4
0.2
0.0
-0.90.1
0.3
0.7 Average Accuracy of Model Pair
0.1
0.5
0.6
Figure 18. LM Similarity (xp) vs Capabilities in Instruct-tune tuned and Base models on MMLU pro and BBH. After applying the same model binning stratergy and pairwise similarity, a steeper trend is observed the instruct-tuned models compared to base models for both datasets.
Instruction-tur models are base models that have been fine-tuned on instruction datasets and their corresponding outputs, enhancing their ability to follow user instructions accurately. Among the models analyzed for the capability-similare trend, we categorize them into instruction-tuned and base models. Using the same binning strategy as discussed in the previous section, we first assign all models to bins based on their accuracy percentiles. When computing pairwise similarity and accuracy, we restrict to pairs of the same model type- base-base and instruct-instruct models. As illustrated Fig. 18, instruction- model pairs exhibit stronger similarity trend with a steeper slope compared base models. This can likely be attributed to the fact that instruction-tune tuned models may have been fine-tuned similar instruction datasets, leading to higher similarity trend among them.
D.2.2. IS THE TREND CONFOUNDED BY QUESTION DIFFICULTIES?
To address the potential confounder that models might exhibit higher similarity as their capability increases simply due to their inherent ability solve harder problems, we analyze the relationship between question hardness and model similarity
52

---

Great Models Think Alike and this Undermines AT Oversight
MMLU Pro and BBH. Question hardness is determined by the percentage of models that answer question correctly, with harder questions being those that fewer models answer correctly. We split the data samples into percentile bins based on question hardness and compute the average similarity across all capability bins of the initial setting, as illustrated in Fig. 19)..
Fig. 19(a) demonstrates that the overall average similarity remains consistent across different levels of question hardness, with only a slight increase observed for the hardest questions (100th percentile). This consistency indicates that the hardness of the questions does not significantly confound the observed trend of increasing similarity with model capability. These findings reinforce the hypothesis that the growing similarity among models is not merely byproduct of their ability to solve harder problems but reflects a deeper trend in model behavior as their capabilities improve.
LM Similarity vs Capabilities
A Average Similarity vs Hardness
MMLU Pro
BBH
Läusar Fi MMLU Pro; slape = 0.68
MMLU Pro
Average Similarity
Linear Fit BBII; slope 0.56
0.2
0.1
0.0
-0.1
0.1
0.2
0.3
0.4 0.5 0.6
0.7
40
60
80
100
A Average Accuracy for Model Pair (a)
Hardness of Q in percentile (b)
Figure 19. Role question difficulty in similarity-capabiiiyy trend. We plot in parallel (a) Scatter plot with model pairs, illustrating the increasing trend of similarity (Kpp) with model capability. (b) Average similarity (Kpp) across all capability bins for different levels of question hardness. CAPA is mostly consistent across question hardness, with a slight increase on the hardest questions. This shows that question difficulty is not significant confounder for increasing similarity in mistakes.
D.2.3. CAN CHANGING ARCHITECTURE REDUCE MODEL SIMILARITY?
To study the effect of model similarities across different architectures, we analyze Falcon3 Mamba 7B base and instruct models by computing their CAPA values with Falcon3 7B transformer, Mistral 7Bv0.3, and Llama 3.1 8B base and instruct models. We ensure that the accuracies of the non-Falcon3 transformers are within +5% of the Mamba model to ensure comparable capabilities.
In Table 13, Similarity presents the CAPA between the Falcon3 Mamba and Falcon3 Transformer, Similarity the CAPA between Falcon3 Mamba and Llama/Mistral Transformer, and Similarity, between Falcon3 Transformer and Llama/Mistral Transformers. The results reveal that base models exhibit lower overall similarity compared to instruction- tuned models, with pairwise similarity between Falcon3 Mamba and non-Falcon Mistral/Llama Transformers showing the least similarity. Within the base model category, Falcon3 Mamba and Falcon Transformers demonstrate the highest similarity. For instruction-tuned models, Falcon3 Transformer and Mistral/L Transformer pairs exhibit the highest similarity, followed closely by Falcon3 Mamba and Falcon3 Transformer.
The Falcon Mamba-Falcon Transformers maintain higher similarity overall, potentially due to their shared model family, despite differences in their underlying architectures. This observation highlights that architectural differences may play a less significant role in model similarity compared to factors such as training data and fine-tuning procedures. From the earlier section, instruction-tuned models exhibit stronger similarity trend, similar to as observed in this setting.
D.3. Alternative Similarity Metrics
As discussed in earlier sections, multiple metrics can be employed to quantify the similarity between models, each with its own strengths and limitations. this analysis, we evaluate several alternative metrics under the same experimental setting
53

---

Great Models Think Alike and this Undermines AT Oversight
Table Analyzing effect of difference architecture on CAPA Kip Using base and instruct variants Falcon3 Mamba and Falcon3 Transformer of size 7B, compare it with transformers with similar size and accuracy from a different model family- LLama Mistral. Similarity: consistently has an overall higher similarity models belonging the same family. In instruct-tuned models, Similaritys is the highest, possibly due to the instruct-tuning.
Falcon Mamba
Falcon Transformer
Transformer Model
Similarity:
Similarityz
Similaritys
Llama 3.1 8B
0.0167
0.0422
7B Base
7B Base
0.0619
Mistral v0.3 7B
0.0105
0.0235
Llama 3.1 8B Instruct
0.0665
0.173
7B Instruct
7B Instruct
0.1111
Mistral v0.3 7B Instruct
0.0582
0.1584
JSD
Error Consistency
1.0
BEM
BEI
MMLU Pro
MMLU Pro
0.8
0.6
Similarity
0.4
0.0
0.2
0.1
0.2
0.3
0.4
0.5
0.7
0.2
0.3
0.4
0.5
0.6
0.7
Average Accuracy of Model Pair
Figure 20. Error consistency and JSD for model similarity on BBH and MMLU Pro. The y-axis represents the similarity computed using JSD and Error consistency. JSD exhibits high variance flat trend, whereas Error Consistency shows an increasing trend with model capability, similar to the trend observed in Kp.
used for CAPA, including the binning and averaging strategies, for the two benchmark datasets. Fig. 21a presents the results for discrete Kp, which does not utilize logit information, while Fig. 21b demonstrates similar trend using Kp for M >> 2 (Ki extended for more than 2 models). Additionally, Fig. 20 includes results for Jensen-Shannon Divergence (JSD) and Error Consistency (Geirhos et al., 2020).
Discrete Kp exhibits an increasing trend with model capability, closely mirroring the trend observed with Kp. Similarly, Kp for M > 2, which leverages probabilistic information, unlike Discrete Ky, shows strong upward trend. Unlike other metrics, Kp for M > 2 provides a direct measure of similarity that quantifies agreement among all models within bin, eliminating the need for averaging pairwise similarities. Models of same family within the same bin are retained when computing the metric. contrast, JSD does not exhibit clear trend and remains flat with high variance across the capability axis. Error Consistency, however, aligns with the upward trend observed in other metrics, further supporting the hypothesis that model similarity increases with capability.
D.4. Model capability vs similarity across domains
The scatter plot Fig shows the increasing similarity trend after aggregating across the subjects (MMLU pro) and tasks (BBH). Fig 22 and Fig 23 show the observed trend within each individual subject and task for MMLU Pro and BBH respectively.
In the MMLU Pro dataset, the trend of increasing average similarity within each bin as model capability improves is consistently observed across all individual subjects. For the BBH tasks, while the trend is not as pronounced in some tasks, remains significant in the majority of them. This weaker trend in certain BBH tasks can be attributed the limited number
54

---

Great Models Think Alike and this Undermines AT Oversight
0.40
MMLU Pro
MMLL
Pro
0.30
BIH
0.33
Similarity
Similarity
0.33
0.15
0.30
-0.00
6.29%
30-10%
10-00%
60 SIX
80 100%
0,00
0.20%
20 90%
10-60%
00 50%
80 100%
Average Accuracy of Model Pair
Average Accuracy of Model pairs
(a) LM Similarity (r, for M 2) Average Accuracy of Model Pairs in each Capability bin
(b) LM Similarity (Discrete ky) Average Accuracy of Model Pairs in each Capability bin
Figure 21. Discrete Kp and Kip for M > 2 values computed the MMLU Pro and BBH dataset. An increasing trend in similarity is observed across both datasets in accordance with the hypothesis. Discrete Kp uses similar averaging idea as used in Kp while in rip for M 2, the similarity computed using all models in capability bin at once.
of questions per task for each model, with maximum of 250 questions per task, which reduces the reliability of the results compared to the more robust MMLU Pro dataset. This is also visible through the high confidence interval in the BBH tasks, unlike MMLU pro subjects.
D.5. List of models
55

---

Great Models Think Alike and this Undermines AT Oversight
Table 14. Models from OpenLLM leaderboard used to study the capahilityssimilarie trend. Models different families,
architectures, size and versions were used to ensure robustness in the experimental results. The models included are both base and fine-tuned versions where available.
Dev
Model Family
Size
Type
Yi-1.5
9B, 34B
Base, Instruct
01-ai
Yi
34B
Base, Instruct
pllud
Base, Instruct
CohereForAl
aya-expanse
32b
Base
EleutherAI
Pythia
160m, 410m, 2.8b, 6.9b, 12b
Base
Gemma
2b, 7b
Base, Instruct
Gemma- 1.1
2b, 7b
Instruct
Google
Gemma-2
2b, 9b, 27b
Base, Instruct
Flan-T5
Small, Base, Large, XL, XXL
Base
Llama-2
7b, 13b, 70b
Base, Instruct
Llama-3.2
IB, 3B
Base, Instruct
Llama-3
8B, 70B
Base, Instruct
Meta
Llama-3.1
8B, 70B
Instruct
Llama-3.3
70B
Instruct
Mistral-7B
v0.1, v0.2, v0.3 v0.1
Base, Instruct
Mistral AI
Mixtral-8x7B
Base, Instruct
Mistral-Large
Instruct
Nvidia
Mistral-NeMo-Mininee
8B
Base, Instruct
Qwen2
0.5B, 1.5B, 7B, 72B
Base, Instruct
Qwen2.5
0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B
Base, Instruct
Qwen2-Math
7B, 72B
Base, Instruct
Qwen
Qwen2-VI.
7B, 72B
Instruct
Qwen2.5-Coder
7B
Base, Instruct
Qwen1.5
32B, 110B
Base, Chat
Falcon
7b, 11B, 40b
Base, Instruct
Tiiuae
Falcon3
7B, 10B
Base, Instruct
Falcon-mamba
7b
Base
Upstage
solar oolar-pro-preview
Instruct
56

---

Great Models Think Alike and this Undermines AT Oversight
philosophy
psychology
biology
0.40
0.49
0.40
0.33
0.33
0.3
0.90
0.90
0.90
0.25
0.85
0.5
0.30
0.10
3.10
0.05
aus
0.13
0.00
6.23%
20-00%
00 61%
63-ITS
S0-1002
3.00
0.20%
20-
38-00%
43-62%
00-817
Su. SI-10155
0.00
0.275
21-40%
40 0055
02-917
S0-1005
oconomics
chemistry
computer science
0.30
0.25
0.25
0.25
0.30
0.30
0.20
0.15
0.19
0.13
0.10
3.10
0.10
0.00
0.00
physics
business
math
0.40
0.40
0.40
0.35
0.55
0.3
Similarity
0.30
0.90
0.90
0.15
0.10
0.10
0.10
0.03
0.03
0.0
0.00
0.23%
20-005
00 60%
00-30%
80 100%
0.00
0.20%
20
30-00%
00 40%
00-0%
80 100%
0.00
0.20%
20 00-009
00 01/
00-80%
50 JOYZ
other
engineering
health
0.55
0.30
0.25
0.29
0.25
0.30
0.20
0.20
0.13
0.15
0.10
0.00
0.00
law
history
0.00
0.00
0.85
0.55
0.90
0.20
0.15
0.10
0.10
0.05
0.05
0.00
0-23%
20-10
6000%
09-50%
50-
100%
0.00
0-20R 20-00%
30-0%%
05-80% 50-1008%
Avg Capability Percentile Bins
Figure 22, LM Similarity (kp) vs Capability on MMLU pro for each subject. The increasing trend holds for all 14 subjects in MMLU pro. The similarity trend therefore consequence of particular domain or subject in MMLU Pro.
57

---

Great Models Think Alike and this Undermines AT Oversight
boolean expressions
causal judgemennt
date understanding
0..0
0.1
00-005
40-M01
00-505
disamhigaation qu
Fallacies
genmetric shapes
0.2
hyperbaton
logical deduction obojects
logical deduction seven objects
3.4
0.2
10-60%
logical deduction three objects
movie recommendation
navigate
0.0
0.2
Similarity
40-027
40-027
pengruins in table
reasoning about colored objects
ruin nanmes
0.2
40-00%
00-005
20-40%
50-100%
0-20%
40-0055
00-005
10.1005
salient translation error detection
snarks
sports understanding
0.3
0.2
3.00%
70-10%
40-00%
00-907
10-10
6-00%
70-00%
+0-00%
00-90%
80-100%
0.00%
70-10K
40-005
00-00%
50-
10/%
Semporal sequenoce
tracking shuffled objects five objects
tracking shuffled obojects seven objects
0.3
3.30%
20 40%
60%
00 M035
100%
0-30%
20-40/
40-6
90%
M0. 10-100%
0.30%
20-40%
60%
Q3 MPS
100%
tracking shufiled objects Uhrees objects
web of lies
0.3
3-20%
20-40%
40-00%
00 50.5
50.100%
00 90% Avg Capability Percentile Bins
0-20%
20-40%
40-00%
S0- J00%
Figure 23. LM Similarity (s,,) Capability each Big-Bench Hard task. The increasing trend holds for most BBH tasks. Each task has atmost 250 questions, resulting in minimal data to compute similarity for the individual tasks.
58


Reviews:
[{"review_id": "H9Klmxz3Nv", "reviewer": "Reviewer_RC4b", "summary": "This paper introduces a probabilistic metric for model similarity that adjusts for chance agreement due to accuracy, distinguishes different types of mistakes, and incorporates confidences. Using this metric, the authors reveal three key insights:\n1. LM judges demonstrate affinity bias, favoring models similar to themselves.\n2. Greater gains in weak-to-strong generalization occur when training strong student models with annotations generated by weak supervisors that are more different from the student.\n3. Model errors get more correlated as model capabilities increase, raising concerns about correlated failures in AI oversight.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that you found our metric well-motivated (with comparisons to alternatives), experiments comprehensive, and observations insightful.\n\n> The proposed similarity metric seems to be tailored for MCQ tasks. It would be insightful if some discussions on possible extensions to other tasks (e.g., regression) could be provided, even just as future directions.\n\nIndeed, our metric as described in the main paper is designed to address a key challenge in MCQ tasks—specifically, the lack of coherent classes needed to compute the marginal distributions for inter-annotator agreement metrics like Cohen’s κ [L976-979]. You might find Appendix A.3 interesting, where we extend the metric to classification, along with a brief discussion of challenges for exact match settings, for which we only provide a discrete version. \n\nOn the other hand, for free-response tasks like creative writing, one could use both perplexity and model-based metrics, which have their own challenges as discussed in L392-403. That said, we ran experiments where we compute similarity on MMLU-Pro MCQs and plot Elo ratings assigned to evaluated models by making an LLM-judge pick between open-ended chat responses on AlpacaEval -- [New Plot](https://imgur.com/a/CIKQOmf). The strong correlation shows initial evidence that MCQ similarity might transfer across tasks as a predictor. \n\nFor tasks like regression, for the observed agreement, $c_{obs}$, one could measure a distance metric over the two models' predictions, aggregating across samples. Once again, models with lower error would have lower distance in prediction, so the challenge lies in defining chance agreement, $c_{exp}$, for a model with a given error. We would have to make appropriate assumptions about the distribution of errors, such as gaussian errors, based on which $c_{exp}$ can then be computed. Thanks for this interesting question; We will add this discussion to Appendix A.3. We are excited about adapting the metric for other tasks in future work!\n\n> The negative correlation between weak-to-strong generalization and model similarity is arguably surprising. It would be helpful to provide more insights or possible explanations for this phenomenon.\n\nThe result can seem surprising if we view weak and strong models purely through the lens of accuracy. This is where we think our framing of model similarity (or difference) at a sample level is insightful! Lower accuracy does not imply that the knowledge of weak models is a strict subset of stronger ones. Rather, weak models can have complementary knowledge, and we hypothesise the transfer of this also contributes to weak-to-strong generalization. Model similarity provides a way to measure complementary knowledge in terms of the difference in samples they get right. Lesser complementary knowledge to transfer might be the explanation for the seemingly surprising trend of lower weak-to-strong generalization when model similarity is higher (negative correlation). We tried to motivate this in L248-260. We will be sure to utilise the extra page allowed in the revision to expand this section, by including content currently in Appendix C.1 and C.2.\n\n> “Regarding the definition of \"observed error overlap\" on the right of line 87, it may be worth remarking why \"the fraction of samples on which both models are correct or both models are wrong\" is a more reasonable metric than things like \"the fraction of samples on which the two models agree (in the multi-class setting)\". Is the binary nature of the definition crucial here?”\n\nIn L87, we are stating the definition used for error consistency defined in Geirhos et al. (2020). Our own metric modifies this to measure “the fraction of samples on which two models agree”, just as you proposed. We agree it’s a better definition, as it distinguishes differing mistakes [L129-133]. As you correctly noticed, our metric $\\kappa_p$ is equivalent to error consistency (Geirhos et al.) for binary classification [L894-896, Appendix A.1]. Great minds do think alike ;)\n\nThanks for your question and suggestions. We hope our response increases your support for our work and we are happy to discuss further!", "claims_and_evidence": "Yes. While I couldn't verify all the materials in the appendix, the main results in the paper seem reasonable.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria proposed in the paper are reasonable. The proposed similarity metric is well-motivated, with the relation and difference compared to the existing metrics clearly explained.", "theoretical_claims": "The proofs in Appendix A are reasonable.", "experimental_designs_or_analyses": "The experimental designs and analyses that I reviewed are comprehensive and demonstrative of the claims made in the paper.", "relation_to_broader_scientific_literature": "This work introduced a probabilistic metric for model similarity that is built upon the existing metrics while addressing their limitations. Some insightful observations on AI oversight are provided based on the proposed metric, which further solidifies the broader impact of this work.", "essential_references_not_discussed": "To my knowledge, the paper discussed the essential references in the field.", "other_strengths_and_weaknesses": "Strengths:\n- The proposed similarity metric $\\kappa_p$ is well-motivated and addresses the limitations of existing metrics.\n- The observation on negative correlation between weak-to-strong generalization and model similarity is quite insightful.\n- The observation on the potential risk of correlated failures in AI oversight is important and thought-provoking.\n\n(Minor) weaknesses:\n- The proposed similarity metric $\\kappa_p$ seems to be tailored for MCQ tasks. It would be insightful if some discussions on possible extensions to other tasks (e.g., regression) could be provided, even just as future directions. \n- The negative correlation between weak-to-strong generalization and model similarity is arguably surprising. It would be helpful to provide more insights or possible explanations for this phenomenon.", "other_comments_or_suggestions": "Regarding the definition of \"observed error overlap\" on the right of line 87, it may be worth remarking why \"the fraction of samples on which both models are correct or both models are wrong\" is a more reasonable metric than things like \"the fraction of samples on which the two models agree (in the multi-class setting)\". Is the binary nature of the definition crucial here?", "questions_for_authors": "Some questions are raised in the \"Other Strengths And Weaknesses\" section. I do not have further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TtKSQfqOJR", "reviewer": "Reviewer_WJR8", "summary": "This paper investigates the challenges of AI oversight when using Language Models (LMs) to evaluate and supervise other LMs. It highlights how model similarity can undermine oversight, as similar models tend to make correlated mistakes and exhibit affinity bias—where they rate outputs from similar models more favorably.\n\nTo address this, the authors introduce a novel metric, κp, that accurately measures the functional similarity of LMs based on overlapping mistakes while accounting for chance agreement and output probabilities. The study also demonstrates that greater diversity between supervising and student models enhances generalization and learning.\n\nThe paper emphasizes the risk that, as LMs become more capable, their mistakes become more similar, which may amplify oversight blind spots. It calls for better diversity in oversight models, transparent reporting of model similarities, and more robust evaluation frameworks to ensure reliable AI oversight.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad you liked the motivation and experimental design. We hope our new analyses address your concerns.\n\n**(W1)**\n> Comparison with alternative similarity metrics (e.g., KL Divergence, JSD, RSA)\n\nDivergence metrics have desirable information theoretic properties, but **we did not find any way to adjust the effect of higher accuracy inflating the similarity reported by these metrics** [L88-99 (R), Table 1]. We re-plot Figs. 6-8 with JSD, a symmetric and bounded variant of KL, to demonstrate this -- [new plots](https://imgur.com/a/XL7OBwH), e.g., the second plot studies 90% accurate models with independent errors, where $JSD > 0.75$, but as desired, $k_p \\sim 0$. \n\nApp. A4 discusses design choices for $\\kappa_p$ by comparing with many alternate similarity metrics, including why we use probabilistic overlap instead of JSD [L1029-1031]. Here, we will also include that **alternative priors, if known, can be used to compute chance agreement $c^p_{exp}$** on *``tasks where wrong answers aren’t uniformly distributed''*, thanks for this suggestion. Finally, we focus on functional similarity metrics and not model-dependent representational similarity analysis (RSA) [L60-80 (R)]. \n\n**(W2, W3)**\n> Some confounders are not sufficiently controlled …, including: training data, architectural similarities; Limited casual evidence in similarity trends\n\nWe agree deeper analysis of the causal factors is an important next step [L386-390 (R)]. **We tried our best not to make causal claims in the paper, and will rephrase any we overlooked**.\n\nWe will acknowledge training data overlap as a potential reason for our observations. Unfortunately, training data mixtures are not known even for open-weight frontier models to control for this. As requested, we now exclude *``same-family pairs''* for Fig. 2 in [new plots](https://imgur.com/a/qKr2yi3). **Judge scores remain highly correlated** with similarity $r \\in \\[0.89, 0.95\\]$. Further, [new table](https://imgur.com/a/y06LkCT) controls for *model size* as requested by reviewer nzM3. **Similarity still has high partial correlation**.\n\nFor similarity trends with increasing capability, we already excluded same-family pairs in Fig. 5. In Sec 5.2 we mention how switching to **state space models (Mamba) does not reduce similarity**. App D.2 has more details and observations, showing **sample-hardness has a small effect on this trend, while instruction tuning exacerbates it**. We are happy to demonstrate further controls if you have any suggestions.\n\n> the benchmark datasets are limited. In section 3, there is only dataset MMLU-Pro\n\nIn Sec 3, we were constrained to tasks which can be evaluated as both free-response and MCQ [L199-203]. This is why we cannot use benchmarks like BBH in Sec 5 and the 15 NLP tasks in Sec 4. **MMLU-Pro itself is quite diverse, and [new plots](https://imgur.com/a/grG06X1) show consistent results across its 14 domains ranging from law to engineering**. In fact, could this diversity lead to similarities on MMLU-Pro predicting judgements on more open-ended tasks? **[New plot](https://imgur.com/a/CIKQOmf) shows MMLU-Pro similarity has high correlation with LM judge based elos on the chat response dataset AlpacaEval 2.0** [1].\n\n**(Evidence for W2S)**\n> conclusions about complementary knowledge are entirely dependent on the κp metric\n\nWe forgot to add a pointer in our main paper to Fig 13 which shows that the **complementary knowledge conclusion consistently holds with 1 - JSD, though $\\kappa_p$ explains more variance ($r=0.77$ vs $0.85$)** and has better normative grounding for analyzing model similarity trends as discussed in Sec 2. We will mention the *``selection criteria for weak vs strong models''* is model size, consistent with Burns et al. (2024) and Scherlis et al. (2024) in the revision.\n\n**(W4)**\n> no evaluation of the qualitative nature of mistakes.\n\nQualitative analysis of mistakes is an interesting related, yet complementary direction [L360-369 (R)]. We respectfully disagree that this implies a *``lack of insights to connect to/improve the real LMs applications''*. Our quantitative measurement of similarity provides a **necessary foundation for applications like evaluating qualitative descriptions of model differences [L431-437 (R)], debiasing LLM-as-a-judge based leaderboards [L195-205]**, and many exciting directions for future work discussed in Sec 7.  \n\n> For 3.2 Q1, provide the table where the data is from.\n\n[New tables](https://imgur.com/a/7XSrXt3) have the data of all 351 model pairs studied. To make this data easier to interpret, we reported it as scatter points in Fig 2. In the final version, we will link our **PyPi package and interactive tool** to aid readers in exploring similarities of chosen model pairs.\n\nThanks! Your detailed feedback has helped us greatly improve the paper. We hope this increases your support for our work.\n\n[1] Dubois, Yann, et al. \"Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators.\" *COLM* (2024)", "claims_and_evidence": "•\tClaim 1: “LLM judges exhibit affinity bias, favoring models similar to themselves.”\nIt is supported with relatively convincing analysis. However, While Figure 2 suggests family-level bias, it lacks a clear statistical comparison between “same family” and “different family” pairs.\n\n•\tClaim 2: “Training gains from weak-to-strong generalization are higher when the supervisor and student models are less similar.”\nThe authors provide analysis to support this claim. However, the paper’s conclusions about complementary knowledge are entirely dependent on the κp metric. Moreover, the selection criteria for “weak” vs. “strong” models are not transparently discussed.\n\n•\tClaim3 : “Model mistakes are becoming more similar with increasing capabilities, which is concerning for AI safety.”\nThe study only shows a correlation between model capability and similarity of mistakes. There is no causal evidence to prove that increasing capabilities cause more similar mistakes. Additionally, it fails to account for key confounders that could explain the increasing similarity trend. For example, training data overlap: many LLMs use similar datasets, which naturally biases them towards similar mistakes.", "methods_and_evaluation_criteria": "•\tWhile the authors provide evaluations in two different usage scenarios for LMs, the benchmark datasets are limited. In section 3, there is only dataset MMLU-Pro being used. \n•\tAdditionally, there is no evaluation of the qualitative nature of mistakes such as factual errors, reasoning errors etc. Thus, it’s lack of insights to connect to/improve the real LMs applications.", "theoretical_claims": "•\tThe paper lacks theoretical proofs explaining why κp captures probabilistic differences better than divergence metrics.\n•\tThe derivation of chance agreement (cpexp) simplifies the assumption by uniformly distributing probabilities across incorrect options. However, it doesn’t consider tasks where wrong answers aren’t uniformly distributed or where some options are systematically more likely than others.", "experimental_designs_or_analyses": "•\tIn section 3, the authors control for accuracy using partial correlations and regression but neglect other possible confounding factors such as: training data overlap. Models from the same family may have been trained on overlapping datasets, leading to similar outputs and thus higher judgment scores.\n•\tIn section 5, there is no real-world experiment demonstrating how these similarities lead to oversight failures, so increasing similarity is a “safety concern” is questionable.", "relation_to_broader_scientific_literature": "The κp metric extends traditional agreement metrics like: Cohen’s κ (Cohen, 1960): Measures inter-annotator agreement while adjusting for chance; Krippendorff’s α (Krippendorff, 1970): A more flexible metric for measuring agreement with chance correction; Error Consistency Metrics (Burns et al., 2022): Focused on measuring consistency in model mistakes.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n•\tNovel Contribution: κp Similarity Metric\n•\tUtilizes partial correlations and multiple regression to control for accuracy when analyzing judgment bias\nWeakness:\n•\tNo rigorous comparison with alternative similarity metrics (e.g., KL Divergence, JSD, RSA).\n•\tSome confounders are not sufficiently controlled in the experimental designs, including: training data, architectural similarities.\n•\tLimited casual evidence in similarity trends\n•\tLack of qualitative error analysis\n\nOverall, while the paper shows the strong motivation and good effort in experiment design, there are many critical information and analysis missing to convince the audience and provide better insights", "other_comments_or_suggestions": "•\tFigure 2 & 3 are really informative, but also overwhelming. It could be hard to comprehend all the details and interpret the analysis based on it. \n•\tFor 3.2 Q1, it will be better to provide the table where the data is from. The single example is difficult to validate the analysis.", "questions_for_authors": "As listed in previous parts and weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6VHfKYl2hg", "reviewer": "Reviewer_qUdH", "summary": "The paper proposes a similarity metric between LLMs based on their logits, which measures the similarity of mistakes two models make on a task. The authors use this similarity metric to perform a variety of analyses. In particular they find similarity can predict scores from LLM judges, weak-to-strong generalization results, but also model capability predicts similarity, i.e., more capable models tend to be more similar to each other.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your strong support of our work. We are glad you that you found our claims nuanced and well-supported by comprehensive, rigorous experiments.\n\nOn your question about how we compute the “elicitation $\\cup$ complementary” ceiling estimate in Table 3, we should indeed have made this clearer in the main paper itself. We take the union of samples either the strong model fine-tuned on ground-truth labels (elicited) or the weak supervisor (complementary) get right. This is likely an overestimate of what can be realised from weak-to-strong training, but we think that is a lesser harm for a ceiling estimate. The previously proposed ceiling of ground-truth elicitation on the strong model ignores potential gain from complementary knowledge, which can be significant as indicated in Figure 4 and 12. We think such underestimation is worse for a ceiling estimate, as it might explain concurrent work [1] reporting >100% PGR, beating the “ceiling” of ground-truth elicitation on the strong model. Thus, we avoid estimating the ceiling by methods that can introduce any suboptimalities. For example, if we sequentially finetuned on ground-truth labels and then the weak supervisor's predictions or vice-versa, this could lead to catastrophic forgetting of the first part of training instead of fully combining their knowledge as desired. \n\nWe agree with you that many important details about the weak-to-strong setup are in the Appendix. This is due to the limited space for the initial submission. In the camera ready, we will utilise the extra page available to shift setup details for the weak-to-strong experiments currently in Appendix C.1 and C.2 back to Section 4.\n\nWe are happy to discuss any further questions about the weak-to-strong and other experiments!\n\n[1] Shi, Junhao, et al. \"How to Mitigate Overfitting in Weak-to-strong Generalization?.\" *arXiv* (2025).", "claims_and_evidence": "The authors make nuanced claims and I think all claims are well supported by evidence.", "methods_and_evaluation_criteria": "Yes all experiments and analyses make sense.\n\nThe LLM-as-a-judge experiments would benefit from looking at a wider range of settings / datasets.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "I looked at all experiments and results and did not find any major issues with the analysis. Overall I find the experiments comprehensive, convincing and clearly presented. The authors are very clear about the claims they make and how the experiments test these claims. The experimental rigour is significantly greater than for a typical ICML paper.\n\n\nMinor issue: I found the experimental setup for the weak-to-strong generalization experiments difficult to follow. Most relevant details are described in the appendix but I think the clarity in the main paper could be improved.\n\nA specific detail I wasn't able to understand from the paper is: \"as shown in Table 3, the actual ceiling is significantly higher if complementary knowledge of the weak supervisor is fully leverage\"\n\nDo you finetune the model first on ground truth labels and then on the outputs of the weak supervisor?", "relation_to_broader_scientific_literature": "The paper significantly improves upon previous work measuring model similarity, both in terms of the actual similarity metric proposed (which is a \"method\" contribution) as well as in the breadth and depth of the experimental analysis based on this metric. For example the weak-to-strong generalization experiments provide insides about some of the key open questions discussed in Burns et al. 2024.\n\nOverall, I think this paper is a strong contribution and will be of great interest to the ICML community.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Some results around the weak-to-strong generalization experiments are difficult to understand from the main paper.", "other_comments_or_suggestions": "none", "questions_for_authors": "* Could you explain the training of the complementary knowledge ceiling model in the weak-to-strong generalization experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hOrW0PWgxM", "reviewer": "Reviewer_nzM3", "summary": "The authors design a metric for measuring model similarity, using a combination of model performance, the type of mistakes as well as probabilistic decisions made by the model. The authors demonstrate that for various pairs of models, there is a high correlation between the measured similarity between the pair as well as the evaluation score when using one model to judge another, as well as a bias for models to judge a similar model better. They then explore the setting of weak-to-strong supervised training with a teacher model. Finally, the authors make a final observation that as models become more capable, mistakes simultaneously become more similar across a number of different model families.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad you found our **metric useful, experiments well-designed, evidence sufficient, and motivation clear**. Given these positive points, we were a bit surprised by the recommendation. We hope the new analyses and clarifications below address the concerns.\n\n**(W1)**\n> The tasks that are used for measuring similarity appears to only be based off of accuracy. However, various tasks do not directly use hard ground-truth labels\n\nOur metric uses ground truth labels to prevent the inflation of similarity scores as we illustrate with [new plots](https://imgur.com/a/XL7OBwH). **In applications where ground-truth labels are not available, one can still use the observed agreement $c_{obs}^p$ as computed by our metric**. Evaluating open-ended tasks is an exciting direction of future work [L392-404].\n\n**(W1)**\n> I believe a diversification in the number of tasks that are being evaluated for can better demonstrate the benefits of the method\n\nWe appreciate the reviewer's suggestion regarding task diversification. To further confirm the broad relevance of our method, **we extend our analysis to AlpacaEval, open-ended chat responses which do not have objective ground truth**. We find that MMLU-Pro benchmark similarity correlates strongly ($r=0.75$ to $r=0.87$) with judge preferences on AlpacaEval in [new plots](https://imgur.com/a/CIKQOmf). This reinforces conclusions in Sec 3, so we will include it in App. B of the revision. **We believe our experiments across MMLU-Pro (14 categories), BBH (23 categories, Sec 5), and 15 NLP tasks (Sec 4) are comprehensive, as noted by Reviewers RC4b and qUdH**.\n\n**(W2)** \n> In Table 2, the correlation scores can vary quite a bit based on the model size… better understand the relationship between the metric and model size.\n\nTo check whether model size is a possible confounder, **we extend our multiple regression results by including model size as a predictor -- [new plots](https://imgur.com/a/y06LkCT)**. We find that similarity remains a significant predictor of judgement score when controlling for both model size and accuracy. Furthermore, the effect size of model size when controlling for similarity and accuracy is close to 0, confirming that our focus on model capability rather than characteristics is a sound approach, consistent with prior work [1].\n\n**(W3)** \n> I feel that claims about model similarity have been extant for an extended period of time. I do not particularly believe that this work offers a significantly novel perspective on this front”\n\nWhile there might have been a common feeling in the community about models being similar, **our work is the first to quantify it comprehensively (as highlighted by reviewer qUdH) at scale** — showing a concerning increase in similarity with improving capability by analyzing 130 LLMs from various developers. Our analysis of LLM-as-a-judge extends existing self-preference results showing mitigations like using separate judges [4] are not enough. In Sec 6, we discussed previous work that explored model differences, and we will also cite theoretical results like [3] in our revision, which study the effects of algorithmic monoculture on fairness. \n\n**(W3)**\n> many of the claims remain speculative in some part\n\nWe are happy to clarify our claims and provide additional support in our revision. Could you please indicate which specific claims you found speculative?\n\n**(W3)** \n> while the metric the authors introduce can be useful, it doesn't provide a direct solution towards mitigating the problem of AI oversight\n\nWe acknowledge your observation, our metric does not directly mitigate the impact of model similarity on AI oversight. Our paper's **primary goal was to establish a robust framework for quantifying this effect, which is a crucial first step** in addressing the issue. Specifically, our findings (1) shed light on the limitations of LLM-as-a-judge systems, popular in many leaderboards [L195-204], (2) offer insights into open problems in weak-to-strong generalization, as highlighted by reviewer qUdH, and (3) reveal an inverse scaling phenomenon [2] where increasing capabilities exacerbates these issues. By quantifying the problem, we set the stage for future research on potential mitigation strategies, as discussed in Sec 7.\n\nWe thank you for the valuable feedback, which has made our paper stronger! Please let us know if there are any further questions or concerns we can resolve to increase your support for our work.\n\n**References**\n\n[1] Huh, Minyoung, et al. \"Position: The platonic representation hypothesis.\" ICML 2024. \n\n[2] McKenzie, Ian R., et al. \"Inverse scaling: When bigger isn't better.\" TMLR (2023).\n\n[3] Bommasani, Rishi, et al. \"Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?.\" *NeurIPS* (2022)\n\n[4] Decentralized Arena via Collective LLM Intelligence, [https://de-arena.maitrix.org](https://de-arena.maitrix.org/)", "claims_and_evidence": "### Claim\n\nModels favour similar models when used as a judge.\n\n### Evidence\n\nI believe that there is sufficient evidence to provide preliminary support for this claim; the LLM-as-a-Judge setting appears to demonstrate that models tend to favour similar models appears to be consistent, supporting this claim.\n\n-----\n\n### Claim\n\nThe model similarity metric, $\\kappa_p$, is an accurate predictor of model similarity.\n\n### Evidence\n\nThe authors use the model judgement score to show that the model similarity metric correlated well with judgement scores.", "methods_and_evaluation_criteria": "The methods for evaluation are appropriate and the benchmarks/datasets are reasonably chosen based on the construction of the metric.", "theoretical_claims": "N/A. Appendix A.6. computes some bounds but these are generally elementary mathematics therefore I do not see any issues with their correctness.", "experimental_designs_or_analyses": "The design is appropriate. I believe that evaluating judgement scores and comparing them against the similarity metric is a sensible experiment. In Section 4, the experimental setup is appropriate (first evaluate for similarity, then train in a student-teacher setup) and there are no claims that appear problematic. In section 5, the authors again compare accuracy on two tasks (MMLU and BBH) and compare against accuracy, demonstrating a strong correlation between accuracy and similarity.", "relation_to_broader_scientific_literature": "The results are complementary to work in evaluating the trustworthiness or robustness of LLMs, which have also shown a degree of bias in LLM-as-a-judge settings as well as overlap in LLM responses. The authors provide a new way to explicitly measure similarity, providing an interesting alternative interpretation to this setting.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "### Strengths\n\n- The work is clearly motivated and the empirical experiments are well designed.\n- There is a clear existing use-cases for the metric the authors propose.\n\n---\n### Weaknesses\n\n- The tasks that are used for measuring similarity appears to only be based off of accuracy. However, various tasks do not directly use hard ground-truth labels, such as those for summarization or instruction following. I believe a diversification in the number of tasks that are being evaluated for can better demonstrate the benefits of the method, or reveal some potential areas that need to be taken into consideration (ex. measuring similarity within reasoning chains, etc.).\n- In Table 2, the correlation scores can vary quite a bit based on the model size. This may be the direct result of model capability, as the authors explore in Section 5, but I believe there may be a need to either account for model size within the metric computation or potentially better understand the relationship between the metric and model size.\n- While I appreciate the comment that having similar models undermines oversight, I feel that claims about model similarity have been extant for an extended period of time. I do not particularly believe that this work offers a significantly novel perspective on this front, especially as many of the claims remain speculative in some part and as a result make it difficult to draw broader claims from. Furthermore, while the metric the authors introduce can be useful, it doesn't provide a direct solution towards mitigating the problem of AI oversight. Hence in my opinion, it may be better to reframe this perspective of the work in some respect.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper offers two contributions. First, it introduces a novel metric called Chance Adjusted Probabilistic Agreement (CAPA), which improves upon the previously used error consistency metric. CAPA is effective because it can distinguish between different types of errors and captures probabilistic aspects of model predictions. Second, utilizing this new metric, the paper provides analyses demonstrating affinity bias in LLM-as-a-judge scenarios, evidence that complementary knowledge offers advantages beyond elicitation in the weak-to-strong generalization setup, and an observation that models tend to make more similar mistakes as their capabilities increase.

Following the rebuttal phase, three reviewers expressed positive opinions (WA, A, SA), and the authors adequately addressed earlier concerns such as lack of benchmarking, evaluation and experimental clarity, and the novelty of their contributions compared to existing literature and metrics. One reviewer remained skeptical with a WR score due to the speculative claims and lack of direct solution towards mitigating the problem of AI oversight. In my view, the contributions presented in this paper are sufficiently significant, even considering any remaining concerns such as lack of a direct solution. Therefore, I recommend acceptance.
Decision: Accept (spotlight poster)