Monoculture or Multiplicity: Which Is It?
Authors: Mila Gorecki
Moritz Hardt
Venue: NeurIPS 2025 poster ()
Keywords: monoculture
multiplicity
large language models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We systematically evaluate the concerns of multiplicity and monoculture in a suite of large language models and prediction tasks.

Abstract: Two narratives about machine learning ecosystems grew out of recent algorithmic fairness discourse. In one, dubbed \emph{monoculture}, algorithmic ecosystems tend toward homogeneity akin to a single model making all decisions. Individuals then face the risk of systematic exclusion with no recourse. In the other, \emph{model multiplicity}, many models solve the same task with similar accuracy, causing excessive variation in outcomes. Both narratives are compelling, yet, seemingly at odds: model multiplicity can’t exist in a strict monoculture. In this work, we conduct a comprehensive empirical evaluation to test both claims. We work from the premise that increasingly decision makers will use large language models for consequential prediction tasks. We therefore examine 50 language models, open source models ranging in size from 1B to 141B parameters and state-of-the-art commercial models, under 4 different prompt variations, and across 6 different prediction tasks. Evaluating both new and old quantitative measures of monoculture and multiplicity, we find the empirical landscape sits between the two extremes. Each narrative finds some empirical support, but neither is dominant. Systematic exclusion with no recourse is rare, but model similarity is real. Even when starting from a single model, prompt variation induces some diversity in predictions. Our results contribute critical empirical grounding to ongoing debates and point toward a middle ground between monoculture and multiplicity as the most realistic outcome.

Monoculture or Multiplicity: Which Is It?
Mila Gorecki
Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen Tübingen AI Center
{mgorecki, hardt Stardttttueinggen.mppg de
Abstract
Two narratives about machine learning ecosystems grew out of the recent algo- rithmic fairness discourse. In one, dubbed monoculture, algorithmic ecosystems tend toward homogeneity akin to single model making all decisions. Individuals then face the risk of systematic exclusion with no recourse. In the other, model multiplicity, many models solve the same task with similar accuracy, causing excessive variation in individual outcomes. Both narratives are compelling, yet, seemingly at odds: model multiplicity can't materialize in strict monoculture. In this work, we conduct a comprehensive empirical evaluation to test both claims. We work from the premise that increasingly decision makers will use large language models for consequential prediction tasks. We therefore examine 50 language models, open source models ranging size from 1B to 141B parameters and state-of-the-ate commercial models, under 4 different prompt variations, and across 6 different prediction tasks. Evaluating both new and old quantitative measures of monoculture and multiplicity, we find the empirical landscape sits between the two extremes. Each narrative finds some empirical support, but neither tightly fits the observations. Systematic exclusion with no recourse is rare, but model similarity is real. Even when starting from a single model, prompt variation induces some diversity in predictions. Our results contribute empirical insights that illuminate the underappreciated middle ground between monoculture and multiplicity.
Introduction
Bureaucratic decisions have always provoked two distinct anxieties. One is the fear of systematic exclusion with no recourse, discretion, or alternatives [Weber] 2019 Merton
1940
Eubanks
2019
The other is a fear of haphazard, inconsistent, and arbitrary decisions [Lipsky In one case decision making too rigid, the other it too loose.
1980
Pasquale 2016
These concerns extend to algorithmic ecosystems networks of multiple institutions that each use algorithms for consequential decisions about individuals. In this context, the two fears map onto growing academic discourse about algorithmic monoculture and model multiplicity. In monoculture, the algorithmic ecosystem collapses to single rule leading to homogeneous outcomes and reduced welfare [Creel and Hellman 2021 Kleinberg and Raghavan 2021 Bommasani et al., 2022]. In contrast, model multiplicity proliferates the algorithmic ecosystem with inconsistent outcomes [Breiman 2001 Marx et al. 2020]. Both narratives issue compelling warnings of what society might face, but there apparent tension between the two. One problem, severe enough, is an antidote to the other [Black et al.) 2022 Ganesh et al. 2025 Gur-Arieh and Lee 2025
In this work, we systematically study the degree which either narrative fits the empirical landscape. In doing so, we anticipate a near future where decision makers prompt large language models for risk assessments and predictions about individuals. Traditionally, institutions have procured special-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

language ecnxystem
ACSIncome
fraction positive predictions
.75
good models
.5
observed
random error
.25
random prediction
monoculture
strong multiplicity random errors
strict monoculture
25
5
75
75
aligned errors
fruction model pairs
fraction positive instances
Figure 1: Left. Schematic our setup. We empirically measure the degree of multiplicity and monoculture present in recent large language models. We zero-shot models and select those with accuracy within c = 0.05 of the best. Middle. Agreement curve for ACSIncome: x% of model pairs agree
most y% of the positive instances. Observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray), but well below strict monoculture (red). Right. Recourse curve for ACSIncome: x% of positive instances are accepted by more than y% of models; e.g., 25% of instances are accepted by no more than 66% of the models. Under strict monoculture (red, dotted) all models collapse to one, individuals either accepted rejected by all; the accepted fraction equals to the model's true positive rate (here: mean TPR). The y-axis bar plot shows the distribution of recourse levels.
purpose predictive models for consequential Decisions-prriariin regression models trained on task-specific tabular records. Pre-trained language models promise tempting alternative: these models work with any data encoding and flexibly solve numerous tasks, thus mitigating inefficiencies in the old machine learning pipeline.
How the use large language models influences the situation, however, is neither obvious nor has been studied systematically to date. Large language models share overlapping training data -much of the accessible internet-anc and there are relatively few model providers due to excessive training costs. These factors seem to promote monoculture. On the other hand, much research on evaluation points to the inconsistencies and instabilities of answers generated from large language models.
We broadly evaluate 50 large language models across seven prediction tasks and under different prompting strategies. What emerges is a rich empirical landscape that defies either of the two narratives. Language models show significant variation in their predictions, largely contradicting the idea of a monoculture. When rejected by one model, individuals consistently find recourse other models. At the same time, model similarity is real and model errors are far from independent. To the extent that model multiplicity occurs, however, its presence isn't strong enough to fully mitigate the problem of systematic exclusion for non-negligible fraction of the population. Our findings point at the unglamorous middle ground between monoculture and multiplicity as the most likely place for algorithmic ecosystems to end up in.
1.1 Our contributions
We empirically measure the degree of multiplicity and monoculture present in recent large language models under zero- and few-shot prompting. We evaluate 50 large language models, both open source models ranging in size from IB to 141B parameters and state-of-the-art commercial model, across 7 standard prediction tasks, six based on U.S. Census Data and one from the medical domain.
Models share inductive biases. We evaluate the extent to which different models agree in their predictions under identical zero-shot prompts. focusing on the Rashomon ser medels achieving accuracy near the best (Figure [ left panel). We consider two extremes: Under strong multiplicity, all models err randomly. For instance, two models with 80% accuracy would agree in 68% of their predictions errors were randomly located. Higher agreement rates indicate model similarity. Under strict monoculture, agreement rates approach 100%. Figure (midlee panel) shows that the

---

empirically observed agreement lies solidly between the two extremes, far from monoculture, but also far from strong multiplicity.
Individuals generally have high recourse. We define the recourse level of an individual with respect to model set the fraction of models that accept that individual, capturing the number of ways they have in finding a positive prediction within the model ecosystem. We focus on positive instances, i.e., those individuals that should be accepted. Figure I (right panel) shows the recourse curve, corresponding to the recourse level for each population quantile. For example, 7% are wrongly rejected by all models, thus having recourse, while 51 % are correctly accepted by all models. Recourse is generally high with 79% of positive instances accepted by at least half the models. The observed recourse curve is far from what it would look like under either monoculture or multiplicity.
Prompt variation provides recourse even in a single model. Previously, we gave different models identical prompts. In reality, it is extremely unlikely that two institutions would prompt models in exactly the same way. Different institutions likely use different data encodings or follow different practices. We consider only minor prompt variations including changes in feature order, granularity, and formatting. Even these minor variations provide recourse levels that rule out monoculture.
All findings are robust to few-shot prompting. We repeat our analysis under 10-shot prompting. Few-shot prompting generally improves accuracy. However, the results about model agreement and individual recourse are consistent and suggest the same conclusions.
Summary. Our evaluation suggests that neither monoculture nor multiplicity tightly fits the empiri- cal observations. Recourse levels are far from perfect, but sufficiently high to rule out monoculture. This is true even though our setup favors monoculture: we evaluate similar language models under identical use or minor prompt variations. Model multiplicity is a real phenomenon, but models nevertheless share strong inductive biases. Their predictions exhibit high agreement rates, well above those we'd see under randomly located errors.
1.2 Related Work
The study of model multiplicity and the notion of the Rashomon set go back to Breiman [2001] In recent years, this phenomenon has again gained traction [Black et al. [022 Creel and Hellman 2021
D' D'Amour et al. 2020 Jain et al., 2024]. Much prior research examined multiplicity in traditional
models such of regression, linear classification, tree-based models, and small neural networks
in controlled settings MMrxx et al. 2020 Bommasani et al.) 2022 Semenova et al.) 2022 Castillo
al. 2008 Hsu and Calmon 2022 Huu al. 2024]. By systematically varying either training data
or hyperparameters, these studies examine the effects that different training practices have on the
Rashomon set. To address the concerns of arbitrariness and justifiability, other lines of work either
mitigate multiplicity by aggregating models from the Rashomon set [Black et al.] 2021 [022 Cooper
al.
2024| Roth et al. 2023 Behzad al., 2025 or leverage
incorporate additional selection
criteria Semenova et al. 2022 Coston et al. 2021 Black et al. 2024].
Creel and Hellman 202 argue that monopolistic power of few companies may lead algorithmic
leviathans, resulting in arbitrary decisions deployed at scale. They propose to intentionally introduce
variance the model, that is, counteract monoculture by intentionally introducing multiplicity.
Kleinberg and Raghavan| [2021] coin the term algorithmic monoculture and provide a theoretical model where it can occur. Other theoretical models explore the relationship of competition and monoculture [Jagadeesan et al. 2023 RRghavan 2024). Black et al. 2022 argue that multiplicity can be a bulwark against monoculture, providing opportunities to choose the model that best promotes fairness or welfare among equivalent ones. [annesh et al. (0055 give a comprehensive discussion of
the moral concern of arbitrariness and its relationship to multiplicity and monoculture.
Closely related to monoculture, Bommasani et al. 2022 formalize outcome homogenization as the amplification of systemic denial relative to what would be expected under independent model predictions. The authors argue that we may see growing homogenization due to the rise of foundation models and conduct experiments
vision-text model (CLIP) and early language model (RoBERTa-baae Toups et al. (2024) analyze systemic failure and outcome homogenization for three commercial APIs used for emotion recognition and sentiment analysis.
There is broader debate about the diversity of content generated by large language models that's beyond the scope of our work, as we focus on predictions and risk assessment.
3

---

2 Measuring monoculture and multiplicity
(< yt)] be dataset of N i.i.d. samples from joint distribu- tion P(x, ))), where each I2 E X is feature vector and E (0, 1} is the corresponding label. A binary predictor is a function h: X' - {0, 1). As algorithmic monoculture and model multiplicity arise algorithmic ecosystems, we consider a model set M = {h,, .......hh hM ). which consists of M binary predictors.
Preliminaries. Let D
Prior work on multiplicity focuses on a set of good models that achieve accuracy close to that of a reference model ho, typically the best model found by empirical risk minimization. In reference to the Rashomon effect [Breiman| 2001], this model set is called Rashomon set.
Definition 1 (Rashomon set, c-level set). Given baseline model ha, performance metric 8, and error tolerance e, the Rashomon set given by R. = (he H 0(h) 2 0(ha)) -e).
Throughout the paper we use accuracy as performance metric and refer to the parameter e as accuracy deficit. We call the set of available models that deviate in performance at most by from the best model found the empirical Rashomon set to indicate that we don't 't have the full Rashomon set R.. Note that characterizing the full Rashomon set becomes computationally infeasible for large hypothesis classes H. rendering efficient search a core issue of predictive multiplicity [Marx et al. 2020 Hsu and Calmon, 2022 Hsu et al. (023).
Definition 2. Given two classifiers h, h', their pairwise agreement over dataset DD is given by the
fraction of individuals their predictions agree
agree(h,, h²) - # 1[ = h (xx))
Pairwise agreement quantifies model similarity based on their predictions. Note that models can agree on both correct and incorrect predictions. We take strict monoculture to correspond to agreement 1P] Strong multiplicity corresponds to the agreement rate we'd d expect if errors were randomly located. For two models h / with accuracies a and a', respectively, this corresponds to the expression agreCrand (h, h') ad' + (1 a1 - a').
In this work, we focus on predictive multiplicity situations where competing models assign conflict- ing predictions to the same individual. For our analysis, we also adopt two commonly used measures, ambiguity and discrepancy. Note that several other forms of multiplicity have been studied, see Ganesh et al. [2025] for a comprehensive overview.
Definition 3 (Ambiguity, [Marx et al. 2020)) The ambiguity of prediction problem over the model set M is the proportion of points V; in a dataset that can be assigned a conflicting prediction by a competing classifier h compared baseline classifier ho E M:
(hha)= E MaXHEM 1 (1x))) * hz()))]
(2)
Ambiguity captures the extent to which individuals are affected by the choice of model, because they would receive conflicting prediction from another model in the Rashomon set. When switching the baseline model for competing model, the conflicting predictions must all be realized by that particular model. Discrepancy captures the maximum change that can be realized by any other model in the Rashomon set relative to the baseline classifier.
Definition 4 (Discrepancy, (Marx al., 2020)). The discrepancy of prediction problem over the model set M is the maximum proportion of conflicting predictions between the baseline classifier ho and any competing classifier h:
8(hp) = maxieM N
1 ΣN (M(a)) + ho():))
Note that ambiguity and discrepancy are monotonic, meaning they can only increase as additional models are considered. Given that our empirical Rashomon set may omit some models achieving accuracy similar to the best model, monotonicity ensures that our measures provide lower bounds on those computed from the full Rashomon set.
"Acurracy differences within the Rashomon set can oppose strict monoculture: models with accuracies a A a' can agree on at most 1 la a'l the total population. This bound, however, applies only at the population level. Our analysis focuses on positive instances. Thus, the bound does not directly apply without additional assumptions about conditional agreement. If the positive class constitutes at least an e-fraction of the data, perfect agreement within this subset remains possible, even if the models' overall accuracies differ slightly.

---

Recourse with respect to model set. The concept of monoculture implies homogenization of predictions, wherein, in the extreme case, all peccisin--maken rely on a single model. To measure its impact on individuals, BBomasaan et al. [2022] consider the fraction of individuals receiving negative prediction from all models. We generalize this idea by introducing the notion of recourse with respect to a model ser. It captures the ability of an individual to obtain a favorable outcome by switching to a different model within the model set.
Definition 5 (Recourse with respect model set), For an instance x E X, we define recourse level with respect to model set M as rea(x)- [M] EnEm 1[((x) = y²), where y" denotes the
favorable outcome.
Note that this notion of recourse of an individual is property of a model set. In contrast to the single-model notion of algorithmic recourse (Ustun et 2019], we consider favorable predictions obtained by changing the model, not the individual features. For our analysis, we distinguish:
Full recourse: rec(z) =
Substantial recourse: rec(x) 0.5, the majority of models yield favorable outcome.
Limited recourse: rec(x) v 0.5, the majority of models yield unfavorable outcome.
No recourse: rec(2) 0, the individual faces systematic exclusion.
The fraction of individuals having no recourse corresponds to the observed rate of systemic failure defined by [Bommasani al. (022). They further define the notion of outcome homogenization as the observed ratio of individuals with no recourse relative to the fraction of individuals that would have no recourse, if model errors were random.
The notion of a favorable outcome depends on the prediction task and context. may correspond to predicting 1 (e.g. loan approval), 0 (e.g. fraud detection), to correct prediction irrespective of the label. In this work, we focus on the recourse level of positive instances - individuals who, according to their ground-truth -ruuh label, should be granted opportunity. We treat y" 1 as the favorable outcome, allowing us to analyze cases where such opportunities are denied. Under this framing, the average recourse level among positive instances natural generalization of the true positive rate from a single model to a model family.
We generalize the random error baseline used by Bommasani et al. [2022 to varying recourse levels. Recall, strict multiplicity corresponds to the case where errors are randomly located. Assuming that model hm has true positive rate pm: the fraction of models accepting fixed positive instance follows Poisson- Poisson-Bimomial distribution so that
M (m))
E[rec(=)] M
1
Σ
II
Pm (1 Pmm 1-x(m)))
44)
m=1
where 1R describes the set of distinct permutations successes and M k failures. Further let (m) E {0, 1} indicate whether for permutation model m provides success (positive prediction) or a failure (negative prediction). Note that the success probability in the baseline should reflect the outcome deemed favorable. In our case, since we are focusing on positive instances achieving positive outcomes, Pm corresponds to the true positive rate. More generally, when considering correct classifications, the appropriate metric would be accuracy. Conversely, a negative outcome (i.e., class 0) is considered favorable, the corresponding success probability is the true negative rate.
3 Experimental Design
Working from the premise that decision makers will increasingly use language models for decision making, we evaluate the consistency of model predictions across diverse set of language models on multiple binary classification tasks. In each task, models are prompted with natural-tes representation of the features x using standard multiple-choice format. We then compare similarity of model outputs and their impact on individuals varying either the models or the prompt.
Prompting language models. We evaluate 50 language models, including both open-source models-rane cadein-ranging from IB to 141B parameters-ann state-of-the-art commercial models. A complete
5

---

Table 1: Recourse levels and multiplicity measures across tasks using identical prompts.
substantial recourse
full
task
Re
ambiguity
discrepancy
ACSIncome
27
0.07
0.80
0.51
0.43
0.19
ACSEmployment
8
0.06
0.82
0.64
0.30
0.15
BRESS Blood Pressure
23
0.03
0.59
0.14
0.83
0.38
SIPP
16
0.05
0.77
0.48
0.47
0.18
ACSTravelTime
12
0.04
0.72
0.24
0.72
0.35
ACSPublicCoverage
21
0.14
0.38
0.00
0.86
0,47
ACSMobility
0.68
0.00
0.00
0.32
0.20
list is provided in Appendix A.1 For open-source models, we include both base pretrained variants and their instruction-tuned counterparts available. We use the folktexts package [Cruz et al. 2024) to zero- and few-shot the language models in a standardized multiple-choice format and to extract predicted risk scores. Since language models are widely miscalibrated on non-rcalizable tasks, adopt the approach from Cruz al. [2024] to calibrate the model predictions. For each model and task, we fit a decision threshold on n 2000 samples from validation set to maximize balanced accuracy. The threshold is then applied to turn the risk scores into class predictions.
Prediction tasks. We evaluate model predictions on seven binary classification tasks derived from three data sources. Five tasks are based on the American Community Survey (ACS) Public Use Microdata Sample (PUMS), high-quality dataset from the U.S. Census Bureau [Flood et al., 201 8). Ding et al. [2022] construct several predictions tasks on this dataset, for which folktexts [Cruz et al. 2024] provide natural-text mappings for prompting. We use ACSIncome (individual's income is above $50, 000), ACSEmployment (individual is employed), ACSPublicCoverag! (individual is covered by public health insurance) and ACSTravelTime (commute time to work is greater than 20 minutes) and ACSMobility (individual moved in the last year). To complement these, we extend folktexts by two additional tasks. BRESS Blood Pressure is a health-related prediction task introduced by Gardner ctt al. [2024) on lcage surveys from the Behavioral Risk Factors Surveillance System [BRFSS, Centers for Disease Control and Prevention (CDC) 2021]. The task is to predict if individual has been diagnosed with hypertension. SIPP is defined on the longitudinal Survey of Income and Program Participation [SIPP, U.S. Census Bureau 2014). Here, the goal is to predict whether person's ' income is significantly above the Official Poverty Measure (OPM). See Appendix (..2 for a detailed description of all tasks. We randomly sample 10% of each dataset for evaluation, yielding test sets ranging in size from approximately 60,000 to 320,000 instances for most tasks. The SIPP task has smaller test set of almost 4,000 instances.
Our analysis focuses on the impact on positive instances, that is individuals with a positive label y = 1. We refer the reader to Appendix [D.] for results for negatives instances. In the next two sections we examine similarity of model predictions for identical prompts across differrnt models (Section 4 and, starting from single model, explore the impact of prompt variations on monoculture and multiplicity (S
4 Models disagree on identical prompts
central concern increasingly interconnected algorithmic ecosystems is the potential conver- gence toward homogeneity where different models yield highly similar outputs that, if flawed, could systematically limit individuals' opportunities. To explore the degree of homogeneity and its consequences, evaluate our base set of 50 language models using identical zero-shot prompts. Ag- gregate performance varies considerably (Appendix B) We thus restrict our analysis to the Rashomon mod-msdels whose accuracy falls within 0.05 of the best-performing model. Across tasks, the empirical Rashomon set comprises models from at least four providers, encompassing different sizes as well both base and instruction-tunee variants.
Model predictions are highly similar. As shown in Figure (left), pairwise comparisons on the ACSIncome task reveal that all model pairs exhibit substantially higher agreement (blue line) than would be expected if errors were randomly distributed (orange line). Specifically, model pairs

---

Income (M=27)
Employment (M=8)
Blood Pressure (M=23)
SIPP (M=16)
fraction of positive predictions
.75
-5
.25
.5
.75
1
0
.25
-5
.75
0
.25
5
.75
0
.25
5
75
Traction of positive instances
TravelTime
PovlicCoverage (M=21)
Mobility (M=5)
(M=12) 12)
observed
random error monoculture
-5
.25
0
.25
.75
25
5
.55
0
.2 .75
Figure 2: Recourse curves across tasks: x% of positive instances are accepted by no more than y% of models. We zero-shot models and select those achieving accuracy within 0.05 from the best. For example, on ACSIncome we observe (blue) 20% of positive instances being accepted by at most 50% of the models. Under random errors (orange) this would rarely happen. Under strict monoculture (red, dotted) individuals only experience no or full recourse. Here, the mean TPR is used for illustration. The bar plot shows the distribution of recourse levels.
agree on least 75.34% of positive instances, with mean agreement of 88.37%. In contrast, the random-error baseline yields agreement rates between 61 11.69% and 66.49% (mean 63. 66%). These results demonstrate that models share considerable inductive biases. However, agreement remains below perfect consensus, which would be observed in a strict monoculture. This suggests that models are highly similar, but not perfectly aligned their predictions. This pattern holds when disaggregating by model provider, model variant, or demographic groups (Appendix C Extending this analysis across tasks (Figure 19 top panel) shows a consistent pattern of high predictive similarity, with empirically observed agreement rates consistently falling between the two extremes of strong multiplicity and strict monoculture.
Individuals generally have high recourse. Pairwise agreement rates unless exactly 0 or do not directly capture the individual-level impact of predictions within a model ecosystem. In particular, they do reveal the extent to which entire model sets concur on an individual's outcome. To address this, we examine recourse: the fraction of models that accept an individual, reflecting the individual's ability to obtain favorable outcome by switching to another model. The recourse curve in Figure (right) shows these values across population quantiles for ACSIncome. We find that 79% of positive instances are accepted by at least half of the models. The median individual is accepted by 96% of the models, compared to approximately 78% under the random-erro baseline. These results suggest that recourse levels are generally high. Notably, 7% of positive instances are rejected by all models, meaning they receive no recourse. Conversely, 51% are correctly accepted by all models. Relative to baseline of random errors, we observe a higher likelihood of both extreme outcomes, no recourse and full recourse, reflecting the influence of shared inductive biases. At the same time, the distribution recourse levels differs considerably from a strict monoculture, wherein the recourse curve would collapse into step function determined by the false negative rate of the sole remaining model. Thus, despite substantial predictive similarity, individuals largely retain the opportunity to obtain favorable outcomes by switching models.
Extending this analysis across tasks (Figure 2 top panel, and Table reveals a generally consistent pattern of high recourse levels, with small fraction of individuals experiencing no recourse. Across tasks, the distribution recourse levels largely falls between the extremes of strong multiplicity and strict monoculture. Two tasks, ACSMobility and ACSPublicCoverage deviate from this pattern, exhibiting higher rates of no recourse and smaller fractions of individuals with substantial recourse. Both tasks characterized by low predictive signal and high class imbalance, which also affect the
7

---

Blond Pressure
Travel Time
Public Coverage
Income
Employment
SIPP
Mobility
[R.] prevalence
0.9
0.6
rea(a)
0.3
0.0
0.45
0.30
0.15
24
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
accuracy deficit.
observed
random error
Figure 3: Severity of monoculture and predictive multiplicity as a function of accuracy deficit from the best model. Each column corresponds to one task. Predictions are obtained via O-shot prompting. Top. The fraction of positive instances that experience recourse (blue) is consistently higher than what would expected under random errors (orange). While stable across different Rashomon sets, the gap between the observed and expected fraction of no recourse decreases with increasing e. Middle. Discrepancy increases with e, potentially affording opportunities for recourse for some individuals. Due to monotonicity, their values are likely higher for the full Rashomon set. Bottom. Number of models the empirical Rashomon set.
Rashomon set. See Appendix D.1 for a detailed discussion. We further assess recourse levels for different demographic groups defined by sex, race, and age, examine whether predictive similarity affects these groups differently. While levels of agreement and recourse are comparable between female and male individuals, observe notable differences in multiple tasks when disaggregating by race or age. These observations provide preliminary evidence that certain groups may face disproportionate barriers finding recourse. Differences between tasks suggest that disparities may also be dusk-dependent. A more comprehensive understanding of these dynamicc remains an important direction for future work. Further details and analysis are provided in Appendix C.2
Multiplicity is high, but might still underestimate it. As our analysis limited available pre-trained models, we cannot influence training choices or exhaustively explore the parameter space. Nonetheless, actually deployed by decision makers. Among those models, we find substantial discrepancies: on ACSIncome, 19% of positive instances receive different classification when switching another model (Table D: across tasks, at least 15% do. Both ambiguity and discrepancy are monotonic with respect the size the Rashomon set [Marx et al., 2020). Therefore, exhaustive search over the parameter space would likely yield even higher values for both measures.
empirical Rashomon likely includes models representative of those
Impact of accuracy deficit The accuracy deficit specifies the maximum performance gap among models the Rashomon set. To assess its impact on our findings, we compute discrepancy and the prevalence of no recourse across varying values of (Figure 3). Even small increases in rapidly expand the empirical Rashomon set and observed discrepancy & indicating that many models achieve comparable accuracy while generating divergent individual- predictions. This diversity creates potential avenues for recourse and reduces the likelihood that decision makers converge on single model. Nevertheless, whenever the Rashomon set includes more than one model, non-negligite fraction individuals experiences no secourse-ubsstntiiaa higher than expected under random errors. Full recourse curves for varying values of are provided in Appendix D.2
Consistent under few-shot prompting We repeat our analysis under 10-shot prompting, providing each model with 10 class-balanced examples. We find that aggregate performance increases slightly, resulting in overall larger empirical Rashomon sets. Observed recourse levels remain at similar order of magnitude with slightly decreased discrepancy, indicating a mild trend towards monoculture. See Appendix E for detailed results.

---

feature order
format
connector
granularity
0.79
accuracy
0.77
0.75
0.73
Qwen 2.5 7213 (it)
Qwen 2.5 721 (ic)
Qwen 2.5 72B (it)
Qwen 2.5 7B (it)
Qwen 2.5 7B Liama 3
70B
Qwen 2.5 7B
Liama 3.8
7013
Qwen 2.5 7
7013
Liama 3 8B
Llana 70B
Qwen 2.5 72B
Llama 3.3
Liama
Liama 3.3
Llana
Figure 4: Minor prompt variations induce changes in accuracy of up 3 percentage points, con- sistently across models on ACSIncome. Each subplot varies a single aspect of the prompt, keeping the others fixed to default. Light blue dots show accuracy for individual variations, dark blue dots indicate the mean accuracy with error bars.
5 Recourse through minor prompt variations
Under monoculture, algorithmic ecosystems tend toward homogeneity resembling a single model making all decisions. To simulate this in the context of LLMs used for consequential dlecisionmmaking we select the two best-performing models on ACSIncome - -Llama 3.3 70B (it) and Qwen 2.5 72B (it) along with smaller variants, Llama 8B (it) and Qwen 2.5 7B (it). Even after training. LLMs accept a wide range prompt formats, styles or even modalities. This flexibility means that decision- may naturally vary in how they compose prompts. In line with prior work on prompt sensitivity of language models [Petroni et al., 2019, Shin et al. 2020 [elar et al.) 2023] Voronov et al. 2024], we find that minor prompt variations cause significant changes in model predictions.
Prompt variations. We vary four minor aspects of prompt construction: feature order, testing five orders in which features of an individual are presented the default order given by folktexts, its reverse, and three random permutations: format, presenting features as buller list, a commu-separated list plain text in the form '<feature name> <connector> <feature value>'; connector, choosing the symbol linking <feature name> and <feature value> among 'is', "=' and ":"; and granularity, using either the original feature mapping provided by folktexts a lowerrccollution version such as age groups instead of the exact age. Examples are provided in Appendix G
We find that varying single aspect of the prompt construction can shift accuracy up to 3 percentage points (Figure 4) and with it, induce predictive multiplicity. The effect on accuracy is consistent across models, with no clear trend by size or model family. While granularity changes alter the information content, we find their impact on accuracy to be comparable to that of other variations.
Prompt variations still induce substantial multiplicity. Since decision- makers may vary in more than one aspect of how they construct prompts, we evaluate all four models on the cross-product of these variations, resulting in V = 90 distinct prompting styles per model. To compare effects of prompt variations and model changes directly, we fix the number of prompts and models to be the same: we randomly sample M prompt styles (M 27 for ACSIncome) and evaluate agreement and recourse across 100 independent repetitions (Figure 5joop panel). Pairwise agreement across models increases under prompt variations (blue), though it remains similar to the agreement observed when varying the model under identical prompting (gray line), suggesting that substantial disagreement persists in both settings. Comparing recourse curves (bottom panel), we find that prompt variation leads to a higher fraction of positive instances with full recourse, while the rate of instances with no recourse remains similar. This suggests mild trend toward monoculture; nevertheless, prompt variations alone still enable recourse for a considerable fraction of individuals, with the majority experiencing substantial recourse.
Prompt variations and few-shot prompting Few-shot prompting introduces additional sources of variation, such as the choice and ordering of examples. Consistent with prior work (Zhao et al., 2021 Lu et al. 2022 Gao et al. 202T Schick and Schütze, [02T], we find that these factors impact model performance and add variability model predictions, impacting recourse observe (Appendix F

---

Qwen 2.5 7B (it)
Llama 8B (it)
Llama 3.3 70B (it)
wenn 2.5 72B (it)
.5
25
.75
25
25
.75
fraction of model variant pairs
fraction of variants accepting
0
.25
.5
.75
1
0
.25
-5
-10
0
.25
-5
.75
1
0
.25
55
.75
1
fraction positive instances
observed
random error
same prompt
random prediction
monoculture
Figure 5: Agreement and recourse across prompt variations on ACSIncome. We zero-shot models with varying prompting styles, subsampling V 27 prompt variations per model to match the number of different models Figure 1 Top. Agreement curve: x% of prompt variation pairs agree on up to y% of the positive instances. Observed agreement rates (blue) surpass those under random errors (orange), but remain similar to agreement rates observed with identical prompting across different models (gray, solid). Bottom. Recourse curve: x% of the positive instances are accepted by at most y% of the prompt variations. For example, 25% of the positive instances are accepted by at most 50% of the variations. Observed recourse (blue), random errors (orange). Under strict monoculture (red, dotted) individual only experience no or full recourse. Here, the mean TPR is used for illustration, Bar plot the y-axis shows density function of recourse level in the population.
Discussion
Our evaluation suggests that neither monoculture nor multiplicity fits the empirical landscape. Every way we look at it, the observations fall strictly between the two extremes of strict monoculture and strong multiplicity. Still, predictive similarity among the evaluated models high: across tasks, notable fraction of individuals face no recourse, yet disagreement between models provides opportunities for the majority to find recourse. Disaggregating by demographics reveals that these effects are unevenly distributed, with certain groups disproportionately affected. These findings underscore the need for deeper understanding of the system-levels dynamics and the individual harms arising from them. We believe that conducting such system-level analyses can serve as an important initial step in this direction. Some might contend that our experimental setup is biased toward monoculture. After all, we evaluate all models on exactly the same dataset. In reality, two different institutions almost certainly collect different datasets. The records that two different institutions have about the same individual are likely quite different in most cases. This introduces an additional source variation that we don't capture. As a result, our empirical findings likely understate how far reality is from monoculture. We study monoculture in the sense of systematic exclusion from opportunity in the context of consequential decision making using predictive risk assessment. We do not address broader sociological questions about how the use of language models might homogenize culture and expression.
In our study, we work from the assumption that increasingly decision makers will prompt language models for consequential decisions. Admittedly, this is look into the future rather than the present. But it's highly plausible near future given current trends of adoption of language models. Language models must strike institutions like perfect for the mundane bureaucratic processes in which consequential decisions take place. These models can sift through volumes of messily encoded data and come up with some answer to any question. It's urgent, then, to ask: What harms might individuals face in language model ecosystem that governs consequential decisions? Our work points to a troubling conclusion: the harms individuals face resist capture by either the lens of monoculture or that of multiplicity.
10

---

Acknowledgments and Disclosure of Funding
We thank Yatong Chen, Ricardo Dominguez-Olmedo, Karolin Frohnapfel, Mina Remeli, and Mara Seyfert for invaluable feedback earlier versions of this paper. We also thank André Cruz and Vivian Nastl for their continued support with implementation and data-related aspects. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Mila Gorecki.
References
OL.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengeheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Open Foundation Models by 01.AI, Jamuary 2025.
Barry Becker and Ronny Kohavi. Adult Dataset, 1996.
Tina Behzad, Silvia Casacuberta, Emily Ruth Diana, and Alexander Williams Tolbert. Reconciling Predictive Multiplicity in Practice ICML Workshop Version, January 2025.
Emily Black, Klas Leino, and Matt Fredrikson. Selective Ensembles for Consistent Predictions. In Intermational Conference on Learning Representations, October 2021.
Emily Black, Manish Raghavan, and Solon Barocas. Model Multiplicity: Opportunities. Concerns, and Solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22. pages 850-863. New York, NY, USA. June 2022. Association for Computing Machinery. ISBN 378-1-440 378-1-4500-9352-22 doi: 10. 145//5311 146.3533149.
Emily Black, Logan Koepke, Pauline Kim, Solon Barocas, and Mingwei Hsu. The Legal Duty to Search for Less Discriminatory Algorithms, June 2024.
Rishi Bommasani, Kathleen A. Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?, November 2022.
Leo Breiman. Statistical Modeling: The Two Cultures. Statistical Science, 16(3):199-215, 2001. ISSN 0883-4237.
Enrique Castillo, Roberto Minguez, Carmen Castillo, and Antonio S. Cofiño. Dealing with the multiplicity of solutions of the El and foo regression models. European Journal of Operational Research, 188(2):460-484, July 2008. ISSN 0377-2217. doi: 10. (016//eejor::000 .0.0000..
Centers for Disease Control and Prevention (CDC). BRESS survey data (2015, 2017, 2019, 2021), 2021.
A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grim- melmann, Jon Kleinberg, Siddhartha Sen, and Baobao Zhang. Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification, March 2024.
Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing Fairness Over the Set of Good Models Under Selective Labels. In Proceedings of the 38th International Conference on Machine Learning, pages 21 2144-2155. PMLR, July 2021.
Kathleen Creel and Deborah Hellman. The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision Making Systems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT 21, page 816, New York. NY, USA. March 2021. Association for Computing Machinery. ISBN 978- 978-1-4503-8-88 1-5503-8309-7 doi: 10.1145/34421 011144444188..4455000 3445942..
André Cruz, Moritz Hardt, and Celestine Mendler-Dünner. Evaluating language models as risk scores, July 2024.
Alexander Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang. Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecifiiction Presents Challenges for Credibility in Modern Machine Learning. November 2020.
11

---

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning Advances in Neural Information Processing Systems, 34, January 2022.
Virginia Eubanks. Automating Inequality: How High-Teck Tech Tools Profile, Police, and Pumish the Poor. Picador St. Martin's 's Press, New York, first picador edition edition, 2019. ISBN 978-1-250-077333 90111-7777------222 1-250-21578
Sarah Flood. Miriam King, Renae Rodgers, Steven Ruggles, and J Robert Warren. Integrated Public Use Microdata Series, Current Population Survey: Version 6.0. Minneapolis, MN: IPUMS, 2018.
Prakhar Ganesh, Afaf Taik, and Golnoosh Farnadi. The Curious Case of Arbitrariness in Machine Learning, January 2025.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 3816-3830, Online, August 2021, Association for Computational Linguistics. doi:
Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking Distribution Shift in Tabular Data with TableShift, February 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khy- athi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nis- hant Subramani. Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer. Jesse Dodge, Kyle Lo., Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the Science of Language Models, June 2024.
Shira Gur-Arich and Christina Lec. Consistently Arbitrary or Arbitrarily Consistent: Navigating the Tensions Between Homogenization and Multiplicity in Algorithmic Decision-Making. Proceedings of the 2025 ACM Conference on Fairness, Accountabilit, and Transparency, FAceT '25, pages 3336-3349, New York, NY, USA, June 2025. Association for Computing Machinery. ISBN 979-8-4007-1482-5. doi: 10.1145/
3715275.3732215.
Moritz Hardt and Michael Kim. Backward baselines: Is your model predicting the past?, June 2022.
Hsiang Hsu and Flavio du Pin Calmon. Rashomon Capacity: A Metric Predictive Multiplicity in Classification. In 36th Conference on Newral Information Processing Systems (NeurIPS 2022). arXiv, October 2022.
Hsiang Hsu, Guihong Li, Shaohan Hu, and Chun-Fo Fu Chen, Propout-Basse Rashomon Set Exploration for Effi- cient Predictive Multiplicity Estimation. In The Twelfth International Conference on Learning Representations, October 2023.
Hsiang Hsu, Ivan Brugere, Shuhham Sharma, Freddy I Lecue, and Chun-Fu Chen. RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting. 2024.
Meena Jagadeesan. Michael Jordan, Jacob Steinhardt, and Nika Haghtalab. Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition. In Seiety-Sevnnth Conference on Neural Information Processing Systems, November 2023.
Shomik Jain, Vinith Suriyakumar, Kathleen Creel, and Ashia Wilson. Algorithmic Pluralism: A Structural Approach To Equal Opportunity. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. pages 197-206, June 2024. doi: 011114/53630100.65588899
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts, January 2024.
12

---

Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):2018340011 18, June 2021. ISSN 0027-8424, 1091-6490. doi:
10.1077/pnas.2015 201340118.
Michael Lipsky. Srret-Levee Bureaucracy: Dilemmas of the Individual in Public Services. Publications of Russell Sage Foundation. Russell Sage Foundation, New York, 1980. ISBN 663-1.
978-1-98811111 1-644-4
Yao Lu, Max Bartolo, Alastair Moore, Sehastian Riedel, and Pontus Stenetorp. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Com- putational Linguistics (Volume I: Long Pupers), pages 8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10118655/v1/2022..2 1ol..o55...6.
Charles Marx, Flavio Calmon, Berk Ustun. Predictive Multiplicity in Classification. In Proceedings of the 37th International Conference on Machine Learning, pages 6765 6674. PMLR, November 2020.
Robert K. Merton. Bureaucratic structure and personality, In Social Theory and Social Structure, pages 195-206. Free Press, 1940.
MetaAI.
Introducing Mcta Llama 3: http://aal.meta.commbbommmttt 1lama,3/, 2024.
The most capable
openly
available
LLM
date.
Mistral AI. Mistral small 3, 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang. Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri. Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannanch Hajishirzi. 2 OLMo 2 Furious, January 2025.
OpenAI, GPT-3.5 turbo, 2024.
OpenAL. Introducing GPT-4.1 in the API, April 2025.
Frank Pasquale. The Black Box Society: The Secret Algorithms That Contrul Money and Information. Harvard University Press, Cambridge, Massachusetts London, England, first harvard university press paperback edition edition, 2016. ISBN 978-0-674-97084-7 978-0-674-36827-99
Fabio Petroni, Tim Rocktäschel. Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.P-JCNIIP), pages 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10, 10.18553v1/D19 1250.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei. Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li. Mingfeng Xue, Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, January 2025.
Manish Raghavan. Competition and Diversity in Generative AI, December 2024.
Aaron Roth, Alexander Tolbert, and Scott Weinstein. Reconciling Individual Probability Forecasts. Proceed- ings of the 2023 ACM Conference on Fairness, Accountabiliry, and Transparency, FAceT '23, pages 101-110, New York, NY, USA, June 2023. Association for Computing Machinery. ISBN 979-8-4007-0199-11 doi: 10.1145/3593013.3592 35593980.
Timo Schick and Hinrich Schütze. Exploiting Queze-Questions for Few-Shot Text Classification and Natural Language Inference, In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255 269, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18853/v1/20 eecl- main. 20.
13

---

Melanie Sclar. Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations October 2023.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the Existence of Simpler Machine Learning Models. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency. FAceT '22, pages 1627-1858, New York, NY. USA, June 2022. Association for Computing Machinery. ISBN 978- 978-1-4503- 9352-2. doi: 10.1 145/3531 146.3533232.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh, AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages +222-4235. Online, November 2020. Association for Computational Linguistics. doi: 10.1 10.18653/v1/2000 w/w.220em0lll----.mmm main.346.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélic Héliou, Andrea Tacehetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquelte-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christinn Muraru, Grigory Rozhdestvenskiy. Henryk Michalewski, lan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptii Baptiste Lespiau, Jeff Stanway. Jenny Bremnan, Jeremy Chen, Johan Ferret, Justin Chiu. Justin Mao-Jones, Katherine Lee. Kathy Yu, Katie Millican, Lars Lowe Sjoesund. Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Matco Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruiho Liu, Ryan Mullins, Samuel I.. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri. Soham De, Ted Klimenko. Tom Hennigan, Vlad Feinberg. Wojciech Stokowiee, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals. Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathlecn Kenealy, Gemma: Open Models Based on Gemini Research and Technology, April 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin. Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic. Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlic Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogoziíska, Dustin Herbison, Elisa Bandy, Emma Wang. Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Plucifskke Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway. Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji. Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre. Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park. Mofi Rahman. Mohit Khatwani. Natalie Dao. Nenshad Bardoliwalla, Nesh Devanathan. Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton. Pradeep Kuppala. Ramona Comaneseu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébasticn M. R. Arnold, Scbastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving Open Language Models at a Practical Size, October 2024b.
14

---

Connor Toups, Rishi Bommasani, Kathleen A. Creel. Sarah H. Bana, Dan Jurafsky. and Percy Liang. Ecosystem- level analysis of deployed machine learning reveals homogeneous outcomes. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, pages 51178-51201, Red Hook, NY, USA, May 2024, Curran Associates Inc.
U.S. Census Bureau, Survey Income and Program Participation (SIPP), 2014.
U.S. Census Bureau, American Community Survey (ACS) Public Use Microdata Sample (PUMS), 2018.
Berk Ustun, Alexander Spangher, and Yang Liu. Actionable Recourse in Linear Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT '19, pages 10-19, New York, NY, USA. January 2019. Association Computing Machinery. ISBN 978- 1 1-4503-611555. 3287566.
doi: 10.1145/3287560.
Anton Voronov, Lena Wolf, and Max Ryabinin. Mind Your Format: Towards Consistent Evaluation In- Context Learning Improvements. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties: ACL 2024. pages 1018, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10. 10.1886550 18653/1120224 http://002222./nneeeeeee findings-acl.375.
Jamelle Satson-Danicls, David C. Parkes, and Berk Ustun. Predictive Multiplicity in Probabilistic Classification, June 2023.
Max Weber. Economy and Suciety: A New Translation. Harvard University Press, 2019.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu. Fei Huang, Guanting Dong, Haoran Wei, Huan Lin. Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang. Mei Mingfeng Xue, Na Ni, Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenhin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report, September 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use:
Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, pages 12697-12706. PMLR, July 2021.
15

---

Technical Appendices and Supplementary Material
A
Additional experimental details
A.1
Models
We evaluate 50 language models, including both open-source models- ranein-ranging from IB 141B parameters- and state-of-the-arr commercial models. A complete list of the models is provided in T For open-source models, we include both base pretrained variants and their instruction-tuna counterparts if available. Instruction-tuned models may be appealing to decision makers as they are trained specific instruction-rese respooss pairs to enhance their ability follow natural language instructions. As a result, they often show better alignment with human preferences realizable tasks. However, on unrealizable tasks such as those studied our work, they have also been observed to produce overconfident risk scores [Cruz al. 2024 Accordingly, to facilitate a more comprehensive characterization model behavior, we also consider the corresponding base model variants. Base models can be naturally attractive to decision makers for several reasons: Unlike instruction-tuned models, they do not impose an instruction-respee responns format, offering greater flexibility be adapted in custom workflows in deployment settings where particular alignment may be undesirable. As detailed Appendix B the two model variants exhibit comparable performance, resulting in models from both variants being included in the Rashomon set of each task.
Table 2: Models evaluated for this work. Model size in billions of parameters is indicated by N. Model weights for open-source models were retrieved from the corresponding HuggingFace (HF) repositories.
Language Model
N
HF Repository
Citation
Gemma 2B
2.51
google/gemma-28
Team al.
2024a
Gemma 2B (it)
2.51
google/gemma- 1at.rmmma----------- 1.1-2b-it
Team
2024a
Gemma 7B
8.54
google/gemma- 7b
Team
Gemma 7B (it)
8.54
google/gemma- 1.1-7b-it
Team ct al.
2024a
Gemma 2 9B
9.24
google/gemmat 2-9b
Team ct al.
2024b
Gemma 2 9B (it)
9.24
google/gemmm-2299e 2-9biit
Team
20240
Gemma 2 27B
27.2
google/gemma-22277
Team
2024h
Gemma 2 27B (it)
27.2
google/gemma-22222
2-22bb-i
Team
20244
Llama 3 8B
8.03
meta-
meta-laama/Meta-LLaaa-3333
MetaA
2024
Llama 3 8B (it)
8.03
MetaA
2024
Llama 70B
70.6
methe-lama/MMettLLLnaa3-7777
MetaAl
2024
Llama 70B (it)
70.6
mothallam//MMiaa-Laaaa333000
Llama 3.1
8.03
mote-llama/MMcta-LLaaa333--13
MctaAl
2024
Llama 3.1 8B (it)
8.03
mota-llana//VcaaaLLaaaa333------
MctaA
2024
Llama 3.1 70B
70.6
neta-llamaMMetta-Laaa--------101
MetaA
2024
Llama 3.1 70B (it)
70.6
meta-laama/MMcaa-Laaaa33--------
--ntruttttt
MelaAl
2024
Llama
1.24
mett-laama/Neta--Lamma333...111
MetaA
2024
Llama 3.2 IB (it)
1.24
meta-
IB-Instruct
MctaA
2024
Llama 3.2 3B
3,21
meta Maana/Meta-Llam-33
MetaA
2024
Llama 3.2 3B (it)
3.21
meta- Boete-llam/MMttt-aaaaaa--
-.t-:mmmaaa.-------------- Instructt
MetaA
2024
laama 3.3 70B (it)
70.6
MelaA1
2024
Mistral 7B
7.24
mistralai/Mfistral--8--.111
Jiang
2023
Mistral 7B (it)
7.24
mistrllai/Miissaa------isssssss
Jiang ct al
2023
Mixtral 8x7B
46.7
Jiang et al.
2024
Mixtral 8x7B (it)
46.7
Jiang
2024
Mixtral 8x22B
141
Jiang al.
2024
Mixtral 8x22B (it)
141
mistralai/Mixxtral 2aa-
Jiang ct al
2024
Mistral Small 24B
23.6
-wwiiiitttaa-a-ll------- --4--Basee-200
Mistral A1
2025
Mistral Small 24B (it)
23.6
mistralai/Mo SiaillaaiMMitrra--mmaaaaa 1.-t------ttuut-25
Mistral A1
2025
Yi
6.06
Ol-ai/Yi-6B
U1.AI ct al.
2025
Yi (chat)
6.06
-iii/i--B--hha
01.AI et al.
2025
Yi 34B
34.4
O1-ai/Yi-34B3
01.AI et al
2025
Yi 34B (chat)
34.4
01-ai/Yi-34B-Chal
01.A1 et
2025
Qwen 7B
7.62
Qwen/Qwen2-7B
Yang al.
2024
Continued next page
16

---

Table 2 Continued from previous page
Language Model
N
HF Repository
Citation
Qwen 7B (it)
7.62
Qwen/Qwwnn--7-Innsne
Yang al.
2024
Qwen 72B
72.7
Qwen/Qwen2-722)
Yang ct al.
2024
Qwen 2 72B (it)
72.7
QwvenQQwen2-72B-Imme
Yang cl al.
2024
Qwen 2.5 7B
7.62
Qwen/Qwen2.5577B
Owen et al.
2025
Qwen 2.5 7B (it)
7.62
wwem//Qwen25577--nnsstt
Owen et al.
2025
Owen 2.5 72B
72.7
Qwen/Qwen2.5-722B
Owen al. 2025
Qwen 2.5 72B (it)
72.7
Qwen/Qwen2.5-725
Qwen ct al.
2025
OLMo IB 0724
1,28
allenai/OLMo- 1---00MM------- 1B-0724-hi
Groeneveld ct al.
2024
OL IB
1.18
allenai/OI.Me-IB-hff
Groeneveld ct al.
2024
OL 7B 0724
6.89
allonai/O1.Mo-77--7224h
Groeneveld
2024
ML
6.89
allenai/OO.Mo-7B-hff
Groeneveld ct al.
OLMo 7B (it)
6.89
allena/OOLMo-7B-Instteee
Groeneveld et
2024
OLMo 2 7B
7.3
allenai/OL.Mo-2211222
OLMo et al.
2025
OLMo 2 7B (it)
7.3
OLMo al.
2025
gpt-3.5-tubo-01225
unknown
OpenA
gpt-4,1
unknown
OpenAl 2025
A.2 Prediction Tasks
We evaluate model predictions on seven binary classification tasks derived from three data sources. Census Bureau's 's American Community Survey (ACS) Public Use Microdata Sample (PUMS) (U.S]
Census Bureau 2018], the Behavioral Risk Factors Surveillance System [BRFSS, Centers for Disease
Control and Prevention (CDC), 2021 I] and the Survey Income and Program Participation [SIPP,
U.S. Census Bureau 2014]. In this section we will provide more details about the data sources and
each of the the prediction tasks used.
ACS prediction tasks. Five of the prediction tasks used for this work are based ACS PUMS data, which is derived from US Census data and provides a rich, diverse, and high-quality representation of the US population. While wide range of prediction tasks could be defined on this data source, we adopt five prediction tasks predefined the popular folktables package (Ding et al.) 2022)). These tasks span broad range of prediction challenges from high predictive signal more difficult low-signal settings (such as ACS Mobility)- - and collectively make up a diverse benchmark suite. While some features (e.g., age, race, and sex) appear across tasks due to their relevance for fairness analysis, their overall feature sets differ, and none is strict subset of another. Further, each task is constructed on a distinct subpopulation (c.g., adults, employed individuals). As a result, individuals do not consistently appear across tasks: The maximum pairwise overlap corresponds to at most 10.1% 1% relative to each task's size. No individual appears in all five tasks.
The folktexts package by Cruz et al. 2024 complements each task with a natural-text mapping for every feature, ready be used for language model prompting. To enable comparison with existing benchmarks and prior work on multiplicity, we use the task definitions including feature sets and population filters as provided. Following Ding et al. [2022) and (ndz al. [2024] analyze data from the 2018 1-year-horizon person-level survey, although any ACS survey year could be used. The
following paragraphs detail each ACS prediction task. Table 3 provides description and exemplary natural language encoding for all features.
ACSIncome The goal of the ACSIncome task is to predict whether person's 's yearly income is above $50,000, given by the PINCP column. The ACS columns used as features are AGEP, COW. SCHL, MAR, OCCP, POBP, RELP, WKHP, SEX, and RACIP. The column PINCP is binarized and used as target. The sub-population over which the task is conducted is employed US residents with age greater than 16 years. The ACSIncome prediction task was fur-forth as the successor to the popular UCI Adult dataset [Becker and Kohavi 1996), used extensively the algorithmic fairness literature.
17

---

The goal of the ACSEmployment's is to predict whether an individual is employed, given by the ESR column. The ACS columns used as features are AGEP, SCHL, MAR, RELP, DIS, ESP, CIT, MIG, MII., ANC, NATIVITY, DEAR, DEYE, DREM, SEX and RACIP. The sub- population over which the task is conducted is US residents.
ACSEmployment
ACSTravelTime The goal of the ACSTravelTime task is to predict whether a person's commute time work is greater than 20 minutes, given by the JWMNP column. The ACS columns used as features are: AGEP, SCHL, MAR, SEX, DIS, ESP, MIG, RELP, RACIP, PUMA, ST, CIT, OCCP, JWTR, POWPUMA, and POVPIP. The cub-populttion over which the task is conducted is employed US residents with age greater than 16 years.
ACSPublicCoverage The goal of the ACSPublicCoverage task is to predict whether individual is covered by public health insurance, given by the PUBCOV column. The ACS columns used as features are: AGEP, SCHL, MAR, SEX, DIS, ESP, CIT, MIG, MIL,, ANC, NATIVITY, DEAR, DEYE, DREM, PINCP, ESR, ST, FER, and RACIP. The sub-population over which the task is conducted is US residents with age below 65 years old, and with personal income below $30,000.
ACSMobility The goal of the ACSMobility task is to predict whether an individual has changed their home address in the last year, given by the MIG column. The ACS columns used as features are: AGEP, SCHL, MAR, SEX, DIS, ESP, CIT, MIL, ANC, NATIVITY, RELP, DEAR, DEYE, DREM, RACIP, GCL, COW, ESR, WKHP, JWMNP, and PINCP. The sub-population over which the task is conducted is US residents with age between 18 and 35.
BRFSS prediction task. The tableshift package [Gardner et al. 2024 provides a unified API to 15 prediction tasks, including some of the above mentioned ACS prediction tasks. For this work, we complement the ACS prediction tasks by one additional health-related prediction task, BRFSS Blood Pressure. The data comes from the Behavioral Risk Factor Surveillance System [BRFSS, Centers for Disease Control and Prevention (CDC) 2021]. BRESS is US-wide system of telephone surveys that assess health-related risk behaviors, chronic health conditions, and the use of preventive services by US residents. BRESS collects data in all 50 states well as the District of Columbia and three US territories. Following the implementation in tableshift, we use the default year range (2015-2021) of biannual BRFSS survey data. Further, we use the dataset preprocessed in the tableshift package, with the exception of feature normalization, which is omitted to preserve the raw data entries used for mapping to natural text. Following the codebook available for BRFFS, define a natural-text mapping for every feature. Table provides a description and exemplary natural language encoding for all features.
BRESS Blood Pressure The goal of the BRESS Blood Pressure task is to predict whether an individual has been told by a health professional that they have high blood pressure (hypertension). The features used include several risk factor like age. family history, other medical conditions, race, sex and social and economic factors. The BRESS columns used features are: BMISCAT, AGEG5YR, FRUIT ONCE PER DAY. VEG ONCE PER DAY, DRNK PER WEEK, REBINGS, TOTINDA, SMOKE 100, SMOKDAY2, CHCSCNCR, CHCOCNCR, DIABETES, POVERTY. EM- PLOY1, IYEAR, STATE, MEDCOST, PRACEI and SEX. The subpopulation over which the task is defined includes all US residents, with additional filters.
SIPP prediction task. Following Hardt and Kim 2022], we define one task based on the Survey of Income and Program Participation (SIPP) data [U.S. Census Bureau 2014). SIPP is an important longitudinal survey administered by the US Census Bureau that provides information on the dynamics of income, employment, household composition. and government program participation. The survey is considered nationally representative and interviews individuals for several years, providing monthly data about changes in household and family composition and economic circumstances over time. As Hardt and Kim [0022], we consider Wave and Wave 2 of the SIPP 2014 panel data for the prediction task and adopt their data cleaning and pre-processing steps. Following the codebook available for SIPP, we define natural-text mapping for every feature. Table 55 provides a description and exemplary natural language encoding for all features.
SIPP
The goal of the SIPP task is to predict whether an individual's income is well above the Official Poverty Measure (OPM), a aush-income-based measure of poverty. The target variable is
18

---

calculated using the OPM ratio from Wave 2 data. In total, set of 50 features is constructed from one or multiple variables appearing the Wave raw data are used for the SIPP task. These include features of the individual related to
demographics (AGE, GENDER, RACE, EDUCATION MARITAI_STATUUS, CITIZEN- SHIP.
FAMILY SIZE SIZE_AVG, ORIGIN),
income (INCOME, HOUSEHOLD INC,
RECEIVED COMP. IN-
COME_FROM, CROOMASSISTANCEE SAVINGS BOAISs_N_AAAMM INV_AMOUNT),
health
MEDICAID AEEDICAID_ASSISTAACCE, PRESCRIPTION_ MEDS,
HEALTHDISAB, DAYS SICK, HIGHTTAL
HEALTH INSURANCE PREMIUMS, IMSMIUS,
HEALTH OVER THE COUNTER PRODUCTS PAY,
HEALTH MEEHHEEEIAALAAREEE.. HEALTH SEEING,
HEALTH HEARING,
HEALTH E
HEALTH AMBULATORY,
HEALTH SELF CARE,
HEALTH ERRANDS DIFFICULTY,
HEALTH CORE DISABILITY,
HEALTH HEALTH_SUPPLEES SUPPLEMENTAL CITENNAL-DISBBILIIY
VIST__DOTTOR_NUM,
VISIT_ DENTIST_NUME
hardship (LIVING_QURL) QUARTERURRRRRRREE STPPE TYPE, LIVING_OWNERSHIIP FOOD_ASSISTANCE,
WIC_ASSISTANCE, SNAP ASSISTANN),
and details program participation
(TANF ASSISTANCE .IANF .TRANS-
PORTATION ASSISTANCE, .CMPP UNEMPLOYMENT COMP AMOUNT,
SOCIAL __CC____________________ BENEFITS,
VA VA_BENEHITS, _AMOUNT, LEL__INCOOEAAAMCA DISABILITY BENEFITS AMOUNT, AMOUUT, CHILD_CARE_AA SHILD_SPPPORTLAAA AMT, ALIMONY ALIMONY_AMT),
RETIRE-
MENT INCOME INCOME_AMOUNT,
SEVERANCE PAY PENSION,
FOS-
19

---

Table 3: Description of all column-to-text mappings implemented for ACS features. The variable part of each example is shown in typeset gray font. See the ACS PUMS data dictionary for the full list of available variables.'
Column
Description
Example
AGEP
Age
age is 29 years old
COW
Class of worker
class of worker Working for non-profit organization
SCHL
Educational attainment
highest educational attainment Bachelor's degree marital status is Karried
MAR
Marital status
OCCP
Occupation
occupation is Human Resources Manager place of birth is E1 Salvador
POBP
Place of birth
RELP
Relationship
relationship the reference survey respondent in the survey Brother or sister usual number hours worked per week 40 hours sex is Female
WKHP
Work-hours per week Sex
SEX
RACIP
Race
race is Black or African American total yearly income is $75,000
PINCP
Total yearly income
DIS
Disability status
disability status is With a disability
ESP
Employment status of parents
employment status of parents is living with tuo parents, only Father is employed citizenship status is Born in the United States mobility status the last year lived in the same house year ago military service status is Never served in the military ancestry Single ancestry nativity is foreign born hearing status is No hearing difficulty
CIT
Citizenship status Mobility (lived here year ago)
MIG
20
MIL.
Military service
ANC
Ancestry
NATIVITY
Nativity
DEAR
Hearing
DEYE
Vision
vision status is With vision difficulty. cogmition status is cognitive difficulty employment status Civilian employed, at work. resident state is Colorado
DREM
Cognition
ESR
Employment status
ST
State
FER
Person has given birth within the last year
person has given birth within the last year is Person has not given birth within the last year public health coverage status is Not covered by public health insurance means transportation to work is Bicycle Public Use Microdata Area (PUMA) code is Southeast Colorado Public Use Microdata Area (PUMA) code for the place of work is Southeast Colorado income-te to poverty ratio is 40% of the poverty 1ine income, which is below the poverty line commute time minutes grandparent living with grandchildren Household does not include grandparents living with grandchildren
PUBCOV
Public health coverage status
JWTR
Means transportation to work
PUMA
Public Use Microdata Area (PUMA) code
POWPUMA
Place of Work PUMA
POVPIP
Income-to Lomne-to-Porne to-Poverty Ratio
JWMNP
Commute time
GCL
Household includes grandparents
https://wwel http://///////.////////////////a

---

Table 4: Description of all column-to-text mappings implemented for BRFSS features. The variable part of each example is shown in typeset gray font. See the details BRESS Documentation for the full list of available variables.
Column
Description
Example
BMISCAT
Body Mass Index (kg/v) (ig/m") category
body index (kg/m²) category is normal weight age group (in intervals years) is 62-64 years consumption of fruit one or more times per day is Yea consumption of vegetables one or more times day is Yes
AGEGSYR
Age group (in intervals
years)
FRUIT LONE PER DAY VEG REG_ONCE DAY
Consumption of fruit one or more times per day Consumption of vegetables one or more times
per day
DRNK PER_ WEEK
Total number alcoholic beverages consumed
total number alcoholic beverages consumed per week 23 alcoholic beverages per week binge drinking behavior is Yes
per week
RFBING5
Binge drinking hehavior (i.e. 2: 5 drinks per occasion for males, 2 drinks per occasion for females)
TOTINDA
Leisure-time physical activity in past 30 days
leisure-time physical activity in past 30 days is Yes, had physical activity or exercise during the past 30 days other than regular job. history of smoking at least 100 cigarcttes in their lifetime is No
SMOKE100
History of smoking at least 100 cigarcttes in their lifetime
SMOKDAY2
Current frequency of cigarette smoking
current frequency of cigarette smoking is some days prior diagnosis skin cancer Yes prior diagnosis other cancer than skin cancer is No prior diagnosis of diabetes is No binary indicator of whether individual's income falls below 2021 poverty guideline for family four is Yes current employment status is Retired year of survey is 2017
21
CHCSCNCR
Prior diagnosis of cancer
CHCOCNCR
Prior diagnosis of other cancer than skin cancer
DIABETES
Prior diagnosis of diabetes
POVERTY
Binary Indicator: Individual's income falls be- low 2021 poverty guideline for family four Employment Status
EMPLOYI
IYEAR
Survey year
STATE
State of residence
state of residence is Georgia
MEDCOST
Unmet medical need due the last 12 months
unmet medical need due costs in the last months No
PRACEI
Preferred race category
preferred race category is White sex is Female
SEX
Sex

---

Table 5: Description of all column-to-bee mappings implemented for SIPP features. The variable part of each example is shown typeset gray font. See the SIPP codebook and the pre-processing script provided by Hardt and Kim 2022 for the full list of available variables.'
Column
Description
Example
demographics
AGE
age
age is 67 years old
GENDER
gender
gender of person is Male
RACE
race
races the person identifies with White only
EDUCATION
highest educational attainment
highest level of education completed is 9th grade marital status is Divorced US citizenship status is Yes average number of persons in family is persons Spanish, Hispanic, or Latino origin is No
MARITAL STATUS
marital status
CITIZENSIPP STATUS
STATUS
US citizenship status
FAMILY _SILE_SSIE_AGG
average family size the reference year of Spanish, Hispanic, or Latino origin
ORIGIN
income
INCOME
total individual income from earnings, investment and property in the reference year total household income the reference year receives worker's 's compensation payments monthly amount received from assistance total value of IRA, KEOGH, 401k, 403b, 503b, and Thrift Savings Plan accounts
total personal income is $6288
HOUSEHOLD INC
total monthly income all household members is $13500 received worker's compensation payments is No total income from public assistance, benefits or compensation is $6288 total value of retirement accounts is $0
RECEIVED WORK COMP
22
INCOME, FROM LOMSSSSSTNNC
SAVINGS INV INV_AMOUNT
hardship
LIVING_ QUARTERS_TYEE TYPE
type of living quarters living quarters are owned, rented or occupied with- payment
type of living quarters is house, apartment, flat ownership status of living quarters is owned or being bought by in the household received food assistance percentage of the year the respondent received WIC assistance is 0.00%%
LIVING OWNERSHII
FOOD ASSISTANCE
received some form food assistance pereentage of year in which individual received assistance from WIC (Women, Infants, and Chil- dren supplemental program)
WIC_ASSISTANCE
ASSISTASIT
percentage of year in which individual received assistance from SNAP (Supplemental Nutrition Assistance Program)
percentage of the year the respondent received SNAP/food stamps assistance is 1 00%%
health MEDICARE CECICARRASSISSANCC
percentage of year in which individual received assistance from MEDICARE percentage of year which individual received assistance from MEDICAID
percentage of the year the respondent covered by Medicare is 100.00%
HEDICAID_LSSISTAAN
percentage the year the respondent was covered by Medicaid is 0%
(Continued on next page)

---

(Continued from previous page)
Column
Description
Example
HEALTHDISAB
has physical, mental or other health condition limiting the amount of work they do number sick days
has a physical, mental other health condition that limits the kind or amount of work they do Yes
DAYS SICK HOSPITAL_ NIGHTS MEDS
number of days sick in the last year 10 days number nights in a hospital is nights
number of nights in hospital did take any prescription medications expenditures for comprehensive health insurance premiums
PRESCRIPTION LEDDS
uses prescription medications is Yes
HEALTH SHALTHLINSURANCE PREMIUMS
amount paid for comprehensive health insurance premiums is $0
SAALTH_OVER_TTE [HHE_CCUNTER COUNTER_PRODUCCTS PAY
out-of -oocket expenditures for over-the-counter health-relatee products
amount paid over-the-counter health-related products is $100
HEALTH MEDICAL CARE PAY
HEALTH_MEDICAL
out-of- ofpokket expenditures for on medical care
amount paid for non non-premium medical out-of-poccke expenditures medical is $300
HEALTH _EARING
has serious difficulty hearing
deaf of has hearing difficulties is is blind or has seeing difficultics is Yes
HEALTH HEALTH_SEEING
serious difficulty seeing
HEALTH_COGNITTV
has serious difficulty concentrating, remembering,
has serious difficulty concentrating, remembering, or making decisions is No
making decisions
HEALTH AMBULATORY
has serious difficulty walking climbing stairs difficulty dressing or bathing
has serious difficulty walking climbing stairs Yes has difficulty with self-care such dressing or bathing has difficulty doing errands alone is No has a core disability is Yes, with core disability answered positively to at least tions,
23
HEALTH SELF CARE
HEALTH ERRANDS DIFFICULTY
BRRANDS,DIFFICULLYY
has difficulty doing errands alone
HEALTH COREH_COR_EUEE DIR_DDISAIIITY
has at least one of six core disability measures answered positively to least
MAAIHHSSUPLEEMETTA DISABILITY
questions, work dis-
questions, three child disability ques-
three child disability questions,
work disability questions is Yes, with disability
ability questions
WISTTDDCCTO_NIT NUM
number of visits to a doctor, nurse, or any other type of medical provider
number of visits to a doctor, nurse, or any other type of medical provider is 25
visits
VISIT DENTIST NUM
number of visits dentist other dental pro- fessional
number of dentist visits is visit
program participation details TTANFASSISTANCCS
percentage of year in which individual received assistance from TANF received some kind of transportation assistance received unemployment compensation payments at any time during the reference year total amount of unemployment compensation pay- ments
percentage of the year the respondent received TANF benefit is O. 0.00%
TRANSPORTATION. ASSISTANCE E UNEMPLOYMENT, COMP
receives transportation assistance is No
receives unemployment compensation payments No
UNEMPLOYMENT COMMPNECCMMMPPLLL COMP_AMOUUU AMOUNT
amount of unemployment compensation per month is $0
SOCIAL SECNEEIIS
received social security benefits for themselves or on behalf of a child
received social security benefits
(Continued on next page)

---

(Continued from previous page)
Column
Description
Example
VA BENEFITS AMOUNT RETIREMENT INCOME AMOUNT
total monthly amount of VA benefits
total amount of VA benefits month is $0 total amount of retirement income per month $0
total monthly amount of retirement income
SURVIVOR JURVIVORJINCOME.C INCOLE_ AMOUNT
total monthly amount survivor income
DISABILITY LEENEEBTT
total monthly amount payments due to sickness,
total amount disability benefits or income per month $0
accidnnt or disability
SEVERANCE, PAY PENSION
received any severance pay lump sum payments from pension retirement plan during the ref-
receives any severance pay or lump sum payments from pension retirement plan No
crence period
FOSTER CHILD CAREE AMT
amount of foster child care payments received in each month
amount of foster child care payments received per month $0
CHILD SUPPORT_ AMT ALIMONY_AMP
amount of child support payments in each month amount of alimony payments received in each month
amount of child support payments received per month is 80 amount of alimony payments received per month is
24

---

Table 6: Dataset statistics. Test set size is reported as samples features.
Test set size
Positive instances
Negative instances
Task
(samples features)
(y
(y
ACSIncome
166, 450 10
61, 233
105, 217
ACSEmployment
323, 611 16
146, 740
176,871
ACSMohility
62, 094 x 21
16, 446
45, 648
ACSTravel
146, 665 16
64, 285
82, 380
ACSPublicCoverage
113, 829 19
33, 971
79, 858
BRESS Blood Pressure
87,676 x 19
44, 586
40, 090
SIPP Poverty
3, 3,972 50
035
1, 937
A.3
Experimental Details
We follow the default configuration provided by folktexts, adopting random 80/10/10 split for training, validation, and test sets. All evaluations are performed exclusively on the test set; no models trained in this study. For few-shot prompting experiments, from the training set to construct the prompt context. These same examples are reused across all few-shot prompts to ensure that each model receives an identical prompt.
randomly sample 10 examples
To report performance of XGBoost, we use the implementation provided by scikit-lears with default hyperparameters. No additional hyperparameter tuning performed.
Table summarizes the datasets used in this study. The main text focuses on results for positive instances (i.e., samples with true label y = 1). Results for negative instances (y 0) are presented in Appendix D.4
Code. We provide the code necessary to reproduce our analysis, along with a step-by-step guide for obtaining model predictions, available here: https////gtaaab.cooooooooiaaaaaa mono-mule multii
Resources used. shot evaluation of all models required approximately 1000 GPU hours, while 10-shot prompting added an additional 2, 500 GPU hours.
We use an internal compute cluster with NVIDIA A100 and H100 GPUs. Zero-
B Aggregate Model Performance
Aggregate performance varies considerably across models and tasks. See Figure 6 for accuracy and F balanced accuracy. Thresholds for all models where chosen to maximize balanced accuracy. For task with imbalanced label distributions, such as ACSMobility and ACSPublicCoverage only a few models outperform the constant majority-class baseline when selected for inclusion the Rashomon set based overall accuracy (Figure 6). Generally, parameters tend to perform better and are frequently included in the Rashomon set (highlighted in black), trend that even more pronounced under balanced accuracy. Variation in performance across models is lower for balanced accuracy, likely because model predictions are explicitly optimized for this metric (Figure Z Finally, we find that high performance on one task rarely translates other tasks, indicating limited cross-task task generalizability.
observe that models with more
25

---

labels
fraction segater prodlcted
ACSibann
de WEETs
Figure 6: Zero-shot performance on the test set. Each row is a task; columns show the ground truth label distribution, accuracy, and the fraction of negative predictions. Models in the middle and right panel are ordered by parameter size. In the accuracy panel, the blue line marks the constant majority-class predictor, and the green line indicates XGBoost performance. Models included in the Rashomon set (e = 0.05) are shown in black; others in gray.
26

---

labels
fraction segater prwdlcted
ACSSioan
Milice
Sieers
2IR2E Biuo Prosest
and
Tha me
Figure 7: Zero-shot performance on the test set. Each row is a task; columns show the ground truth label distribution, balanced accuracy, and the fraction of negative predictions. Models in the middle and right panel are ordered by parameter size. In the accuracy panel, the blue line marks the constant majority-clas predictor, and the green line indicates XGBoost performance. Models included in the Rashomon set (e = 0.05) based on balanced accuracy are shown in black; others in gray.
27

---

C Zero-Shot Prompting with Identical Prompts: Disaggregated Results
In the main paper, we report agreement and recourse levels aggregated across base and instruction- tuned models from different model providers, and across individuals in the test dataset to assess the model landscape. In this section, we break down our results along several axes. The first section focuses on model-specific characteristics: model provider and model variant. In the second section, we disaggregate results by demographic groups (sex, race and age) to examine how predictive similarity among language models affects these groups.
C.1 Results by Model Attributes
In this section, we present fine-grained results by model attributes, focusing on model provider and model variant. To enable fair comparisons of how agreement rates and recourse levels vary across different model groups, we subsample larger groups match the size of the smaller group. Reported values then correspond to the mean and standard error computed over 1,000 independent repetitions, or over maximum number unique subsamples possible given the group sizes. Results on ACSMobility task are omitted, as its Rashomon set is small to begin with, and further breakdowns would produce prohibitively small model groups
models
models
models
models
fraction of positive prodictions
1
.75
.25
0
25
.5
.75
1
.5
75
0
25
5
.75
1
0
.5
5
.75
1
fraction positive instances
models
models
20 models
27 models
.75
25
0
25
5
75
1
.25
.5
75
0
25
5
755
1
0
.55
15
.75
1
fraction positive instances
observed
random error
Figure 8: Recourse curves are sensitive to the model set size. Illustrative example based on the Rashomon set for ACSIncome (c = 0.05).
Recourse is sensitive to the model set size. To motivate the subsampling, we illustrate how recourse curves depend on the size of the model set. By expectation, full agreement more likely in smaller sets, and the recourse curves are correspondingly coarser with fewer distinct values. In larger model sets, achieving full agreement requires a high degree predictive similarity across all models. Even single model deviating from the majority directly reduces the fraction of individuals experiencing either full or no recourse. At the same time, the impact on the individual is minimal: an individual who experiences nearly full recourse is still very likely to receive a favorable outcome in large model ecosystem, without relying on recourse. We demonstrate this on the ACSIncome task by varying the number of models (not the restrictiveness of the Rashomon set!) and reporting the mean and standard error in Figure Recall, that models in the Rashomon vary by up to accuracy points.
To summarize, our metric recourse with respect to a model set is inherently sensitive to the set size, is the random error baseline. To ensure comparability across model groups resulting from disaggregation, we subsample the larger groups without replacement to match the size of the smallest group.
C.1.1 Disaggregation by Model Provider
Results in the main paper are aggregated over models from different families and developers to assess the empirical landscape of the current language model ecosystem. Technical reports for various open-source models reveal that their pre-training data mixes frequently draw from overlapping
28

---

Income
Blood Pressure
SIPP
1
agreement
.75
.25
.5
.75
1
.25
.5
.75
1
.25
.5
.75
fraction model pairs
TravelTime
PublicCoverage
random prediction
agreement
random error
all
.75
Alibaba
Meta Mistral
.5
0
.25
.
.75
0
.25
.5
.75
1
fraction of model pairs
Figure Agreement rates by model provider: x% of model pairs agree on at most y% of positive instances. Models grouped by provider, showing only groups with at least four models, subsampled to match the smallest group. Mean and standard error are reported; group sizes are the task legend. Across all groups, observed agreement (solid colored) exceeds random-error (dotted) and candom-predictee
(dashed gray) baselines, but remains below strict monoculture (agreement = 1). For most tasks, within- group agreement is slightly higher than across the full Rashomon set (solid gray).
sources such as CommonCrawl, Wikipedia and Wikibooks, GitHub, ArXiv, and Semantic Scholar, indicating considerable overlap in the training datasets even across developers. In this section, break results down by model developer, yielding subgroups whose models are likely to share not only larger portions of training data, but also similar architecture choices and training pipelines. Under the component-sharing hypothesis proposed by |Boomasani et al.] [2022] such increased overlaps may lead higher within-group agreement. Examining these patterns sheds light on the effects of increased component sharing, and also helps anticipate the potential outcomes when models from a single developer dominate domain or when training pipelines converge across the ecosystem.
Note that for tasks, the Rashomon set includes models from many different providers, often resulting in very small groups with only one or two models, which limits interpretability. To address this, we report results only for groups with at least four models; as a result, the disaggregated results presented here do not cover the full Rashomon set. For ACSEmployment, all groups fell below this threshold, so this task is excluded from the analysis in this section.
Agreement rates. When adjusting for the size of the model sets, we find that within- provider agreement is elevated for some groups but generally comparable to the overall level of agreement computed across models (Figure In particular, on ACSIncome, within- group agreement is nearly indistinguishable from the overall average. In contrast, within-group agreement varies notably on BRFSS Blood Pressure. Notably, agreement among Mistral models slightly below the overall level and considerably lower than that of other providers, which may reflect the heterogeneity of the Mistral group, which includes both dense and mixture-of-cxperts (MoE) models. On ACSPublicCovrrag. the Meta group exhibits high standard error, suggesting that some models provided by Meta more similar to each other than others. Across tasks, we find that models from Meta and Alibaba are consistently contained in the Rashomon set and display slightly elevated within-grow group agreement. Nevertheless, there is no consistent pattern indicating that a single provider's 's models consistently show exceptionally high tithin-group agreement, and very few model pairs reach full agreement. Thus, even restricted to a single provider, agreement remains well below the level of strict monoculture.
Baseline agreement under random errors remains relatively stable across subgroups, suggesting that overall accuracy differences alone cannot explain the observed disparities. Instead, the slightly elevated agreement among models from the same model provider likely reflects shared training
29

---

Income
Blood Pressure
SIPP
fraction of positive prediction
.25
.25
.75
0
.25
5
.75
1
.25
5
.75
fraction of positive instances
TravelTime
PublicCoverage
fraction of posilive predictions
random error
.75
all
.5
5
Alibaba
.25
Meta
Mistral A1
0
.25
.75
0
.25
.75
1
fraction of positive instances
Figure 10: Recourse curves by model provider: x% of positive instances are accepted by at most y% of models. Only provider groups with 2 4 models are shown; all groups are subsampled to match the smallest group. Mean (solid colored) and standard error (shaded) are reported, with group sizes in the task legends.
data, architecture choices training pipelines, in line with the shared component hypothesis of Bommasani et al. [2022..
Recourse curves. When disaggregating recourse levels by model provider, the overall picture remains stable: both recourse and full recourse occur much more frequently than would be expected under random errors (Figure IOP. Even when restricted to a single provider, disagreement between models still creates opportunities find recourse in other models. Across all tasks, the majority of positive instances experiences substantive or full recourse, receiving a favorable outcome by the majority of models.
Comparing within-group recourse levels to those computed across all models in the Rashomon set shows a tendency toward more homogeneous outcomes, with larger fractions of positive instances experiencing either no or full recourse. On BRFSS Blood Pressure, for example, the higher agreement among models from Alibaba translates into increases in both extremes, with no recourse as well as more achieving full recourse compared to the fraction computed across all models in the Rashomon set. On ACSPublicCoverags the pattern differs: for models from Alibaba the prevalence no recourse remain comparable to that across all models. Increased agreement levels mostly stem from true positives, visible from higher levels of full recourse. Similarly, full recourse also increased among models from Mistral AI, but in combination with a decreased fraction of individuals experiencing
individuals facing
recourse resulting in overall higher recourse levels. Models from Meta show the reverse pattern: higher fraction of no recourse and comparable levels of full recourse suggest that agreement among Meta models arises primarily from shared false negatives. Because these observations vary by task and by provider, consistent generalizations are difficult. We find some evidence that restricting to a single provider can amplify the extreme levels of recourse compared to those computed across all models in the Rashomon set.
C.1.2 Disaggregation by Model Variants (base vs. instruction-tuned)
In our work we include both base and instruction-tune variant when available because both variants might be attractive for fecision-maken for various reasons. Importantly, as shown in Figure 6 we find that across many tasks, there is often no clear performance gap between base models and instruction-tunet variants. Consequently, models of both types are present the Rashomon sets of all tasks. Including both variants thus provides more comprehensive picture. In this section, we disaggregate the results from the main paper by model variant as it can serve to analyze the impact of instruction-tuus -uunnng model agreement and recourse opportunities experienced by individuals.
30

---

Income
Employment
Blood Pressure
1
agreement
.75
15
13
.25
5
.75
.25
.5
.75
1
.25
.5
.75
fraction model pairs
SIPP
TravelTime
PublicCoverage
agreement
.75
.5
0
25
5
.75
.25
.5
.75
0
.25
5
75
fraction of model pairs
random prediction
random error
base
instruct
Figure 11: Agreement rates grouped by model variants: x% of model pairs agree on at most y% of the positive instances. The Rashomon is split into base and instruction-tuned models. For both groups, observed agreement (solid colored) is higher than under random errors (corresponding dotted) and random predictions (dashed gray). but remain below strict monoculture (red). The solid gray line depicts agreement among all models in the Rashomon set, while the dotted gray line shows the corresponding random-ers baseline. For most tasks, average agreement among instruction-tune variants in the Rashomon set is slightly above that among base models.
Agreement rates, When comparing agreement rates between base and instruction-uue -unne models, find them largely comparable, with slight tendency for pairwise agreement to be higher among citruction-tuneed models (Figure ITI. This is most pronounced for ACSIncome and ACSEmployment Across tasks, instruction-tune models account for least half of each group and, in most cases, con- stitute the majority. This suggests that instruction-tuning improves balanced accuracy. However, the effect on predictive similarity across models is moderate. A notable exception is ACSPublicCoverage where a subset base model pairs exhibits distinctly low, partially anti-correlated predictions with agreement rates below random, while other models shows relatively high agreement (mean 65.6%). In contrast, most instruction-tune tunde model pairs are highly correlated, with an mean agreement rate of 72.62%. These differences may related low predictive signal and class imbalance of this task.
Recourse levels. When disaggregating recourse levels by model variant, the overall picture remains stable: both no recourse (receiving an unfavorable outcome from all models) and full recourse (receiving favorable outcome from all models) occur much more frequently than would be expected under random errors (Figure 12]. Even when restricted specific model variant, disagreement between models creates opportunities for individuals to find recourse in other models. Across tasks, recourse curves are largely comparable to those computed across all models in the Rashomon set. On ACSIncome and ACSEmployment, instruction-tune models show slightly increased fraction of individuals correctly receiving a favorable outcome from all models, suggesting that higher agreement rates stem primarily from shared true positives. The opposite pattern is observed for BRFSS Blood Pressure, where full recourse is slightly more prevalent among base models. For ACSPublicCoverage instruction-tuni models tend to produce positive predictions more frequently. resulting in larger fraction of individuals experiencing substantive recourse compared to base models. Nevertheless, within both base and instruction-tume tundd models, substantial fraction of individuals still faces recourse.
31

---

Income
Employment
Blood Pressure
fraction of positive prediction
15
13
.25
0
.25
5
.75
.25
.
.75
1
0
.25
15
.75
fraction of positive instances
SIPP
Travel
PoblicCoverage
posilive predictions
fraction of
.75
.5
.25
0
.25
.
.75
0
.25
.5
.75
1
0
.25
.75
fraction of positive instances
random error
basc
instruct
Figure 12: Recourse curves grouped by model variants: x% of the positive instances are accepted by most y% of the models. The Rashomon is split into base and instruction-tuned models. The solid gray line depicts observed recourse among all models in the Rashomon set, while the dotted gray line shows the corresponding baseline under random errors. Observed recourse curves for base and instruction-tun -andedee---teedddee models are depicted in blue and orange respectively.
C.2 Results by Demographic Attributes
In this section, we provide fine-grained results for different demographic groups: sex, race and age, features that are present among all tasks. For the plots we define unified mapping to obtain similar groups (if present and defined) across tasks. We report results for demographic groups that account for at least 5% of the available data.
C.2.1 Disaggregation by Sex
Across all tasks, sex (or gender in the case of SIPP) is encoded as a binary variable with two categories: male and female. When stratifying individuals by this variable, we observe that agreement rates are largely comparable across tasks, with models showing slightly higher agreement for male individuals in most cases (Figure (3). This is also the case when looking at recourse levels (Figure T4p. The largest group-level disparity is observed in the BRESS Blood Pressure task. Although agreement rates between male and female individuals are nearly identical, an examination of recourse levels reveals systematically higher recourse for males. This pattern may indicate a shared bias among models that could disadvantage female individuals downstream decision makinn... Alternatively, the observed pattern could reflect intersectional effects between and age the prevalence of blood pressure, which we have not investigated further.
C.2.2 Disaggregation by Race
To analyze results by race, we standardize the encoding across tasks resulting in the follwing categories: American Indian Alaska Native, Asian, Black, Multiracial, Native Hawaïan or Other Pacific Islander, Other, and White. We report outcomes only for race groups representing at least 5% of the test data, but disparities are also prevalent for smaller groups. Note that, following the preprocessing procedure from tableshift, only binarized race information is available for BRESS Blood Pressure. In this case, individuals get mapped the categories White and Non- White,, with the latter labeled as "Other" in the plots, since does not correspond to a specific racial group.
Disaggregating results by race reveals substantial variation between groups in both within- grrup agreement rates (Figure [5) and within-group recourse levels (Figure [6). Across tasks, the majority of individuals in the datasets White. For most tasks, agreement rates for this majority group closely
32

---

Income
Employment
Blood Pressure
SIPP
agrocmont
.75
0.38
0.18
0,46
0.50
0.62
0.52
0.54
0.50
25
25
.75
fraction of model pairs TravelTime
PublicCoverage
Mobility
random prediction
agrocmenn
random error
0.45
0.48
0.50
all
0.55
0.54
0.50
Female
Male
25
25
.75
fraction of model pairs
Figure 13: Agreement rates by x% of model pairs agree most y% of the positive instances, with data grouped by sex. For both groups, observed agreement (solid colored) exceeds that expected under random errors (dotted) and random predictions (dashed gray), but remains below strict monoculture (red). The solid gray line represents agreement across all models in the Rashomon set, while the dotted gray line indicates the corresponding random-error baseline. Overall, agreement rates are comparable between groups across tasks. Task legends indicate group sizes.
Income
Employment
Blood Pressure
STPP
fraction of positive predictions
.75
0.38
0.48
0.46
0.50
.5
0.62
0.52
0.54
0.50
.25
.75
.25
.25
fraction of positive instances
fraction of posilive predictions
TravelTime
PublieCoverage
Mobility
random error
0.45
0.46
0.50
0.55
0.54
0.50
Female
Male
25
.5
75
0
.2
.75
1
fraction of positive instances
Figure 14: Recourse curves grouped by sex: x% of the positive instances are accepted by at most y% of the models, with data grouped by sex. Task legends indicate group sizes. Observed recourse curves comparable between groups
33

---

Income
Employment
Blood Pressure
SIPP
agrocmont
0.06
0.06
.75
0.07
0.08
0.15
0.09
0.85
0.83
0.52
0.78
25
25
775
25
25
775
fraction model pairs
TraeelTime
PublicCoverage
Mobility
random prediction
random error
oigreement
0.07
0.07
0.07
all
.75
0.10
Asian
0.76
0.73
Black
White
Other
25
.75
25
.75
.2
.75
fraction model pairs
Figure 15: Agreement rates grouped by race: X% of model pairs agree on at most y% of the positive instances, with data grouped by race. For all groups, observed agreement (solid colored) is higher than under random errors (corresponding dotted) and random predictions (dashed gray), but remain below strict monocultrre (red). The solid gray line depicts agreement among all models the Rashomon set. For most tasks, average agreement among instruction-tung variants in the Rashomon is slightly above that among base models.
fraction of posilive predictions
Income
Employment
Blood Pressure
SIPP
.75
0.06
0.06
0.15
0.09
.5
0.85
0.83
25
25
.75
.25
.75
25
fraction of positive instances
fractionn of positive predictinn
TravelTime
PublicCoverage
Mobility
random
all
0.07
0.07
0.07
Asian
0.10
0.17
0.11
Black
0.76
0.06
0.73
White
Other
.25 5 .75
25
5
.75
0
.25
.75
fraction of positive instances
Figure 16: Recourse curves grouped by race: x% of the positive instances are accepted by at most 5% of the models, with data grouped by race. Task legends indicate group sizes. Observed recourse curves vary systematically between groups.
34

---

Income
Employment
Blood Pressure
SIPP
agroomenn
0.07
0.16
0.16
.75
0.21
0.21
0.22
0.31
0.72
0.62
0.53
.5
.25
.2
.75
.25
fraction of model pairs TravelTime
PublicCoverage
Mobility
random prediction
aggoceennt
random error
0.15
0.14
all
0.19
0.35
0.75
30-60 years
0.66
0.51
60 years and older
below 30 years
25
.75
25
.5 .75
fraction of model pairs
Figure 17: Agreement rates grouped by age: x% of model pairs agree most y% of the positive instances, with data grouped by age. Task legends indicate group size. Observed agreement (solid colored) lies between consistently the random-error baseline (corresponding dotted) and strict monoculture (red). Agreement rates vary systemically between groups.
match overall agreement rates. An exception is ACSMobility, where overall pairwise agreement rates are elevated for individuals identified as White. Recourse levels generally follow the same pattern. For tasks including individuals identified as Black, agreement rates for this group tend to be lower, a trend consistent across all tasks except ACSTravelTime. This suggests that Black individuals experience greater multiplicity in the examined model set. In terms of recourse, observe that for ACSIncome, ACSEmployment and SIPP, Black individuals face systemic exclusion (no recourse) at much higher rates and receive full recourse lower rates than other groups. This pattern is particularly pronounced in ACSIncome, despite this task having the largest empirical Rashomon set. By contrast, ACSTravelTime and ACSPublicCoverage higher fraction of Black individuals experiences high levels of recourse. The elevated agreement rates on ACSTravelT can be explained by larger proportion of individuals receiving full recourse. When present, individuals identified as Asian generally experience higher agreement rates among models in the Rashomon set compared both the overall agreement rates and those of other groups. This is primarily driven by an increase in shared true positives, as reflected in the fraction of individuals receiving full recourse. Correspondingly, the fraction of Asian individuals facing no opportunity for recourse tends to be lower than for other groups.
Overall, the disaggregated results indicate that agreement and recourse patterns vary systematically by race: White and Asian individuals often experience higher agreement and recourse levels, whereas Black individuals exhibit lower agreement and more frequently subject systemic exclusion across several tasks.
C.2.3 Disaggregation by Age
We categorize individuals into three age brackets: younger than 30 years, 30-60 years, and 60 years or older. Note that some tasks are restricted to specific age ranges. For instance, most ACS tasks consider only individuals aged 16 and older, ACSMobility restricts the population to those younger than 35, and the BRESS Blood Pressure dataset includes only individuals aged 50 and above.
Disaggregating results by age reveals systematic variation in both within- grriin agreement rates (Figure (7) and within-group group recourse levels (Figure Across tasks, the majority of individuals are aged 30-60, with the exceptions of BRESS Blood Pressure and ACSMobility due to the aforementioned age restrictions. For most tasks, agreement rates within this age group are higher and closely match or exceed overall agreement rates. Recourse levels follow similar pattern, with larger fraction of individuals in this age range receiving full recourse. Individuals younger than 30 tend to experience lower recourse levels across multiple tasks, although these disparities are generally
35

---

Income
Employment
Blood Pressure
SIPP
fraction of positiv predictions
.75
0.07
0.16
0.16
.5
0.21
0.21
0.31
.25
0.72
0.62
0.53
.75
.75
.75
.25
.75
fraction of positive instances TravelTime
fraction of positive predictions
PublicCoverage
Mobility
random error
.75
0.15
0.14
.5
0.19
0.35
30-60 years
.25
0.66
0.51
years older below 30 years
.75
.25
.75
.25
.5
.75
fraction of positive instances
Figure 18: Recourse curves grouped by age: x% of the positive instances are accepted by most y% of the models, with data group by age. Task legends indicate group sizes. Observed recourse curves vary systematically between groups.
less pronounced than those observed for the 30-60 age group. The most notable disparities occur in ACSEmployment, where younger individuals are disproportionately classified as false negatives, receive lower levels of recourse overall, and exhibit lower agreement rates. Individuals aged 60 years or older show recourse curve with slope similar that of younger individuals, but a substantially larger fraction of positive instances (nearly 25%) experience no recourse. Overall, the disaggregated results indicate that agreement and recourse patterns vary systematically by age: individuals aged 30-60 generally experience higher agreement and greater access to recourse, whereas younger individuals face lower agreement and more limited recourse across several tasks. The relationship of the recourse curve for individuals aged 60 years older to those of other age groups appears task-dependent.
The disaggregated analyses may serve as an initial step toward understanding how recourse patterns vary across demographic groups, providing indicative evidence that some groups may face disproportionate barriers to recourse in the present language model ecosystem. It remains an important avenue for future research further investigate the mechanisms underlying such disparities and ways to mitigate them.
D Zero-Shot Prompting with Identical Prompts: Additional Results
In this section, provide additional details with identical prompts. Section D.I presents a more detailed discussion of results across tasks, Section D.2 shows the effect of varying e, which determines the restrictiveness of the Rashomon set, and Section D.3 examines the impact of selecting models based on their balanced accuracy.
results when zero-shotting different models
D.1 Agreement and Recourse Levels Across Tasks
To complement the results presented in the main paper, provide more detailed analysis across tasks in this section, including agreement rates between model pairs (Figure 19)
As reported in detail for ACSIncome, we observe generally high recourse levels among the positive instances, with a small, but considerable fraction experiencing no recourse (Figure [9] bottom panel). Compared to random-error baseline, extreme outcomes either no or full recourse are more likely, reflecting shared inductive biases between models. Nevertheless, levels remains far from strict monoculture, such that individuals largely retain the opportunity to obtain favorable outcomes by switching models. We find these patterns to be generally consistent across tasks (Figure and 19 Empirically observed agreement rates significantly exceed those
distribution recourse
36

---

Blood Pressure
Travel Time (M-12
Public Coverage (M-21)
Income (M-27)
Employment (M-8)
SIPP (M-16)
Mobility (M=5)
(M 23)
.75
.5
1.25.
.5.75 1
0 .25 .5.5
1.25 .5.5
0 .25 25.5.75
0 .2 125.5.75
.25 .5 .75
0 1.25.5.7
fraction of model pairs
fraction of positive predictions
.5
.25
0.25.5.78 (5.5]
0 .25.3.75
25.22.55.5 5.75
0 2..5555
75.5..5
0 0..5.5.75
.2..5.75
0 .5 5
5.75
fraction of positive instances
observed
random error
random prodiction
monoculture
Figure 19: Agreement rates and recourse curves for accuracy-based selection of models. Top. Agreement curves across tasks: x% of model pairs agree on most y% of the positive instances. Observed agreement (blue) is higher than under random eπrors (orange) and random predictions (dashed gray), but well below monoculture (red). Bottom. Recourse curves across tasks: x% of the positive instances are accepted by at most y% of the models. We vero-shot models and select those that achieve accuracy within e 0.05 from the best. For example, on ACSIncome we observe (blue) 20% of positive instances being accepted by at most 50% of the models. Under random errors (orange) this would rarely happen. Under strict monoculture (red, dotted) individuals only experience no or full recourse. Here, the mean TPR is used for illustration. The bar plot on the y-axis shows density function of recourse level the population.
expected under random errors, showing substantial predictive similarity across language models, but generally remain below complete consensus. As as result, both agreement rates as well as recourse curves consistently occupy the middle ground between the extremes of strong multiplicity and strict monoculture, reflecting some degree of diversity in model behavior even within the Rashomon set.
Two tasks, ACSMobility and ACSPublicCoverage deviate notably from the overall pattern, showing higher rates of no recourse and smaller proportions of individuals with substantial recourse. These deviations likely reflect the influence two underlying characteristics common to both tasks, whose combination may further amplify the observed differences. First, they exhibit low predictive signal [Dig al. 2022]. We assess this by examining the performance gap between a majority-clas predictor and XGBoost, a strong baseline for tabular data (Figure 6). Small performance gaps indicate that models hardly improve over trivial baselines. Second, both tasks display high class imbalance. with low prevalence of the positive class. Because the Rashomon set is constructed based on overall accuracy, tends favor models that prioritize performance on the majority class. This effect is particularly pronounced for ACSMobility, where the (small) Rashomon set includes mainly models behaving highly similar to the constant majority-class predictor. Consequently, agreement among some models is close perfect and large fraction of individuals receive no recourse. Notably, for ACSMobility, recourse levels align closely with those observed under the respective the random-crroo baseline. For ACSPublicCoveraal the high negative prediction rates models in the empirical Rashomon set (see Figure 6 right panel). Nonetheless, 38% of positive instances receive substantial recourse. This task also exhibits elevated levels of ambiguity and prediction discrepancy (Figure 20). consistent with unexpectedly low agreement rates, that for a subset of models even fall below the baseline of random predictions. Notably, the lowest agreement rates are observed for model pairs comprising one large and one small model, suggesting that differences in model capacity may contribute the divergence in predictions.
recourse levels are generally lower than in other tasks, reflecting
As discussed in Appendix [D.3 (Figure 22), selecting models using balanced accuracy rather than overall accuracy partially mitigates these effects, reducing the fraction of individuals receiving
37

---

Blond Pressure
Travel Time
Public Coverage
Income
Employment
SIPP
Mobility
prevalence rec(a) =0
0.9
0.6
0.0
0.75
0.50
0.00
0.45
0.30
0.00
24
[2_
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
accuracy deficit
observed
random
Figure 20: Severity of monoculture and predictive multiplicity as function of accuracy deficit e from the best model. Each column corresponds to one task. Predictions are obtained via O-shot prompting. Models are selected based on overall accuracy. Top. The fraction of positive instances experiencing recourse (blue) is consistently higher than would be expected under random errors (orange). Although this gap persists, decreases as e increases. Second Row. Ambiguity increases with e, highlighting the fraction of individuals affected by model choice. Due monotonicity, ambiguity likely higher for the full Rashomon set. Third Row. Discrepancy increases with potentially affording opportunities for recourse for some individuals. Due to monotonicity, discrepancy is likely higher for the full Rashomon set. Bottom. Number models in the Rashomon set as set increases.
recourse and increasing substantial full recourse. Taken together, these observations suggest that lask-dependent factors such as class balance, the predictive signal of the prediction task, and the selection criterion for the Rashomon can influence the observed levels of recourse individuals
receive.
D.2 Effect of the Choice of Accuracy Deficit e
In the main analysis, we consider models whose accuracy lies within an absolute margin of 0.05 of the best-performing model. This choice aligns with ranges commonly explored in prior work [Marx et al. 2020 Hsu et al.) 2023 Watson-Danicls et al. 2023] and strikes a practical balance between mncludig plausible deployment candidates from multiple major language model providers and avoiding overly narrow empirical Rashomon sets. Because 0.05 difference in accuracy can correspond to substantial change in error depending the task and the best model's performance, this section explores the sensitivity our findings to more restrictive values c.
Figure 20 reproduces results from the main analysis, showing the prevalence of no recourse and discrepancy across varying values e, now also including ambiguíty as an additional measure of multiplicity. At 0.0, the Rashomon set contains only the best model found. Across tasks, we observe that small increases in e rapidly expand the empirical Rashomon set as well as the observed discrepancy (&.) and ambiguity (a.), indicating that many models achieve comparable accuracy while producing divergent individual-Ieww predictions. This diversity creates potential avenues for recourse and reduces the likelihood that decision makers converge for all e values where the Rashomon set contains more than one model, nog-nggli fraction of individuals experiences recourse cicourse-substattiaaly errors.
single model. Nevertheless,
substantially higher than would be expected under random
To further examine the effect of provide agreement and recourse curves for varying values in Figure21 The results show pattern consistent with the main analysis. For small e, the empirical Rashomon set typically contains very few models, often only the best-performing one (see also
38

---

Blood Pressure
Travel Time
Public Coverage
Income
Employment
SIPP
Mobility
agreement
.75
0.25.5.75
1.25.5.75
0 25.55.75
0 0.25.5.75 1
0
0.25.5.75
0.25.5.75
0.25.5.75
fraction of model pains
positive predictions positive predictions positive predictions positive predictions positive predictions
M=5
M-
M=1
M=1
M=3
M=5
M=3
fraction of
== 0.01
.75
..5
.25
0.25.5.75.
0..5..5.7
0 1..55..75
0 1.255..755 1
0
0..5.5.75
0.25.5.75
25
125.5.75
M=7
M=5
M=
M=6
M=3
M=7
M=4
fraction of
c= 0.02
.5
.25
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
M=13
M=5
M=15
0.03
fraction
.75
c
1.25
25..
O 0.25.5.75
0 0.25..5.7 1
0 0.25.5.751
0.25.5.75
0.25.5.75
0. 0.255.0 .5.5...75
M=22
M=7
M=18
M=13
M=9
M=18
M=4
fraction of
== 0.04
.5
.25
0
TITT
T
T
TT 0.25.5.75
0.25.5.75
0.25.5.75
0 0.25.5.75
0 0.25.5.751
0 0.25.5.75
.5.55.775
M=27
M=8
M=23
M=16
M=12
M==1
M=5
fraction of
c = 0.05
.75
-5
.25
25.5..
0..5.5.75
0 0.25.5.75
0.25.5.75
0 0.25.5.75
75
0 0.25.5.75
0.25.5.75
fraction of positive instances
observed
0.02
= 0.04
random error
monoculture
0.01
0.03
0.05
random prediction
Figure 21:
Agreement rates and recourse curves for different values of c. First row. Agreement
curves: x% of model pairs agree at most y% of the positive instances. Observed agreement (blue) consistently lies between strong multiplicity (orange) and strict monoculture (red). Rows 2-6. Recourse curves for varying values of x% of the positive instances accepted by most y% of the models. We zero-shot models and select those that achieve accuracy within from the best. For small e, the empirical Rashomon can become restrictively small.
39

---

Table Recourse levels and measures multiplicity for all tasks under zero-shot prompting when using balanced accuracy as selection criterion for the Rashomon set.
substantial recourse
task
[R.]
ambiguity
discrepancy
recourse
recourse
ACSIncome
32
0.05
0.81
0.56
0.39
0.15
ACSEmployment
8
0.06
0.82
0,64
0.30
0.15
BRFSS Blood Pressure
25
0.03
0.59
0.11
0.86
0.37
STPP
15
0.05
0.78
0.49
0.46
0.18
ACavaeel/Timee
13
0..3
0.77
0.29
0.67
0,25
ACSeablicCoverage
17
0.09
0.51
0.22
0.69
0.37
ACSMobility
14
0.12
0,46
0.07
0.81
0.43
Figure 20] bottom row). When the Rashomon set includes at least two models, agreement rates exceed what would be expected under random errors, indicating substantial predictive similarity across language models. Nevertheless, agreement generally remains below complete consensus, even more restrictive Rashomon sets. Consistently across varying levels and across tasks, the observed recourse curve lies in between the extremes of strong multiplicity and strict monoculture (see Appendix for a discussion task differences). extreme outcomes, no recourse and full recourse, is clevated compared to the random-enee bascline even for more restrictive, smaller Rashomon sets. Unsurprisingly, we do observe, that opportunities for recourse increase and thus the size of the Rashomon set increases. Nevertheless, distribution recourse levels differs considerably from a strict monoculture, wherein the recourse curve would collapse into step function determined by the false negative rate of the sole remaining model.
particular, the observed likelihood for both
observe
Across varying values of and tasks, the observed recourse curve lies between the extremes of strong multiplicity and strict monoculture (see Appendix [.]] for a discussion of task differences). recourse and full recourse is elevated
In particular, the likelihood of both extreme outcomes compared to the random-crror baseline, even for smaller Rashomon sets. As expected, opportunities and thus the size of the Rashomon set grow. Nonetheless, the distribution
for recourse increase of recourse levels differs substantially from a strict monoculture, even for more restrictive values of c, reflecting some degree of diversity in model behavior within the Rashomon set.
D.3 Impact of Thresholding and Model Selection Metries
In the main paper, we take a conservative approach: to obtain model predictions, the decision threshold i is tuned to maximize balanced accuracy, converting continuous risk scores into binary outcomes. However, when constructing the Rashomon set, models are selected based on overall accuracy rather than balanced accuracy. An alternative would be to align the selection criterion with the objective used to fit the threshold by employing the same metric for both steps. In the following sections, report additional results obtained when from using either balanced accuracy for both threshold tuning and model selection, or overall accuracy both steps. We note that the objective used by decision makers convert risk scores into discrete predictions may vary across contexts and generally unknown.
D.3.1 Thresholding and Model Selection based on Balanced Accuracy
In the main analysis, we optimize model predictions by tuning threshold / on validation subset of = 2000 samples to maximize balanced accuracy, which is then used to convert risk scores into binary class predictions. Models are subsequently selected for inclusion in the Rashomon set based on their overall accuracy. In this section, we use balanced accuracy for both threshold tuning and model selection for the Rashomon set. Applying it also as the selection criterion ensures that the chosen models perform well across both classes. The selected models, along with their corresponding balanced accuracy scores, highlighted in black in Figure 7
Overall, the findings remain consistent (Figure p2p. Across tasks, agreement rates consistently fall between two extremes: (i) strong multiplicity, as expected under random errors, and (ii) striet monoculture, in which all model predictions are identical. Likewise, the observed recourse curves clearly lie between these two extremes across tasks, with elevated likelihood of no and full recourse.
40

---

Blood Pressure
Travel Time (M=13)
Public Coverage (M=17)
Income (M-32)
Employment (M-8)
SIPP (M-15)
Mobility (M=14)
(M 25)
.75
.5
0.25.5
.5.75 1
0 .25 .5.5
1.25 .5.5
.25 15.5..5
0 .2 .2255.75
0 1.25 1.2.5.77 1.2 .75
fraction of model pairs
fraction of positive predictions
.75
.5
0.25.5.77 0 .25.3.75
25.22.55.5 5.75
0 25 .5....5
0 222.5.75
0 .22 ..75 1
fraction of positive instances
observed
random error
random prodiction
monoculture
Agreement rates and recourse curves under zero-shot prompting when selecting models based on balanced accuracy. Top. Agreement curves across tasks: x% of model pairs agree at most y% of the positive instances. Observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray). but well below monoculture (red). Bottom. Recourse curves across tasks; x% of the positive instances are accepted by at most y% of the models. We zero-shot models and select those that achieve balanced accuracy within c 0.05 from the best. For example, on ACSIncome we observe (blue) 20% of positive instances being accepted by at most 50% of the models. Under random errors (orange) this would rarely happen. Under striet monoculture (red, dotted) individuals only experience or full recourse. Here, the mean TPR is used for illustration. The bar plot on the y-axis shows the density function of recourse level in the population.
Figure 22:
Furthermore, we observe that the size of the Rashomon set increases for most tasks compared to the accuracy-based sets. This likely because not only includes models that are already high- performing overall, but also models that achieve more balanced performance across classes, placing more emphasis on the minority class. Despite this increase in set sizes, recourse levels remain stable across those tasks with balanced datasets (see Table[Jand Figure 23p. We observe stronger differences for tasks with highly imbalanced datasets, particularly ACSMobility and ACSPublicCoverage For both tasks, smaller fractions of individuals experience recourse, while larger fractions experience substantial or even full recourse. On ACSMobility, discrepancy increases compared to accuracy- based selection, indicating that the Rashomon set includes models with more diverse predictive behavior. In contrast, discrepancy decreases for ACSPublicCoverage. Recall that under accuracy-brat basedd selection, subset of models exhibits anti-correlated predictions. When selection is based balanced accuracy, model agreement rates are consistently higher than expected under random errors, suggesting that the inductive biases of models in the empirical Rashomon set more aligned. This alignment also results in lower discrepancy.
D.3.2 Thresholding and Model Selection based on Accuracy
In the main analysis, we optimize model predictions by tuning threshold on a validation subset of n = 2000 samples to maximize balanced accuracy, which is then used to convert risk scores into binary class predictions. Models are subsequently selected for inclusion in the Rashomon set based on their overall accuracy. In this section, we instead use the same metric, overall accuracy, for both threshold tuning and model selection. Overall, the results remain largely consistent, except for ACSMobility, where the empirical Rashomon set includes all evaluated models and very high agreement rates up to perfect agreement are observed. As discussed Appendix D.I this dataset highly imbalanced and has low predictive signal. Consequently, most models basically behave like the constant majority class predictor, further amplified by tuning the threshold for overall accuracy. This example illustrates that the choice of metrics for thresholding and the Rashomon set should
41

---

Income
EmploymentBlood Pressure
SIPP
TravelTime PublicCoverage Mobiliyy
prevalenoe
0.45
rec(2)
0.15
0.75
0.50
0.25
0.00
0.45
0.30
0.15
0.00
[21
20
10
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
0.0
0.05
accuracy deficit
observed
random error
Figure 23: Severity of monoculture and predictive multiplicity as a function of accuracy deficit from the best model. Each column corresponds one task. Predictions are obtained via O- shot prompting. Models are selected based on balanced accuracy. Top. The fraction of positive instances that experience no recourse (blue) is consistently higher than what would be expected under random errors (orange). Although this gap persists, it decreases as increases. Second Row. Ambiguity increases with c, highlighting the fraction of individuals affected by model choice. Due to monotonicity, ambiguity is likely higher for the full Rashomon set. Third Row. Discrepancy increases with e, potentially affording opportunities for recourse some individuals. Due to monotonicity, discrepancy is likely higher for the full Rashomon set. Bottom. Number of models in the Rashomon set as set increases.
Table 8: Recourse levels and measures of multiplicity for all tasks under zero-shot prompting when using overall accuracy selection criterion for the Rashomon
substantial recourse
full recourse
task
R.
ambiguity
discrepancy
recourse
ACSIncome
34
0.13
0.64
0.31
0.56
0.25
ACSEmployment
8
0.07
0.78
0.61
0.32
0.15
BRFSS Blood Pressure
22
0.02
0.72
0.22
0.76
0.35
SIPP
14
0.06
0.78
0.51
0.44
0.17
ACSTravelTime
15
0.08
0.49
0.10
0.82
0.37
ACSPublicCoverage
22
0.38
0.27
0.03
0.59
0.26
ACSMobility
50
0.88
0.00
0.00
0.12
0.05
be considered in task-dependent manner. For completeness, Table & reports the exact fractions of individuals receiving no, substantial or full recourse as well as ambiguity and discrepancy as measured on the empirical Rashomon set.
D.4 Results for Negative Instances
In this work, we focus on positive instances, individuals with true label accepted or approved. Nevertheless, the notion of (favorable outcome) varies depending on the context. For instance, the ACSIncome dataset, positive prediction may indicate loan eligibility, where approval is the advantageous outcome. Conversely, when the same data is used to assess eligibility for financial aid, which case negative prediction (reflecting lower income) would instead result in beneficial outcome for the individual. Similarly, in tasks like fraud detection.
who should be
42

---

Blood Pressure
Travel Time (M=15)
Public Coverage (M-22)
Income (M-34)
Employment (M=8)
SIPP (M-14)
Mobility (M=50)
(M-21 22)
.75
.5
0.25.8
.5.75 1
0 .25 .5.5
1.25 0.25.5.70 .5.5
.25 5.
0 .2 .2.5..7
0 .22 25.5.75 .22 .75
fraction of model pairs
fraction of positive predictions
.75
.5
0.25.5.75 (5 0 .25.3.75
25 5..5
0 25 75.
0 0..5.5.75
11.25.5.78
0 25 5.
fraction of positive instances
observed
random error
random prodiction
monoculture
Figure 24: Agreement rates and recourse curves under zero-shot prompting when selecting models based on overall accuracy. Top. Agreement curves across tasks: x% model pairs agree most y% of the positive instances. Observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray), but well below monoculture (red). Bottom. Recourse curves across tasks: x% of the positive instances are accepted by at most y% of the models. We zero-shot models and select those that achieve balanced accuracy within e 0.05 from the best. For example, on ACSIncome we observe (blue) 20% of positive instances being accepted by at most 50% of the models. Under random errors (orange) this would rarely happen. Under strict monoculture (red, dotted) individuals only experience The bar plot on the y-axis shows the density function of recourse level in the population.
full recourse. Here, the mean TPR is used for illustration.
Table Recourse levels and measures of multiplicity for negative instances. We zero-shot models and use their overall accuracy selection criterion for the Rashomon set.
task
R.
no
substantial recourse
full
ambiguity
discrepancy
recourse
recourse
ACSIncome
27
0.08
0.76
0.52
0.40
0.14
ACSEnploymmnnt
8
0.13
0.75
0.61
0.26
0.16
BRESS Blood Pressure
23
0.04
0.66
0.13
0.83
0.39
SIPP
16
0.07
0.76
0.45
0.48
0.20
ACSTravelTime
12
0.09
0.53
0.20
0.70
0.29
ACSPublicCoverage
21
0.00
0.91
0.33
0.67
0.22
ACSMobility
5
0.00
1.00
0.77
0.23
0.14
the desirable outcome corresponds negative prediction (i.e., being classified not fraudulent). In such settings, becomes more appropriate to evaluate recourse for negative instances, as these individuals represent those potentially excluded from desirable opportunities favorable decisions.
In this section, we present results for zero-shot prompting under the assumption that the favorable outcome corresponds to negative prediction. Consequently, our analysis focuses negative instances. In this context, full recourse generalizes the notion the true negative rate across models, that is, it captures the extent to which all models consistently classify individual as belonging to the negative class. Models are included the empirical Rashomon set their overall accuracy lies with margin of 0.05 of the best rest-performing model.
Results for negative instances mirror those for positive ones: The empirical pairwise agreement rates fall between the two extremes substantially higher than what would be expected under strong multiplicity, yet notably lower than under full strict monoculure (Figure top panel). The corresponding recourse curve reveals overall high level of recourse, with most negative instances
43

---

Blood Pressure
Travel Time (M-12
Public Coverage (M-21)
Income (M-27)
Employment (M-8)
SIPP (M-16)
Mobility (M=5)
(M= 23)
.75
.5
0.25.5
.5.75 1
0 .25 .5.5
1225 25.5
.25 5.
0 .2 16.55.75
0 25 25.5.77 1.25.5 .75
fraction of model pairs
fraction of negaliiv predictiios
.5
.25
0.25.5.77
0 .25.5.77
25 5.75
0 25
.5...7
0 0..5.5.75
.2..5.5... 35..5.75
0 .25 5.75
fraction of uogative instances
observed
random error
random prodiction
monoculture
Agreement rates and recourse curves under zero-shot prompting for accuracy- based selection of models. Results are shown for negative instances. Top. Agreement curves across tasks: x% of model pairs agree on at most y% of the negative instances. For most tasks, observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray), but well below monoculture (red). Bottom. Recourse curves across tasks: x% of the negative instances are correctly rejected by at most y% of the models. We zero-shot models and select those that achieve accuracy withine = 0.05 from the best.
Figure 25:
experiencing substantial recourse potential (Figure 25 the results for positive instances, recourse levels for ACSMobility and ACSPublicCoverage are shifted toward higher values; recourse does not occur. Across tasks, both extremes, no recourse and full recourse, occur more frequently than expected under the random-error baseline. Nevertheless, the overall recourse curves remain situated between the poles of strict monoculture and strong multiplicity, underscoring the potential for many individuals to find recourse by turning to a different model.
bottom panel, and Table 9). Complementary
E Few-Shot Prompting with Identical Prompts
We repeat our analysis using 10-shot prompting, providing each model with the same class-balanced set of examples five labeled positive and five labeled negative. This uniform prior is chosen to ensure that models are exposed to both classes. Note, prior work has shown that language model behavior can be sensitive to the specific composition of few-shot examples (Zhao et al. 2021], which might further increase variability in model predictions (see also Section F). Consistent with the zero-shot analysis, we report results based on two selection strategies for constructing the empirical Rashomon set: one defined by overall accuracy and the other by balanced accuracy, corresponding to criterion used to tune the decision threshold.
E.1 Impact of the number of shots
Moving from zero-shot to few-shot prompting has been found to generally improve performance; here, we examine its effects on predictive similarity among models. Since the exact number of shots is design choice, we tested 4-, 8-, and 10-shot prompting on ACSIncome and observed consistent results (Figure 26). with a slight increase average accuracy among models selected for the Rashomon set as the number of shots increased. For the main analysis, we focus on larger number of shots to allow for in-context variations and ensure class balance. Beyond these considerations, the choice of the number of shots appears to have little impact the observed patterns.
44

---

4-shot
8-shot
10-shot
.5
.25
.5
0
.25
.25
5
.75
fract ion of model pairs
fraction of positive predictions
.75
.5
.25
.25
.75
1
0
.25
-
.75
1
0
.25
.5
.75
fraction positive instanoes
observed
random error
random prediction
monoculture
Figure 26: Agreement rates and recourse curves under few-shot prompting for varying numbers of shots. Top. Agreement curves across tasks: x% of model pairs agree on at most y% of the negative instances. For most tasks, observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray), but well below monoculture (red). Bottom. Recourse curves across tasks: x% of the positive instances are correctly accepted by at most y% of the models. We few-shot models and select those that achieve accuracy within = 0.05 from the best.
F,2 Aggregate Performance
As in zero-shot prompting, aggregate performance under 10-shot prompting varies considerably across models and tasks (Figure 27 for accuracy; Figure 28 for balanced accuracy). All thresholds were tuned maximize balanced accuracy. We observe tendency for additional context in few-shot prompting to benefit larger models, while also reducing performance variability among them. Effects on smaller models are mixed. Consequently, Rashomon sets are mostly comprised of large models.
At the task level, accuracy improves slightly for most models on ACSIncome and ACSEmployment. In contrast, tasks with lieCoverage show substantial variability in performance, with some models experiencing notable declines accuracy under 10-shot prompting. We hypothesize that this degradation may be attributed to the use of uniform label prior when selecting few-shot examples. This might be further amplified thresholds are tuned to maximize balanced accuracy.
imbalanced label distributions particularly ACSMobility and ACSPub-
E.3
Agreement and Recourse Levels Across Tasks
E.3.1 Selection based on overall accuracy
Overall, the analysis of predictive similarity under 10-shot prompting yields results consistent with those observed in the zero-shot setting (Figure[9). Across tasks, we observe slightly higher agreement rates, though they continue fall short of complete consensus as would expected under strict monoculture. Notably, for ACSPublicCoverage a small subset of models exhibits distinctly low agreement rates. Closer inspection of the Rashomon set shows that these model pairs all involve the single small model in the Rashomon set, Llama 3.2 IB, which effectively behaves like constant majority-class predictor. As shown in Figure (0 this pattern can be addressed by choosing a more restrictive value of without affecting the overall results.
Similarly, recourse curves under 10-shot prompting exhibit characteristies comparable to those observed under zero-shot prompting. The curves consistently lie between the two extremes of strong multiplicity and strict monoculture, while distribution of recourse levels shows higher likelihood of observing both extremes relative to the random-error baseline. For the tasks with class imbalance and low predictive signal, ACSeublieCoverage and ACSMobility, we observe more varied pattern of model similarity. In particular, notably larger fractions of positive instances receive substantial
45

---

sagrtie predited
ACShocoo0
by
of
wall
by
112
the th
WRRSS Blood Preetete
the DO
the
de paa
d0 by o
03-
dic
the Kecdin MNC
In comte
Lan
or
0.
Figure 27: 10-shot performance on the test set. Each row is a task; columns show the ground truth label distribution, accuracy, and the fraction of negative predictions. Models are ordered by parameter size. In the middle panel, the blue line marks the constant majority-class predictor, and the green line indicates XGBoost performance. Models included in the Rashomon set (e = 0.05) are shown in black; others in gray. For reference, the&f-so performance is shown in round markers.

---

ragerte predited
ACShoco00
of
by
the OFF by
HIS will
112 Mha
the toy
Of
MRRSS Blood Preetete
the DH
the OF
de taae
by
03-
de
MNC the Keadin
In comt
de
Searr
Hit i wildy
Lan
or the
05
H
Figure 28: 10-shot performance on the test set. Each row is a task; columns show the ground truth label distribution, balanced accuracy, and the fraction of negative predictions. Models are ordered by parameter size. In the middle panel, the blue line marks the constant majority-class predictor, and the green line indicates XGBoost performance. Models included in the Rashomon set (c = 0.05) are shown in black; others in gray. For reference, the99-hott performance is shown in round markers.

---

Blood Pressure (M)
Travel Time (M-17)
Public Coverage (M=14)
Income (M=35)
Employment (M=14)
SIPP (M-19)
Mobility (M-11)
.75
.5
1.25.
55.55 1
0 .25 .5.5
1.2 .5.5
.25 25.5..5
0 .2 22.5..5
1.25.5 .75
0 12255.78
fraction of model pairs
fraction of positive predictions
.75
.5
0.25.5.77 0 .25.3.75
25 5.75
1
0
25
.5.55.5
0 225 255.5.75
0..25.5.75
0 .22 .5775
fraction of positive instances
observed
random error
random prodiction
monoculture
Figure 29: Agreement rates and recourse curves under 10-shot prompting for accuracy- selection of models. Top. Agreement curves across tasks: x% model pairs agree on at most y% of the positive instances. For most tasks, observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray). but well below monoculture (red). Bottom. Recourse curves across tasks: x% of the positive instances are accepted by at most y% of the models. We zero-shot models and select those that achieve accuracy withine 0.05 from the best. For example, on ACSIncome we observe (blue) 20of positive instances being accepted by at most 50% of the models. Under random errors (orange) this would rarely happen. Under strict monocultre (red, dotted) individuals only experience or full recourse. Here, the mean TPR is used for illustration. The bar plot on the y-axis shows density function of recourse level in the population.
Table 10: Recourse levels and measures of multiplicity for all tasks under 10-shot prompting. Models are selected for the empirical Rashomon set based on overall accuracy.
substantial recourse
full recourse
task
[Ra
ambiguity
discrepancy
recourse
ACSIncome
35
0.07
0.80
0.43
0.50
0.24
ACSEmployment
14
0.07
0.83
0.64
0.28
0.12
BRESS Blood Pressure
28
0.07
0.59
0.21
0.72
0.27
SIPP
19
0.06
0.81
0.57
0.37
0.15
ACSTrave/Time
17
0.10
0.69
0.33
0.56
0.25
ACSPublieCoverage
14
0.24
0.00
0.76
0.39
ACSMobility
11
0.35
0.20
0.00
0.65
0.26
recourse. Comparing the Rashomon sets for these tasks between the zero-shot and 10-shot settings reveals that larger models are more frequently included under the 10-shot condition, likely because they benefit most from the additional context (see Section E.2).
While 10-shot prompting largely mirrors zero-shot results, the data reveal mild trend toward monoculture, as higher fractions of positive instances fall into one of the extremes for most tasks (Table IOP. This is particularly notable given that the empirical Rashomon sets are larger than those identified under the zero-shot setting, making it less likely that larger set of models will agree unless they are highly aligned. This pattern is also reflected the overall lower discrepancy observed across Rashomon sets.
Together, these results suggest that while 10-shot prompting slightly aligns model behavior, substantial heterogeneity in model predictions remains, preserving opportunities for individuals to find recourse by switching to another model.
48

---

Blood Pressure
Travel Time
Public Coverage
Income
Employment
SIPP
Mobility
agreement
.75
0 0.25.5.75
1.25.5.75
0 2..55..75
0 0.25.5.75
0 0.25.5.75
0.25.5.75
0.25.5.75
fraction of model pains
positive predictions positive predictions positive predictions positive predictions positive predictions
M-6
M=
M=4
M
M=5
M=4
M=1
fraction of
== 0.01
0.25.5.75
0.25.5.75
0 1.255..75
0 0.25.5.75 1
0
0..5.5.75
0.25....5
1..25.
.25.5.77
M=11
M=
M=11
M=10
M=10
M=5
M=4
c= 0.02
fraction of
.75
.5
.25
TITT
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
0.25.5.75
M=23
M=7
M=19
M=15
M=
M=10
0.03
fraction
.5
c
0. 0.25.55 .5.75
0 0.25.5.75
0 25.55.75 1
0 0.25.5.75
0.25-5.75
0.25.5.78
0 ...5...75
M=32
M=10
M=24
M=18
M=14
M=13
M=7
fraction of
== 0.04
.75
.5
.25
TITT
0.25.5.75
0.25.5.75
0 0.25.5.75
0 0.25.5.751
0.25.5.75
0.
0.25.5.751
0.25.5.75
M=35
M=14
M=28
M=19
M=17
M=14
M=11
fraction of
c = 0.05
.5
.25
25.5..7
.25.5.75
0 0..5.55.5
0.25.5.75
0.25.5.75
0.25.5.751
0.25.5.75
fraction of positive instances
observed
0.02 = 0.04
random error
monoculture
0.01
0.03
0.05
random prediction
Figure 30: Agreement rates and recourse curves under 10-shot prompting for different values of First row. Agreement curves: x% of model pairs agree at most y% of the positive instances. Observed agreement (blue) consistently lies between strong multiplicity (orange) and strict monoculture (red). Rows 2-6. Recourse curves for varying values of x% of the positive instances are accepted by at most y% of the models. We zero-shot models and select those that achieve accuracy within from the best. For small e, the empirical Rashomon can become restrictively small.
49

---

E.3.2 Selection based on balanced accuracy
In the main analysis, model predictions are optimized by tuning threshold & on validation subset of n = 2000 samples to maximize balanced accuracy, which is then used to convert continuous risk scores into binary class predictions. Models are subsequently selected for inclusion in the Rashomon set based on their overall accuracy. Given that few-shot examples are provided in a manner, this section presents 10-shot prompting results when the model selection criterion is aligned with the optinization objective- is, when the empirical Rashomon set defined based on balanced accuracy.
Analogous to observations in the zero-shot setting, selecting models based on balanced accuracy pri- marily affects tasks with highly imbalanced datasets, namely ACSMobility and ACSSublicCoverage. For both tasks, we observe distribution shift toward higher levels of recourse, with fewer individuals experiencing recourse and larger fractions experiencing substantial or full recourse. In contrast to accuracy-basec selection, agreement rates for ACSPublicCoverage are more homogeneous and higher, likely reflecting the emphasis minority-class performance enforced by balanced accuracy. As result, the Ilama 3.2 IB model, which was included in the accuracy-based Rashomon set and effectively behaves like constant majority-class predictor, is longer part of the Rashomon set. While the Rashomon set size does not noticeably increase compared to accuracy-based selection in the 10-shot setting, it increases relative the corresponding zero-shot balanced-accuracy based Rashomon sets. Pairwise agreement rates are generally higher, and despite the larger set sizes, instances of no recourse and full recourse are more frequent, indicating a mild trend toward monocul- ture. Recourse curves remain situated between strong multiplicity and strict monoculture, showing that opportunities for recourse persist for large fraction of positive instances.
Taken together, these results largely mirror those observed under zero-shot prompting. Slightly higher alignment in model predictions under 10-shot prompting suggests mild trend toward monoculture, but substantial disagreement between models remains, preserving opportunities for individuals to find recourse by switching another model.
Blood Pressure (M 25)
Travel Time (M=19)
Public Coverage (M=19)
Income
Employment (M=16)
SIPP (M=19)
Mobility (M=10)
(M=36)
1.25. 25.5.75
0 1..5...75
5.75 1.25..77
25.5..7
15.55. 35.33.5
0 0.25.5.75
fraction of model pairs
fraction of positive prodictions
.25
0.25.5.75
0 .25 35.5
.2..5.77 1
0 0.255.575
0.25.5.75
1..2555.70 .5.55.751
0 1.25.5.7
fraction of positive instances
observed
random error
random prediction
monoculture
Figure 31: Agreement rates and recourse curves under 10-shot prompting for models selected by balanced accuracy. Top. Agreement curves across tasks: x% of model pairs agree on at most y% of the positive instances. For most tasks, observed agreement (blue) is higher than under random errors (orange) and random predictions (dashed gray), but well below monoculture (red). Bottom. Recourse curves across tasks; x% of the positive instances are accepted by at most y% of the models. We zero-shot models and select those that achieve balanced accuracy within e = 0.05 from the best. For example, on ACSIncome we observe (blue) of positive instances being accepted by most 50% of the models. Under random errors (orange) this would rarely happen. Under strict monoculture (red, dotted) individuals only experience no or full recourse. Here, the mean TPR is used for illustration. The bar plot on the y-axis shows density function of recourse level in the population.
50

---

Table 11: Recourse levels and measures of multiplicity for all tasks under 10-shot prompting. Balanced accuracy is used as selection criterion for the Rashomon set.
substantial recourse
task
[R.
ambiguity
discrepancy
recourse
recourse
ACSIncome
36
0.06
0.80
0.50
0.44
0.17
ACSEmployment
16
0.07
0.84
0.64
0,29
0.13
BRFSS Blood Pressure
25
0.10
0.58
0.24
0.66
0.27
STPP
19
0.06
0.81
0.57
0.37
0.15
ACTarave/Time
19
0.70
0.31
0.62
0,23
ACSeablicCoverage
19
0.14
0.47
0.25
0.61
0.27
ACSMobility
10
0.17
0.48
0.15
0.67
0.34
F
Few-Shot Prompting with Varying Prompts
Few-shot prompting introduces additional sources of variation, such as the choice, ordering, and
class composition of examples. To investigate this, repeat our analysis of prompt variations under
10-shot prompting. Consistent with prior work (Zhao et al. [021 Lu et al. 2022 Gao et al. 202T
Schick and Schütze 202T], we find that both the order of examples and the class composition of
few-shot examples affect overall model performance (Figure 32). resulting in accuracy differences
of up three percentage points even when changing a single aspect. In addition, models remain sensitive to the same minor prompt variations tested in the zero-shot setting, exhibiting similar fluctuations accuracy.
Since makerss may vary in more than one aspect of how they construct prompts, we evaluate all four models on a subsample of the variations possible for few-shot prompting. To compare effects of prompt variations and model changes directly, we fix the number of prompts and models to be the same: randomly sample M prompt styles (M 35 for ACSIncome) and evaluate agreement and recourse across 100 independent repetitions (Figure 33). We observe that pairwise agreement across models increases under prompt variations (blue), though it remains similar to the agreement observed when varying the model under identical prompting (gray line), suggesting that substantial disagreement persists in both settings. Comparing recourse curves (bottom panel), we find that prompt variation leads to higher fraction of positive instances with no and full recourse. This suggests mild trend toward monoculture; nevertheless, prompt variations alone still enable recourse for considerable fraction of individuals, with the majority experiencing substantial recourse.
feature order
example order
example composition
format
connector
granularity
0.79
accuracy
0.77
0.76
0.74
Qwen 2.5 7B (It)
Llama 3 8B (t)
Llama 3.3 701 (1))
Qwen 5 72B (it)
Qwen 2.5 TB (16)
Qwen 2.5 72B (16)
7013 (1))
Qwen 2.5 78 (16)
(1)
(6)
(1t)
7B (16)
72B (16)
Quen 2.5 7B (1t)
70B (6)
(1)
T2B lit)
Qwen 2.5 TB (it)
(10)
8B
8B
70B
72B
7013
Llama SB
Llamna 3.3 70B
Qwen 2.5 72B
Llama
Llama 3.3
3
Llama 3
Llama 3.3
Qwen 2.5
Qwen 2.5
Llama
Llama 3.3
Qwen 2.5
Llama 3
Llnma 3.3
Qwen
Figure 32: Minor prompt variations induce changes in accuracy of up to percentage points, consistently across models on ACSIncome. Each subplot varies single aspect of the prompt, keeping the others fixed default. Light blue dots show accuracy for individual variations, dark blue dots indicate the mean accuracy with error bars.
51

---

the reference person the survey is Brother or Qwen 2.5 7B (it)
Llama 8B (it)
Llama 3.3 70B (it)
wen 2.5 72B (it)
.5
25
.75
25
.75
fraction of model variant pairs
fraction of variants accepting
.25
.5
.75
1
.25
-10
0
.25
-5
.75
1
0
.25
-5
.75
1
fraction positive instances
observed
random error
same prompt
random prediction
monoculture
Figure 33: Agreement and recourse across prompt variations on ACSIncome. We 10-shot models with varying prompting styles, subsampling number of different models in Figure 29 Top. Agreement curve: x% of prompt variation pairs agree on up to y% of the positive instances. Observed agreement rates (blue) surpass those under random errors (orange), but remain similar to agreement rates observed with identical prompting across different models (gray, solid). Bottom. Recourse curve: x% of the positive instances are accepted by at most y% of the prompt variations. For example, 25% of the positive instances are accepted by most 50% of the variations. Observed recourse (blue), random errors (orange). Under strict monoculture (red, dotted) individual only experience no or full recourse. Here, mean TPR used for illustration. Bar plot the y-axis shows density function of recourse level in the population.
35 prompt variations per model to match the
G Prompt Variations
In this section we provide further details and examples for the prompt variations tested in this work. By default, information about individual is presented as bulleted list, where each feature name followed by its value using the verb is. The default feature order and feature-to-text mapping are taken provided from folktexts. As running example consider:
Information :
age is 48 years old
class of worker is Working for non-profit organization highest educational attainment Doctorate degree marital status is Married occupation is Education and childcare administrators place of birth is California
relationship
sister
usual number of hours worked per week is 45 hours sex is Female race is Asian
To assess how sensitive models are to subtle changes in prompting, we modify four minor aspects of prompt construction specifically, the way information about individuals is presented. The following paragraphs detail each variation.
We test five arbitrary orders in which features of individual presented the default order given by folktexts, its reverse, and three random samples from the d! possible orders, where d the number of features. On ACSIncome, the resulting feature orders are:
Feature order:
default: AGEP COW, SCHI., MAR, OCCP, POBP, RELP, WKHP, SEX, RACIP
reverse: RACIP, SEX, WKHP, RELP, POBP, OCCP, MAR, SCHL, COW, AGEP
52

---

random 1: RACIP, WKHP, AGEP, SCHL, MAR, SEX, RELP, POBP, COW, OCCP
random 2: WKHP, OCCP, RACIP, MAR, AGEP, RELP, SCHL, POBP, COW, SEX
random 3: AGEP, SCHI,, OCCP, MAR, COW, WKHP, RACIP, RELP, SEX, POBP
The rows in the above example reordered according to the given feature order.
Format. Features are presented either bullet list, comma-separated list (CSV style) as simple text using the format '<feature name> <connector> <feature value>.".
Information provided as comma-separated list:
Information:
age is 48 years old, class of worker is Working for a non profit organization, highest educational attainment is Doctorate degree, marital status is Married, occupation is Education and childcare administrators, place
birth is California, relationship reference person the survey is Brother or sister, usual number hours worked per week is 45 hours, sex is Female, race is Asian
the
Information provided simple text
Information:
The age is 48 years old. The class of worker is Working for non Doctorate
profit organization. The highest educational attainment degree. The marital status Married. The occupation is Education childcare administrators. The place of birth is California. The relationship to the reference person in the survey is Brother or sister. The usual number of hours worked per week is 45 hours. The is Female. The race Asian.
Connector. We vary the symbol used between <feature name> and <feature values, choosing among 'is", "=' or''"""
Information provided with ' connector:
Information:
age: 48 years old
class of worker: Working a non non-profit organization highest educational attainment: Doctorate degree marital status: Married
occupation: Education and childcare administrators
place of birth: California
relationship the reference person the survey: Brother or sister
usual number of hours vorked per week: 45 hours
sex: Female
Asian
Information provided with '=' as connector:
Information:
age 48 years old
class of vorker Working for a non non non-profit organization highest educational attainment Doctorate degree marital status Married occupation Education and childcare administrators
place of birth California
relationship to the reference person in the survey = Brother sister
usual number hours worked per week 45 hours sex Female
race Asian
53

---

Granularity. We toggle between the original feature mapping provided by folktexts and a lower resolution mapping. For most features, the rower-resolution version aggregates categories bins numerical values for example, exact age is grouped into age ranges, and detailed occupational statuses are mapped broader occupational categories as defined in the ACS data documentation'
Information provided in lower resolution:
Inf ormation:
age is 40-49 years old
class of worker is Employed
highest educational attainment is Graduate or professional degree marital status is Married
occupation is Management Occupations
place of birth
West USA
relationship to the reference person the survey is Siblings usual number of hours worked per week is 40-49 hours sex is Female is Asian
54

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are empirically backed in Section and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Scope and assumptions are clearly stated, we discuss limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
55

---

Justification:
Paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Steps reproduce all experimental results are provided in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
56

---

Answer: [Yes]
Justification: Code and data to reproduce the main results are provided with the supplemen- tary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl cc/public/guu
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 3 and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Provided in Figure 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
57

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper includes details on computational resources used in the supplemen- tary material in Section A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines: Societal impact discussed in
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
58

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper is focused on evaluation and does not release any new assets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Original paper that produced the code package dataset is cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
59

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Paper does release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
60

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs primarily used for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
61


Reviews:
[{"review_id": "A8lTJhgYj4", "reviewer": "Reviewer_6KfP", "summary": "Different LLMs can make different classification decisions when prompted in the same way. In this paper, this \"multiplicity\" is seen as an opportunity, an antidote to \"monoculture\": if an individual was denied for a mortgage by one bank based on their income levels or other traits, they have \"recourse\" in that they go to another bank, who might use another LLM, and potentially be approved for the mortgage by that bank. The experiments in this paper quantify the degree of multiplicity across 50 LLMs and six tasks. They find that there's some consistency across models (more than expected by chance), but most individuals still have recourse - the consistency doesn't rise to the level of monoculture. Variation in prompts and few-shot examples increases multiplicity.", "questions": "Could you spend a little bit of space in the paper explaining the following issue: \"Since language models are widely miscalibrated on non-realizable tasks, we adopt the approach from Cruz et al. [2024] to calibrate the model predictions. For each model and task, we fit a decision threshold t on n = 2000 samples from a validation set to maximize accuracy. The threshold is then applied to turn the risk scores into class predictions.\" What is a non-realizable task? Should we be concerned about the fact that this adjustment is necessary? Does it affect the conclusions of the paper?\n\nThe models being tested do not all have the same accuracy levels (there's an epsilon tolerance of 0.05 for inclusion in the \"rashomon set\"). Doesn't this on its own mean that we can't possibly expect to have full monoculture? Is there a way to quantify the highest level of monoculture given the 0.05 tolerance? \n\nCould you justify this particular value of epsilon? I was a little surprised to see such a high tolerance level as the difference between e.g. 90% accuracy and 95% accuracy seems very substantial to me (half the error rate), though it wasn't clear what the absolute accuracy was for all tasks.", "limitations": "Yes, there is a reasonable discussion of the limitations in section 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "STRENGTHS\n\nThe paper has very clear motivation and definitions: the first few pages are a crash course on this topic for someone like myself who doesn’t know much about this area.\n\nThe experiment is comprehensive: the authors experiment with a range of LLMs from different companies, with multiple tasks, and with prompt variations.\n\nWEAKNESSES\n\nIt is hard for me to gauge the quantitative significance of the results. Neither of the extreme outcomes - fully uncorrelated errors across models (full multiplicity) and fully correlated errors across models (monoculture) - seem plausible or even mathematically possible given non-insignificant differences in accuracy across models (see questions below). So it's hard to know what to make of the fact that the truth is somewhere in the middle. I wonder if there's a way to come up with a \"multiplicity index\", to get a feel for how strongly the errors are correlated across LLMs compared to, say, other ML models (I suspect there will be more multiplicity for the LLMs, especially in the zero-shot setting, but I'm not sure). Perhaps the increased variability due to prompts and/or few-shot examples can help here.", "was_revised": true, "final_justification": "I again thank the authors for their engagement with my comments, and for clarifying the more minor issues I brought up in my review. Our discussion didn't convince me that monoculture isn't a bit of a strawman (this was also raised by another reviewer) and/or a mathematical impossibility given variable accuracy levels across models, and I'm still unsure how encouraged we should be with, say, 8% of individuals not having recourse (as opposed to 1%, or 50%). This could still be a good first paper that would advocate for a more rigorous quantitative framework for evaluating variability in predictions across models.", "rebuttal": "Dear Reviewer 6KfP, \n\nThanks a lot for the time you spent reviewing our work and for your valuable feedback! We appreciate your assessment of our work as accessible and our experiments to be comprehensive. \n\nWe address your questions in the following paragraphs:\n\n> 1. What is a non-realizable task? Should we be concerned about the fact that this adjustment is necessary? Does it affect the conclusions of the paper?\n\nA prediction task is non-realizable when no model, regardless of its complexity, capacity or training data can achieve perfect performance. This typically stems from inherent stochasticity in the data generating process, underdetermined problem formulations or a lack of ground truth. This is a relevant distincting from a body of literature that focuses on the evaluation of LLMs on realizable tasks where there is a unique correct label for each data point (such as factual question-answering). \n\nThreshold adjustment is a standard component of classification tasks, not a cause for concern. Since models often output probabilities, adjusting the threshold allows alignment with trade-offs between sensitivity and specificity, particularly in imbalanced or cost-sensitive settings. In the main paper, we report results optimizing for balanced accuracy, which we consider appropriate given the class imbalance of some tasks.\n\nTo address your concerns, we also **re-ran the analysis optimizing thresholds for *overall* accuracy**. **Results were largely consistent**: observed recourse levels remained between strict monoculture and strong multiplicity. However, for highly imbalanced tasks (e.g., ACSMobility and ACSPublicCoverage), accuracy optimization favors models that resemble the majority-class predictor, resulting in high fractions of individuals with no recourse—models we consider unlikely to be deployed in practice. We will include those results in the Appendix of the paper for full transparency.\n\n\n> 2. [Does the tolerance of $\\epsilon=0.05$ for inclusion in the \"Rashomon set\" imply] we can't possibly expect to have full monoculture? Is there a way to quantify the highest level of monoculture given the 0.05 tolerance? \n\nWe appreciate the reviewer’s observation that **differing accuracy levels already act as a force against monoculture**. Indeed, two models with accuracies $a \\neq a’$ cannot agree on more than $1-|a-a’|$ of the overall population. However, this bound applies at the population level. \n\n\nOur analysis **focuses on positive instances**, where it does not directly apply without assumptions about the conditional agreement. Assuming the positive class constitutes at least an $\\epsilon$-fraction of the dataset, perfect agreement on this subset (i.e., strict monoculture) remains theoretically possible—even when overall accuracies differ slightly. We will clarify this in the revised paper.\n\nEven given this additional information, we maintain that strict monoculture provides a natural and interpretable baseline, capturing the limiting case of full predictive homogeneity.\n\n> 3. Could you justify this particular value of epsilon? [...] 90% accuracy and 95% accuracy seems very substantial to me [...] wasn't clear what the absolute accuracy was for all tasks.\n\nThank you for this question. We fully agree that a 5% accuracy difference (e.g., 90% vs. 95%) is meaningful, as it reflects a substantial relative change in error. In prior work, $\\epsilon$ values between 0.01 and 0.5 are commonly explored (see e.g. Marx et al., 2020, Hsu et al., 2023, Watson-Daniels et al., 2023). \n\nOur particular choice of $\\epsilon$ is not intended as a normative threshold, but reflects a practical trade-off: including plausible deployment candidates from multiple major LLM providers while avoiding overly narrow empirical Rashomon sets. In fact, for some tasks, lowering $\\epsilon$ to 0.01, reduces the empirical Rashomon set to a single model.. \n\nThat said, our key findings remain consistent even with more restrictive $\\epsilon$ values. Currently, **Figure 3 provides some insights** into that, showing how the fraction of individuals with no recourse is affected by different $\\epsilon$ values. In the revised version of the paper, we **will further expand on this by adding a plot on the distribution of recourse for different ϵ values for greater clarity**. \n\nFinally, we agree that the Rashomon set abstracts away from absolute performance, which is why we report overall performance metrics in the Appendix. We will make this connection clearer in the revision.\n\n> hard to know what to make of the fact that the truth is somewhere in the middle [...] I wonder if there's a way to come up with a \"multiplicity index\"\n\nThank you for raising this important point. Our goal is to illustrate what a system-level analysis of model ecosystems might look like, highlighting the degree of monoculture or multiplicity and its implications for individual recourse. In light of our findings occupying the middle ground between strict monoculture and strong multiplicity, we **recommend conducting such system-level analyses and create transparency by reporting the level to which individuals have recourse**. \n\nWe appreciate your suggestion to develop a “multiplicity index”  to compare different model ecosystems. This could be a valuable next step, and we hope our work provides a good starting point for such efforts. Importantly, we caution against viewing multiplicity as a solution concept to monoculture and vice versa. A more nuanced framework is needed – one that considers ecosystem dynamics and captures individual harms arising from it.\n\nIf you have any further questions or concerns, we will be happy to address them. \n\nThe authors"}, {"review_id": "TIJgSCUg56", "reviewer": "Reviewer_tNUx", "summary": "The paper addresses the presence of two phenomena; monoculture and model multiplicity. Monoculture indicates that models trained on similar data are going to have similar decision making, indicating that certain individuals could potentially be discriminated against by all models used by decision-makers for high-consequential tasks. Model multiplicity indicates that models trained in the same (or similar) circumstances, so on similar data, while achieving similar performance (e.g., in terms of accuracy) might still have diverging predictions. This paper investigates the degree to which models lean toward monoculture or multiplicity. \n\nThe authors evaluate 50 pretrained base and instruction-tuned LLMs, open and closed-source, on six high consequential tasks. This is done through zero-shot prompting and 10-shot prompting. A pairwise agreement rate is used, which measures the overlap in individuals that two models would both accept, and recourse, which is the fraction of models that give an individual a positive outcome; acceptance. \n\nResults indicate that models do not lean toward either monoculture or multiplicity, but sit somewhere between. The recourse levels do not support monoculture yet models still have strong shared biases underscored by the high agreement rates.", "questions": "- Why specifically 10-shot learning? Do you notice different results when reducing the amount of samples in few-shot learning? \n- Does $x_i$ always represent a different individual or can there be overlap between individuals across the ACS tasks?  \n- Why do you use base pre-trained versions of the model as well? I would imagine that most decision-makers would use the instruction tuned version. \n- Line 217: Is it that bad that models yield highly similar outputs? For such high-consequential tasks, it might actually be a good thing that there is consistency in predictions across different models? Of course, this is with the assumption that there is no bias against specific individuals due to unrelated factors. Maybe this is also exactly what you mean and I might have missed this point, else it would be great to clarify this.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper tackles an important and interesting dichotomy of model behavior which is monoculture and model multiplicity. Findings can be highly impactful for tasks with high consequences and give better insights into model robustness and informed deployment choices into the real world. \n- The evaluation setup is well-designed. The tasks chosen make sense, which are highly-consequential individual based prediction tasks. The authors conduct a large scale of experiments with the amount of models used (50). The evaluation metrics used correspond to the objective of the paper. \n- The results are well-presented, also additional analyses are well-motivated and give a holistic understanding of monoculture or multiplicity in models, such as the prompt variation analysis. \n- I think in general, the paper really thinks about the practical usage of LLMs and tailors its setup accordingly. \n\n**Weaknesses**\n- I feel the that the Introduction section could benefit from a bit more clarity and conciseness. Model multiplicity and monoculture could be explained a bit better in the introduction. A direct link to the task of risk assessment for individuals could make it clearer. What are the consequences of model multiplicity and monoculture for this task? Maybe a figure that represents the problem statement would be nice? Also, I am not sure whether the result figure has the desired effect in the Introduction and in the contributions section certain details can be left out and explained in the respective sections directly. \n- The Results section could be made stronger with certain analyses being discussed. For instance, since this is such a highly consequential task, it would be highly valuable if there are insights included on whether there is a pattern regarding who the ones are in the 7% that have 0 recourse. Further, the results for the recourse are currently aggregated, but it would be very interesting to know how this changes across model size and/or family and base pre-trained vs. instruction-tuned. \n- The results clearly showcase that there is no clear cut case of monoculture or multiplicity. It would really strengthen the paper if there is a discussion on whether there are specific instances (e.g. specific types of inputs) where there models perform rather monotonously versus this multiplicity.", "was_revised": true, "final_justification": "The authors have addressed many of my questions and concerns. I do agree with other reviewers that some takeaways for LLM practitioners and researchers could be discussed more clearly (regardless of the task) and hence did not raise my score. I think there is still merit in including something like this in this paper but I am not sure whether that could be added in this iteration.", "rebuttal": "Dear Reviewer tNUx, \n\nthanks a lot for the time you spent reviewing our work and your detailed and valuable feedback! We are pleased to read your positive comments on our empirical analysis and the value of our findings, contributing to a “holistic understanding of monoculture or multiplicity”. We address your remaining questions in the following paragraphs:\n\n\n> 1. Why specifically 10-shot learning? Do you notice different results when reducing the amount of samples in few-shot learning? \n\nWe had performed a **sensitivity analysis using 4-shot and 8-shot learning on ACSIncome and noticed** that there was no difference—we’ll include these checks in the Appendix of the paper. We ended up showcasing the results for more samples to allow for considerations related to in-context variations and class balancing. Other than that, we acknowledge that the exact number of shots is somewhat arbitrary. \n\n> 2. Does $x_i$ always represent a different individual or can there be overlap between individuals across the ACS tasks?\n\nThank you for the question. While all five ACS tasks adopted in his work are derived from the same data source, US Census data from 2018, each task is constructed on a distinct subpopulation (e.g., adults, employed individuals). As a result, **individuals $x_i$  do not consistently appear across tasks**: The maximum overlap in terms of individuals is at most 10.1% for any pair of tasks, 0.8% for any triplet, and 0.08% for any combination of four tasks—measured relative to each task’s size. No individual appears in all five tasks.\n\n> 3. Why do you use base pre-trained versions of the model as well? I would imagine that most decision-makers would use the instruction tuned version.  \n\nWe thank the reviewer for raising this point. As shown in Appendix B.1, we find that across many tasks, there is **often no clear performance gap between base models and instruction-tuned variants**. Including base models thus provides a more comprehensive picture. In line with your comment below, they also serve as a useful reference for analyzing the impact of instruction tuning on multiplicity and monoculture. We will update the paper to **include results separated for base and instruction-tuned models**.\n\n**Base models can be naturally more attractive to decision makers** for several reasons. Unlike instruction-tuned models, they do not impose an instruction-response format, offering greater flexibility to be adapted in custom workflow or in deployment settings where a particular alignment may be undesirable. Moreover, while instruction-tuned models often show better alignment with human preferences on realizable tasks, they have also been observed to exhibit overconfident risk scores (Cruz et al, 2024) on the tasks we study.\n\n> It would be very interesting to know how [recourse] changes across model size and/or family and base pre-trained vs. instruction-tuned. \n\nWe appreciate the reviewer’s suggestions and have **added additional results conditioned on either model developers or model type (base vs. instruction-tuned)**.\n\nWhen grouping by model developer, we find no consistent trend across tasks for specific model families. Notably, all Rashomon sets contain models from at least four different developers, though for some tasks and families, the number of models is restrictively small (only one or two models), limiting interpretability. In terms of model agreement, the finer-grained results align with our aggregate-level findings. Among model families with at least three members in the Rashomon set, agreement rates consistently fall between the extremes of strict monoculture and strong multiplicity.\n\nExtending the analysis to include agreement rates across all models—not just those in the empirical Rashomon sets—we observe that Qwen models exhibit slightly higher in-family agreement than others. In contrast, OLMo and Gemma models tend to show slightly lower agreement within their respective families. \n\nWhen grouping by instruction tuning, we observe slightly higher agreement rates among instruction-tuned models across tasks. However, recourse levels remain comparable between base and instruction-tuned models.\n\nWe will update the revised paper to include these more fine-grained analyses and their implications.\n\n> it would really strengthen the paper if there is a discussion on whether there are specific instances [...] where the models perform rather monotonously versus this multiplicity.\n\nThanks for raising this point. We absolutely agree that this would strengthen the paper and **will include a plot showing recourse levels by demographic attributes (e.g., age, sex, race)** in the revised version to examine more fine-grained disparities. \n\nBetter understanding of which individual-level factors might contribute to whether an individual is more likely to have no recourse or many opportunities for recourse in a (language) model ecosystem would be an interesting endeavour for future work. \n\n> Introduction section could benefit from a bit more clarity and conciseness\n\nThank you for your detailed feedback and suggestions—we will revise the introduction accordingly. Regarding your comment on Figure 1, could you kindly elaborate on which aspects you find non-ideal? Any additional detail would be greatly appreciated and will help us improve this figure more effectively.\n\n> Line 217: Is it that bad that models yield highly similar outputs? For such high-consequential tasks, it might actually be a good thing that there is consistency in predictions across different models? \n\nThanks for bringing this up. Models agreeing on consequential tasks can be seen as an indicator of consistency, on the other hand this is exactly what literature on monoculture warns about. It may mean that individuals have no opportunity for recourse. In particular for unrealizable tasks this might be undesirable. Our point here is not to provide a normative framework, but to evaluate whether this occurs in practice in language model ecosystems. We will adjust the opening of Section 4 to clarify this. \n\n\n\nFinally, we thank the reviewer for several concrete suggestions for improvements of the flow and layout of the manuscript (e.g. shortened introduction, link to risk assessment earlier, improved opening of Section 4). We agree with all points made by the reviewer and will update the manuscript accordingly.\n\n\nIf you have any further questions or concerns, we will be happy to address them. \n\nThe authors"}, {"review_id": "azywkDDqSv", "reviewer": "Reviewer_CF6P", "summary": "This paper provides an evaluation of current LLMs in terms of the degree of monoculture and multiplicity. As an evaluation study, the experimental work covers 50 LLM models across 6 different prediction tasks with 4 different prompt variations. The paper concludes that LLMs as predictors fall between the extremes of monoculture and multiplicity.", "questions": "1. What are the possible factors that cause the significant differences in the sample proportions corresponding to rec(x) > 0.5 across different datasets, as shown in Figure 2?\n\n2. Could the authors discuss  more insights from their evaluation conclusions for the design of future LLMs, or provide recommendations on how to organize LLMs models to avoid harming individuals?", "limitations": "The work can be further improved in terms of the variety and number of tasks.  Additionally, the paper could convey deeper insights from the evaluations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S:\n1. This work presents a novel evaluation direction regarding the homogenization of LLMs.\n2. The experimental designs are sound, and the analysis is well-reasoned.\n3. The paper concludes that  racism faced by individuals in LLM predictions cannot be fully addressed from the perspectives of monoculture or multiplicity.\n\nW：\n1. The paper seems to lack deeper insights from the evaluation results. Specifically, the conclusion that LLMs as predictors fall between monoculture and multiplicity does not provide clear guidance for researchers on how to use LLMs effectively. While the authors mention that the harms faced by individuals resist capture by either monoculture or multiplicity, there is still a gap in the work regarding what actions should be taken moving forward.\n\n2. The tasks used for evaluation are highly limited. Both the ACS series tasks are actually derived from a single dataset, with only different labels. Apart from that, the only other dataset used is the BRFSS Blood Pressure task. In addition to the limited task variety, the task modalities are also restricted, as both tasks are based on structured tabular datasets.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CF6P, \n\nThanks a lot for the time you spent on providing valuable feedback on our work and helpful suggestions on improving the paper. We appreciate your  positive comments on the novelty of our evaluation and our sound experimental design.\n\n\nTo address remaining concerns regarding our manuscript, we have **revised the text where suggested** and, to improve the empirical evaluation, **added additional results on the Survey of Income and Program Participation (SIPP)**. Please find below a detailed discussion of the points you have raised. \n\n\n>  tasks used for evaluation are highly limited\n\n\nTo address your concerns regarding task diversity, we have **added empirical results for one additional task** defined on the longitudinal Survey of Income and Program Participation (SIPP). Here, the goal is to predict whether a person's income is significantly above the Official Poverty Measure (OPM). Summarizing the results we find similar patterns as observed for the other tasks. There is substantial predictive similarity across models. We observe a mean agreement rate of 0.84 that clearly exceeds the baseline agreement rates (63%). Still, agreement is generally below full agreement for all model pairs. In terms of recourse, 5% of individuals experience no recourse, 77% substantial recourse and 48% full recourse. \n\nWhile we acknowledge your point that all ACS tasks are derived from the same underlying data source (US-wide ACS PUMS data), we want to point out that ACS, being derived from US Census data, is one of the most comprehensive surveys worldwide. As such, it provides a rich, diverse, and high-quality representation of the US population, making it a strong foundation for a wide range of prediction tasks such as employment, education or housing.\n\nTo ensure comparability with prior work, we adopt five tasks predefined by Ding et al. (2022). These tasks span a broad range of prediction challenges—from high predictive signal to more difficult low-signal settings (such as ACS Mobility)—and collectively make up a diverse and valuable benchmark suite.  While some features (such as age, race, and sex) appear across tasks due to their relevance for fairness analysis, their overall feature sets differ, and none is a strict subset of another. Further, each task is constructed on a distinct subpopulation (e.g., adults, employed individuals), with no individual appearing in all five tasks and a maximum pairwise overlap of only 10.1% of individuals relative to the task’s size. Importantly, we view the partial overlap in individuals as an opportunity to study model multiplicity across different contexts (see Bommasani et al., 2022).\n\n\n\n>   task modalities are also restricted\n\nWe appreciate the reviewer’s suggestion to include additional task modalities. We agree that extending the analysis to a broader range of modalities is a valuable direction. However, this falls outside the scope of the current paper as it would require a new model suite and task set. We want to note that our focus on tabular data reflects both practical and normative considerations: Tabular data is the primary modality for risk assessment in high-stake domains such as hiring, credit, criminal justice or other bureaucratic contexts. As a result, much of the normative discourse about the concerns of model multiplicity and algorithmic monoculture as well as recourse has centered on this modality. In this sense, our choice aligns with a broader body of research that has largely concentrated on tabular data. Nevertheless, we see expanding to other modalities as a promising direction, and we expect that our evaluation framework could be readily adapted for that purpose.\n\n>   1. What are the possible factors that cause the significant differences in the sample proportions corresponding to rec(x) > 0.5 across different datasets, as shown in Figure 2?\n\nWe assume you're in particular referring to the ACSMobility and ACSPublicCoverage tasks, where we observe notably higher rates of no recourse affecting also the fraction of individuals that experience substantial recourse ($\\text{rec}(x) > 0.5$). We provide some discussion on both tasks in the Appendix, but will revise the text to address and clarify this earlier in the paper. \n\nBoth tasks are characterized by high class imbalance, with a low prevalence of the positive class. Since the Rashomon set is constructed based on overall accuracy, it tends to favour models that prioritize the majority class. Especially pronounced  for ACSMobility, the Rashomon set includes mainly models behaving highly similar to the constant majority-class predictor (see also the general model performance on ACSMobility), resulting in a high fraction of individuals receiving no recourse. This may be exacerbated by low predictive signal of the task, which makes it challenging for models to improve over trivial baselines.\n\nWhen models are instead selected using balanced accuracy (see Appendix B.2.2, Figure 10), which accounts for class imbalance, the picture changes: fewer individuals experience no recourse, and more receive substantial or full recourse. A similar trend is seen for ACSPublicCoverage.\n\nThis suggests that **factors such as class balance, the predictive signal of the prediction  task, and the selection criterion for the Rashomon set can all influence the observed levels of recourse**. We hope thas adequately addresses your question. If anything remains unclear, we’d be happy to clarify or discuss further.\n\n\n>  2. Could the authors discuss more insights from their evaluation conclusions for the design of future LLMs, or provide recommendations on how to organize LLMs models to avoid harming individuals?\n\nThank you for raising this point. We will **incorporate it more clearly into the discussion**. Our work illustrates what a system-level analysis might look like to assess the degree of monoculture or multiplicity within a model ecosystem. We believe that conducting such system-level evaluations and increasing transparency about the extent to which individuals have recourse is both valuable and necessary. This kind of analysis could serve as a starting point for developing something like a \"multiplicity index,\" as suggested by Reviewer 6KfP. Accordingly, we advocate for auditing entire model ecosystems, not just individual models. If homogeneity is identified as a significant concern, then concrete (policy) interventions, such as randomization, may be warranted. For a detailed discussion of the policy implications surrounding monoculture and multiplicity, see also Gur-Arieh et al. (2025).\n\nWith regard to individual harm, we caution against treating model multiplicity as a straightforward remedy for the risks posed by monoculture. Both extremes carry distinct concerns, and neither should be viewed as a solution concept for addressing harm in language model ecosystems. Simply increasing diversity among models does not guarantee better outcomes for individuals or society. Instead, our findings suggest that the kinds of harm individuals may experience are not adequately captured by focusing solely on either monoculture or multiplicity. A more nuanced framework is needed to understand and mitigate these risks effectively.\n\nThank you for your thoughtful comments. We believe the revisions have strengthened our work, and we would be happy to address any further questions or concerns.\n\nThe authors"}, {"review_id": "U2V72dRlJ8", "reviewer": "Reviewer_t4zS", "summary": "This paper provides empirical evidence to test two theories about machine learning ecosystems: monoculture and model multiplicity. Monoculture hypothesizes that the ecosystem will tend toward a single model making all decisions, leading to homogeneous outcomes for individuals. Multiplicity suggests that instead there will be many different models, leading to large variation in individual outcomes. THe paper tests 50 language models on social science prediction tasks to see whether outcomes are monolithic or varied, as measured by metrics like ambiguity, discrepancy, and recourse of an individual. The results suggest that real ecosystems lie between the two extremes. Model outcomes are quite similar, but not as much as monoculture suggests, suggesting that it may be rare that individuals be systemically excluded by all models.", "questions": "1) I wonder if \"individuals overwhelming have recourse\" is too strong a statement. If 8% of positive instances are rejected by all models, doesn't that seem to be a problem?  and the numbers are much much higher for the two ACS tasks in figure 8, or for Table 6. Prior theories of monoculture like Bommasani et al 2022 do seem to emphasize that recourse is a problem even if only a small fraction of individuals don't have recourse.  Should your discussion and claims be more nuanced?\n\n2) Related, I do worry that an unsympathetic reader might see the \"pure monoculture\" hypothesis as a straw man. Some of your results on monoculture and recourse do seem socially problematic, even if we consider monoculture as a tendency rather than an absolute. Can this discussion and claim also be more nuanced? This probably requires some changes throughout the paper.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: The paper brings a nice empirical lens to an important theoretical question in the literature. The findings are actionable, there are natural followups, and the paper is extremely clearly written.\n\nWeaknesses: I worry that \"strict monoculture\" is a bit of a straw man; the literature talks about algorithmic homogeneity more as a continuum or a tendency. So the finding that there is not \"strict monoculture\" seems less exciting; I am more excited by the details of empirical findings themselves, which are fascinating. So I might suggest that the discussions, especially around recourse, might want to have more nuance.", "was_revised": true, "final_justification": "This paper brings a nice empirical lens to an important theoretical question in the literature. The findings are actionable, there are natural followups, and the paper is extremely clearly written.", "rebuttal": "Dear Reviewer t4zS, \n\nThank you for the time you spent reviewing our work and your valuable feedback and suggestions! We appreciate your positive comments on our empirical analysis and the value of our findings. \nWe address your remaining questions or concerns in the following paragraphs: \n\n> \"strict monoculture\" is a bit of a straw man [...] suggest that the discussions, especially around recourse, might want to have more nuance\n\nWe thank the reviewer for raising this point. We find it helpful to contrast both extremes, strict monoculture and strict multiplicity, as a way to evaluate and reason about a given model ecosystem. To address your comments, we will more clearly term strict monoculture as the far end of a spectrum encompassing model multiplicity and monoculture and **will revise our discussion of recourse in order to better reflect the nuances of our findings**.\n\n\n> \"individuals overwhelming have recourse\" might be too strong a statement. \n\nWe agree that the fraction of positive instances experiencing no recourse is indeed significant and hence this phrasing might come across as too strong. We will update the text to reflect this better. \n\n> numbers [of individuals experiencing no recourse] are much much higher for the two ACS tasks in figure 8\n\nThank you for bringing this up. We assume you're referring specifically to the ACSMobility and ACSPublicCoverage tasks, where we observe notably higher rates of individuals experiencing no recourse. We provide some discussion on both tasks in the Appendix, but will revise the text to address and clarify this earlier in the paper. \n\nBoth tasks are characterized by **high class imbalance**, with a low prevalence of the positive class. Since the Rashomon set is constructed based on overall accuracy, it tends to favour models that prioritize the majority class. Especially pronounced for ACSMobility, the Rashomon set includes mainly models behaving highly similar to the constant majority-class predictor (see also the general model performance on ACSMobility), resulting in a high fraction of individuals receiving no recourse. This may be exacerbated by **low predictive signal** of the task, which makes it challenging for models to improve over trivial baselines.\n\nWhen models are instead selected using balanced accuracy (see Appendix B.2.2), which accounts for class imbalance, the picture changes: fewer individuals experience no recourse, and more receive substantial or full recourse. A similar trend is seen for ACSPublicCoverage.\n\nThis suggests that the elevated rate of instances without recourse observed for these tasks may be driven by a combination of class imbalance, low signal, and accuracy-based model selection for the Rashomon set.\n\nIf you have any further questions or concerns, we will be happy to address them. \n\nThe authors"}]

Author rebuttal:

Meta-review: ## Summary
The paper seeks to investigate the degree to which language models tend towards one or the other of two "poles": monoculture, where all models essentially behave the same, and multiplicity, where all models, while solving tasks with similar accuracy, exhibit huge variance in outcomes. The authors test a number of different language models with different prompts and different tasks, and determine that the reality of model behavior lies somewhere between these two extremes. 

## Strengths
Reviewers appreciated the creative problem formulation, the clarity of presentation, and the detailed and thorough experimental evaluation. 

## Weaknesses
Reviewers had mixed feelings about the framing of the paper, especially regarding the idea of a 'strict monoculture'. Some reviewers felt that the argument was somewhat of a strawman, and that in reality we would never see a strict monoculture. There were also a number of detailed questions about the experiments.

## Author Discussion
There was extensive and productive discussion between authors and reviewers, with the paper adding experiments, changing framing to soften some of the bolder language, and generally being improved. Most reviewers were quite satisfied with the outcome of the discussion. 

## Justification
This paper tees up an interesting discussion of importance in an environment where we are using multiple models and are concerned about algorithmic monoculture. While there is room for improvement in the delivery of the paper itself, the work is solid and is likely to spark further discussion in the community.
Decision: Accept (poster)