Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Authors: Rishi Bommasani
Kathleen Creel
Ananya Kumar
Dan Jurafsky
Percy Liang
Venue: NeurIPS 2022 Accept (2022)
Keywords: systemic harms of ML
sharing
fairness
algorithmic monoculture
foundation models
AI Ethics

Abstract: As the scope of machine learning broadens, we observe a recurring theme of *algorithmic monoculture*: the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers.  While sharing offers advantages like amortizing effort, it also has risks.  We introduce and formalize one such risk, *outcome homogenization*: the extent to which particular individuals or groups experience the same outcomes across different deployments.  If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy.  We relate algorithmic monoculture and outcome homogenization by proposing the *component sharing hypothesis*: if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes.  We test this hypothesis on algorithmic fairness benchmarks, demonstrating that increased data-sharing reliably exacerbates homogenization and individual-level effects generally exceed group-level effects.  Further, given the current regime in AI of foundation models, i.e. pretrained models that can be adapted to myriad downstream tasks, we test whether model-sharing homogenizes outcomes across tasks.  We observe mixed results: we find that for both vision and language settings, the specific methods for adapting a foundation model significantly influence the degree of outcome homogenization.  We also identify societal challenges that inhibit the measurement, diagnosis, and rectification of outcome homogenization in deployed machine learning systems.

Picking on the Same Person: Does Algorithmic
Monoculture lead to Outcome Homogenization?
Rishi Bommasani"
Kathleen A. Creel
Ananya Kumar
Computer Science Stanford University
Philosophy, Computer Science
Computer Science Stanford University
Northeastern University creel@hortheasterrn edu
nl mlprish1@stanforde Ott.asstaannddedooooooooo
ananya@cs stanford.
Dan Jurafsky
Percy Liang
Linguistics, Computer Science
Computer Science
Stanford University
Stanford University
jurafsky@stanf ord edu
pliangecs stanf oord.ord...o edu
Abstract
As the scope of machine learning broadens, we observe a recurring theme algorithmic monoculture: the same systems, or systems that share components (e.g. datasets, models), are deployed by multiple decision-makers, While sharing offers advantages like amortizing effort, it also has risks. We introduce and formalize one such risk, outcome homogenization: the extent to which particular individuals groups experience the same outcomes across different deployments. If the same individuals or groups exclusively experience undesirable outcomes, this may institutionalize systemic exclusion and reinscribe social hierarchy. We relate algorithmic monoculture and outcome homogenization by proposing the component sharing hypothesis: if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes. We test this hypothesis algorithmic fairness benchmarks, demonstrating that increased data- Rata-sharing reliably exacerbates homogenization and individual-level effects generally exceed group-level effects. Further, given the current regime in Al of foundation models, i.e., pretrained models that can adapted to myriad downstream tasks, we test whether model-sharing homogenizes outcomes across tasks. We observe mixed results: find that for both vision and language settings, the specific methods for adapting foundation model significantly influence the degree of outcome homogenization. We also identify societal challenges that inhibit the measurement, diagnosis, and rectification outcome homogenization in deployed machine learning systems.
Introduction
Machine learning is built on strong traditions sharing: we share datasets (e.g. ImageNet), models (e.g. BERT), libraries (e.g. PyTorch), optimizers (e.g. Adam), evaluations (e.g. SuperGLUE) and much more. This ethos of sharing serves the field well: we are able to repeatedly capitalize on the effort required to build high-quality assets (c.g. ImageNet has supported thousands of researchers in computer vision), and improvements to these assets have sweeping benefits (c.g. BERT raised all boats in NLP). Yet does sharing also have risks? Could this central tenet of the field lead undesirable outcomes?
"Corresponding author.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

We observe that certain forms of sharing can be reinterpreted as monoculture: Kleinberg and Raghavan [02T] define algorithmic monoculture as the state "in which many
all rely on the [exact] same algorithm." In parts of society where algorithmic systems are ubiquitous, we see trends towards such monoculture [Moore and Tambini| 2018 Engler] 202T]. Monocultures often pose serious risks: Kleinberg and Raghavan| [2021] show monoculture ture is suboptimal for decision-makers when their decisions are interconnected. as when they compete to hire job candidates. In ML, our sharing practices often are more complex than sharing the entire algorithmic system: should we think of our practices of sharing assets in ML as monoculture and, if so, what harms should we worry about?
We investigate this question by proposing one potential risk we call outcome homogenization, i.e. the phenomenon of individuals (or groups) exclusively receiving negative outcomes from all ML models they interact with. For example, job applicant may be rejected from every job they apply to due to the use of similar algorithmic resume screening systems at all companies. We view outcome homogenization as an important class of systemic harms that arise when we study social systems, i.e. harms that require observing how individuals are treated by many decision-maie conceptually motivate outcome homogenization in the context of algorithmic hiring. In $3
P In $2 we
introduce the first mathematical formalism for outcome homogenization: we measure homogenization as the observed probability of systemic failure normalized by the base rate.
To link the practice of sharing in ML with the proposed harm of homogenization, we pose and test the component sharing hypothesis: algorithmic systems built using the same underlying components, such as training data and machine learning models, will tend systematically fail for the same individuals or groups. We see component sharing specific form of algorithmic monoculure, which broadens the initial definition in Kleeiinberg and Raghavan| (20TT) from aicision-maker deploying the same system to deploying similar systems in terms of how they are constructed. We investigate how two types of shared components - training data and foundation models - contribute to homogeneous outcomes.
In [4 we demonstrate that data-sharing reliably homogenizes outcomes for individuals and for racial groups, especially for small training datasets involving US census data. In [3] we discuss how the rise of foundation models [Bommasani et al. [202]]. i.e. pretrained models that can be adapted to myriad downstream tasks, could yield unprecedented homogenization. Based on experiments with foundation models for vision (CLIP) and language (RoBERTa), to our surprise, we find the use of foundation models does not always exacerbate outcome homogenization. Instead, we find the specific mechanism for adapting the foundation model to the downstream task significantly influences homogenization for example, linear probing consistently leads to more homogeneous outcomes than finetuning for both modalities. Through these experiments, it is clear that the relationship between sharing and homogenization is not fully explained by our hypothesis, but that there is some evidence that sharing homogenizes outcomes. To advance the study of homogenization in practice, where systemic harms are most consequential, we conclude by identifying key challenges for diagnosing, measuring, and rectifying homogenization in society (86).
2 Outcome Homogenization in Resume Screening
To illustrate outcome homogenization and its potential causes (including algorithmic monoculture), we will use the example of algorithmic resume screening. Companies use resumes to screen job applicants, choosing which candidates to interview and which to reject. Maximum homogenization occurs when every company makes the same decision about each candidate, such that each lucky candidate is interviewed by all companies and each unlucky candidate by no companies. We say that the unlucky candidates who receive interviews experience a systemic failure F
What factors might homogenize outcomes in human makiing? Even in the absence of the algorithms, we observe homogeneous outcomes in many setting. In hiring, historically, hiring
IIn fact, outcome homogenization is systemic harm that may arise even in the absence of algorithmic monoculture, though this work is restricted to settings where monoculture is present.
"A fundamental consideration is that the right just outcomes in hiring are contested: the notions of merit and ground truth are much more subjective than, say, classifying images as dogs cats. For this illustrative example, we do not delve into this, though we it makes explicit that some individuals can be justifiably rejected from all opportunities (e.g. those attempting to become lawyers without passing the bar exam). Hence, the interpretation of homogeneous outcomes will need to be contextual, and is likely to be value-laden in allocative contexts such as hiring, education, lending, and health.

---

managers at each company decided who to interview and often agreed in their decisions. This agreement can be attributed multiple sources: first, if the needs of each company were identical, then managers at different companies may be incentivized to interview the same candidates, thereby homogenizing outcomes. Second, if hiring managers' choices are influenced by the same social biases, they will mistakenly reject the same people, thereby homogenizing their errors. Bias in resume screening is well-documented and remains significant [Jowell and Prescott-Clarke 1970 Bertrand and Mullainathan 2004 Kline et al. 2021 inter alia].
homogeniziess
However, neither explanation implies that systemic failures are inevitable. Since companies have different needs and resumes are imperfect predictors of success in role, the "best" candidates will likely differ across companies. Further. bias is not uniform across companies: Klinne al. (2021] find that 21% % of firms were responsible for 46% of the racial bias in interview decisions. Even if decisions are influenced by the same group-level biases, different companies may choose different individual members of the advantaged and disadvantaged groups. Variance in company needs, in prevalence of bias, and in individual hiring manager preferences all make more likely that different resumes survive the screening stage different companies, ensuring some diversity in resume screening outcomes.
How do these dynamics change with the introduction of automated decision-makee maainn? Most large companies now use automated resume screening software to parse resumes and decide which applicants advance. As stylized example, if every company deploys the same deterministic system and has the same hiring criteria, then outcomes will be necessarily homogeneous: individuals will either receive interviews at every company or be rejected by all of them (i.e. systemic failure). This example is not far from reality: a few major vendors dominate the marketplace for algorithmic resume screening with 700 companies, including over 30% of Fortune 100 companies, relying on Hirevue [Hirevue 2021]. Thiractice of different companies deploying the same system is defined as algorithmic monoculture by Kleinberg and Raghavan (202T].
More generally, different companies may instead deploy similar, but non-identical, systems. We expand the definition of algorithmic monoculture to encapsulate this broader setting, which is also alluded to in Kleinberg and Raghavan [202T] Engler [2021] describe this as the reality for college enrollment management algorithms, writing "there are a relatively small number (between five and 10) of prominent vendors in the enrollment management algorithm market, ... their process and analytics are markedly similar. Since their processes seem relatively consistent, the outcomes might be as well potentially leading to consistently good results for students who match the historical expectations of colleges, and consistently poor results for students who don't".
Component Sharing Hypothesis. In this work, we study systems that are related in how they are constructed, akin to what is described by Engler [202I] We pose the component sharing hypothesis that relates such algorithmic monoculture with outcome homogenization: If deployed algorithmic systems share components, outcome homogenization will increase (i.e. there will be more systemic failures). In this work, we empirically test this hypothesis for two prominent forms of component sharing: (i) the sharing of training data in training all deployed systems ($4) and (ii) the sharing of the same foundation model for building all deployed systems $55)
3 Formalizing Outcome Homogenization
While prior work [Kleinberg and Raghavan 2021 Creel and Hellman| 2022 alludes to outcome homogenization, here we provide the first mathematical formalism of outcome homogenization In line with our running example of resume screening, we formalize outcome homogenization for individuals in terms of failures (i.e. every algorithmic system fails for an individual). We then generalize to the group setting, where groups are systemically excluded rather than individuals, with a discussion of how these metries relate to established fairness, robustness, and accuracy metries ($3.4)
3.1 Formalizing Outcome Homogenization for Individuals
Notation, Since we define outcome homogenization as a systemic phenomenon, we consider a social system (1r ) where every individual j interacts each deployed models h², As an example, indi-
'The formal model of Kleinberg and Raghavan 2021 is related, but substantially distinct, Concretely, their formalism considers harms experienced by decision becision-makers, makers, whereas we center decision-subject.
3

---

vidual jE [N] submits features 2² (e.g. their resume) as input to company i [k] to receive an outcome h' (x())))= Ú0 (e.g. interview). Let D' be the empirical distribution of inputs xf for company O
To define notion of failure, let 7 (xx) indicate if új is a negative outcome, i.e. individual j experiences a negative outcome from model h². The failure rate for model h²
FAIL
E at wi
(==)==
Pr x'>Di ( [1'x=)=-]
Experimentally, we consider classification errors as failures (i.e. 1 (x & (1""(x')) *V') but other negative outcomes (e.g. rejections from hiring or educational opportunities; (xx)) - [[ also can be studied under our framework.
Systemic failures for individuals. If an individual exclusively experiences failure, we say they experience systemic failure. The observed rate of systemic failure SYSTEMIC FAILURE(h),...
SYSTEMIC FAILURE( kt
A E
('(x)) = -
(2)
Homogenization metric for individuals. SYSTEMIC FAILURE quantifies homogeneous outcomes, but is difficult to compare across systems with different underlying accuracies: SYSTEMIC FAILURE will in general be higher for less accurate systems independent a specific tendency to pick (i.e. fail) on the same person. While may sometimes want combine accuracy and outcome homogenization into implicitly does,
an overall measure of utility or social welfare, which SYSTEMIC FAILURE(hh,..... we focus on a relative measure of homogenization that disentangles accuracy from homogenization. In particular, we are interested in outcome homogenization even, and perhaps especially, in systems that are highly accurate.
As result, we measure individual-level outcome homogenization for social system { by normalizing the observed rate of systemic failure by the expected rate of systemic failure.
E (a))
SYSTEMIC FAILURE ..... .bk)
Hindividual (h h)
(3)
(FAIL
II (r))
This measure is the ratio between (i) the probability that individual experiences systemic failure and (ii) the probability that randomly sampled outputs for each model are all failures. That is, the measure captures how the rate of systemic failure changes when attend to the structure of individuals.
3.2 Formalizing Outcome Homogenization for Groups
In addition to individual- homogenization, we also measure group-level homogenization. While our individual-leve metric individualizes harm, complementing work on group-level biases, we may also want to identify the extent to which (possibly marginalized) social groups (e.g. Black women) are systemically excluded. Further, we often lack individual-lewe information (e.g. due to privacy concerns; see (6) or study algorithmic deployments that do not share individuals (e.g. hiring different states).
For each input x², denote the associated group as G(æ*) Eg. Group identity can correspond to the data producer (e.g. the age of a user querying search engine) or the data subject (e.g. the race of an individual subject to face recognition). Let D' be the empirical distribution of inputs for group
Notation.
g (i.e. {xr G(++)) = )) The group failure rate FAIL. (h*) is
FAII. (h²)
E ('xx)). x Di
(4)
Homogenization metric for groups. To measure group-level homogenization, we modify our individual-level metric with weighted average over groups in place of simple average over individuals.
'Note that our framework is general: we permit the deployed models to be for different tasks and for the individual's 's inputs to not be the same, though in our resume screening example all the models perform the same task and applicants often submit the same resume different companies.

---

Σ
W [
HG (h² ....."")-
(5)
[FAIL(h')
Weights. We consider three weighting schemes, specified by categorical probability distributions W distributed over 9 (full definitions SA.1
Average (Hagg) W weights each group proportional its frequency across all deployments.
Uniform (Humtt) W the uniform distribution, so W (g) TBT
Worst wora)) W assigns weight I to the group Sworst with the highest systemic failure rate and 0 to all other groups. This reduces the numerator to simply be the systemic failure rate for Gworst-
We introduce these weight functions to clarify that, much like having both individual-level metrics and group-level metrics, we may want to weight groups differently in different circumstances. For example, weighting by frequency may provide a useful overall measurement of homogenization but obscure systemic exclusion experienced by minority groups specifically the worst-off group.
3.3 Understanding our metrics
As ratio of probabilities, our metrics take values in [0,cc) where 0 indicates no systemic failures, 1 indicates the observed rate matches the expected rate, and values greater than indicate some degree of outcome homogenization. In the individual setting, we assume each individual generates exactly one input per deployment, which may not hold in practice (e.g. people may submit multiple resumes or not apply to every company). We appropriately generalize our individual-leve metric to address this in Ap pendix A] Further, in the group setting, werooee the individual- metric using the uniform weight- ing (or the average weighting) if each individual's ''s inputs are treated as belonging their own group.
3.4 Relationship with other metries
Since we introduce (several) metrics, we consider how they relate to metries for related constructs (e.g. accuracy, fairness, robustness). This speaks the convergent and divergent validity of our metrics [Cambell and Fiske 1959 Messick 1987 Jacobs and Wallach 2021]. i.e. whether they are adequately correlated with metrics of similar constructs and adequately uncorrelated with metrics of dissimilar con- structs. Here, we discuss theoretical relationships, whereas in $5.2 we look at the empirical correlations.
Accuracy. When failure are errors, we design our metrics to minimize canti-ycorrelation with accuracy. While not theoretically guaranteed, we empirically demonstrate this in Table T With that said, we expect there will be settings where the two correlated: correlation, but to ensure that we do not neglect homogencous outcomes in highly accurate systems (i.e. neglect the individuals who are systemically failed even when the overall picture may seem rosy).
goal is not technically ensure
Fairness and Robustness. Beyond accuracy, outcome homogenization closely related to fairness and robustness. However, we emphasize that outcome homogenization is fundamentally about correlated outcomes for social systems, whereas almost all robustness or faimess metries are defined for single model. Recent work [Zhao and Chen 2019 D' Amour et al.) 2020 Wang al.) 2021 has initiated the study of fairness multi-task learning, however these works focus on favorable overall trade-off' across tasks as opposed to systemic modes of failure. Conversely, our metrics cease to be interesting (c.g. H iinddiiduull is always 1) in the single-mode setting as systemic failures are single-mon model failures.
At a more fine-grained level, algorithmic fairness metrics [e.g. Dwork et al. 2012 Hardt et al. 2016) emphasize discrepancies between individuals/groups. these differences: we are interested in the observed rate of systemic failures (and whether this exceeds the expected rate). Performance differences are not sufficient for outcome homogenization: performance disparities for each deployment do not align across deployments, then outcomes may not be homogeneous. For robustness metrics, our metric i the worst-case setting closely resembles the metrics studied in work on gorst-group robustness [e.g. [Sagawa* et al. 2020). In particular, when there is only one deployment, our metric recovers the standard worst-group accuracy normalized by the overall accuracy.
contrast, our metrics do not (explicitly) center
the
5

---

3.5 Alternative metries
In An/ppeddix A] we more extensively discuss desiderata for our metric, alternatives we considered, and how we arrived at the metrics we present in the main paper. With that said, we also note conditions where we may instead favor alternatives, as well as connections to familiar quantities like the covariance, Pearson correlation, and (pointwise) mutual information in the binary setting (k=
4 Data-Sharing Experiments
Having stated our mathematical formalism and metrics for outcome homogenization, we test if sharing training data leads to outcome homogenization. We consider widely used algorithmic fairness datasets (Fabris et al. 2022]: systemic failures for these specific datasets may not be of direct social consequence, but the datasets capture relevant social contexts where other forms of inequity have been documented.
Data, We work with two datasets: German Credit [GC; Dua and Graff 2017]. the third most widely used fairness dataset, and ACS PUMS (Ding el. 2021 which was built replace the most widely used fairness dataset, UCI Adult I history, credit amount, credit risk for the individual); following Wang al. [021], we consider two prediction tasks of (i) predicting if the individual receives good or bad loan and (ii) predicting whether their credit amount exceeds 2000. ACS PUMS contains US Census survey data recording 286 features (e.g. self-reported race and sex, occupation, average hours worked per week) for 3.6 million individuals. Ding et al. [2021] construct several predictions tasks of which we use three: (i) predict if an individual is employed, (ii) predict an individual's 's income normalized by the poverty threshold, and (iii) predict if an individual has health insurance.
GC contains information on 1000 German contracts (e.g. credit
Individuals and Groups. For both datasets, we have individual-level information, hence we measure individual-level homogenization across models for each task. For ACS PUMS, we have self-identiffed race across 9 US Census categories (c.g. American Indian. Asian, Black/Affrican American, White, two or more races), hence we measure group-level racial homogenization.
Experimental Design. To test if data-sharing influences outcome homogenization, we execute a controlled comparison by specifying two sampling protocols for the training data: fixed and disjoint. In general, articulating and determining what it precisely means for models to share data is complicated in natural settings: here we make the concept very precise. In the fixed setting, we sample n points without replacement from the entire training dataset, which we use to train all of the & task-specific models (k: = 2 for GC and k 3 for ACS PUMS). In the disjoint setting, we sample kn points without replacement that we randomly partition across the k: task-specific models. In other words, in the fixed setting, the task-specific models share the exact same training data inputs, whereas in the disjoint setting, the task-specific models share the same training distribution, but not the exact data. We emphasize that this is subtle difference between the setting, but it implies fixed shares more than disjoint.
Having specified the training data, we train models for each of the k tasks (we consider all of logistic regression, SVMs, gradient- boosted trees, and small neural networks in [Appendix Cp. To account for randomness, we report results averaged over 25 trials of the experiment (i.e. 5 samples of training data and training runs per sample for every value of n we consider).
Results and Analysis. In Figure (Ieft) we see clear evidence for our hypothesis: the fixed setting reliably shows more homogeneity than the disjoint setting. Overall, across all 3 datasets and 4 learning algorithms we consider, we find that fixed generally leads to more homogeneous outcomes than disjoint, which provides evidence towards to greater outcome homogenization than the use of different (but identically distributed) training data. However, this relationship is not perfect: we do see several instances where the degree of homogeneity is similar or even sometimes greater for the disjoint setting (e.g. regions of the left subplot of Figuue 2). Therefore, data sharing alone does not fully characterize homogeneity (e.g. randomness in training and instability in the number of observed systemic failures are important to consider).
hypothesis: the of the same training data leads
Further, the trends in the enror rates for fixed and disjoint are near-identical (right subplot Figure 1). as we would expect given the relationship between the sampling protocols. That is, only observing the accuracy is not enough in these settings for understanding the observed systemic failures, and provides evidence that our measures correctly identify discrepancies even when the underlying
We include results for third dataset, LSAC Wightman al. 1998 in Appendix C

---

Outcome Homogen zation for Individuals GC (MK)
Expected Systemic for individuels in (NN)
0.99
akgjing
asporrt
food
fxed
0.98
0.10
Vonegeltaes
0.97
0.09
0.36
0.00
0.35
1.50
200
300
350
200
320
350
Number
Number
Figure 1: Data-sharing leads more homogeneous outcomes. Data-sharing results for GC using neural network classifiers, which shows homogenization (Ieft) and expected systemic failure rate (right) as a function of training dataset size (r-axis).
Outcama =omaganizitian for indidieuals
PUMS (Logletic)
Herrage vizatiin Racial Group: In ACS PUMS (Loglaticc
30
diqpare
1.35
digion:
2.5
foou
1.32
MOrIL digp rt. worst, foodd
1.25
1.20
1.15
1.0
3003
2000
Figure 2: Picking on the same person. Data-sharing results for ACS PUMS using logistic regression: individual-level homogenization (left) exceeds group-level (right; avg and unif).
accuracies of each model are (approximately) the same. Finally, as a function of dataset size, we do not see consistent trends on the relationship between dataset scale and outcome homogenization, though we unsurprisingly see the disjoint and fixed settings converge for larger dataset sizes (since the discrepancies in the sampling become negligible due to convergence in measure).
Picking on the same person. For ACS PUMS in Figure 2 we contrast the individual-level and group- level homogenization. (Recall the average and uniform metrics are the group-level analogues of our individual- metric.) Outcomes are consistently more homogeneous at the individual-gne group than for racial groups. In fact, group-level analysis show little homogenization (values near 1) and little change as function of dataset size, whereas individual- level measurement exposes greater homogeneity and more variation (which is unsurprising since group-level quantities are more extensively aggregated). This has significant ramifications for many works on algorithmic fairness, which only consider social groups (e.g. race): these works may miss systemic failures for particular individuals that are obscured at the group-level Kef,[Ka et al. [2018] Hashimoto et al. 20188. Even intersectional approaches may not suffice to surface these systemic failures, unless each intersectional group comprises a single individual.
Model-Sharing Experiments
Having found that data-sharing appears to exacerbate outcome homogenization, we now turn to model-sharing. Specifically, we test how sharing.funddatio models affects outcome homogenization. Bommasani al. [202T] detine foundation models as "models trained on broad data (generally using self-supervision at scale) that can be adapted to wide range of downstream tasks". These models have had a sweeping impact on the Al research community, most notably in NLP, and are increasingly central to deploying ML at both startups (e.g. Hugging Face, Cohere, A121) and established technology companies (e.g. g. Google, Microsoft, OpenAI). Sharing is endemic to the foundation model paradigm: to justify their immense resource requirements, models must be used repeatedly for costs to amortize favorably. In the extreme, entire domain like NIP comes to build almost all downstream systems on or few foundation models, then any biases or idiosyncrasies of these models that pervasively
7

---

manifest downstream could potentially yield unprecedented systemic failures and outcome homog- enization [Bommasani al. 2021 Fishman and Hancox Li 2022]. We see initial evidence for such algorithmic monoculture: BERT was downloaded 10 million times in the past month alone and GPT-3 enables hundreds of deployed apps. Consequently, we believe it is especially timely to understand if, and to what extent, outcomes get homogenized as these models become entrenched as infrastructure.
?
5.1 Experiments
Data. To test how foundation models (c.g. CLIP, RoBERTa) influence homogenization. we run experiments for both vision and language data dn the vision side, we work with the CelebA dataset eL. 2015) of celebrity faces paired with annotations for facial attributes. For each face image, given the associated attributes, we define two tasks (Earrings, Necklace) that involve predicting whether the individual is wearing the specific apparel item. Attribute prediction in CelebA has been studied previously in work on fairness and robustness (Sagawa* et al. 2020 Khani and Liang 2021 Wang et al. [222]. On the language side, we use four standard English text classification datasets followingGururrage Gururangan al. [009]] IMDB [Maas al. 2011], AGNews (Zhang al. 2015). Yahoo
Chang et al. 2008 and HateSpeech18 de Gibert et al. [2018).
Individuals and Groups. Since the vision tasks are all based CclebA, we have individual-level information. However, since the language tasks involve entirely different data (c.g. movie reviews vs. news articles), there is no (shared) individual-level information. At the group-level, for vision we use annotations for hair color and for whether the individual has a beard, whereas for language automatically group inputs by binary gender.
Experimental Design. To test if model-sharing influences outcome homogenization, we contrast setting with differing degrees of model-sharing. In the vision experiments, we produce task-specific models for each task by either (i) training from scratch on CelebA data, (ii) linearly probing by fitting linear classifier features from the CLIP foundation model [Radford et al. [021], (iii) finetuning CLIP. To ensure meaningful comparisons, the models trained from scratch shared the same VIT architecture [Dosovitskiy et al. 2021] used in CLIP but with weights initialized randomly.
In the language experiments, we further hone in on the specific adaptation method used to adapt the foundation model (specifically RoBERTa-base (Liu et al. 2019)) to each task. We consider (i) linear probing, (ii) finetuning, and (iii) BitFiT [Ben Zaken et 2022 which is a recent lightweight finetuning method in NLP that involves freezing all the RoBERTa weights except the bias parameters which are updated as in finetuning. Consequently, BitFit is an intermediary between probing and finetuning, which has been shown to achieve similar accuracy as finetuning while updating very few of the pretrained parameters. For both vision and for language, all models are trained for the same number of epochs and we repeat each experiment for 5 random seeds per adaptation method.
Hypotheses. Much like data data-nharing, model-sharing is graded and not binary: different downstream systems can share varying degrees of underlying models. By design, our experimental design suggests a continuum in sharing: first, downstream system either can share a foundation model or not (scratch). Second, among methods that involve foundation models, all methods initialize the weights using the pretrained weights but differ in which parameters remain the same after adaptation is completed: finetuning changes all the parameters, BitFit only changes the bias parameters, and probing changes none of the parameters. As a result, overall, we can rank methods from most to least sharing as (i) probing, (ii) BitFit, (iii) finetuning, (iv) scratch, which leads us to predict the degree of homogenization will also follow this ranking under our ceapponnnt-shainng hypothesis.
Results. In Figure 3 (left), across all vision settings, we surprisingly find that scratch is the most homogeneous, T.e. more homogeneous than either approach involving shared foundation models. This is the opposite of what we hypothesized: we posit that this may indicate model sharing is not the key explanatory variable for outcome homogenization here, but instead it is more complex form of data sharing. Specifically, we conjecture that since the scratch models are only trained on Celeb A data, whereas the others also are trained on the much larger WebImage via the CLIP foundation model, this may mean that the models based on CLIP are effectively regularized from learning idiosyncrasies of CelebA that the seratch models acquire. This may more generally suggest that a more correct hypothesis around data sharing should factor in the relationship (c.g. distribution shift) between the
/attpe://huggingfaen co/bert- base uncased as of Octoher 2022. Fuull reproducibility details for vision are in BB.2 for language are in |B.3
8

---

Dutcoma lormoge raaion Staue
Mrath
1350
Factuaing
Brotunir
Elaturry
025
pridiig
Figure 3: Model-sharing does not reliably lead to more homogeneous outcomes. Model-sharing results as function of iaaatingaddptattionn method for vision (left three) and language (rightmost). Vision: scratch is the most homogeneous, then probing, then finetuning. Language:
probing is the most homogeneous; finetuning and BitFit are similarly homogeneous.
training data and the evaluation data for each model. Additionally, we find probing is consistently more homogeneous than finetuning, which aligns with our hypothesis. Finally, akin to the census results (54) we once again find that outcome homogenization is significantly higher for individuals than for groups (comparing to Havg and Hunir).
In Figure ( (right), across all language settings, we find the ordering of homogenization matches what our hypothesis predicts. Specifically, we find BitFit and finetuning achieve similar levels of homogene- ity, even though BitFit updates 0.08% of the parameters full finetuning does (i.e. the number of shared parameters for BitFit is more like probing than finetuning), suggesting the number of shared parameters not the right lens for understanding model sharing. More broadly, these results do suggest parameter- sharing effects may contribute outcome homogenization within the foundation model regime, but comparisons between foundation models and no foundation models may be more complex to explain.
5.2 Correlations between Metries
Since we introduce several metrics, we measure the correlations between our metrics. Further, we measure correlations with accuracy (specifically, the expected rate of systemic failure) to test if homog- enization is disentangled from accuracy. Since outcome homogenization is related to fairness, we also measure the correlation between our metrics and standard group fairness metric. Fairness metries are generally defined for single model h, whereas we study entire systems (h') f-- We extend the unfair- ness definition used by Khani et al. [2019) as the variance in the systemic failure rates across groups
UNFAIRNESS h...,hh) Var g
Il FAILL (h')
Results. In Table we report the pairwise correlation between metric pairs, based on the models we
Vision
Language Hase
Hyra
Husl
Huwes
Accumey
Unfairness
Hasil
Accuracy
Unfaimess
Hare
(087,0.93)
(0.0. 0.96)
(0.0. 0.09*)
(0.0.
(0.22.
(0.11.0.56
(0.06 -0.22*) (0.02.
Hust (0.87, 0.93)
(0.0, (0.0. 0.96)
(0.0, -0.02)
0.74
(0.22, -047)
(0.63, -0.53)
(0.0, 0.10% (0.0.-0.01)
Hwant
(0.0,0.9)
(0.05,00 0.1*) (7.0.082) (0.11,0.56) (0.63..0.53)
(0.02.
(0.13,047)
Table 1: Inter-metric correlations. Correlations amidst our metrics as well as with other metrics reported as (Pearson R.2 Spearman p) with significant at 0.005 and italics significant at 0.001.
trained in $5.1 These correlations are for 45 systems (3 methods x 3 groupings x random seeds) of models for vision and 15 systems of 4 models for language. For vision, our metrics are highly correlated with each other, whereas for language. Hunit patterns quite differently (columns 1-3, 6-8). This is to be expected in that the vision groups (e.g. hair colors) all share similar frequencies, whereas the female group is significantly rarer in the language datasets. For both language and vision, we find that our metrics are generally not correlated, or perhaps weakly correlated, with accuracy as we intended (columns 4, 9). With respect to fairness, our worst-case metric H Hworst is strongly correlated for both modalities, but for the other two metrics we see no linear correlations and only monotone correlations for the vision experiments (columns 5, 10). This is line with our broader expectations that fairness and outcome homogenization are indeed related (especially for the porst-performing group), but that given they are distinct theoretical constructs, they should not always be correlated [Campbell and Fiske 1959].

---

5.3 Discussion
Across our experiments, we provide considerable that sharing leads to homogeneous outcomes. but that it is incomplete explanation incompleteepplaaation of homogeneity. This is particularly relevant when the findings in 34 anr contrasted, given model-sharing in the foundation model regime indirectly implies immense data-sharing via the pretraining data (as mediated by the pretrained initialization). We emphasize that the regimes for these findings are quite different: low-dimensional tabular data with simple model families in data-sharing so discrepancies in the findings may be attributable to these differences. More broadly, we believe a more complete explanation requires accounting for the data distributions and the associated distribution shifts (e.g. between pretraining and adaptation) at play. What we believe is clear, however, is that findings provide empirical basis to build our conceptual arguments that sharing in machine learning can increase homogenization. This motivates investigation into real deployments of machine learning: for example, does sharing/monoculuure lead homogenization in algorithmic hiring.
-ighdiimnsiooall images/text with large neural networks in model-sharing,
Societal Considerations and Challenges
To situate work in a broader social context, we identify and discuss core challenges in diagnosing. measuring, and rectifying outcome homogenization in real deployed systems.
Diagnosis. In our work, we posit monoculture yields homogenization: to follow this approach would require knowing which deployments rely on the same vendor, dataset, or foundation model (i.e. know- ing where there is monoculture). Unfortunately, how algorithmic systems are constructed is often so opaque that identifying shared components is nigh impossible. However, if high homogenization were demonstrated, the measurement itself could justify provisions for increased transparency to identify the latent monoculture (i.e. the anti-causal direction). This provides a plausible mechanism for empowering auditors to be granted conditional access otherwise inaccessible proprictary systems.
Measurement. Measuring homogenization only requires black box access, which is often achievable in practice [see Buollawini and Gebru 2018 Raji and Buolamwini, 2019, Metaxa et al. 2021).
However, identifying individual- effects requires linking individual outcomes across deployments.
Due to privacy constraints, linking individuals across different deployments may be challenging or impossible, which motivates group-level homogenization as more generally accessible (see (3.2).
Rectification. Even once outcome homogenization is identified, organizations may not be incentivized to reduce it. In fact, homogenization neither is attributable to any single entity nor can it always be addressed by unilateral action from single organization. In the face of misaligned incentives and col- lective action problems, regulation, policy, or other compliance mechanisms may be required. Potential trade-offs between organization incentives and homogenization are further complicated if the harms of homogeneous outcomes take time to observe/accrue, but the benefits of, say, maximizing accuracy are immediate. More optimistically, KKlinberr and Raghavan| [2021] show (under specific conditions) no trade-off exists between accuracy-maximizing maximizing policies and diversifying outcomes for societal benefit.
7 Limitations and Conclusion
We have introduced, formalized, and measured outcome homogenization as a systemic harm that may arise from practices of sharing in ML. Outcome homogenization is new, understudied, and conceptually compelling topie: its definition, interpretation, statistical estimation, mitigation, and connections to monoculture remain poorly understood in spite of this work. We encourage future work to push in all of these directions. As to our measure, direct optimization may not lead to desirable outcomes, potentially even contributing to ethics- washing: its interpretation must be contextual since the implications of homogeneous outcomes heavily depend on broader socictal context.
We believe homogenization is essential to holistically characterizing algorithmic harm, especially given growing monoculture (e.g. via foundation models). Without scrutiny, its harms may insidiously entrench. Consequently, we believe early intervention is necessary to prevent such harms in society.
Reproducibility. All code, data, and experiments are available on GitHub and CodaLab Worksheets
10

---

References
Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy Sciences, 118(22), 2021. ISSN 0027-8424 doi: 10. 11073//aas.001834000 2018340118
URLttts::://ww pnas.
(Cited on
10)
Martin Moore and Damian Tambini, editors. Digital Dominance: The Power of Google, Facebook, Ama- zon conand Oxford University Press, New York, 05 2018. ISBN 9780190845133 (Cited 2
Alex Engler. Enrollment algorithms are contributing to the crises of higher education. report, The Brookings Institution, September 2021. URL bttp:pppppp//....ooooosss.ss...e
mroolment- aass--algoo htw.amms-are--oocabaaatiiiieeiee chw.totttt---c-re--eeteeeeeeeeso ottp::/www.............oooooooa.
(Cited on 3
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, D. Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel F. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jackson K. Ryan, Christopher R' e. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam. Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258, 2021. URL https://crfm: http::///crssssss.ososssoo stanf edu taaasess/repores pdf (Cited on 778
Roger Jowell and Patricia Prescott-Clarke Racial discrimination and white-collar workers in britain. Race, 11(4):3974177 April 1970. doi: 10.1177/00069663000 10040000110 100401. URL
https://doi. org/ wttpg//..1177///000
(Cited
Marianne Bertrand and Sendhil Mullainathan. Are emily and greg more employable than
lakisha and jamal? field experiment on labor market diserimination. American Eco- nomic Review, ([44)991-10133 August 2004. doi: 10. 10.155700022333442005566 URL
htttp:://do..org//// /ttp//000006/0000222200006
(Cited
Patrick Kline, Evan Rose, and Christopher Walters. Systemic discrimination among large u.s. employers. Technical report, National Bureau of Economic Research, July 2021. URL
Stttp://doi.org// 31//0/0.3386///95533
(Cited 3
Hirevue. Hirevue customers wwndut million video interviews in just 30 days, Oct 2021. URL wtwp://wweecctttttoooeoooocteet. wowduct- over-
customers- wtw.riiwwweuucccceteeeeeeeeeeeee
-ilion-video--inerviiee htt.:reeeeeetttrreeeeeeeeea--a-- (Cited 3
Kathleen Creel and Deborah Hellman. The algorithmic leviathan:
Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philosophy, 52(1):
26-43, 2022. doi: 10. (Cited 3)
Donald T. Camphell and Donald W. Fiske. Convergent and discriminant validation by matrix. Psychological Bulletin, 56(2):81, 1959. URL the .nbii.ddddddddid................ ncbi nlm. nih, gov/13634291//
https://pubmed.
(Cited on
Samuel Messick. Validity. ETS Research Report Series, 1987())i-208, 1987. wRt..ttts::/ ine
library. wiley. com/ doi/ wwwpo////as//ooooo 1002/ 23::://...330888883 1987 tb00244. x (Cited on 18)

---

Abigail Z. Jacobs and Hanna Wallach. Measurement and faimess. In Proceedings of the 2021 Confer- ence on Fairness, Accountability, and Transparency, FAceT FAceT '21, New York, NY. USA, 2021. Associ- ation for Computing Machinery. Sttllhttps://aae wwwpexiiv..rraaaao wwt/
05511, (Cited on 5 [8]
Chen Zhao and Feng Chen. Rank-based multi-task learning for fair regression. In 2019 IEEE Interna- tional Conference on Data Mining (ICDM), pages 916-925, 2019, doi: 20.1000/CD.22009.00000 10000...
(Cited on 5)
Alexander D' Amour, Katherine A. Heller, Dan I. Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi-An Ma, Cory Y. McLean, Diana Mineu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin G. Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. ArXiv, 0bbs22011.03399 2020. (Cited 5
Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H. Chi. Understanding Trade- Offss in Multi-Tast Task Learning, page 1748-1757. Asso- ISBN 978 1450383325. URL
and Improving
ciation for Computing Machinery, New York, NY, USA, 2021.
https://doi org/10. 10. +145/3447548 3467326
(Cited on
5
6 g
21
23
24
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fair- ness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '12, page 214-226, 226. New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450311151
doi: 10. 00.1145/2000233.20000555 URL (Cited on
https://doi. org/ 10. 1145/ 1ww.1111220022000 2090236 2090255
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In NIPS, 2016. (Cited on
Shiori Sagawa*. Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openrevver www.:eeetoooee httt//ooummidrryxuurrtttet
(Cited on 2121 21
Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. Algorithmic faimess datasets: the story so far. Data Mining and Knowledge Discovery, abs/2202.01711, 2022.
(Cited on 6 23
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. //archive. ics. aci.edu/mmi (Cited on 23)
URL
h http
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair
machine learning. In Thirty-Fifth Conference on Neural Information P Processing Systems, 2021.
/oppette http://ss.ppppeeeee.eeeeee//rraa 2dt (Cited on 6 20
24
L.F. Wightman, H. Ramsey, and Law School Admission Council. LSAC National Longitudinal
Bar Passage Study. LSAC research report series. Law School Admission Council, 1998. URL
httts:://book: http://booos.oogleeeeee com/ /ttp///oos....-TTAAAAAAAAAA.
(Cited on 6 23 24
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerryman- dering: Auditing and learning for subgroup fairness. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2564-2572. PMLR, 10-15 Jul 2018. URL htttps://proceedinga. mml. www.m.l....///..mmmmmmmmmmmm.... press/ w0800kearss18a..o html (Cited on
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Faimness without demographics in repeated loss minimization. In ICML, 2018. (Cited on
Nic Fishman and Leif Hancox-Li. Should attention be all we need? the epistemic and ethical implications of unification in machine learning. In Proceedings of the 2022 Conference on Fairness, Accountability, and Transparency, FAceT "22 New York, NY, USA, 2022. Association
for Computing Machinery. URL Utthttpps: //ta:////////////aa org/ abs/ ab0/2205..222.. 08377 (Cited on 8)
12

---

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. 2015 IEEE International Conference on Computer Vision (ICCV), pages 3730-3738, 2015. (Cited
on 8 22
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately In Proceedings of the 2021 ACM Conference on Fairness, Accountability
and Transparency, FAccT '21, page 196-205, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 7781450383997
doi: 10. 1e1111/3442188. 88.444888... URL
https://doi. 1145/3442188 3445883
(Cited 8 21
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. Variational pretraining for semi-supervised text classification. In Proceedings the 57th Ammual Meeting of the Association for Computational Linguisties, pages 5880-5894, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10.18653/v 1/P19- 1590. BRLhhttps://ccaattiii
otg/p19- 1590
(Cited on 8 22
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. 1. Association for Computational Linguistics. URL http://www
aclweb. wwtp:/awwwwb..og/atooooooooooo http://ghhooolgo1111111 1015 (Cited o
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Cortes, N. Lawrence, Lee, M. Sugiyama, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL https: https:://proceeddigg. neuripe pttps/ccpppp/p/p/e/aaapaep /ttp:////////////000000//1333/// 67a9a02-Paper papperr.df]] pdf (Cited on
Ming- Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference Artificial Intelligence Volume 2, AAAI'08, page 830-835. AAAI Press, 2008 ISBN 57735363. (Cited on 8)
Ona de Gibert, Naiara Perez, Aitor Pablls, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop Abusive Language Online (ALW2), pages 1 1-200, Brussels, Belgium, October 2018 8. Association for Computational Linguistics. doi:
(0.18653/v1/W18-. 1002 URLhttps::
awwwwb.wwb........ wwg.ww...g//a.....eoooooo..o.eeo hww..//w...../////oooogg//////// 5102
(Cited on
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, (Cited on 8 21
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL htttp:://openreieee.eeee wttee//fooummiiiiiccaaalaaa
(Cited on 8 21
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.
ArXiv, abs//907..... 1692, 2019. (Cited on
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficieet fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL acl sel-shhort.1 (Cited 8
/tttss://aclanthology wrg.lllooyyooo/////
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy. ArXiv, abs/1906.03518, (,000311882081 2019. (Cited on
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Pro- ceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of
13

---

Proceedings of Machine Learning Research, pages 77-91. PMLR, 23-24 Feb 2018. URL httpps:/prroceeding. mlr. ppress/81/buulllatt
18a. html (Cited on 20
Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly results of commercial ai products. In Proceedings the 2019 AAAI/ACM
naming biased performance results
Conference on AI, Ethics, and Society, AIES 19, page 429-435, New York, NY, USA, 2019. doi: 10. 144/33366118.311122222 (Cited I0
Association for Computing Machinery. ISBN 9781450363242.
URLhhttps:/doi. org/ 10. 0145/3306666611888 3314244
Danaë Metaxa, Joon Sung Park, Ronald Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends@ in Miiman-Computte Interaction, 14(4):272-344, 2021. ISSN 155 1555- (Cited on 10)
doi: 10.1561/1 100000083. httphttp////t http://xxxooi.i... org/ 10w11161111000 1561/1000000088
Jane Loevinger. Objective tests as instruments of
doi: 10. psychological theory.
Psycho- URL
logical Reports, 3(3):635-694, 1957.
https://doi ieaiidiooorgggg/ 10. 2466/pr0. 3.3.7.3..... 635 (Cited on
18)
D.J. Hand. Measurement: A Very Short Introduction. Very short Ooootrttooaccttosooooooood Universtty Press,
2016. ISBN 978019877568.. URL.https://book http://oook..oggll..oooooo///rbs httpg///oooooooe1111188810000000 (Cited on I8
L. E. Dubins and E. H. Spanier. How to cut a cake fairly. The American Mathematical Monthly, 68 (1):1-17, 1961. ISSN 00029890, 19300972. URLp:ttp://www jstor wrggsstable//3115555
(Cited on 19
Monika Henzinger, Charlotte Peale, Omer Reingold, and Judy Hanwen Shen. Leximax approximations and representative cohort selection. ArXiv, abs/2205.01 157, 2022. (Cited on [9)
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthicu Brucher, Matthicu Perrot, and Edouard Duchesnay, Scikit-learn: Machine learning in python. Journal of Machine Learning Research, ) 2830, 2011. URL htttlhttt///mmlll org. www.:rogggpppeee/e/e v12/ pttpppps////pprprrraaaaaaaaeeeee pedregosalla. htmmll (Cited on 20 24
Abeha Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In 202/ IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536-1546, 2021. doi: 10. 1t..1////ACV446302222211 000158. (Cited on 21)
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. ArXiv, abs/2110.019633, 2021. (Cited on 21)
Quentin Lhoest, Albert Villanova del Moral, Yacine Jerniee Abhishek Thakur, Patrick von Platen, Suraj Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François La- gunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language pro- cessing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, 184, Online and Punta Cana, Dominican Republic, Novem- ber 2021, Association for Computational Linguistics. doi: 10.18863/vv 1/202 100553//1/2221.mmlp-odo org 2021. emnlp-demo 21 (Cited on 22
Julien Chaumond, Mariama Drame, Julien
lemnlp-deoo.222
URR_itttp:://caanttooo
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Sfate-of-the-ar natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, October 2020. Association for Computational Linguisties. doi: 10. 1 8653/v /2////20eemmm 1t222..emmlmpeeeeeeee eenop- URL
oonlp-demos... httts://aclanthoo/ org//o//000 20200.00000 emnlp- demos 6 (Cited on 22
14

---

Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. "you are grounded!". Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850-6861, Online, November 2020.
Association for Computational Linguisties. doi: 10.1 100.18633v11/ 1/203/ emnlp-main httips///allanthologg org/ 22200 emnlp-main 556 (Cited on 22
URL.
Alexey Romanov, Maria Dr-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Kalai. What's in a name? Reducing bias in bios without access to protected attributes. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4187-4195, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/v1/N11-14222 URL_htttss:/aaclantbollo. org/N19 1424 (Cited on 22)
Konstantinos Tzioumis. Demographic aspects of first names. Scientific Data, 5, 2018.. (Cited on
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):13335-E33444 2018. ISSN 0027-8424. doi: 10. 10.107 (t7.3///a..72034777 15. URL
https://www pnas. org/ ct/////enennn/////////1/ (Cited on 22 23
Yang Trista Cao and Hal Daumé III. Toward gender-inclusive coreference resolution. Proceedings
of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568-458 Online,July 2020. Association for Computational Linguistics. doi: 10. 1653/v1/2020.acl-mm m11.
URL httpaattssss//lllahhllllleo aclaathoonntooyyoooo org/2020 acl-main 418, (Cited on 22
Maria Antoniak and David Mimno. Bad seeds:
Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1889-19800 1904, Online, August 2021 Association for Computational Linguistics. doi: 10.18653/v org/ wttpg/g//2..0...e 2021 acl acl-lonlgnn.. 148
1/2021.acl-oor long. 148. URLhtttp:
(Cited on 23
Yu Zhang and Qiang Yang. A survey multi-tas leamning ArXiv, hwtp:/ab.//1111111111 2017. (Cited on 23
Tai Le Quy, Arjun Roy, Vasileios losifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12, 05 2022. doi: 10.1000/widm. 1452. (Cited on (3)
Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. parency, FAccT '22, page 1827-1858, New York, NY, USA, 2022. Association for Com- URL
2022 ACM Conference on Fairness, Accountubility, and Trans-
puting Machinery. ISBN 9781450393522. https://doi. org/10. 10. 1145/3531146 2353333381144333332222
doi:
(Cited 27)
Charles T. Marx, Flavio Du Pin Calmon, and Berk Ustun. Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020. (Cited on 27)
L. Breiman. Statistical modeling: The two cultures. Quality Engineering, 48:81-82, 2001. (Cited on 27
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license the code and datasets? [N/A]
15

---

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "f7p5VcdApOw", "reviewer": "Reviewer_o96f", "summary": "In this paper, the authors discuss how model and/or data sharing arising from a monoculture could lead to outcome homogenization, thus giving rise to several individual and group harms at a systemic level. Taking the example of resume screening, if a resume screening model misclassifies the resume of an otherwise well qualified individual and the same model is being utilized at every company, that individual will be experiencing systemic harm as they'll be rejected by every company. Similar case could be made for college admissions. The authors highlight that given recent advances in NLP, with large language models forming the basis of several AI applications, their sharing across applications and across companies could potentially be harmful to users for whom the underlying language model (that is being shared) does not perform accurately. The authors then extend this to group harms (systemic harms that could be faced by minority populations for instance). The research hypothesis that the authors then test is that *if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes*.\n\nThe authors conduct experiments on data sharing and model sharing and find supporting evidence for their hypothesis in the data sharing regime, while finding mixed results in model sharing which suggest that the pretraining strategy used to train the underlying model likely plays a role in outcome homogenization.\n\nWhile overall this is a sound paper, I have several questions. The paper is missing several inferential steps as one has to make some big (and potentially unrealistic) assumptions to arrive at a scenario where the same systems are being used by the same companies and/or are trained on the same data. Furthermore, the results on data sharing are not much different than the scenario where different datasets are used (when you reach closer to 10,000 datapoints). The practical viability of this problem is not established and I hope the authors will take time during the response period to address some of these issues.\n\n-----------------------------------------------------------------------------------\nI have updated my score following discussion with the authors but still have some concerns (reflected in my score) about the practical viability of the assumptions made in the experiments in the paper.", "questions": "- Did you try any experiments on data sharing with pretrained architectures? For example, you share the data you're using to finetune models but you're finetuning different pretrained models---that too is data sharing.\n- Why did you run logistic regression five times on each sample? Correct me if I'm wrong but shouldn't the variance just be coming from different samples in case of a logistic regression model (this wouldn't have changed your plots as you're only plotting means but still curious).", "limitations": "I don't think the paper has done a good job at putting its limitations and assumptions up front.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***:\n- The paper is well situated in prior work on monoculture.\n- The experimental setups are well created and the discussion is particularly strong (except the assumptions).\n- Limited results on data sharing suggest that there could be outcome homogenization in such scenarios and strikingly clear results on model sharing following certain forms of pretraining.\n\n***Weaknesses***:\n- The foundation of this problem is very fragile. It relies on a lot of assumptions being true, where it's unclear how true they might be. Counter example: Chase Bank would likely create its own model on it's own data (that it might merge with the Census data) to predict whether someone should be approved a loan or not. Citi Bank would create it's own. The corporations do not have an incentive to share models or data (or even rely on existing pretrained models when the normal practice in companies is to retrain their models frequently to account for new data) as they are competing against one another. Now let's look at resume screening. Take for example, Companis X and Y. Company X has different needs than Company Y despite both hiring for AI Research Scientists. Would Company X and Y want to use the same resume screening software trained on a generic dataset or have it at least be calibrated to their individual needs?\n- Experiments on data sharing are limited to only Census data and it is unclear whether the trends hold beyond that dataset. In absence of additional results, a possible inference from these results could be that the trends you're showing are special to just the Census data.\n\n***Minor comments***:\n- The paper often makes broad generalizations when there are important qualifiers that need to be attached. For instance, Line 256 should be saying that data sharing appears to exacerbate outcome homogenization for small datasets...\n- When providing examples in introduction (Adam, BERT, Pytorch, etc.), please include citations.\n- Typo on Line 366: align aligns ---> align", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sdgeo-D-bhO", "reviewer": "Reviewer_zUzT", "summary": "The paper formalizes the idea of outcome homogenization, where a specific individual might receive negative outcomes from an array of different (but interconnected) models. The key contribution is to propose formal measures of homogenization and conduct analysis showing how homogenization relates to shared datasets and shared models used to carry out different tasks.", "questions": "Line 89: “not far from reality”. Does a lot of companies using the same provider/software indicate that the software outcomes on a given resume will be the same for all the companies?\n\nLine 232: How are the 5 training runs different from each other? I understood how 5 samples of training data are different (drawn using different seeds), but could not understand the difference with training runs. Is it the model random seed? If yes, that should make no difference since the logistic regression solution is deterministic (owing to convexity).\n\nLine 288: Is the whole transformer trained from scratch? How much time did it take? How did the accuracy compare to the fine tuned models?\n\nLine 309: Do these results also consider different random seeds (there are some error bars in Fig 2 but maybe I missed how these are computed)? This might be quite important given the non-convex nature of these models.\n", "limitations": "In general the paper is quite up front about potential limitations. There are still many places where improvements can be made. For suggestions for improvements, please see the main review.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper formalizes a very interesting and relevant notion of disadvantage which indeed may be prevalent given the widespread use of shared datasets and shared (foundational) models. While the prior works allude to the same notion (as the paper mentions in Section 3), a formalism and dedicated study has indeed been lacking. Formalizing homogenization opens up the doors for important analysis and insights, for instance the insight in Section 4 that even intersectional fairness metrics may not be able to detect systematic disadvantages. The paper is generally very well-written and easy to follow. Assumptions are stated rather clearly and formalism is followed by appropriate reasoning.\n\nUnfortunately, the definitions and experiments seem quite preliminary. Definitions being preliminary is perhaps OK since the paper is only initiating research into a new area and later papers can study the topic in more detail. But drawing insights from the limited set of experiments presented here seems premature. For these reasons, I think the paper should be improved before it is ready for publication. Please see detailed comments for suggestions / questions:\n\n1. The data sharing insights use just one single model (logistic regression) and dataset (ACS PUMS). How do we expect the insights to vary when we use other datasets or other models? Would the insights change if we use high parametrized models like NNs (where memorization is a concern)? Drawing insights from a single dataset seems insufficient.\n\n2. It would also have been good to see some discussion into where the data sharing ends and model sharing starts. After all, two different versions of a foundational model still have a lot of training data in common (since the underlying model was pretrained on the same data). Given the fact that a large part of the training data is still shared, how do we reconcile the two sets of takeaways from Section 4 and 5?\n\n3. I feel that the definition of homogenization might be too strict. Consider for example Eq. 2. Right now the quantity $\\prod_i I^i$ is only 0 or 1 — indicating whether or not *all* systems misclassify the individual in question (even with 1 out of 100 systems going the other way, the takeaway would change completely). Why not use a soft version, for instance, fraction of systems that misclassify the individual?\n\n4. I think the formulation in Section 3.2 can be simplified quite a bit. The weighted average of groups in Eq. 5 seems quite difficult to interpret. Why not simply measure which groups are being disadvantaged at a higher rate than others. In other words, why not measure the gap between the best and worst systematic group failure rates?\n\n5. Line 119 and 120: First the paper mention $x_i^j$ but then the next line mentions $x_j^i$, causing a confusion in the terminology.\n\n6. Looks like Figure 1 is quite sparse in terms of number of points. Given the large change that happens between $10^1$ and $10^3$, it might make sense to add more points in this region. Also, since the results anyway plateau quite quickly, why not consider a non-log scale? Finally, it would be great to add point markers indicating where the observed points are.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HoKURIt1DX8", "reviewer": "Reviewer_h4Gz", "summary": "The paper introduces a mathematical framework for characterizing outcome homogenization wherein homogenization is measured as the observed probability of systemic failure normalized by base rates. A conceptual motivation is also provided with the example of a algorithmic hiring. Experiments on US census data demonstrate that data sharing homogenizes outcomes for individuals and racial groups, this is more prominent for small training datasets. ", "questions": "1. How would the type of learning algorithm influence homogenization? For e.g., studies have shown that supervised algorithms reinforce human bias especially in the context of scientific discoveries. \n\n2. Related to the above question, would the extent of homogenization therefore be a function of the particular use case? \n\n3.  Would the results be similar to tabular data?\n\n4. How will dataset sparsity and measurement errors influence outcome homogenization?\n\n5. Which of the factors mentioned in Section 2 are the most dominant factors in causing outcome homogenization - is there any intuitive or empirical evidence in support of this?", "limitations": " see above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper studies an important topic and can be valuable for further research in the context of fairness, accountability, and transparency of ML systems. \n\n2. The claims made in the paper are backed by reasonable experimentation and exposition of the results\n\nWeakness:\n\nThe paper would be much stronger with a more fine grained analysis across use cases and data modalities- see some pointers under questions below. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept