Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
Authors: Connor Toups
Rishi Bommasani
Kathleen Creel
Sarah H Bana
Dan Jurafsky
Percy Liang
Venue: NeurIPS 2023 poster (2023)
Keywords: homogenous outcomes
societal impact of ML
deployed ML
systemic failure
TL;DR: Characterizing the societal impact of ML requires considering all relevant ML models: we introduce ecosystem-level analysis,  showcasing its ability to generate new insights for holistically characterizing the societal impact of machine learning.

Abstract: Machine learning is traditionally studied at the model level: researchers measure and improve the accuracy, robustness, bias, efficiency, and other dimensions of specific models. In practice, however, the societal impact of any machine learning model is partially determined by the context into which it is deployed. To capture this, we introduce *ecosystem-level analysis:* rather than analyzing a single model, we consider the collection of models that are deployed in a given context. For example, ecosystem-level analysis in hiring recognizes that a job candidate’s outcomes are determined not only by a single hiring algorithm or firm but instead by the collective decisions of all the firms to which the candidate applied. Across three modalities (text, images, speech) and 11 datasets, we establish a clear trend: deployed machine learning is prone to systemic failure, meaning some users are exclusively misclassified by all models available. Even when individual models improve at the population level over time, we find these improvements rarely reduce the prevalence of systemic failure. Instead, the benefits of these improvements predominantly accrue to individuals who are already correctly classified by other models. In light of these trends, we analyze medical imaging for dermatology, a setting where the costs of systemic failure are especially high. While traditional analyses reveal that both models and humans exhibit racial performance disparities, ecosystem-level analysis reveals new forms of racial disparity in model predictions that do not present in human predictions. These examples demonstrate that ecosystem-level analysis has unique strengths in characterizing the societal impact of machine learning.

Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
Connor Toups" *
Rishi Bommasani"
Stanford University
Stanford University
Kathleen A. Creel
Sarah H. Bana
Dan Jurafsky Stanford University
Percy Liang Stanford University
Northeastern University
Chapman University
Abstract
Machine learning is traditionally studied at the model level: researchers measure and improve the accuracy, robustness, bias, efficiency, and other dimensions of specific models. In practice, however, the societal impact of any machine learning model depends on the context into which it is deployed. To capture this, introduce ecosystem-levvel analysis: rather than analyzing single model, we consider the collection of models that are deployed in given context. For example, ecosystem-level analysis in hiring recognizes that a job candidate's 's outcomes are determined not only by single hiring algorithm firm but instead by the collective decisions of all the firms to which the candidate applied. Across three modalities (text, images, speech) and eleven datasets, establish clear trend: deployed machine learning is prone to systemic sailume meaning some users are exclusively misclassified by all models available. Even when individual models improve over time, we find these improvements rarely reduce the prevalence of systemic failure. Instead, the benefits of these improvements predominantly accrue to individuals who are already correctly classified by other models. In light of these trends, we analyze medical imaging for dermatology, setting where the costs of systemic failure are especially high. While traditional analyses reveal that both models and humans exhibit racial performance disparities, ecosystem-lews analysis reveals new forms of racial disparity in model predictions that do not present in human predictions. These examples demonstrate that ecosystem-level analysis has unique strengths in characterizing the societal impact of machine learning."
Introduction
Machine learning (ML) pervasively deployed. Systems based ML mediate our communication and healthcare, influence where we shop or what we eat, and allocate opportunities like loans and jobs. Research on the societal impact of ML typically focuses on the behavior of individual models. If center people, however, we recognize that the impact of ML on our lives depends on the aggregate result of our many interactions with ML models.
In this work, we introduce ecosystem-level analysis to better characterize the societal impact of machine learning on people. Our insight is that when ML model is deployed, the impact users depends not only on its behavior but also on the behavior of other models and decision-makers (left of Figure 1). For example, the decision of a single hiring algorithm reject or accept a candidate does
Equul contribution.
Coonespnnding author: Auprishi@stanteoo edu. All code is available at https:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Failure Matrix
a
Systomic Farkure
Users
G
Decision makers
Figure 1:
Ecosystem-level analysis. Individuals interact with decision-makers (left), receiving outcomes that constitute the failure matrix (right).
not determine whether or not the candidate secures job; the outcome of her search depends on the decisions made by the firms which she applied. Likewise, in selecting consumer products like voice assistants, users choose from options such Amazon Alexa, Apple Siri. or Google Assistant. From the user's perspective, what important is that least one product works.
In both settings, there is significant difference from the user's perspective between systemic failure, which zero systems correctly evaluate them or work for them, and any other state. The difference in marginal utility between zero acceptances and one acceptance is typically much higher than the difference between one acceptance and two acceptances. This con-linearity is not captured by average error metrics. For example, imagine that three companies are hiring and there are ten great candidates. All three companies wrongly reject the same two great candidates for false negative error rate of 20%. Now imagine that each company wrongly rejects different candidates. The second decision ecosystem has the same false negative error rate, 20%, but no systemic failures or jobless candidates.
Ecosystem-leve analysis is methodology that centers on the failure matrix (F;; right of Figure I): F encapsulates the outcomes individuals receive from all decision-makers. Of special interest are systemic failures: exclusively negative outcomes for individuals such as misclassifications or rejections from all decision- makers [Bommasani al., 2022]. To establish general trends in deployed machine learning, we draw upon large- scale audit [HAPI; Chen et al., 2022a] that spans three modalities (images, speech, text), three commercial systems per modality, and eleven datasets overall. Because HAPI contains predictions from some of the largest commercial ML providers including Google, Microsoft, and Amazon and the models evaluated are deployed models that real users interact with, evaluating HAPI has real-world implications.
Across all settings. ecosystem-lever analysis reveals a consistent pattern of homogeneous outcomes. In each of the HAPI datasets, many instances are classified correctly by all three commercial systems and many instances are classified incorrectly by all three systems. The pattern of outcomes across the decision ecosystem is homogenous if the rates of systemic failure and consistent classification success significantly exceed the rate predicted by independent instance-level behavior. Since the commercial systems analyzed in this dataset are popular and widely used, being failed by all systems the dataset has societally meaningful consequences.
Ecosystem-level analysis enriches our understanding not only of the status quo, but also of how models change over time. In particular, it allows us ask, when individual models improve, how do ccosystem-level outcomes change? Since HAPI tracks the performance of the same systems over three-year period, we consider all cases where least one of the commercial systems improves. For example, Amazon's 's sentiment analysis API reduced its етгог rate on the WAIMAI dataset by 2.5% from 2020 to 2021; however, this improvement did decrease the systemic failure rate at all. Precisely 0 out of the model's 303 improvements are on instances on which all other models had failed. These findings generalize across all cases: on average, just 10% of the instance-level improvement of single commercial system occurs on instances misclassified by all other models. This is true even though systemic failures account for 27% of the instances on which the models could have improved. Thus most model improvements do not significantly reduce systemic failures.

---

To build on these trends, we study medical imaging, a setting chosen because the costs to individuals of systemic failure of medical imaging classification are especially high. We compare outcomes from prominent dermatology models and board-certified dermatologists on the DDI dataset (Daneshjou et al., 2022]: both models and humans demonstrate homogeneous outcomes, though human outcomes are more homogenous. Given established racial disparities in medicine for both models and humans, fairness analyses in prior work show that both humans and models consistently perform worse for darker skin tones (e.g. Daneshjou et al. [2022] show lower ROC-AUC on DDI). Ecosystem-level analysis surfaces new forms of racial disparity in models that do not present in humans: models are more homogenous when evaluating images with darker skin tones, meaning that all systems agree in their correct or incorrect classification, whereas human homogeneity is consistent across skin tones.
Our work contributes to a growing literature on the homogeneous outcomes of modern machine learning [Ajunwa, 2019, Engler, 2021, Creel and Hellman, 2022, Bommasani al., 2022, Fishman and Hancox-Li, 2022, Wang and Russakovsky, 2023, Jain et al., 2023]. While prior work conceptual- izes these phenomena, our work introduces new methodology to study these problems and provides concrete findings for a range of ML deployments spanning natural language processing. computer vision, speech, and medical imaging. Further, by centering individuals, we complement established group-centric methods [Barocas and Selbst, 2016, Buolamwini and Gebru, 2018, 8, Koenecke et al., 2020], unveiling new forms of racial disparity. Ecosystem- analysis builds on this existing work, providing new tool that contributes holistic evaluations of the societal impact of machine learning.
Developing better methodologies for ecosystem-level analysis of deployed machine learning systems is important for two reasons. First, systemic failures in socially consequential domains could exclude people from accessing goods such as jobs, welfare benefits, or correct diagnoses. Individuals who are failed by only one model can gain informal redress by switching to another model, for example by seeking second doctor's s opinion or switching banks. Individuals failed by all models cannot. Socially consequential systemic failures can happen due to reliance APIs, such as image recognition APIs used to identify cancers, speech recognition APIs used to verify individuals for banking, or facial recognition APIs used to unlock devices. Systemic failures can also occur in algorithmic decision- making systems such as those used for hiring, lending, and criminal justice.. The social importance of avoiding systemic failures in all of these systems is clear.
Second, as decision-makert makers become more likely to rely on the same or similar algorithms to make decisions [Kleinberg and Raghavan, 2021] or to use the same or similar components in building their decision pipelines [Bommasani et al., 2022], we believe that the prevalence of systemic failures could increase. Measuring systemic failures as they arise with the tools we present in this paper will expand our understanding of their prevalence and likely causes.
2 Ecosystem-level Analysis
How individuals interact with deployed machine learning models determines ML's impact on their lives. some contexts, individuals routinely interact with multiple MI models. For example, when a candidate applies to jobs, they typically apply to several firms. The decision each company makes to accept or reject the candidate may be mediated by a single hiring algorithm. In other contexts, individuals select single model from set of options. For example, when consumer purchases voice assistant, they typically choose between several options (e.g. Amazon Alexa, Google Assistant, Apple Siri) to purchase single product (e.g. Amazon Alexa). Centering people reveals simple but critical insight: exclusively receiving negative outcomes, as when individuals are rejected from every job or unable to every voice assistant, has more severe consequences than receiving even one positive outcome.
2.1 Definition
Recognizing how ML is deployed, we introduce ecosystem-level analysis as methodology for characterizing ML's cumulative impacts on individuals. Consider N individuals that do, or could, interact with i decision-makers that apply ÿ labels according to their deccision-making processes hy,,.., hg. Individual is associated with input s,, label yt, and receives the label i/! = hy(x.) from decision-maker j.
3

---

Outcomes. Define the failure matrix F E {0, 13Nx& such that Fli.j] = / + yi The failure outcome profiles f for individual which we refer
the outcome profile for brevity, denotes F(ē, :]- The failure rate fy for decision-maker j is f, Σm , Fl..j. (i.e. the empirical classification enror in classification). For consistency, we order the entries of decision-makers (and, thereby, the columns of the failure matrix) in order of ascending failure rate: Fl;, 1] is the outcome profile associated with the decision-mant maker with the fewest failures and Fl: is the outcome profile associated with the decisioonmakke with the most failures. The failure matrix is the central object in ecosystem-level analysis (see Figure l).
Systemic Failures. Individual i experiences systemic failure if they exclusively experience failure across the domain of interest: F[i.:] = [I...... 1]. Not only are systemic failures the worst possible outcomes, but they also often result in additional harms. If an individual applying to jobs is rejected everywhere, they may be unemployed. If no commercial voice assistant can recognize an individual's voice, they may be fully locked out of accessing class of technology. In our ecosystem-leve analysis, we focus on systemic failures as a consequential subset of the broader class of homogeneous outcomes [Bommasani et al., 2022].
3 Homogeneous Outcomes in Commercial ML APIs (HAPI)
To establish general trends made visible through ecosystem-level analysis, we draw upon large-scale three-year audit of commercial ML APIs [HAPI: Chen et al., 2022a] to study the behavior of deployed ML systems across three modalities, eleven datasets, and nine commercial systems.
3.1 Data
Chen et al. [2022a] audit commercial ML APIs, tracking predictions across these APIs when evaluated on the same eleven standard datasets over period of three years (2020 (020),002), We consider ML
APIs spanning three modalities (text, images, speech). where each modality is associated with a task (SA: sentiment analysis, FER: facial emotion recognition, SCR: spoken command recognition) and 3 APIs per modality (c.g. IBM, Google. Microsoft for spoken command recognition). The models evaluated are from Google (SA, SCR, FER), Microsoft (SCR, FER), Amazon (SA). IBM (SCR), Baidu (SA), and Face++ (FER). Additionally, each modality is associated with three to four datasets, amounting to eleven datasets total; further details are deferred to the supplement.
To situate our notation, consider the DIGIT dataset for spoken command recognition and the associated APIS (IBM, Google, Microsoft). For each instance (i.e. image) I; in DIGIT, the outcome profile f, E {0, 1 is the vector of outcomes. The entries are ordered by ascending model failure rate: Fl: 1] corresponds the most accurate model (Microsoft) and F(:,3) corresponds to the least accurate model (Google).
Facial emotion recognition AFNET
Spoken command recognition
Sentiment analysis
RAFDB
EXPW
FER+
PLUENT
DIGIT
AMNIST
SHOP
YELP
IMDB
WAIMAL
Dataset size Number of classes
15,3k
287.4k
31,5k
6.4k
30.0k
2.0k
30.0k
62.8k
20.0k
25.0k
12.0k
2
2
failure rate (i.e. erroe)
0.283
0.277
0.272
0.156
0.019
0.217
0.015
0.078
0.043
0.136
0.1
by failure rate (i.e. error)
0.343
0.317
0.348
0.316
0.025
0,259
0.015
0.085
0.111
0,219
0.151
failure rate (i.c. error)
0.388
0.359
0.378
0.323
0.081
0.472
0.043
0.122
0.486
0.484
0.181
Systemic failure rate
0.152
0.178
0.181
0.066
0.01
0.129
0.002
0.039
0.021
0.043
0.065
Table 1: Basic statistics on HAPI datasets including the (observed) systemic failure rate (i.e. fraction of instances misclassified by all models).
Descríptive statistics. To build general understanding of model performance HAPI, we provide
basic descriptive statistics (Table 1). For most datasets, all APIs achieve accuracies within 5-10% of each other (exceptions include DIGIT, YEL.P, IMDB). Interestingly, we often find the systemic failure rate is roughly half the failure rate of the most accurate model h1.

---

3.2 Ecosystem-level Behavior
In order for a measure of systemic failure be useful, must be (i) meaningful and (ii) comparable across systems. A challenge to the meaningfulness of any proposed metric is that systemic failures occur more often in an ecosystem with many inaccurate models. that primarily communicated the aggregate error rates of models in the ecosystem would not be meaningful as an independent metric. It also would not support goal (ii) because we could not compare the rates of systemic failure across ecosystems with varying model accuracies. It would be difficult to identify a system with 'large' rate of systemic failure because the systemic failure properties would be swamped by the error rates of the models in the ecosystem. Therefore, achieving meaningfulness and comparability requires the metric to incorporate error correction.
metric for systemic failure
Assuming model independence is a helpful baseline because it adjusts for model error rates without making assumptions about the correlation between models in the ecosystem. To avoid assumptions and for the sake of simplicity, therefore, we juxtapose the observed behavior with simple theoretical model in which we assume models fail independently of each other. Under this assumption, the distribution of the haseline number of model failures t. E (O....,kk) ..) follows a Poisson-Binomic distribution parameterized by their failure rates (Equation 2).
The baseline of independence also means that our metric does not attempt to quantify whether it is "reasonable" that the models all fail on some instances. For example, some instances might be harder (or easier) than others, making it more likely that all models will fail (or succeed) to classify that instance. However, "hardness" is observer-relative What is hard for one class of models might be easy for another class of model, and likewise with humans with particular capabilities or training. Therefore 'correcting' the metric to account for hardness would relativize the metric the group of humans or class of models for whom that instance is hard. We choose independence as a bascline to be neutral on this point. However, we depart from independence in Appendix $A.3, exploring how baseline that assumes some level of correlation between models can more accurately model the observed distribution of ecosystem- outcomes.
Comparing the true observed distribution of ecosystem-level outcomes with the baseline distribution helps illuminate how correlated outcomes are across models. Below we define Potserved (Equation 1) and P Passeliee (Equation 2).
N Si=l II
Σ F[
Pobserved (t failures)
(1)
N
Prosetine (t failures) = Poisson-Binomiall (f..... J.)1
Finding 1: Homogenous Outcomes. In Figure 2a, we compare the observed and baseline distri- butions for the spoken command recognition dataset DIGIT. We find the observed ecosystem-level outcomes are more clearly homogenous compared the baseline distribution: the fraction of in- stances that receive either extreme outcome (all right or all wrong) exceeds the baseline rate. These findings generalize to all the datasets (Figure 2b): the observed rate always exceeds the baseline rate for the homogeneous outcomes (above the line y = I) and the reverse mostly holds for intermediary outcomes.
4 Do Model Improvements Improve Systemic Failures?
The performance of a deployed machine lcarning system changes over time. Developers serve new versions to improve performance [Chen al., 2022b.a], the test distribution shifts over time [Koh et al., 2021], and the users (sometimes strategically) change their behavior [Björkegren et al., 2020]. In spite of this reality, most analyses of the societal impact of machine learning only consider static models.
Ecosystem-level analysis provides a new means for understanding how models change and how those changes impact people. When models change, what are the broader consequences across their model ecosystem? Do single-model improvements on average improve ecosystem-leve outcomes by reducing systemic failures? And to what extent are the individuals for whom the model improves the same individuals were previously systemically failed?
5

---

Observed Baseline
1.0
Models Correct
Proportion of insttances
0.8
0.6
0.4
Models Correct
Baseline
(a) Foosystem-leved outcomes for DIGIT dataset. (b) Ecosystem-level outcomes for all datasets. Figure 2: Homogeneous outcomes. Ecosystem-level analysis surfaces the general trend of ho- mogeneous outcomes: the observed rates that all models succeed/fail consistently exceeds the corresponding baseline rates. Figure 2a shows that models in the DIGIT dataset are more likely to all fail or all succeed on an instance than baseline. Figure failure (red dots) and consistent success (blue dots) of all three models on an instance are both more common than baseline, whereas intermediate results less common than baseline.
shows that across all datasets, systemic
1.0
improvements (nw672)
0.8
potentaal
Observed Improvements
0.8
Proportion of Instances
0.5
0.6
0.367
0.4
Outcome Profile
0.2
0.160
0.2
[X v]
0.110
0.1
0.074
[v v]
0.04%
0.2
1.0
[X
#1
[ XI
[< x]
Outcome Profile
Potential Improvements
(b) The distribution of outcome profiles for all year-over- when Amazon improves on WAIMAI from 2020 2021. year model improvements across all datasets.
(a) Outcome profile distribution for (Baidu, Google)
Figure 3: Model improvement is not concentrated on systemic failures. When a model improves, we compare the distribution of outcome profiles of the other two models on its initial failures (potential improvements) to the distribution on the instances it improved on (observed improvements). Across all improvements, including Amazon's improvement on WAIMAI (left), there a clear over-improvement right) and under- inder-improvement on [X, X] (below the identity line
on [V. v] (above right).
Setup. Chen et al. [2022a] evaluated the performance of the commercial APIs the same eleven evaluation datasets each year 2020-2022. Of all year-over-year comparisons, we restrict our attention to cases where of the three APIs for given task improves by at least 0.5% accuracy." Let himp denote the model that improved. We identify the instances that himp initially misclassified in the first year potential improvements and the subset of these instances that hamp correctly classified in the second year as improvements. Considering the initial distribution of failures for hmm-- we can ask where does the himp improve? We answer this comparing the distribution of outcome profiles for the other models (besides himp)) between the potential improvement and improvement sets.
Thh supplement $B.3 contains analysis that confirms findings robust to alternate thresholds.

---

Finding 2: Model improvements make little progress systemic failures. As case study, we consider Amazon's improvement on the WAIMAI dataset from 2020 2021. In Figure 3a, from left to right, [X, X] indicates the other APIs (Baidu, Google) both fail, [X, v and IV X] indicate exactly one of the other APIs succeed, and [V. v1 indicates both succeed. The majority (64.7%) of Amazon's 's improvement is on instances already classified correctly by the other two APIs, far exceeding the fraction of potential improvements that were classified correctly by Baidu and Google (34.1%) in 2020. In contrast, for systemic failures, the improvement of 11.6% falls far short of the potential improvement of 36.7%. In fact, since models can also fail on instances they previously classified correctly, the model's improvement systemic failures is even worse in terms of net improvement. Amazon's 's improvement amounts to no net reduction of systemic failures: the model improves on 78 systemic failures but also regresses 78 instances that become new systemic failures, amounting to no net improvement. The Baidu and Google APIs similarly show little improvement on systemic failures even as models improve.
This finding is not unique to Amazon's improvement on the WAIMAT dataset: in the 11 datasets we study, we observe the same pattern of homogeneous improvements from 2020-2022. In Figure 3b, we compare the observed improvement distribution' (y axis) to the potential improvement distributions (æ axis) across all model improvements. We find clear pattern: systemic failures (the [X, X] category) are represented less often in the observed improvement set than in the potential improvement set. This finding indicates that when models improve, they under-improve users that are already being failed by other models. Instead, model improvements especially concentrate on instances where both other models succeeded already.
Ecosystem-levar analysis in the context of model improvements disambiguates two plausible situations that are otherwise conflated: does single-mode model improvement (i) marginally or (ii) substantively reduce systemic failures? We find the reduction of systemic failures is consistently marginal: every case, the reduction fails to match the the distribution seen in the previous year (i.e. every [X, X] red point is below the line in Figure 3b).
Ecosystem-level Analysis in Dermatology (DDI)
Having demonstrated that ecosystem-level analysis reveals homogeneous outcomes across machine learning deployments, we apply the ecosystem-lewe methodology to medical imaging. We consider this setting to be an important use of ecosystem-level analysis because machine learning makes predictions that inform the high-stakes treatment decisions made by dermatologists.
5.1 Data
Daneshjou et al. [2022] introduced the Diverse Dermatology Images (DDI) dataset of 656 skin lesion images to evaluate binary classification performance at detecting whether lesion was malignant or benign. Images were labelled with the ground-truth based on confirmation from an external medical procedure (biopsy). In addition, each image is annotated with skintone metadata using the Fitzpatrick scale according to one three categories: Fitzpatrick I & (light skin), Fitzpatrick & IV (medium skin), and Fitzpatrick V & VI (dark skin). We use the predictions from Daneshjou et al. [2022] on DDI for two prominent ML models (ModelDerm [Han et al., 2020] and DeepDerm [Esteva et al., 2017]) and two board-certifed dermatologists." 4 We defer further details to Appendix C.
5.2 Results
Finding 3: Both humans and models yield homogeneous outcomes; humans are more ho- mogeneous. We compare observed and baseline ecosystem-level outcomes DDI for models (Figure 4a) and humans (Figure 4b). Consistent with the general trends in HAPI, model predictions yield homogeneous outcomes. For human predictions from board-certified dermatologists, we also
'The supplement $B.4 contains analysis comparing 'net improvements' to 'potential improvements' as well; the trends are consistent across both analyses.
*Daneshjou et al. [2022] also evaluated third model [HAMI0K: Tschandl al., 2018] that almost always predicts the majority class in this class-imil iaaasainble setting. We exclude this model since its failures are not interesting, but replicate our analyses in the supplement Appendix C to show the findings still hold if it is included.
7

---

06
66
2.44
03
Proparion Instances
0.4%
0.21
0.3
63
Proparion
0.206
82
0.99
0J137
03721
0.159
0:14
01
61
0.291
Outcome Profiie
Outcome Profie
(a) Outcome profiles for models.
(b) Outcome profiles for humans.
Figure 4: Homogeneous outcomes for models and humans. Consistent with HAPI, model predictions (left) yield homogenous outcomes on DDI. Human predictions (right) are even more homogenous than models.
0.10
0.10
Obeerved khus Boollr
Observed Nhur Baseline
0.00
0.00
Fitepwrick (N-208)
Hitpwerik 18 N (4-241)
Fiteperick V (1-2777
-0.05
-0.05
(NH4566)
-0.10
Profibe
(a) Model outcome profiles by skin tone.
(b) Human outcome profiles by tone.
Figure 5: Racial disparities for models but not humans. We stratify ecosystem-level analysis in DDI by the three skin tone categories, plotting the difference between observed and baseline rates. Models (lefl) show clear racial disparities, exhibiting the most homogeneity for the darkest skin tones, whereas humans (right) show no significant racial disparity.
see that outcomes are homogeneous. However, in comparing the two, we find that humans yield even more homogeneous outcomes. We take this to be an important reminder: while we predict that models are likely to produce homogeneous outcomes, we should also expect humans to produce homogeneous outcomes, and in some cases more homogeneous outcomes.
Finding 4: Ecosystem-level analysis reveals new racial disparities in models but not humans. Standard analyses of machine learning's impact focus on model performance across groups [e.g. Buolamwini and Gebru, 2018, Koenecke et al., 2020]. In AI for medicine, several works have taken such an approach towards understanding fairness [Obermeyer et al., 2019, Seyyed-Kand Kalantari et al., 2021, Kim et al., 2022, Colwell et al., 2022]. Daneshjou et al. [2022] demonstrate racial disparities for both model and human predictions on DDI and find that predictive performance is worst for darkest skin tones, aligning with broader trends of racial discrimination in medicine and healthcare [Vyas et al., 2020, Williams and Wyatt, 2015, Fiscella and Sanders, 2016].
Ecosystem-level analysis can build on these efforts. We conduct the same analysis from Figure stratified by skin tone. In previous experiments, we had observed systemic failures across the whole population. Here we measure systemic failures for subpopulations grouped by skin tone. In Figure 5, we plot the difference between the observed and baseline rates on the y axis: the All bars (blue) reproduce the homogeneity results from Figure 4.

---

Strikingly, ecosystem-lever level-analysis surfaces an important contrast between model behavior (left) and human behavior (right). Models (Figure 5a) are most homogenous for darkest skin tones (Fitzpatrick V & VI; dark brown) and least homogeneous for the lightest skin tones (Fitzpatrick & II; cream): The observed systemic failure rate for the darkest skin tones 8.2% higher than the baseline, while for the lightest skin tones it is 1.5% lower than the baseline. By contrast, humans (Figure 5b) show no significant variation as a function skin tone.
Ecosystem-level analysis therefore identifies a new form of racial disparity not previously documented. Critically, while prior works show racial disparities for both models and humans, here we find form of racial disparity that is salient for models but not present for humans. We note that, in absolute terms, homogeneity is higher for all racial groups human predictions than model predictions, though human predictions don't t display significant differences in homogeneity across racial group. This demonstrates that ecosystem-level analysis can reveal new dimensions of fairness, allowing stakeholders to identify the metrics that are most relevant in their context, be that model error or systemic failure rate. Our tool helps researchers and stakeholders evaluate the holistic ecosystem of algorithmic and human judgements that ultimately shapes outcomes for those subject to algorithmic judgement.
6 Commentary
While ecosystem-level analysis reveals new dimensions of machine learning's societal impact, also opens the door for further questions. We prioritize two here, deferring further discussion and a longer related work section to the supplement. How can we explain the pervasive homogeneous outcomes we have observed in deployed machine learning? And what are the implications of this work for both researchers and policymakers?
6.1 Explanations for Homogeneous Outcomes
Our findings provide evidence across several settings for homogeneous outcomes in deployed machine learning (Finding 1; $3) that are mostly unabated by model improvement (Finding 2: $4).
Data-centric explanations. We posit that "example difficulty" may give rise to homogeneous outcomes and provide three analyses that instantiate variants of this hypothesis.
First, human ambiguity on the ground-truth may predict homogeneous outcomes. To test this, we make use of the ten human annotations per example in the FER+ dataset within HAPI. We find that the systemic failure rate is monotonically correlated with annotator disagreement with the majority label. This suggests that some systemic failures are correlated with the ambiguity "hardness" of the image. However, we find that even some the least ambiguous images, namely images on which all or most annotators agree, have systemic failures. This indicates that human ambiguity is only partially explanatory of ecosystem-level model outcomes. We explore this further in $A.1.
Second, human error may predict homogeneous outcomes. To test this, we compare human pre- dictions on DDI with the ground truth biopsy results. We stratify ecosystem-level analysis by dermatologist performance, comparing (i) instances both dermatologists get right, (ii) instances they both get wrong, and (iii) instances where they disagree and exactly one is right. We find that when both dermatologists fail, there continues to be outcome homogenization. However, when both derma- tologists succeed, there is no homogeneity and the observed rates almost exactly match the baseline rates for every image. This suggests human error is also partially predictive of ecosystem-level model outcomes. We explore this further in $A.2.
Finally, more expressive theoretical models can better capture the observed trends than our simple full instance-level independence model. We introduce a two-parameter model. a fraction of instances are categorized as difficult and the remaining
a: are easy. A model's failure rate S3 over all examples f, on easy examples. Partitioning examples in this way, while continuing to assume instance-level independence, inherently homogenizes: models are more likely to systemically succeed on easy instances and systemically fail on hard instances. To best fit the HAPI data, the resulting average a (0.2 - 0.3) and A (1 4, meaning these examples are 2 - 5x harder) values are quite extreme. In other words, for this theoretical model to fit the
scales to (1 (1+ A)ff on hard examples and (1 (1- مه (1 1-a

---

data, a significant fraction (+e 25%) would need to be considerably harder (=s 3.5x) than the overall performance. We explore this further in $A.3.
These analyses contribute to an overarching hypothesis that example difficulty partially explains homogeneous outcomes. While we discuss the construct of diffieulty in the supplement, we draw attention to two points. First, difficulty is largely in the eye of the beholder: what humans or models perceive as difficult can differ [e.g. adversarial examples; Goodfellow et al., 2014]. Thus while example difficulty could be caused by inherent properties of the example (such as noisy speech or corrupted images), it could just as well be due model properties, such as all the models having made similar architectural assumptions or having parallel limitations in their training data, Second, whether or not homogeneous outcomes are caused by example difficulty does not change their societal impact. The consequences of systemic failure can be material and serious (c.g. unemployment).
Model-centric Explanations and Algorithmic Monoculture. An alternative family of explana- tions put forth in several works is that correlated outcomes occur when different deployed machine learning models share training data, model architectures, model components. learning objectives or broader methodologies [Ajunwa, 2019, Engler, 2021, Bommasani et al., 2021, Creel and Hellman, 2022, Fishman and Hancox-Li, 2022, Bommasaní et al., 2022, Wang and Russakovsky, 2023]. Such algorithmic monoculture [K [Kleinberg and Raghavan, 2021, Bommasani et al., 2022] in fact appears to be increasingly common as many deployed systems, including for the tasks in HAPI, are derived from the same few foundation models [Bommasani et al., 2023, Mądry, 2023]. Unfortunately, are unable to test these hypotheses because we know very little about these deployed commercial systems, but we expect they form part of the explanation and encourage future work in this direction.
Implications for Researchers. Our paper shows that ecosystem-level research can reveal eeriiously-invisibe social impacts of machine learning. We believe this methodology concretizes the impact of decision maniin in real contexts in which individuals would typically be affected by decisions from many actors (c.g. job applications, medical treatment, loan applications, or rent pricing). As we demonstrate in DDI, our methodology applies equally to human-only, machine-only, and more complex intertwined decision-making We anticipate understanding of homogeneous outcomes arising from any or all of these sources will be valuable.
Implications for Policymakers. Given the pervasive and persisting homogeneous outcomes document, there may be a need for policy intervention. In many cases no single decision-maker observe the decisions made by others in the ecosystem, individual decision-makers (such as companies) may not know that systemic failures exist. In addition, systemic failures are not currently the responsibility of any single decision-maker, so no decision-maker is incentivized to act alone. Consequently, policy could implement mechanisms to better monitor ecosystem-level outcomes and incentivize ecosystem-level improvement. In parallel, regulators should establish mechanisms for recourse or redress for those currently systemically failed by machine learning [see Voigt and von dem Bussche, 2017, Cen and Raghavan, 2023].
Conclusion
We introduce ecosystem-level analysis as new methodology for understanding the cumulative societal impact of machine learning on individuals. Our analysis on HAPI establishes general trends towards homogeneous outcomes that are largely unaddressed even when models improve. Our analysis on DDI exposes new forms of racial disparity in medical imaging that arise in model predictions but not in human predictions. Moving forward, we hope that future research will build on these empirical findings by providing greater theoretical backing, deeper causal explanations, and satisfactory sociotechnical mitigations. To ensure machine learning advances the public interest, should use approaches like ecosystem-level analysis to holistically characterize its impact.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Shibani Santurkar, Mina Lee, Deb Raji, Meena Jagadeesan, Judy Shen, p-lambda, and the Stanford ML group for their feedback on this work. We would like to thank James Zou and Lingjiao Chen for guidance with using the HAPI dataset. We would like to thank Roxana Daneshjou for providing the DDI dataset along with guidance
how to analyze the dataset. In addition, the authors would like to thank the Stanford Center for Research on Foundation Models (CRFM) and Institute for Human-Centeree Artificial Intelligence (HAI) for providing the ideal home for conducting this interdisciplinary research. RB was supported by the NSF Graduate Research Fellowship Program under grant number DGE-165500 18. This work was supported in part by Stanford HAI/icrosoof Azure cloud credit grant and in part by the A12050 program at Schmidt Futures (Grant G G-22-63429).
References
Rishi Bommasani, Kathleen Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL httttps://ppeeeviiiiie
http://oounnii--HoOooe H6KKmDDVo...
Lingiaa Chen, Zhihua Jin, Evan Sabri Eyuboglu, Christopher Ré, Matei Zaharia, and James Y Zou. HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24571-24585. Curran Associates, Inc., 2022a. URL https: h/proceddg///ooooonin/ neurips www.epp.cc/pppprrrre.
https//pppr//2///2 Dbco0al2771eec7/66222009933 Pataaaaaaee--aetaa.eaaaasassaaee and BewcBBennchhh..eeeee pdf.
Roxana Daneshjou, Kailas Vodrahalli, Roberto A. Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M. Swetter, Elizabeth Bailey, Olivier Gevaert, Pritam Mukherjee, Michelle Phung, Kiana Yekrang, Bradley Fong, Rachna Sahasrabudhe, Johan A. C. Allerup, Utako KariaaK, James Zou, and Albert Chiou. Disparities in dermatology ai performance on a diverse, curated clinical image set. Science Advances, 8(32):aaby6147, 2022. doi: 10. 10.1s26/s,.adv. abq6147. URL https: : //www/ science org/ htgg/diiaab///in
Ifcoma Ajunwa. The paradox of automation as anti-bias intervention. Cardozo L. Rev., 41:1671, 2019. URI.
Alex Engler. Enrollment algorithms are contributing to the crises of higher education. report, The Brookings Institution, September 2021, URL cfw.aabstaactttdd2222225 enrollment- al gor are- cewreeccooribbuttii -hteaattin--ttt-ttt-eteeeeeeeeee
sarn. www.co////oo/oo
edu/research/
Kathleen Creel and Deborah Hellman. The algorithmic leviathan: Arbitrariness, faimess, and opportunity in algorithmic decision deaision-making systems. Canadian Journal of Philosophy. 52(1): 26-43, 2022. doi: 10.1017/can 302.........22... URL abstract id=3786377.
sern. com/ cow/sol3//pttcccccccccocoo hwop://ppers..... cfm?
Nic Fishman and Leif Hancox-Li. Should attention be all we need? the epistemic and ethical implications of unification machine learning. 2022 ACM Conference on Fairness, Accountabil- ity, and Transparency, 2022. URL https:///// acm. ht/////ffllHtmmm1100 1145/3531144.
3533206.
Angelina Wang and Olga Russakovsky. Overcoming bias in pretrained models by manipulating the finetuning dataset. arXiv preprint arXiv:2303.06167, 2023. URL https: /ttp::////xxxvorr/aa 2303.06167
Shomik Jain, Vinith M. Suriyakumar, Kathleen Creel, and Ashia C. Wilson. Algorithmic pluralism: A structural approach towards equal opportunity. ArXiv, abs/2305.08155 157,, 2023. URL. https: // //arxiv.. org/ abs//305. 08157.
Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review, 104:671, 2016. URL https:://// bttpp::/wwwyr //www/ http://www.//rr///////////////tt hetttaabeee/57//////
11

---

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In FAT, 2018. URL https::/proceeddnng. mlr fal...preesvv1/ buolamwini /Ba/ Baabaulllaaaaaa www.bbulliwwnn 18a. pd.
Allison Koenecke, Andrew Joo Hun Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences of the United States of Amer- ica, 117:7684- 7689, 2020. URL https: //ww: pnas org/ htgg/di/100 10. 1073/ 10w3//nas..../// pnas 1915768111
Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(2))ee018334011 18, 2021. doi: 20.03///aa..21184400
URL https: / /ww. pnas wwrg/ddi/abb//t/ 1073/ 1073/pmas. pnas 2018340118.
Lingjiao Chen, Matei Zaharia, and James Zou. How did the model change? efficiently assessing machine learning API shifts. In International Conference on Learning Representations, 2022b.
URL https: htttap::///penrvvviie www.:/t..eeeeeeeeeeeeeeeeeeeeeee gf
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irema Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning,
pages 5637-5664. PMLR, 2021. URL https: / /ttp:///axxivvvei htt///aaiv.ooraa/////aaa/////e/e htt.b///211.07222.
Daniel Björkegren, Joshua E. Blumenstock, and Samsun Knight. Manipulation-pron machine learning. 2020. URL https: /wwvvv..vvvvv org abs///0ss0000....////////// 038665.
Scung Han, Ilwoo Park. Sung Chang, Woohyung Lim. Myoung Kim, Gyeong Park, Jebyeong Chac, Chang- Huh, and Jung-Im Na. Augment intelligence dermatology Deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders. Journal of Investigative Dermatology, 140, 03 2020. doi: 10. 1016/f.id.2020.011.1
URL https://www.t http:://wwww..ieecdrreeeee www.seeedeeeeete.....o... www.ssiieeeeeeeeeeecccccccoecoee article/pii/ $0022202X20301366 viw.//bbbb..
Andre Esteva, Brett Kuprel, Roberto Novoa, Justin Ko, Susan Swetter, Helen Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 01 2017. doi: 10.1038/nature210555 URL https: : //www nature. www.oootiieeeoocc articles/naturr210055
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 1000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 08 2018. doi: 10. 10.100//sdata22000 2018. URL https:
nature. com/ commaariclee////oooooocooaeeeeet sdata2018161.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447-453, 2019, URL https: //www. science. org/doi/ 100o/111//0 /10. 2/22///ciencee aax2342.
Laleh Seyyed- Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27(12)2116--188, 2021. URL. https: //www. nature. com/ http//arrieesssseeeeeeeeeeeeeeee #41591-02211 01191-00
Yong-hun Kim, Ajdin Kobic, and Nahid Y. Vidal. Distribution of race and fitzpatrick skin types in data sets for deep learning in dermatology: A systematic review. Journal of the American Academy of Dermatology, 87(2):460 461, August 2022. doi: 10.1016/j.jaad.2000 2021.10.11. URL https:/ /ttp://///ooooo....././ 1016/ j. jaad. 2021. 10.0100 010.
Rebecca L. Colwell, Anand K. Narayan, and Andrew B. Ross. Patient race or ethnicity and the use of diagnostic imaging: systematic review. Journal of the American College of Radiology, 19 (4):521-528, April 2022. doi: ja.r.066j.acr..2222.0001
URL /ttpp:://oi. org/ 10////00.006//
jacr 2022.022200 008.
Darshali A. Vyas, Leo G. Eisenstein, and David S. Jones. Hidden plain sight - reconsderring the use of race correction in clinical algorithms. New England Journal of Medicine, 383()): 874-882, August 2020. doi: 10. 00.10056/ejjmm2204470.. URL httpp:///do http::://doi..og////// wtw./o../.......1111 1056/ nejmms2004740.
12

---

David R. Williams and Ronald Wyatt. Racial bias in health care and health. JAMA, 314(6):555, A 2015. doi: 10.100//jma..2015.9222 URL https: http:://ddiioor////// 10000/1111/11111 jaaa. 201.. 9260.
Kevin Fiscella and Mechelle R. Sanders. Racial and ethnic disparities in the qual- ity of health care. Annual Review of Public Health, 37(1):375-394, March 2016. doi: 10. 0wwp annurev- httppppbbbbaaaalt---1--------00-
URL. httpp::/bol.org/tO...
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572, 12 2014. URL
wwtpiiiv.org//bbo. wwtg/as///1///a./ 6572.
Rishi Bommasani. Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, Erik Brynjolfsson, S. Buch, D. Card, Rodrigo Castellon, Niladri S. Chatterji, Annic Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani. Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jackson K. Ryan, Christopher R' e'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258, 2021. URL https: / /crfm.mm.. a stanford edu/ wwwaass reportt. pdf.
Rishi Bommasani, Dilara Soylu, Thomas I Liao, Kathleen A Creel, and Percy Liang. Ecosystem graphs: The social footprint of foundation models. arXiv preprint arXiv:2303.15772 2023, URL https://arxiv.. org/ abs/ 2303. 15772.
Alcksander Mądry. Advances in ai: Are we ready for a tech revolution? Cybersecurity, Information Technology, and Government Innovation Subcommittee, 2023. URL https: // //wrrsigtt..../..../o www.errrtttttte.e..... house.
http:pppccateett/pplloaaa//////a o pdf.
Paul Voigt and Axel von dem Bussche. The general data protection regulation (gdpr). 2017. URL
https://link www.iinksppinggnn www.nnge....boooooo wwt.cooo//oo//ecccc 10. 10011100777----10 311--33--199-9999-1
Sarah H Cen and Manish Raghavan. The right be an exception to data-driven rule. 2023. URL
https://arxiv.ot wttp:////////b/////////o// httpg///b//222213/////3
Emad Barsoum, Cha Zhang, Cristian Canton-Ferrer, and Zhengyou Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. CoRR, abs/ fiss/160..n 00001,, 2016. URL http://arxiv: wrgg/ba////0eeee/ 01041.
Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. Emo- tional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1): 2019. doi: 10. 00.1177/1521100019922200 PMID: 31313636.
URL https: //tpi///.oioo/ org/10 10. 1177/112910061188222888
Tuan Le Mau, Katie Hoemann, Sam H. Lyons, Jennifer M. B. Fugate, Emery N. Brown, Maria Gendron, and Lisa Feldman Barrett. Professional actors demonstrate variability, not stereotypical expressions, when portraying emotional states in photographs. Nature Communications, 12 (1), August 2021. doi: 10.1038/641447-011-2233 s41467-021. 021467-021-25352-66
URL. ftttp::/ddo..org100
13

---

Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizztion exacerbates spurious correlations. Hal Daumé III and Aarti Singh, ed- itors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8346-8356. PMLR, 13-18 Jul 2020. URL https: https://prroe http://///ocoeeeese///ee/ mlr. prwpprrsssse.... v119 sagawa20a. htmml..
Shan Li,
Weihong Deng, and Junping Du. Reliable crowdsoureing and deep Docality-preservvie learning for expression recognition in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 2584-2593, IEEE Computer Society, 2017. doi: 10.1 (0.109/9RP.2201.2277 URL https: /do::///oiooo.oo org/ axgg10.1109. @VPR.2017 .777
Ali Mollahosseini, Behzad Hassani, and Mohammad H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput., 10(1):18-31, 2019, doi: 10.1100/AAFFC:220777749992 URL https://doi. org/ 1109 2017. 2740923.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial expression recognition to interpersonal relation prediction. CoRR, abs/1609 (tbb/1699.06466, 2016. URL. http:://axixii.. org/ abs/1609. 06426.
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communica- fion Association, Graz, Austria, 15-19 September 2019, pages 814-818. ISCA, 2019. doi: 10.21437/ Interspeech.2019-2222 URL https: /ttp////.ooo//// org/ 10. 214224447722 Inte/eeeeeeereeeeeeeeeeeeee 2019-2396.
Zohar Jackson, César Souza, Jason Flaks, Yuxin Pan, Hereman Nicolas, and Adhish Thite. nukoborskk/frre-spoken--- -ii-s-ooiiddid-dtasttttt v1.0.8, August 2018. URL https://dolt org/ 10/1101.111 zenodo. 1342401.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, 0ts//807.03418, 2018 URL http:////// / /taxxviivvv..///// http://twww..aaa..aa.//.../.rtet
orgooog///oo hwtgg//a////// 1807 03418.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, The 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19.24 June, 2011 1, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics, 2011. URL. https: / //aclanthology.com or../11-1//1///
14

---

Disagroement Pencentage = 0.0 (n=1810)
Disagremment Percenlage 10.0 (1=1142)
Disagroement Percentage 20.0 (n:948)
0.7
Preporton of instamees
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Observed Basclino Chserved
Desggrement Percentage 30.0 (nw672)
Disegremmet Perceniage 40.0 (n=806)
Desggremert Percenlage 50 (n=784)
0.7
Propeden Ineirnee
0.8
0.5
0.4
0.
02
0.1
0.0
Mocels Conrect
Models Correct
Models Correct
Figure 6: We stratify ecosystem-leved analysis on FER+ by instance-leved annotator disagreement which we take proxy for the ambiguity inherent the input instance and plot the difference between observed and baseline rates for each instance subset. We observe homogeneous outcomes for all subsets of data, regardless of the level of annotator disagreement on the instance.
A Data-centrie Explanations for Homogeneous Outcomes
Prior work has explored model-centric explanations for homogeneous outcomes [Ajunwa, 2019, Engler, 2021, Bommasani et al., 2021, Creel and Hellman, 2022, Fishman and Hancox-Li, 2022, Bommasani al., 2022, Wang and Russakovsky, 2023]. However, data-centric explanations comparatively less explored. We posit that properties of the underlying data could contribute to the homogeneous outcomes that observe in all of the datasets examine. Intuitively. if believe that some examples are 'hard' and others are 'easy", then we might expect to see models all fail for the 'hard' examples and all succeed for the "easy' examples.
We test three variants of this hypothesis. In $A.1, we examine how the level of annotator disagreement in the ground truth label impacts ecosystem-leved behavior. In $A.2, test how the accuracy of human dermatologists in predicting the malignancy neous outcomes. Finally, to build on these finer-grained empirical analyses, in $A.3, we introduce more express theoretical model. Under this model, parameterized by two difficulty parameters, we compute different baseline rate for ecosystem-level outcomes, showing it can better recover the observation distribution.
skin lesion image correlates with homoge-
A.1 Annotator disagreement
To study the effects of annotator disagreement, we make use of the FER+ dataset. Each instance of the FER+ dataset, facial emotion recognition dataset, contains emotion annotations from 10 human annotators; the emotion label is determined by majority vote [Barsoum et al., 2016]. Because each instance has been annotated by multiple annotators, can caleulate the annotator disagreement for each instance and use this as a proxy for the 'ambiguity' of the instance. For example, an instance where all 10 annotators agree that the label is 'sad" is less ambiguous than instance where annotators vote the label should be 'fear' and 4 vote that the label should 'surprise'.
The test set of FER+ provided in HAPI contains instances with disagreement percentages ranging from to 50%. We stratify on the disagreement percentage of the instances and compare baseline and observed ecosystem-level outcomes for each subset of instances.
15

---

mocetto comrot, Emittion Bod.
nodes coreet. Erictin Sad.
mocos conrcc, Errction luAngy
moddd coreet Emrotin is Angry
O O cornect Emodon Angy
Figure 7: Examples of homogeneous outcomes Instances that are sampled uniformly at random from "0 models correct" (top row) "3 models correct" (bottom row) in FER+ The systemic failures (top row) do not appear to be inherently harder for humans to classify; more extensive analysis appears in the supplement.
Strength Measurement (Observed Baseline) Strength Measurement (Observed Baseline)
0.05
8
Disagreement Percentage
0.0
Strength
0.00
10.0
20.0
30.0
-0.05
40.0
50.0
2
-0.10
0
Models Correct
Models Correct
Figure Interpretation of the relationship between instance-level annotator disagreement and homogeneous outcomes depends on if the strength of the effect is quantified as the ratio between observed and baseline rates (left plot) or as the difference between observed and baseline rates (right plot).
Examples. To build further intuition, we present several randomly sampled instances from the FER+ facial emotion recognition dataset in Figure 7.5 We emphasize that while systemic failures may share structure, we do not believe these instances are inherently harder than ones on which models perform well. The authors did not have difficulty labelling these examples, nor would other human labelers. We address the question of why systemic failures arise in $6.
Homogenous Outcomes manifest regardless of annotator disagreement. In Figure 6, we find that homogenous outcomes appear for all levels of annotator disagreement. While more ambiguous examples exhibit higher model error rates and systemic failure rates, the observed rate of homogeneous outcomes exceeds the baseline rate of homogeneous outcomes for all instance subsets. This suggests that instance-level ambiguity does not (fully) explain the existence of homogenous outcomes at least in FER+
The intensity of the effect varies by disagreement level, but the direction of the relationship depends on how strength is quantified. In light of the observed existence of homogeneous
*e acknowledge the task of facial emotion recognition has been the subject of extensive critique [e.g. Barrett et al., 2019, Mau et al., 2021]). We provide examples for this task due to ease of visualization, but our claims also hold for examples from the text and speech modalities that provided in the supplement.
16

---

Outcomo
200
P-1
(a) Subset of instances that both dermatologists classify correctly.
(b) Subset instances that pre- (c) Subset of instances that both
cisely one dermatologist misclas- dermatologists misclassify.
sifies.
Figure Homogeneous outcomes exists for the subset instances where both dermatologists misclassify the image but doesn't It exist for the subset where both dermatologists correctly classify the image.
outcomes across all levels of disagreement, now examine how the intensity of this effect varies with annotator disagreement. The relationship between annotator disagreement and the intensity of homogeneous outcomes depends the quantification method used to measure homogeneous outcomes intensity. When quantifying the intensity as the difference between observed and baseline rates, the effect becomes more pronounced as disagreement increases. However, when considering the intensity as the ratio between observed and baseline rates, as in the homogenization metric introduced by Bommasani et al. [2022], the effect is most pronounced at lower levels of disagreement.
A.2 Human accuracy
Each instance of the DDI dataset - a medical imaging skin lesion dataset - contains predictions from 2 models and 2 humans with the ground truth generated from external process (in this case, a biopsy of the lesion). To understand how numan-perceiin outcomes, stratify instances the dermatologist accuracy of that instance (each instance has dermatologist accuracy of 0%, 50%, or 100%) and examine homogeneous outcomes for each subset of instances.
difficulty relates to homogeneous
Model outcomes are homogenous for instances which dermatologists fail and heterogeneous for on which dermatologists succeed. In Figure 9, we find that model outcomes exhibit homoge- neous outcomes for the subset of instances that both human dermatologists fail to an even greater extent than observed at the population level. In contrast, the observed and bascline distributions match each other (meaning there no homogeneous outcomes, heterogeneity or other form of deviation between the distributions) for the subset of instances that both dermatologists get right. Note that in the instances that both dermatologists fail at, both models systemically fail more than the baseline, but both also succeed more than the baseline. While the correlation between human systemic failures and model systemic failures seems intuitive, it's less clear why models jointly succeed more than the baseline on human systemic failures.
Given these surprising and unintuitive findings, encourage future work explain what features of these instances lead models to pattern together. We speculate that these instances lack sufficient informational cues, prompting the models place excess emphasis on narrow selection of features, consistent with the literature of machine learning models being susceptible to spurious correlations [e.g. Sagawa et al., 2020]. At face value, these results would suggest that human-level difficulty can be predictive of outcome homogeneity, but we emphasize two caveats: (i) the DDI dataset's 's sample size limited (155 instances of dermatologists both failing and 356 instances of dermatologists both succeeding) and (ii) we rely on the judgments of just two human domain experts, meaning the findings may not generalize to larger annotator pools of non-experts (as are common in many NLP or computer vision datasets). In general, we caution against overgeneralizing this finding. We provide it as an initial foray into understanding example difficulty in a unique setting where we have data that supports approximation of human- perceived difficulty.
17

---

A.3 More expressive theoretical models
Finally, while we consistently find that observed ecosystem- levell behaviors yield more homogeneous outcomes than the instance-level independent model predictions would predict, we might intuit there exists instance-level structure that should be encoded into the prior. Therefore, we consider theoretical framework to encode richer priors on what we might expect of models. As a simple model, we will assume some instances are universally 'hard', meaning all models will perform worse on average across these instances, and others are conversely 'easy', meaning all models will perform better on average across these instances.
As before, we note that there is not a universal standard according to which examples can be considered 'hard.' Some examples are easy for humans but hard for models; other examples are easy for models but challenging for humans; and still other examples are challenging for some models but not others. The three hypotheses we consider in this section explore these observer-relative dimensions of hardness.
We parameters (a, A) to parameterize this model, thereby adjusting the baseline rate that calculate for ecosystem-level outcomes. a specifies the composition of 'hard' vs "easy" instances in dataset and A controls how much harder easier the hard easy examples, respectively, expected to be. Concretely, a fraction of instances are categorized as difficult and the remaining
1 a: are easy. A model's failure rate J5 over all examples scales f, examples and f, /, (1 (1-
hard (1 A)ʃf on hard
F, on easy examples.
The distribution of the baseline number of model failures E {0,............... k} follows a weighted sum of two -anoomall distributions parameterized by the scaled hard ʃ hard error rates.
and easy Sz easy model
I phard baseline (f failures) =
(a:) Poisson Poissonn-inoomiaaa Binoomial hard ..... fk hand
(3)
P baseline (t failures)
= (1 - a)
easy
f easy
[
(4)
P Abaseline (f failures) phaadd
baseline [t] + p prensy
[t]
(5)
Identifying a and A values that recover the observed ecosystem-level outcomes in HAPI
We utilize this framework to identify which (a: A) combinations generate baseline distributions that recover the observed distributions in the HAPI datasets. We perform grid search for a e [0.1, 0.5] with step size of 0.1 and A [0.2, 5) with a step size of 0.2. Note that certain (a, A) combinations can result in invalid error rates depending on the original error rates of models i.e.
when (1 (1- --) T <.
For each dataset, we identify the (a, A) combination that minimizes the LI distance (equivalently the total variation distance) between the observed and baseline distributions. In Figure 10, we visualize the observed and baseline distributions for the distance-minimizinn (a, A) pair for each dataset, and in Figure 11 we plot the L1 distance for each dataset as function of a and A. The majority of the best values are 0.2 or 0.3 while the A values can range from to 4 indicating that the f, hard be up 5x higher than fy.
can
High A and low a represents a small group of very difficult instances that all models struggle at. The combination of high A and low values performing well suggests that would need to expect that a relatively small fraction of the dataset contains instances that are significantly harder for all models to perform well at. Note that this framework assumes that all models consider the same instances to be 'hard': this agreement is a form of homogenization that could be caused about something inherent the data something about how the models are constructed.
B HAPI Experiments
In the main paper, we work extensively with the HAPI dataset of Chen et al. [2022a]. While we defer extensive details about the dataset to their work, we include additional relevant details here as well as any relevant decisions made in using the HAPI dataset.
18

---

Astio =
Apte- -03
Apti 0.3
L1 cistannce 0.0168
csAance 0203
(isoiice 094922
LI ossance 0.0436
02
Dstaur Hart
Dranee uotee
Dota 4.0
Deta 1.0
D3la-332
Eata 0.3
E4th 022
Dath-110 distance
ciatence DGG5
cisannce D696
Figure 10: Observed and baseline distributions of ecosystem-level outcomes for the (a, A) combina- tion that yields the lowest L1 distance for each dataset.
Aphhe 0.1
02
Apha-
0.7
044
0/2
svebl
divo
0.7
0.5
D-90000
00
Figure 11: L1 distance between observed and baseline distributions for all datasets as a function of and A.
19

---

Effect conditooing foiling distribtion
hy (most acourate)
Rg
Fty (lessl ascourale)
1225
wwwkown........
00
4.245
-0.37
-04
-06
Otnen-Moool Ounzori Profku
Obei-Mide Outoome Prolii
Cttor ModulMadil Outoons Phofii
Figure 12: Change in the probability of observing each outcome profile for the other models upon observing h; fail.
B.1 Data
We work with subset of HAPI, a dataset introduced by Chen et al. [2022a] which contains predictions from commercial MI. APIs on variety of standard benchmark datasets from 2020-2022. HAPI contains benchmark datasets for three classification tasks and three structured prediction tasks. We only work with the classification tasks spoken command recognition (SCR), sentiment analysis (SA), and facial emotion recognition (FER) because they contain a single ground truth label where the notion 'failure' clear (i.e. a misclassification).
The three classifiation tasks span 3 modalities (text, images, speech), and each task is associated with 4 datasets. However, we exclude one of these datasets: the COMMAND dataset has duplicate example IDs with differing predictions from the same ML API provider in the same time period. This makes calculating ecosystem-level outcomes impossible. After excluding the COMMAND dataset, there are I datasets that we conduct ecosystem-level analysis on: RAFDB [Li et al., 2017], AFNET [Mollahosseini al., 2019], EXPW [Zhang et al., 2016), FER+ [Barsoum et al., 2016], FLUENT [Lugosch al., 2019], DIGIT [Jackson et al., 2018), AMNIST [Becker al., 2018), SIOOP, YELP," 7, IMDB [Maas et al., 2011] ] and WAIMAL.
Each dataset contains predictions from 3 commercial ML APIs in 2020, 2021, and 2022; however the HAPI API did not return predictions from the Face++ model AFNET in 2022, so we use 2021 predictions for AFNET when conducting experiments that only use predictions from single year.
The HAPI dataset is distributed at htttps::/gttttbb.cc////eee/// httpgg/tta..../m///ooooo1111111e under Apache License 2.0.
B.2 Leader Following Effects in Systemic Failure
One consequence of homogeneous outcomes is that concentrates failures on the same users, so a user who interacts with a model and experiences a failure from that model is now more likely to experience failure from every other model in the ecosystem. To quantify the strength of this effect, we examine how the probability of a user experiencing each outcome profile changes after that user experiences one failure from a model.
In Figure 12, we find that, consistent with the observed homogeneous outcomes in HAPI, observing single model failure significantly increases the probability that the user will now experience failures from all other models in the ecosystem. However, we also find that the strength of this effect is strongly graded by the accuracy of the model for which initially observe failure. Upon observing the most accurate model in the ecosystem fail for a user, the probability of that user experiencing systemic failure increases by 37% whereas it only increases by around 16% when we observe the least accurate model fail.
/ttp Shopping_10__catt
hhttpa://wwt htgggh....oo/aaaaaaaaaaaaaaaaaar
hhttps://gite tht.b//.ttttttt//./eee/ httpa///ooootooeeee/e/////////ee
20

---

Improved:
Threshold:
Improved
0.8
impovements
0.8
0.4
0.2
Ourcome Profie
[X
2.00%, Improved:
3.00%, Improved:
Thresho 00%, Improved
[r X]
[< v
0.8
0.8
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
1.00
0.00
0.25
0.50
0.75
1.00
potential improvements
potential improvements
Potential Improvements
Figure 13: We replicate the graph from Figure 3 but with various thresholds for how much model must improve for us to include it in the analysis. The patterns discussed $4 are consistent across choice of threshold.
This result suggests that instances that the most accurate model fails likely be failed by the less accurate models as well. This 'leader following' phenomenon has implications for users in model ecosystem: users failed by the most accurate model likely have few options for alternative models that could work for them.
B.3 Model improvement analysis insensitive to threshold
In $4, we study how the improvement of a single model, in the sense that it becomes more impact, manifests at the ecosystem-level. To define improvement, we set the (slightly arbitrary) threshold that the model's accuracy improve by 0.5%. which we found to be large enough to be material to model's performance while small enough capture most model improvements. In Figure 13, we plot the outcome profile distribution in the observed improvements set against the distribution over the potential improvements (as in Figure 3b) for 6 different thresholds of change. Across all thresholds, the patterns we discuss in $4 hold: namely, models consistently under-impree on systemic failures. This confirms that the findings and qualitative understanding we present not particularly sensitive to the exact value of this threshold.
B.4 Net Improvements
In $4, we define improvements as the specific instances that himp misclassifies in the first year and correctly classifies in the second year. However, the size this set set- hereafter, gross improvements is always larger than the number of net improvements the model makes because updates to the model tend to improve on some instances and regress on others. That is, there are two competing forces when models change: the instances the model flips from incorrect to correct, but also the instances the models from correct to incorrect. The difference of (i) and (ii) is the number of net improvements.
In Figure 14, we use net improvements instead gross improvements, replicating the plot from in Figure 3. To calculate the outcome profile distribution over net improvements, we subtract the number of gross declines from the number of gross improvements each outcome profile: the denominator the number of gross improvements minus the number of gross declines across all outcome profiles.
21

---

0.848
1.2
improvements
0.8
potential improvements
Observed Net Improvements
1.0
Proportion Instances
0.7
0.8
0.5
0.6
0.4
0.367
0.341
0.4
Outcome Profile
0.3
0.219
0.2
[x
0.2
0.135
0.0
[x v
[V X]
0.1
0.074
0.000
0.017
[ v
[X X]
IX v
[< X]
[ M
0.0
0.2
0.4
0.6
0.8
1.0
Outcome Profile
Potential Improvements
(a) Outcome profile distribution using net improvements instead of gross improvements for (Baidu, Google) when
(b) The distribution of outcome profiles net improve- ments instead of gross improvements for all year-over- year model improvements across all datasets.
Amazon improves on WAIMAT from 2020 to 2021. Figure 14: We replicate the graphs in Figure but using 'net improvements' instead of "gross improvements'. The overall trends consistent between the two experiments: namely, hmp under-improves on systemic failures.
The trends are generally consistent with what we observed when using gross improvements: himp tends make little progress on systemic failures. In fact, when considering net improvements, models make even less improvement on systemic failures than before. We highlight WAIMAI as striking case study: there no improvements on systemic failures, despite 2.5% decrease in model error at the population level.
Overall, emphasize that recommend future research conducts analyses with both notions of improvements. While we expect in many cases, as we have seen here, that the qualitative trends will be similar, the interpretations may differ. For example, gross improvements more directly attend to the concern that there are some individuals who, year-over-year, continue to be failed by some or all models in the system. contrast, net improvements more directly matches the sense in which models are improving.
B.5
When Models Get Worse
As related question to what we examine in Figure 3b, we examine what happens to ecosystem-level outcomes when model gets worse. We find that, when models get worse, they disproportionately introduce new systemic failures into the system by 'over-declining on instances that other models were already failing for. This further highlights how single-model measurements often fails to align with ecosystem-lese outcomes.
Dermatology Experiments
C.1
Data
work with DDI (Diverse Dermatology Images), a dataset introduced by Daneshjou et al. [2022], which contains predictions from 3 models and board-certified dermatologists on 656 skin lesion images; the task is to predict whether a lesion is malignant or benign. The ground truth label comes from an external-source: in this case, biopsy of the lesion, which is considered the gold-standard labeling procedure this domain.
The 3 evaluated models include ModelDerm [Han et al., 2020], publicly available ML API, and two models from the academic literature DeepDerm [Esteva et al., 2017] and HAM10K [Tschandl et al., 2018] that were chosen by Daneshjou et al. [2022] on the basis of their "popularity, availability, and previous demonstrations of state-of-the-arr performance." " Note that, this case, none of the models have been trained any portion of the DDI dataset; the entire dataset serves test set.
22

---

1.0
0.8
Observed Declines
0.6
0.4
Outcome Profile
[X X]]
[X
0.2
[V X]
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Potential Declines
Figure 15: The distribution of outcome profiles for all year-over- year model declines across all datasets
In addition, each image is annotated with skintone metadata using the Fitzpatrick scale according of three categories: Fitzpatrick & (light skin), Fitzpatrick III & IV (medium skin), and Fitzpatrick V & VI (dark skin). For all instances, the Fitzpatrick classification was determined using consensus review of two board-certified dermatologists. Additionally, separate group of dermatologists rated the image quality of each image and discarded any low quality images; there was significant difference image quality ratings between images of different FST classifications.
Data on model and dermatologist predictions was graciously provided by Daneshjou et al. [2022], subject the terms of their standard research use agreement described in https: //ddiddaddddaee...../. dataset. gi thub. i index. htmc#access.
C.2 Analyses are insensitive to including/exclludng HAM10K
In $5, we don't include predictions from Ham10K because the model predicts negative almost all instances: it has precision of 0.99 but a recall of 0.06.
We decided to remove HAM10K because the pattern of near-universal negative predictions does not reflect model behavior we would expect of models deployed clinical settings. Namely, even deployed, the structure of the model eтrors are not particularly interesting and are largely predictable (in direction). Beyond these fundamental reasons for excluding the model, we also removed the model for reasons unique to our analysis. Including the model would have introduced an explicit class correlation systemic failures (i.e. almost all systemic failures would malignant instances and none would be benign instances) and would have complicated the comparison with humans since there would be three models but only two humans.
To confirm that our findings hold independent of this choice, in Figure 16, we replicate Figure 4 and Figure but with outcomes from HAMI0K included. We find that inclusion of HAM10K exacerbates the homogeneous outcomes of model outcomes and the racial disparities in model outcomes.
23

---

CEonvic (1966)
Expanaalo -iMe
Proporton of Instances
104
Osconed Miue
0.06
008
Flapstick
Flzpotrckk (N-241)
0.00
Fizzpotrok & (N--207) (Nmn000)
0.102
0.11
135
0007
0350
WXX
XXX
2 41
[eX+]
Outcome Profii
Outcome Proto
(a) Model outcomes when including HAMIOK yield even more homogenous outcomes on DDI than in Fig- ure
(b) The inclusion HAM10K yields more pronounced racial disparities than in Figure 5.
Figure 16: We replicate the graphs in Figure and Figure but with the inclusion HAM10K. Homogeneous outcomes and racial disparities in models more pronounced when including HAM10k.
24


Reviews:
[{"review_id": "ztnVLRsMr2", "reviewer": "Reviewer_kuru", "summary": "This paper introduces the concept of \"ecosystem-level analysis,\" which evaluates the performance and impact of a collection of machine learning models in a given context rather than analyzing individual models in isolation. This approach could significantly contribute to understanding the broader impacts of deployed machine learning models on society. The authors identify a new phenomenon called \"systemic failure\" in machine learning models. This happens when a user is consistently misclassified by all available models in the ecosystem, pointing towards potential systemic biases in machine learning deployments.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides empirical evidence of these systemic failures across three different modalities (text, images, speech) and 11 datasets, establishing a broader trend that transcends specific applications or domains.", "weaknesses": "The study under review makes an ambitious attempt to analyze the societal impact of machine learning (ML) from an ecosystem-level perspective. Although the authors should be commended for their unique approach, the paper has notable gaps that prevent its acceptance, particularly for a high-standard conference such as NeurIPS. The paper's theoretical framework lacks rigor and the theoretical grounding is insufficient, making it more suitable for a venue like FAccT, which is more focused on ethical and societal aspects of ML.\n\nThe authors propose the idea of systemic failure in deployed machine learning, which is indeed intriguing. However, the theory behind this proposition appears to be weak and not well-articulated. The paper lacks robust theoretical to support this claim, as the analysis seems to be limited in terms of the ML models considered. For example, what are the factors that could contribute to system failure? Does these factors differ in terms of features distribution and machine learning algorithm used? Furthermore, the implications of such systemic failures on society, an aspect of paramount importance, is not adequately addressed, making it difficult to comprehend the real-world significance of these findings.\n\nThe authors conducted a large-scale audit covering three modalities and 11 datasets. While this might sound impressive initially, the appropriateness of the methodology is questionable. There is a lack of clarity about the selection criteria for these modalities and datasets, and their representativeness of the ML ecosystem as a whole. This omission undermines the credibility of the results. This is again because of the lack of theoretical analysis, that may make the result questionable -- does the result only hold on the datasets at hand? Had another ML method, data distribution changed, would there still be system failure? \n\nThe claim that improvements in individual models do not significantly reduce systemic failures is an interesting observation. However, the paper fails to provide a detailed theoretical examination and mechanism analysis of this phenomenon. A stronger theoretical grounding for this observation would help the NeurIPS community to better appreciate and understand this claim.\n\nThe study on medical imaging, especially in dermatology, opens up a promising area of research. However, the introduction of a new form of racial disparity unseen in human predictions is a serious claim that requires more in-depth explanation and rigorous validation across a broader range of ML methods. Without comprehensive exploration, the validity of this claim remains dubious. I actually think that the paper can focus on this unique dataset, provide more mechanism analysis using one single dataset. And then validate the conjecture on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to reviewer kuru07 for their helpful review. Overall, we firmly disagree with many of the vague claims made in the review, and ask the reviewer to provide specific evidence for us to be able to better engage with their claims. For example, at many points there are claims of lacking empirical rigor that are unsubstantiated: the work studies 3-4 datasets, 11 ML systems, and 12 datasets in total, so we are very surprised to hear concerns about empirical rigor. \n\nBelow, we provide a more specific response to the weaknesses provided.\n\n> The paper's theoretical framework lacks rigor and the empirical evidence is insufficient, making it more suitable for a venue like FAccT, which is more focused on ethical and societal aspects of ML.\n\nWork on the ethical and societal aspects of ML is directly within scope for NeurIPS 2023, named explicitly in the call for papers as “Social and economic aspects of machine learning (e.g., fairness, …)”. In fact, the award-winning paper of Birhane et al. has provided a direct critique of NeurIPS having historically undervalued the societal dimensions of machine learning. As machine learning is deployed to millions of people, including through the very deployed systems we have studied, it is absolutely essential that work on its impact is presented and discussed at top ML venues like NeurIPS.  \n\n> The authors propose the idea of systemic failure in deployed machine learning, which is indeed intriguing. However, the theory behind this proposition appears to be weak and not well-articulated. The paper lacks robust empirical evidence to support this claim, as the analysis seems to be limited in terms of the ML models considered. \n\nThis paper contributes an extensive empirical analysis of deployed machine learning systems provided by major vendors (e.g. Amazon, Baidu, Google, IBM) across a range of modalities spanning 10+ datasets. We are unsure of what empirical evidence is lacking (our findings of polarized profiles, for example, hold consistently in all contexts) or in what sense the models are limited. This paper does not claim to make a theoretical contribution, instead building on the theoretical framing presented in [Bommasani et. al. 2022](https://arxiv.org/abs/2211.13972), which formalizes outcome homogenization in terms of systemic failures experienced by users interacting with multiple machine learning systems.\n\n> Furthermore, the implications of such systemic failures on society, an aspect of paramount importance, is not adequately addressed, making it difficult to comprehend the real-world significance of these findings.\n\nWe strongly disagree: we do explicitly discuss the importance of these failures in several places, and point more extensively to literature that has already presented these arguments such as section IV.C.4 of Ajunwa 2021, section 6.1 of Bommasani et al. 2022, and Creel & Hellman 2022. As stated, our objective in this work is not to further argue this point, but to provide robust empirical evidence of the harms that these prior works have conjectured and conceptualized.\n\n>There is a lack of clarity about the selection criteria for these modalities and datasets, and their representativeness of the ML ecosystem as a whole. This omission undermines the credibility of the results.\n\nWe strongly disagree, having provided explicit justification for both the HAPI dataset in lines 108-117 and the DDI dataset in lines 189-201. \n\nNamely, the HAPI dataset is the only dataset we know of that includes predictions from multiple deployed commercial models from the largest ML as a service (MLaaS) providers (Google, Amazon, Microsoft, IBM) on the same set of instances. According to one market research report, the ML as a service market was reported to be 21.55 billion dollars in 2022, with a compound annual growth rate of over 30 percent. The major cloud players (Google, Microsoft, and Amazon) are expected to be major players in the ML as a service market, and are all present in our dataset. Beyond this, the dataset ensures we cover 3 modalities (text, image, speech) with 10+ datasets, which amounts to our findings having substantial generalizability. Further, the dataset is longitudinal, enabling out analysis of systemic failures over time. \n\nSimilarly, the DDI dataset is the only dataset we know with instance-level predictions for multiple machine learning models in dermatology (namely some of the most high-profile models in the domain) and multiple board-certified dermatologists. Additionally, as described by Daneshjou et al, the dataset is one of the most diverse in the entire dermatology area, enabling our study of race-based effects. Finally, the dataset has ground-truth labels determined on the basis of a highly reliable medical procedure rather than human-based judgment, which proves to be essential for meaningfully compared model-based and human-based outcomes.\n\nGiven the unique properties of this dataset, its inclusion of the major players in the market, and the importance of the ML services provided by these providers, we believe that findings of systemic exclusion are important on their own. "}, {"review_id": "YL5hMZYaQP", "reviewer": "Reviewer_N5w8", "summary": "The paper introduces ecosystem-level analysis where instead of a single model, a collection of models will be evaluated together and their common failures/successes will be identified. Across three modalities, the paper shows a pattern of polarization where an instance is either more likely to be misclassified by all the models or correctly classified by all of them. Similar patterns are observed in medical imaging. The paper suggests potential mechanisms responsible for polarization and concludes with a discussion of implications for researchers and practitioners. ", "questions": "No specific questions. Feel free to address the weaknesses I mentioned if you think they are not the case.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The framework appears promising to me because it centers people in the analysis and highlights how repeated failures of machine learning models for an individual can result in harm.\n- It provides concrete findings for previously mostly conceptualized homogenous outcomes.\n- It discovers a new form of racial parity that couldn't be identified without ecosystem analysis and is not present in human judgments.", "weaknesses": "1. It could be more interesting if systematic failures were studied by linking modalities rather than each modality in isolation. But I understand the available dataset does not support this.\n2. As discussed in Sec 6.1, the baseline of independent models is very unrealistic, for example, difficulty may confound the outcomes. In the current baseline, the majority vote of different models significantly improves their accuracy which is not at all expected.\n3. Sometimes it's not clear what level of polarization is significant. For example, Fig 5a looks as no polarization to me. \n4. I was expecting more models to make a conclusion about polarization. In many places, only two models are used to draw conclusions.\n5. Since the number of models is always small, I have no good sense of how the effect scales with more models.\n6. To address the last two points, authors may want to increase the number of models by evaluating some models on the same datasets themselves. This will be also helpful in experimenting some conjectures about the polarization mechanisms.\n7. The new form of disparity (Finding 4) is interesting, but the framing seems to obscure the fact that humans are still more polarized on all skin tones. I'd recommend emphasizing this.\n8. As a general comment, I'd like to see an expanded discussion on the causes of polarization and its implications.\n\n\nTypos: line 19: remove do, table 1: $h^1 \\rightarrow h_1$ ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer N5w8’s thorough review, highlighting the core framework and associated findings as key strengths. The questions they pose are definitely of interest but, overall,we unfortunately cannot answer several of them due to the lack of available data. Nonetheless, we provide more specific answers when possible. \n\n> It could be more interesting if systematic failures were studied by linking modalities rather than each modality in isolation. But I understand the available dataset does not support this.\n\nWe agree that cross-modal relationships would be interesting, but the HAPI dataset of Chen et al. does not track the same individuals across modalities and we are unaware of any such linked dataset auditing commercially deployed models currently exists.\n\n> As discussed in Sec 6.1, the baseline of independent models is very unrealistic, for example, difficulty may confound the outcomes.\n\nWe believe there is a misunderstanding in regards to the independent baseline; we would be happy to reword the paper to make this clearer. Namely, we neither claim nor intend for the independent baseline to be seen as a realistic prior for the amount of correlation we will see in practice. Instead, we provide it as an example to contextualize and contrast the observed behavior against. In particular, it accounts for the underlying error rates of the different models.\n\nOverall, we find this baseline to be a simple and natural starting point for building intuition on a new phenomenon (i.e. the failure matrix and associated ecosystem-level outcomes). As we develop in 6.1 and the associated appendix A, we can consider many more sophisticated models of model correlation: for example, the alpha-delta model we introduce in Appendix A (and describe in section 6.1 lines 255-264) generalizes independence as a simple two-parameter model that incorporates a notion of “difficulty” as the reviewer suggests. \n\nWe wonder if the issue rests on the use of the term “expected”, which we use to contrast with “observed” (as is conventional in several fields); we are very open to suggestions on how to better frame this baseline rate -- including a change in the name. Rather than calling this rate “expected” we could call it “baseline” or “normalized.” \n\n> In the current baseline, the majority vote of different models significantly improves their accuracy which is not at all expected.\n\nCould the reviewer clarify what they mean by “majority vote of the different models” so we can address this concern?\n\n> I was expecting more models to make a conclusion about polarization. In many places, only two models are used to draw conclusions. \n\nTo clarify, for almost all results (i.e. all HAPI results), they are based on 3 commercial models, not two. Overall, these are the 3 most widely used APIs for their respective tasks/modalities as described in the HAPI paper of Chen et al. We do agree that in the future, more controlled analyses that understand how the phenomena scales in the number of models would be interesting. However, for our focus on deployed ML systems, we believe the setting is very realistic where a few major ML-as-a-service (MLaaS) providers dominate each of the respective markets. \n\nAs a potential point of clarification, we note that the analyses for model improvements depict two models because the third model (i.e. the one that improved) is being excluded to understand how its improvements relates to the remaining two models.\n\n> Since the number of models is always small, I have no good sense of how the effect scales with more models. To address the last two points, authors may want to increase the number of models by evaluating some models on the same datasets themselves. This will be also helpful in experimenting some conjectures about the polarization mechanisms.\n\nAs we note above, we agree that understanding the scaling behavior is valuable. However, our focus is on the realities of deployed ML systems in these domains: it is a market reality that in many of these domains, only a few MLaaS providers dominate the respective markets (as was also described by Chen et al. in the HAPI paper). The models we study are created by the largest ML providers and, while precise market understanding remains opaque, the evidence we have found shows that these deployed models have major impact: for example, in 2019, Microsoft -- whose API systems are included in HAPI -- [claimed](https://azure.microsoft.com/en-us/blog/companies-of-all-sizes-tackle-real-business-problems-with-azure-ai/) that its ML APIs processed 5 billion ‘cognitive search transactions’ (their term for an API call) each month. While we could have trained our own models and applied them to these datasets, we believe that the societal impact of these deployed models and the interest in understanding their risks from policymakers and the public warranted our initial focus on deployed models.\n\nWe should also keep in mind that the relevance of scaling behavior will depend on the ecosystem being evaluated; in the context of job hiring, for example, scaling behavior could be very relevant since job seekers tend to apply to many jobs, while in automated speech recognition systems it might be less relevant since users are likely to only interact with systems from a limited number of companies.\n\n > framing seems to obscure the fact that humans are still more polarized …  I'd like to see an expanded discussion on the causes of polarization and its implications. … typos\n\nGiven the page constraint, we were forced to include expanded discussion on both of these topics in the appendix - see lines 511-532, 593 - 608. We will uplift these discussions, directly addressing these concerns, when afforded an additional page upon acceptance."}, {"review_id": "Zwoj4nlKyo", "reviewer": "Reviewer_HXE5", "summary": "The paper empirically investigates ecosystems consisting of many machine learning models. Rather than analyze each model in isolation, the paper instead focuses on the extent to which all models in an ecosystem incorrectly classify an individual (the systemic failure rate), building upon recent work on *algorithmic monoculture* (Kleinberg and Raghavan, 2021) and *outcome homogenization* (Bommasani et al., 2022). \n\nThe paper draws upon a large-scale audit (HAPI, Chen et al. 2022) that consists of 3 commercial systems per modality and 3 different modalities. They show that model improvements do not significantly reduce systemic failures: in particular, model improvements often occur on instances that are already classified by the other 2 systems. The paper then investigates two ML models (ModelDerm and DeepDerm) and two dermatologists on the dermatology dataset (DDI, Daneshjou et al., 2022). They show that the humans are more polarized (i.e. humans all agree or humans all disagree) than models. However, models (unlike humans) are more polarized for darkest skin tones than for lighter skin tones. \n\nThe paper concludes by proposing several data-centric and model-centric explanations for the observed trends.  \n", "questions": "Questions: \n- How do the systemic failure rates compare for darker skin tones and lighter skin tones for the models? What about for the humans? How much larger is the gap between systemic failures that the gap between failure rates for these two groups on an individual model/human?\n- Please comment on which of the factors in Section 6 are likely to most responsible for systemic failures in different contexts. What factors of a given ecosystem might influence the level of systemic failure observed? \n\n", "limitations": "The authors adequately addressed the limitations of their work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper provides an interesting empirical investigation into how interactions between models within an ecosystem can affect outcomes for individuals. The results are a compelling extension of previous work on *algorithmic monoculture* (Kleinberg and Raghavan, 2021) and *outcome homogenization* (Bommasani et al., 2022). \n\nSystemic failures is a natural metric to study, and the empirical analysis of systemic failures in model ecosystems is rich and fairly thorough. Altogether, the analysis of systemic failures on the HAPI dataset and DDI datasets provide a fairly convincing argument that a model ecosystem can exhibit different behavior a single model in isolation. \n\nThe paper is well-written and the key ideas are clearly presented. ", "weaknesses": "There seems to be a gap between the results in Finding 4 and the implications for fairness. While Figure 6 focuses on polarization, it seems that the metric of interest for fairness should be systemic failures. This raises the question of how the systematic failure rates compare for darker skin tones and lighter skin tones. See question below.\n\nThe commentary section (Section 6) is somewhat high-level and does not provide significant insight into which factors may be driving systemic failures in a given ecosystem. As a result, it is not entirely clear what factors drive polarization and whether one should expect to see polarization in different contexts. Understanding what types of ecosystems are more susceptible to systemic failures seems like an important question for policymakers and researchers. See question below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer HXE5 for a thoughtful review and set of questions. At the heart of the matter, we see two core concerns raised in the review:\n1. **Finding 4.** We agree that systemic failures are definitely a phenomenon of direct interest, but even beyond systemic failures, the racial disparities in polarization pose substantive concerns. We provide a longer-form discussion below that we had to abridge in the paper due to space, and would be keen to include this discussion with the additional page granted upon acceptance. \n2. **Discussion in Section 6.** This is an oversight on our part: every paragraph in Section 6 corresponds to a detailed analysis presented in the Appendix. We encourage the reader to read Appendix A, as we believe it has all of the desired rigor in these analyses, and have replicated some of the points below. \nOverall, we hope our responses are clear and convincing answers to the questions posed.\n\n> There seems to be a gap between the results in Finding 4 and the implications for fairness. While Figure 6 focuses on polarization, it seems that the metric of interest for fairness should be systemic failures. This raises the question of how the systematic failure rates compare for darker skin tones and lighter skin tones. See question below.\n\nSystemic failure rate is certainly of direct interest to fairness: we do not intend for this point to be undermined by our discussion of finding 4. However, we believe there are many important contexts where it is not enough. Namely, we believe the racial disparities in polarization warrant interest, even if they do manifest in racial disparities in the systemic failure rate.\n\nWe focus on *polarization* instead of the systemic failure rate because the systemic failure rate is partially driven by the error rate of the models being considered. If all of the models under consideration have high error rates, systemic failures would occur just because of the collisions of those errors. We agree that outcomes are ethically important, no matter how they are produced. However, we hope to distinguish systemic failure driven by very high error rates from systemic failure driven by similarity between models.  The first phenomenon is consequential but has been extensively studied in literature on fairness metrics; the second phenomenon is part of the contribution of this paper. \n\n> How do the systemic failure rates compare for darker skin tones and lighter skin tones for the models? What about for the humans? How much larger is the gap between systemic failures that the gap between failure rates for these two groups on an individual model/human?\n\nThank you for the opportunity to elaborate on this point. In the DDI data, the observed systemic failure rate for models is .045 higher for dark skin tones than light skin tones. However, this difference in even more dramatic when considering that the expected number of systemic failures for dark skin is actually *lower* than the expected number of systemic failures for light skin -- a result of the models having lower error rates for dark skin than light skin.  Considering the systemic failure rate without additionally considering the error rates would obscure this disparity. \n\nWe note that the original study used ROC-AUC with the models’ outputted probabilities to measure disparities across race groups and found that the models’ ROC-AUC was higher for light skin tones than dark skin tones.  By contrast, we are considering classification error and, due to the class imbalance in this dataset, we observe a different pattern.\n\nIn humans, the systemic failure rate is actually 0.023 lower for dark skin than light skin; however, the expected failure rate is also 0.024 lower -- nearly equivalent -- due to the lower error rate for dark skin than light skin. \n\nAt a high level, we believe that polarization is an additional dimension of fairness that stakeholders should consider. In this one setting, we’ve seen that three different metrics paint very different pictures: using ROC-AUC suggests that both humans and models perform *worse* for dark skin than light skin; using classification error suggests that both humans and models perform *better* for dark skin than light skin; and using polarization suggests that models perform *worse* for dark skin than light skin while humans don’t exhibit any racial disparity. Stakeholders already consider various forms of fairness when evaluating decision makers -- automated or not -- and we hope that profile polarization can add one additional dimension to existing measures of fairness.\n\n> Please comment on which of the factors in Section 6 are likely to [be] most responsible for systemic failures in different contexts. What factors of a given ecosystem might influence the level of systemic failure observed?\nAppendix A presents three initial explorations of different data-centric explanations for systemic failures. These explanations complement the several prior works that are cited, which pose model-centric explanations (e.g. the sharing of model components in [Bommasani et al. 2022](https://arxiv.org/pdf/2211.13972.pdf)). Unfortunately, due to the absolute opacity of these deployed ML APIs, we have no real means for testing model-centric explanations at the moment.\nMore deeply, for data-centric explanations, we explore how annotator disagreement in the ground truth label impacts homogenous outcomes, how human error correlates with outcome profiles in models, and how more expressive models that account for different levels of difficulty in input instances can better recover the observed distribution of ecosystem-level outcomes. \n\nIn practice, determining what is “most responsible” will be challenging as with any multi-faceted causal analysis: we suspect that both data-centric explanation and model-centric explanations play a role in homogenous outcomes, and we encourage future work to clarify these relationships."}, {"review_id": "k2Rgp6zsPS", "reviewer": "Reviewer_KctL", "summary": "This paper studies how particular examples in different benchmark datasets perform across multiple models using the HAPI and DDI datasets. It finds first that many examples are “systemic failures,” meaning that they are misclassified by all of the commercial models evaluated. Then, it also finds that model improvements over time do not tend to improve the rate of systemic failures. Even though some examples are no longer failing on all models, other examples can become systemic failure and there is overall nearly no net change. Finally, in the DDI dataset, the authors find differing rates of error by skin tone in human annotations, but show that the model does not have the same disparity. Overall, the paper is one of the first to study performance from this cross-model, cross-task “ecosystem-level” view. \n", "questions": "When the paper refers to “negative outcomes”, is it referring to a misclassification or, more generally, a harmful event to a person (like being rejected for a job)? In the introduction, it wasn’t clear to me and it might be worth explicitly defining what it is for the purposes of this paper. \n\nIn section 4, could you also include information about successes that turn into failures? For example, you observe that examples that are improved on are usually ones that were already passing for another model. Is there a similar effect that successes that become failures are also examples that were already failing for another model? \n\nIn lines 242 to 243, there is a statement that systemic failure rate is monotonically correlated with annotator disagreement. I think this is a significant point towards potentially explaining the nature of systemic failures. Would it be possible to include a figure or more details on this result?\n", "limitations": "The authors do a great job of acknowledging their limitations in section 6. My only comment is that some of that material might work better at the ends of the specific sections they address. There were a couple of cases where I found myself with questions at the end of a results section that ended up getting answered in section 6. \n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper’s primary strengths are its originality, quality, and clarity. First, this paper is an original addition to the emerging homogeneous outcome literature. To my knowledge, there has yet to be a study that understands whether individual model improvements actually improve outcomes for people for whom other models have failed. The finding here that they do not is a valuable contribution to the field and important supporting evidence for the dangers of algorithmic monoculture. Second, the authors do an excellent job of clearly defining the technical limitations and strengths of their work, in particular in section 6. I found no major technical errors, and the authors make great use of the two datasets. For example, this seems like exactly the kind of study that HAPI was collected for. Finally, the paper is exceptionally clearly written.\n", "weaknesses": "This paper’s primary weakness, in my view, is specifically the model put forth in section 3.2 to compute the expected rates of failure. Though the authors do a good job in section 6 of addressing some of these concerns, I am not sure that assuming independence of the ML models is the correct probability model for this case. Because each model is tested on the same dataset, their failures by definition cannot be independent of one another. The authors’ test in section 6 with a subsample of harder examples and subsample of easier examples does offer an alternative failure model, but I’m not sure that fully captures the correlation of having the same example passed through each of the models. At the same time, I don’t necessarily feel that an “expected rate” of failure is really essential for the overall message of this paper. One option would be to remove section 3 and the corresponding results (or at least the part about expected failure rates). Another would be to just replace the model in section 3 with the slightly more detailed model of section 6, since that is at least capturing some of the correlation. I still feel that some more treatment of correlation of model performance across individual instances would be necessary though. I also recognize that this correlation is really what the paper is trying to measure empirically, which is again why I feel that a computation of expected failures is unnecessary, and most models of that expected rate are inherently unrealistic. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer KctL for a very thorough and insightful review. First, we will address the issue of the expected failure rates and discuss what we hope this rate captures and what it does not; then we answer the specific questions raised by the reviewer.\n\n> Though the authors do a good job in section 6 of addressing some of these concerns, I am not sure that assuming independence of the ML models is the correct probability model for this case\n\nTo clarify, the purpose of these ‘expected failure rates’ is not to make an accurate ex-ante prediction of what the distribution of ecosystem-level outcomes will look like. Instead, we posit the baseline of independent models as a natural way to account for the error rates of the models in each ecosystem. Because the rate of systemic failure is correlated to the error rates of the model in an ecosystem (i.e. less accurate models will fail more often and thereby systemic failures will occur more often), it can be difficult to compare the rate of systemic failure across ecosystems with varying model accuracies. Additionally, it can be difficult to get a sense of what a ‘large’ rate of systemic failure is since this will depend on the error rates of the model in the ecosystem. The independence assumption is a helpful baseline because it makes no assumptions about the correlation between models in the ecosystem (which, in a sense, is what we hope to measure in our analysis), but enables us to adjust for the model error rates in the system. This is exactly what is proposed by [Bommasani et al. 2022](https://arxiv.org/pdf/2211.13972.pdf) in their formalization of outcome homogenization. We are very open to suggestions on how to better frame this baseline rate -- including a change in the name. Rather than calling this rate “expected” we could call it “baseline” or “normalized.” \n\n> When the paper refers to “negative outcomes”, is it referring to a misclassification or, more generally, a harmful event to a person (like being rejected for a job)? In the introduction, it wasn’t clear to me and it might be worth explicitly defining what it is for the purposes of this paper.\n\nThis is a very good question. We intentionally use the broad term of ‘negative outcomes’ because the definition of a failure can vary by ecosystem. In our setting, we are studying classification ML models, so we concretely define the negative outcome in our ecosystem as a misclassification. However, as the reviewer points out, the term is flexible to the ecosystem; in the job hiring setting, we could define a ‘negative outcome’ as a job rejection (especially because there may be \"ground truth\" of whether an individual should or should not be hired); in a LM setting, we might define it as a model hallucination or misinformation produced. Once we are allowed to edit the paper, we can clarify this point in the introduction.\n\n> In section 4, could you also include information about successes that turn into failures? For example, you observe that examples that are improved on are usually ones that were already passing for another model. Is there a similar effect that successes that become failures are also examples that were already failing for another model?\n\nAnother very good question; we ran this experiment, but we had to cut it from the paper due to page constraints. The figure can be viewed at the anonymous link: https://imgur.com/a/qsE3hbH. \nOur analysis confirms the reviewer’s hypothesis: when models get worse, they tend to get worse at examples that other models were already failing at. At https://imgur.com/a/qsE3hbH, we’ve attached the same graph as figure 4b, expect it displays “Observed Declines” on the y-axis and “Potential Declines” on the x-axis (in place of “Observed Improvements” and “Potential Improvements”, respectively); it does so only for periods where there is a net decline in the accuracy of a model in the ecosystem. This graph suggests that models tend to disproportionately decline on instances that other models in the ecosystem were already failing at. \n\n>In lines 242 to 243, there is a statement that systemic failure rate is monotonically correlated with annotator disagreement. I think this is a significant point towards potentially explaining the nature of systemic failures. Would it be possible to include a figure or more details on this result?\n\nFor a further discussion of this result and figures on this result, please refer to Appendix section A.1. It is true that annotator disagreement is monotonically correlated with the systemic failure rate, but annotator disagreement is also monotonically correlated with the error rate of the models. Regardless of the level of annotator disagreement, the observed systemic failure rate still exceeds the independence baseline systemic failure rate – suggesting that model failures still pattern together. However, when only considering the ‘topline’ systemic failure rate and not the context of the error rates of the models, it is true that annotator disagreement can strongly predict the degree of systemic failure observed in the ecosystem."}]

Author rebuttal:

Meta-review: The paper empirically investigates the systemic failure of multiple models for certain segments. This is an important phenomenon that is studied well in the paper. It makes a valuable addition to the area. A consistent concern was whether the empirical insights can be augmented with theoretical one. To the extent this can be fleshed out within the scope of the paper that would add further value.
Decision: Accept (poster)