SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Connor Toups*
Rishi Bommasani*†
Kathleen A. Creel
Sarah H. Bana
Dan Jurafsky
Machine learning is traditionally studied at the model level: researchers measure and improve the accuracy, robustness, bias, efficiency, and other dimensions of specific models. In practice, however, the societal impact of any machine learning model depends on the context into which it is deployed. To capture this, we introduce ecosystem-level analysis: rather than analyzing a single model, we consider the collection of models that are deployed in a given context. For example, ecosystem-level analysis in hiring recognizes that a job candidate's outcomes are determined not only by a single hiring algorithm or firm but instead by the collective decisions of all the firms to which the candidate applied. Across three modalities (text, images, speech) and eleven datasets, we establish a clear trend: deployed machine learning is prone to systemic failure, meaning some users are exclusively misclassified by all models available. Even when individual models improve over time, we find these improvements rarely reduce the prevalence of systemic failure. Instead, the benefits of these improvements predominantly accrue to individuals who are already correctly classified by other models. In light of these trends, we analyze medical imaging for dermatology, a setting where the costs of systemic failure are especially high. While traditional analyses reveal that both models and humans exhibit racial performance disparities, ecosystem-level analysis reveals new forms of racial disparity in model predictions that do not present in human predictions. These examples demonstrate that ecosystem-level analysis has unique strengths in characterizing the societal impact of machine learning.<sup>1</sup>
Machine learning (ML) is pervasively deployed. Systems based on ML mediate our communication and healthcare, influence where we shop or what we eat, and allocate opportunities like loans and jobs. Research on the societal impact of ML typically focuses on the behavior of individual models. If we center people, however, we recognize that the impact of ML on our lives depends on the aggregate result of our many interactions with ML models.
In this work, we introduce ecosystem-level analysis to better characterize the societal impact of machine learning on people. Our insight is that when a ML model is deployed, the impact on users depends not only on its behavior but also on the behavior of other models and decision-makers (left of Figure 1). For example, the decision of a single hiring algorithm to reject or accept a candidate does
†Corresponding author: nlprishi@stanford.edu.
<sup>1</sup>All code is available at https://github.com/rishibommasani/EcosystemLevelAnalysis.
Figure 1: Ecosystem-level analysis. Individuals interact with decision-makers (left), receiving outcomes that constitute the failure matrix (right).
not determine whether or not the candidate secures a job; the outcome of her search depends on the decisions made by all the firms to which she applied. Likewise, in selecting consumer products like voice assistants, users choose from options such as Amazon Alexa, Apple Siri, or Google Assistant. From the user's perspective, what is important is that at least one product works.
In both settings, there is a significant difference from the user's perspective between systemic failure, in which zero systems correctly evaluate them or work for them, and any other state. The difference in marginal utility between zero acceptances and one acceptance is typically much higher than the difference between one acceptance and two acceptances. This non-linearity is not captured by average error metrics. For example, imagine that three companies are hiring and there are ten great candidates. All three companies wrongly reject the same two great candidates for a false negative error rate of  $20\%$ . Now imagine that each company wrongly rejects different candidates. The second decision ecosystem has the same false negative error rate,  $20\%$ , but no systemic failures or jobless candidates.
Ecosystem-level analysis is a methodology that centers on the failure matrix  $(F$ ; right of Figure 1):  $F$  encapsulates the outcomes individuals receive from all decision-makers. Of special interest are systemic failures: exclusively negative outcomes for individuals such as misclassifications or rejections from all decision-makers [Bommasani et al., 2022]. To establish general trends in deployed machine learning, we draw upon a large-scale audit [HAPI; Chen et al., 2022a] that spans three modalities (images, speech, text), three commercial systems per modality, and eleven datasets overall. Because HAPI contains predictions from some of the largest commercial ML providers – including Google, Microsoft, and Amazon – and the models evaluated are deployed models that real users interact with, evaluating HAPI has real-world implications.
Across all settings, ecosystem-level analysis reveals a consistent pattern of homogeneous outcomes. In each of the HAPI datasets, many instances are classified correctly by all three commercial systems and many instances are classified incorrectly by all three systems. The pattern of outcomes across the decision ecosystem is homogenous if the rates of systemic failure and consistent classification success significantly exceed the rate predicted by independent instance-level behavior. Since the commercial systems analyzed in this dataset are popular and widely used, being failed by all systems in the dataset has societally meaningful consequences.
Ecosystem-level analysis enriches our understanding not only of the status quo, but also of how models change over time. In particular, it allows us to ask, when individual models improve, how do ecosystem-level outcomes change? Since HAPI tracks the performance of the same systems over a three-year period, we consider all cases where at least one of the commercial systems improves. For example, Amazon's sentiment analysis API reduced its error rate on the WAIMAI dataset by  $2.5\%$  from 2020 to 2021; however, this improvement did not decrease the systemic failure rate at all. Precisely 0 out of the model's 303 improvements are on instances on which all other models had failed. These findings generalize across all cases: on average, just  $10\%$  of the instance-level improvement of a single commercial system occurs on instances misclassified by all other models. This is true even though systemic failures account for  $27\%$  of the instances on which the models could have improved. Thus most model improvements do not significantly reduce systemic failures.
To build on these trends, we study medical imaging, a setting chosen because the costs to individuals of systemic failure of medical imaging classification are especially high. We compare outcomes from prominent dermatology models and board-certified dermatologists on the DDI dataset [Daneshjou et al., 2022]: both models and humans demonstrate homogeneous outcomes, though human outcomes are more homogenous. Given established racial disparities in medicine for both models and humans, fairness analyses in prior work show that both humans and models consistently perform worse for darker skin tones (e.g. Daneshjou et al. [2022] show lower ROC-AUC on DDI). Ecosystem-level analysis surfaces new forms of racial disparity in models that do not present in humans: models are more homogenous when evaluating images with darker skin tones, meaning that all systems agree in their correct or incorrect classification, whereas human homogeneity is consistent across skin tones.
Our work contributes to a growing literature on the homogeneous outcomes of modern machine learning [Ajunwa, 2019, Engler, 2021, Creel and Hellman, 2022, Bommasani et al., 2022, Fishman and Hancox-Li, 2022, Wang and Russakovsky, 2023, Jain et al., 2023]. While prior work conceptualizes these phenomena, our work introduces new methodology to study these problems and provides concrete findings for a range of ML deployments spanning natural language processing, computer vision, speech, and medical imaging. Further, by centering individuals, we complement established group-centric methods [Barocas and Selbst, 2016, Buolamwini and Gebru, 2018, Koenecke et al., 2020], unveiling new forms of racial disparity. Ecosystem-level analysis builds on this existing work, providing a new tool that contributes to holistic evaluations of the societal impact of machine learning.
Developing better methodologies for ecosystem-level analysis of deployed machine learning systems is important for two reasons. First, systemic failures in socially consequential domains could exclude people from accessing goods such as jobs, welfare benefits, or correct diagnoses. Individuals who are failed by only one model can gain informal redress by switching to another model, for example by seeking second doctor's opinion or switching banks. Individuals failed by all models cannot. Socially consequential systemic failures can happen due to reliance on APIs, such as image recognition APIs used to identify cancers, speech recognition APIs used to verify individuals for banking, or facial recognition APIs used to unlock devices. Systemic failures can also occur in algorithmic decision-making systems such as those used for hiring, lending, and criminal justice. The social importance of avoiding systemic failures in all of these systems is clear.
Second, as decision-makers become more likely to rely on the same or similar algorithms to make decisions [Kleinberg and Raghavan, 2021] or to use the same or similar components in building their decision pipelines [Bommasani et al., 2022], we believe that the prevalence of systemic failures could increase. Measuring systemic failures as they arise with the tools we present in this paper will expand our understanding of their prevalence and likely causes.
2 Ecosystem-level Analysis
How individuals interact with deployed machine learning models determines ML's impact on their lives. In some contexts, individuals routinely interact with multiple ML models. For example, when a candidate applies to jobs, they typically apply to several firms. The decision each company makes to accept or reject the candidate may be mediated by a single hiring algorithm. In other contexts, individuals select a single model from a set of options. For example, when a consumer purchases a voice assistant, they typically choose between several options (e.g. Amazon Alexa, Google Assistant, Apple Siri) to purchase a single product (e.g. Amazon Alexa). Centering people reveals a simple but critical insight: exclusively receiving negative outcomes, as when individuals are rejected from every job or unable to use every voice assistant, has more severe consequences than receiving even one positive outcome.
2.1 Definition
Recognizing how ML is deployed, we introduce ecosystem-level analysis as a methodology for characterizing ML's cumulative impacts on individuals. Consider  $N$  individuals that do, or could, interact with  $k$  decision-makers that apply  $\hat{y}$  labels according to their decision-making processes  $h_1, \ldots, h_k$ . Individual  $i$  is associated with input  $x_i$ , label  $y_i$ , and receives the label  $\hat{y}_i^j = h_j(x_i)$  from decision-maker  $j$ .
Outcomes. Define the failure matrix  $F \in \{0,1\}^{N \times k}$  such that  $F[i,j] = \mathbb{I}\left[\hat{y}_i^j \neq y_i\right]$ . The failure outcome profiles  $\mathbf{f}_i$  for individual  $i$ , which we refer to as the outcome profile for brevity, denotes  $F[i,:]$ . The failure rate  $\bar{f}_j$  for decision-maker  $j$  is  $\bar{f}_j = \frac{\sum_{i=1}^{N} F[i,j]}{N}$  (i.e. the empirical classification error in classification). For consistency, we order the entries of decision-makers (and, thereby, the columns of the failure matrix) in order of ascending failure rate:  $F[:,1]$  is the outcome profile associated with the decision-maker with the fewest failures and  $F[:,k]$  is the outcome profile associated with the decision-maker with the most failures. The failure matrix is the central object in ecosystem-level analysis (see Figure 1).
Systemic Failures. Individual  $i$  experiences systemic failure if they exclusively experience failure across the domain of interest:  $F[i, :] = [1, \dots, 1]$ . Not only are systemic failures the worst possible outcomes, but they also often result in additional harms. If an individual applying to jobs is rejected everywhere, they may be unemployed. If no commercial voice assistant can recognize an individual's voice, they may be fully locked out of accessing a class of technology. In our ecosystem-level analysis, we focus on systemic failures as a consequential subset of the broader class of homogeneous outcomes [Bommasani et al., 2022].
3 Homogeneous Outcomes in Commercial ML APIs (HAPI)
To establish general trends made visible through ecosystem-level analysis, we draw upon a large-scale three-year audit of commercial ML APIs [HAPI; Chen et al., 2022a] to study the behavior of deployed ML systems across three modalities, eleven datasets, and nine commercial systems.
3.1 Data
Chen et al. [2022a] audit commercial ML APIs, tracking predictions across these APIs when evaluated on the same eleven standard datasets over a period of three years (2020 - 2022). We consider ML APIs spanning three modalities (text, images, speech), where each modality is associated with a task (SA: sentiment analysis, FER: facial emotion recognition, SCR: spoken command recognition) and 3 APIs per modality (e.g. IBM, Google, Microsoft for spoken command recognition). The models evaluated are from Google (SA, SCR, FER), Microsoft (SCR, FER), Amazon (SA), IBM (SCR), Baidu (SA), and Face++ (FER). Additionally, each modality is associated with three to four datasets, amounting to eleven datasets total; further details are deferred to the supplement.
To situate our notation, consider the DIGIT dataset for spoken command recognition and the associated APIs (IBM, Google, Microsoft). For each instance (i.e. image)  $x_{i}$  in DIGIT, the outcome profile  $\mathbf{f}_{i} \in \{0,1\}^{3}$  is the vector of outcomes. The entries are ordered by ascending model failure rate:  $F[:,1]$  corresponds to the most accurate model (Microsoft) and  $F[:,3]$  corresponds to the least accurate model (Google).
Facial emotion recognition	Spoken command recognition	Sentiment analysis
RAFDB	AFNET	EXPW	FER+	FLUENT	DIGIT	AMNIST	SHOP	YELP	IMDB	WAIMAI
Dataset size	15.3k	287.4k	31.5k	6.4k	30.0k	2.0k	30.0k	62.8k	20.0k	25.0k	12.0k
Number of classes	7	7	7	7	31	10	10	2	2	2	2
h1 failure rate (i.e. error)	0.283	0.277	0.272	0.156	0.019	0.217	0.015	0.078	0.043	0.136	0.110
h2 failure rate (i.e. error)	0.343	0.317	0.348	0.316	0.025	0.259	0.015	0.095	0.111	0.219	0.151
h3 failure rate (i.e. error)	0.388	0.359	0.378	0.323	0.081	0.472	0.043	0.122	0.486	0.484	0.181
Systemic failure rate	0.152	0.178	0.181	0.066	0.01	0.129	0.002	0.039	0.021	0.043	0.065
Table 1: Basic statistics on HAPI datasets including the (observed) systemic failure rate (i.e. fraction of instances misclassified by all models).
Descriptive statistics. To build general understanding of model performance in HAPI, we provide basic descriptive statistics (Table 1). For most datasets, all APIs achieve accuracies within  $5 - 10\%$  of each other (exceptions include DIGIT, YELP, IMDB). Interestingly, we often find the systemic failure rate is roughly half the failure rate of the most accurate model  $h_1$ .
3.2 Ecosystem-level Behavior
In order for a measure of systemic failure to be useful, it must be (i) meaningful and (ii) comparable across systems. A challenge to the meaningfulness of any proposed metric is that systemic failures occur more often in an ecosystem with many inaccurate models. A metric for systemic failure that primarily communicated the aggregate error rates of models in the ecosystem would not be meaningful as an independent metric. It also would not support goal (ii) because we could not compare the rates of systemic failure across ecosystems with varying model accuracies. It would be difficult to identify a system with a 'large' rate of systemic failure because the systemic failure properties would be swamped by the error rates of the models in the ecosystem. Therefore, achieving meaningfulness and comparability requires the metric to incorporate error correction.
Assuming model independence is a helpful baseline because it adjusts for model error rates without making assumptions about the correlation between models in the ecosystem. To avoid assumptions and for the sake of simplicity, therefore, we juxtapose the observed behavior with a simple theoretical model in which we assume models fail independently of each other. Under this assumption, the distribution of the baseline number of model failures  $t \in \{0, \dots, k\}$  follows a Poisson-Binomial distribution parameterized by their failure rates (Equation 2).
The baseline of independence also means that our metric does not attempt to quantify whether it is "reasonable" that the models all fail on some instances. For example, some instances might be harder (or easier) than others, making it more likely that all models will fail (or succeed) to classify that instance. However, "hardness" is observer-relative. What is hard for one class of models might be easy for another class of model, and likewise with humans with particular capabilities or training. Therefore 'correcting' the metric to account for hardness would relativize the metric to the group of humans or class of models for whom that instance is hard. We choose independence as a baseline to be neutral on this point. However, we depart from independence in Appendix §A.3, exploring how a baseline that assumes some level of correlation between models can more accurately model the observed distribution of ecosystem-level outcomes.
Comparing the true observed distribution of ecosystem-level outcomes with the baseline distribution helps illuminate how correlated outcomes are across models. Below we define  $P_{\mathrm{observed}}$  (Equation 1) and  $P_{\mathrm{baseline}}$  (Equation 2).
$$
P _ {\text {o b s e r v e d}} (t \text {f a i l u r e s}) = \frac {\sum_ {i = 1} ^ {N} \mathbb {I} [ t = \sum_ {j = 1} ^ {k} F [ i , j ] ]}{N} \tag {1}
$$
$$
P _ {\text {b a s e l i n e}} (t \text {f a i l u r e s}) = \text {P o i s s o n - B i n o m i a l} (\bar {f} _ {1}, \dots , \bar {f} _ {k}) [ t ] \tag {2}
$$
Finding 1: Homogenous Outcomes. In Figure 2a, we compare the observed and baseline distributions for the spoken command recognition dataset DIGIT. We find the observed ecosystem-level outcomes are more clearly homogenous compared to the baseline distribution: the fraction of instances that receive either extreme outcome (all right or all wrong) exceeds the baseline rate. These findings generalize to all the datasets (Figure 2b): the observed rate always exceeds the baseline rate for the homogeneous outcomes (above the line  $y = x$ ) and the reverse mostly holds for intermediary outcomes.
4 Do Model Improvements Improve Systemic Failures?
The performance of a deployed machine learning system changes over time. Developers serve new versions to improve performance [Chen et al., 2022b,a], the test distribution shifts over time [Koh et al., 2021], and the users (sometimes strategically) change their behavior [Björkegren et al., 2020]. In spite of this reality, most analyses of the societal impact of machine learning only consider static models.
Ecosystem-level analysis provides a new means for understanding how models change and how those changes impact people. When models change, what are the broader consequences across their model ecosystem? Do single-model improvements on average improve ecosystem-level outcomes by reducing systemic failures? And to what extent are the individuals for whom the model improves the same individuals were previously systemically failed?
(a) Ecosystem-level outcomes for DIGIT dataset.
(b) Ecosystem-level outcomes for all datasets.
Figure 2: Homogeneous outcomes. Ecosystem-level analysis surfaces the general trend of homogeneous outcomes: the observed rates that all models succeed/fail consistently exceeds the corresponding baseline rates. Figure 2a shows that models in the DIGIT dataset are more likely to all fail or all succeed on an instance than baseline. Figure 2b shows that across all datasets, systemic failure (red dots) and consistent success (blue dots) of all three models on an instance are both more common than baseline, whereas intermediate results are less common than baseline.
(a) Outcome profile distribution for (Baidu, Google) when Amazon improves on WAIMAI from 2020 to 2021.
Figure 3: Model improvement is not concentrated on systemic failures. When a model improves, we compare the distribution of outcome profiles of the other two models on its initial failures (potential improvements) to the distribution on the instances it improved on (observed improvements). Across all improvements, including Amazon's improvement on WAIMAI (left), there is a clear over-improvement on  $[\checkmark, \checkmark]$  (above  $y = x$  on right) and under-improvement on [X, X] (below the identity line on right).
(b) The distribution of outcome profiles for all year-over-year model improvements across all datasets.
Setup. Chen et al. [2022a] evaluated the performance of the commercial APIs on the same eleven evaluation datasets each year in 2020-2022. Of all year-over-year comparisons, we restrict our attention to cases where one of the three APIs for a given task improves by at least  $0.5\%$  accuracy. Let  $h_{\mathrm{imp}}$  denote the model that improved. We identify the instances that  $h_{\mathrm{imp}}$  initially misclassified in the first year as potential improvements and the subset of these instances that  $h_{\mathrm{imp}}$  correctly classified in the second year as improvements. Considering the initial distribution of failures for  $h_{\mathrm{imp}}$ , we can ask where does the  $h_{\mathrm{imp}}$  improve? We answer this by comparing the distribution of outcome profiles for the other models (besides  $h_{\mathrm{imp}}$ ) between the potential improvement and improvement sets.
2The supplement  $\S \mathrm{B}.3$  contains an analysis that confirms our findings are robust to alternate thresholds.
Finding 2: Model improvements make little progress on systemic failures. As a case study, we consider Amazon's improvement on the WAIMAI dataset from 2020 to 2021. In Figure 3a, from left to right, [X, X] indicates the other APIs (Baidu, Google) both fail, [X,  $\sqrt{}$  ] and  $[\sqrt{}$  , X] indicate exactly one of the other APIs succeed, and  $[\sqrt{}$  ,  $\sqrt{}]$  indicates both succeed. The majority (64.7%) of Amazon's improvement is on instances already classified correctly by the other two APIs, far exceeding the fraction of potential improvements that were classified correctly by Baidu and Google (34.1%) in 2020. In contrast, for systemic failures, the improvement of  $11.6\%$  falls far short of the potential improvement of  $36.7\%$ . In fact, since models can also fail on instances they previously classified correctly, the model's improvement on systemic failures is even worse in terms of net improvement. Amazon's improvement amounts to no net reduction of systemic failures: the model improves on 78 systemic failures but also regresses on 78 instances that become new systemic failures, amounting to no net improvement. The Baidu and Google APIs similarly show little improvement on systemic failures even as models improve.