SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This finding is not unique to Amazon's improvement on the WAIMAI dataset: in the 11 datasets we study, we observe the same pattern of homogeneous improvements from 2020-2022. In Figure 3b, we compare the observed improvement distribution $^{3}$  ( $y$  axis) to the potential improvement distributions ( $x$  axis) across all model improvements. We find a clear pattern: systemic failures (the [X, X] category) are represented less often in the observed improvement set than in the potential improvement set. This finding indicates that when models improve, they under-improve on users that are already being failed by other models. Instead, model improvements especially concentrate on instances where both other models succeeded already.
Ecosystem-level analysis in the context of model improvements disambiguates two plausible situations that are otherwise conflated: does single-model improvement (i) marginally or (ii) substantively reduce systemic failures? We find the reduction of systemic failures is consistently marginal: in every case, the reduction fails to match the distribution seen in the previous year (i.e. every [X, X] red point is below the line in Figure 3b).
5 Ecosystem-level Analysis in Dermatology (DDI)
Having demonstrated that ecosystem-level analysis reveals homogeneous outcomes across machine learning deployments, we apply the ecosystem-level methodology to medical imaging. We consider this setting to be an important use of ecosystem-level analysis because machine learning makes predictions that inform the high-stakes treatment decisions made by dermatologists.
5.1 Data
Daneshjou et al. [2022] introduced the Diverse Dermatology Images (DDI) dataset of 656 skin lesion images to evaluate binary classification performance at detecting whether a lesion was malignant or benign. Images were labelled with the ground-truth based on confirmation from an external medical procedure (biopsy). In addition, each image is annotated with skintone metadata using the Fitzpatrick scale according to one of three categories: Fitzpatrick I & II (light skin), Fitzpatrick III & IV (medium skin), and Fitzpatrick V & VI (dark skin). We use the predictions from Daneshjou et al. [2022] on DDI for two prominent ML models (ModelDerm [Han et al., 2020] and DeepDerm [Esteva et al., 2017]) and two board-certified dermatologists. We defer further details to Appendix C.
Finding 3: Both humans and models yield homogeneous outcomes; humans are more homogeneous. We compare observed and baseline ecosystem-level outcomes on DDI for models (Figure 4a) and humans (Figure 4b). Consistent with the general trends in HAPI, model predictions yield homogeneous outcomes. For human predictions from board-certified dermatologists, we also
3The supplement  $\S \mathrm{B}.4$  contains analysis comparing 'net improvements' to 'potential improvements' as well; the trends are consistent across both analyses.
$^{4}$ Daneshjou et al. [2022] also evaluated a third model [HAM10K; Tschandl et al., 2018] that almost always predicts the majority class in this class-imbalanced setting. We exclude this model since its failures are not interesting, but replicate our analyses in the supplement Appendix C to show the findings still hold if it is included.
(a) Outcome profiles for models.
(b) Outcome profiles for humans.
Figure 4: Homogeneous outcomes for models and humans. Consistent with HAPI, model predictions (left) yield homogenous outcomes on DDI. Human predictions (right) are even more homogenous than models.
Figure 5: Racial disparities for models but not humans. We stratify ecosystem-level analysis in DDI by the three skin tone categories, plotting the difference between observed and baseline rates. Models (left) show clear racial disparities, exhibiting the most homogeneity for the darkest skin tones, whereas humans (right) show no significant racial disparity.
see that outcomes are homogeneous. However, in comparing the two, we find that humans yield even more homogeneous outcomes. We take this to be an important reminder: while we predict that models are likely to produce homogeneous outcomes, we should also expect humans to produce homogeneous outcomes, and in some cases more homogeneous outcomes.
Finding 4: Ecosystem-level analysis reveals new racial disparities in models but not humans. Standard analyses of machine learning's impact focus on model performance across groups [e.g. Buolamwini and Gebru, 2018, Koenecke et al., 2020]. In AI for medicine, several works have taken such an approach towards understanding fairness [Obermeyer et al., 2019, Seyyed-Kalantari et al., 2021, Kim et al., 2022, Colwell et al., 2022]. Daneshjou et al. [2022] demonstrate racial disparities for both model and human predictions on DDI and find that predictive performance is worst for darkest skin tones, aligning with broader trends of racial discrimination in medicine and healthcare [Vyas et al., 2020, Williams and Wyatt, 2015, Fiscella and Sanders, 2016].
Ecosystem-level analysis can build on these efforts. We conduct the same analysis from Figure 4 stratified by skin tone. In previous experiments, we had observed systemic failures across the whole population. Here we measure systemic failures for subpopulations grouped by skin tone. In Figure 5, we plot the difference between the observed and baseline rates on the  $y$  axis: the All bars (blue) reproduce the homogeneity results from Figure 4.
Strikingly, ecosystem-level-analysis surfaces an important contrast between model behavior (left) and human behavior (right). Models (Figure 5a) are most homogenous for darkest skin tones (Fitzpatrick V & VI; dark brown) and least homogeneous for the lightest skin tones (Fitzpatrick I & II; cream): The observed systemic failure rate for the darkest skin tones is  $8.2\%$  higher than the baseline, while for the lightest skin tones it is  $1.5\%$  lower than the baseline. By contrast, humans (Figure 5b) show no significant variation as a function of skin tone.
Ecosystem-level analysis therefore identifies a new form of racial disparity not previously documented. Critically, while prior works show racial disparities for both models and humans, here we find a form of racial disparity that is salient for models but not present for humans. We note that, in absolute terms, homogeneity is higher for all racial groups in human predictions than model predictions, though human predictions don't display significant differences in homogeneity across racial group. This demonstrates that ecosystem-level analysis can reveal new dimensions of fairness, allowing stakeholders to identify the metrics that are most relevant in their context, be that model error or systemic failure rate. Our tool helps researchers and stakeholders evaluate the holistic ecosystem of algorithmic – and human – judgements that ultimately shapes outcomes for those subject to algorithmic judgement.
6 Commentary
While ecosystem-level analysis reveals new dimensions of machine learning's societal impact, it also opens the door for further questions. We prioritize two here, deferring further discussion and a longer related work section to the supplement. How can we explain the pervasive homogeneous outcomes we have observed in deployed machine learning? And what are the implications of this work for both researchers and policymakers?
6.1 Explanations for Homogeneous Outcomes
Our findings provide evidence across several settings for homogeneous outcomes in deployed machine learning (Finding 1; §3) that are mostly unabated by model improvement (Finding 2; §4).
Data-centric explanations. We posit that "example difficulty" may give rise to homogeneous outcomes and provide three analyses that instantiate variants of this hypothesis.
First, human ambiguity on the ground-truth may predict homogeneous outcomes. To test this, we make use of the ten human annotations per example in the FER+ dataset within HAPI. We find that the systemic failure rate is monotonically correlated with annotator disagreement with the majority label. This suggests that some systemic failures are correlated with the ambiguity or "hardness" of the image. However, we find that even some of the least ambiguous images, namely images on which all or most annotators agree, have systemic failures. . This indicates that human ambiguity is only partially explanatory of ecosystem-level model outcomes. We explore this further in §A.1.
Second, human error may predict homogeneous outcomes. To test this, we compare human predictions on DDI with the ground truth biopsy results. We stratify ecosystem-level analysis by dermatologist performance, comparing (i) instances both dermatologists get right, (ii) instances they both get wrong, and (iii) instances where they disagree and exactly one is right. We find that when both dermatologists fail, there continues to be outcome homogenization. However, when both dermatologists succeed, there is no homogeneity and the observed rates almost exactly match the baseline rates for every image. This suggests human error is also partially predictive of ecosystem-level model outcomes. We explore this further in §A.2.
Finally, more expressive theoretical models can better capture the observed trends than our simple full instance-level independence model. We introduce a two-parameter model.  $\alpha$  fraction of instances are categorized as difficult and the remaining  $1 - \alpha$  are easy. A model's failure rate  $\bar{f}_j$  over all examples scales to  $(1 + \Delta)\bar{f}_j$  on hard examples and  $\left(1 - \frac{\alpha\Delta}{1 - \alpha}\right)\bar{f}_j$  on easy examples. Partitioning examples in this way, while continuing to assume instance-level independence, inherently homogenizes: models are more likely to systemically succeed on easy instances and systemically fail on hard instances. To best fit the HAPI data, the resulting average  $\alpha$ $(0.2 - 0.3)$  and  $\Delta$ $(1 - 4)$ , meaning these examples are  $2 - 5\times$  harder) values are quite extreme. In other words, for this theoretical model to fit the
data, a significant fraction  $(\approx 25\%)$  would need to be considerably harder  $(\approx 3.5\times)$  than the overall performance. We explore this further in  $\S A.3$
These analyses contribute to an overarching hypothesis that example difficulty partially explains homogeneous outcomes. While we discuss the construct of difficulty in the supplement, we draw attention to two points. First, difficulty is largely in the eye of the beholder: what humans or models perceive as difficult can differ [e.g. adversarial examples; Goodfellow et al., 2014]. Thus while example difficulty could be caused by inherent properties of the example (such as noisy speech or corrupted images), it could just as well be due to model properties, such as all the models having made similar architectural assumptions or having parallel limitations in their training data. Second, whether or not homogeneous outcomes are caused by example difficulty does not change their societal impact. The consequences of systemic failure can be material and serious (e.g. unemployment).
Model-centric Explanations and Algorithmic Monoculture. An alternative family of explanations put forth in several works is that correlated outcomes occur when different deployed machine learning models share training data, model architectures, model components, learning objectives or broader methodologies [Ajunwa, 2019, Engler, 2021, Bommasani et al., 2021, Creel and Hellman, 2022, Fishman and Hancox-Li, 2022, Bommasani et al., 2022, Wang and Russakovsky, 2023]. Such algorithmic monoculture [Kleinberg and Raghavan, 2021, Bommasani et al., 2022] in fact appears to be increasingly common as many deployed systems, including for the tasks in HAPI, are derived from the same few foundation models [Bommasani et al., 2023, Madry, 2023]. Unfortunately, we are unable to test these hypotheses because we know very little about these deployed commercial systems, but we expect they form part of the explanation and encourage future work in this direction.
Implications for Researchers. Our paper shows that ecosystem-level research can reveal previously-invisible social impacts of machine learning. We believe this methodology concretizes the impact of decision-making in real contexts in which individuals would typically be affected by decisions from many actors (e.g. job applications, medical treatment, loan applications, or rent pricing). As we demonstrate in DDI, our methodology applies equally to human-only, machine-only, and more complex intertwined decision-making. We anticipate understanding of homogeneous outcomes arising from any or all of these sources will be valuable.
Implications for Policymakers. Given the pervasive and persisting homogeneous outcomes we document, there may be a need for policy intervention. In many cases no single decision-maker can observe the decisions made by others in the ecosystem, so individual decision-makers (such as companies) may not know that systemic failures exist. In addition, systemic failures are not currently the responsibility of any single decision-maker, so no decision-maker is incentivized to act alone. Consequently, policy could implement mechanisms to better monitor ecosystem-level outcomes and incentivize ecosystem-level improvement. In parallel, regulators should establish mechanisms for recourse or redress for those currently systemically failed by machine learning [see Voigt and von dem Bussche, 2017, Cen and Raghavan, 2023].
We introduce ecosystem-level analysis as a new methodology for understanding the cumulative societal impact of machine learning on individuals. Our analysis on HAPI establishes general trends towards homogeneous outcomes that are largely unaddressed even when models improve. Our analysis on DDI exposes new forms of racial disparity in medical imaging that arise in model predictions but not in human predictions. Moving forward, we hope that future research will build on these empirical findings by providing greater theoretical backing, deeper causal explanations, and satisfactory sociotechnical mitigations. To ensure machine learning advances the public interest, we should use approaches like ecosystem-level analysis to holistically characterize its impact.
We would like to thank Shibani Santurkar, Mina Lee, Deb Raji, Meena Jagadeesan, Judy Shen, p-lambda, and the Stanford ML group for their feedback on this work. We would like to thank James Zou and Lingjiao Chen for guidance with using the HAPI dataset. We would like to thank Roxana Daneshjou for providing the DDI dataset along with guidance on how to analyze the dataset. In addition, the authors would like to thank the Stanford Center for Research on Foundation Models (CRFM) and Institute for Human-Centered Artificial Intelligence (HAI) for providing the ideal home for conducting this interdisciplinary research. RB was supported by the NSF Graduate Research Fellowship Program under grant number DGE-1655618. This work was supported in part by a Stanford HAI/Microsoft Azure cloud credit grant and in part by the AI2050 program at Schmidt Futures (Grant G-22-63429).
Rishi Bommasani, Kathleen Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=-H6kM4DVo.
Lingjiao Chen, Zhihua Jin, Evan Sabri Eyuboglu, Christopher Ré, Matei Zaharia, and James Y Zou. HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24571-24585. Curran Associates, Inc., 2022a. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9bcd0bdd2777fe8c729b682f07e993f1-Paper-Datasets_and_Benchmarks.pdf.
Roxana Daneshjou, Kailas Vodrahalli, Roberto A. Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M. Swetter, Elizabeth E. Bailey, Olivier Gevaert, Pritam Mukherjee, Michelle Phung, Kiana Yekrang, Bradley Fong, Rachna Sahasrabudhe, Johan A. C. Allerup, Utako Okata-Karigane, James Zou, and Albert S. Chiou. Disparities in dermatology ai performance on a diverse, curated clinical image set. Science Advances, 8(32):eabq6147, 2022. doi: 10.1126/sciadv.abq6147. URL https://www.science.org/doi/abs/10.1126/sciadv.abq6147.
Ifeoma Ajunwa. The paradox of automation as anti-bias intervention. Cardozo L. Rev., 41:1671, 2019. URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2746078.
Alex Engler. Enrollment algorithms are contributing to the crises of higher education. report, The Brookings Institution, September 2021. URL https://www.brookings.edu/research/enrollment-algorithms-are-contributing-to-the-crises-of-higher-education/.
Kathleen Creel and Deborah Hellman. The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. *Canadian Journal of Philosophy*, 52(1): 26-43, 2022. doi: 10.1017/can.2022.3. URL https://papers.ssrn.com/sol3/papers.cfm? abstract_id=3786377.
Nic Fishman and Leif Hancox-Li. Should attention be all we need? the epistemic and ethical implications of unification in machine learning. 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022. URL https://dl.acm.org/doi/fullHtml/10.1145/3531146.3533206.
Angelina Wang and Olga Russakovsky. Overcoming bias in pretrained models by manipulating the finetuning dataset. arXiv preprint arXiv:2303.06167, 2023. URL https://arxiv.org/abs/2303.06167.
Shomik Jain, Vinith M. Suriyakumar, Kathleen Creel, and Ashia C. Wilson. Algorithmic pluralism: A structural approach towards equal opportunity. ArXiv, abs/2305.08157, 2023. URL https://arxiv.org/abs/2305.08157.
Solon Barocas and Andrew D. Selfst. Big data's disparate impact. California Law Review, 104:671, 2016. URL https://www.jstor.org/stable/24758720.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8346-8356. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/sagawa20a.html.
Shan Li, Weihong Deng, and Junping Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2584-2593. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017.277. URL https://doi.org/10.1109/CVPR.2017.277.
Ali Mollahosseini, Behzad Hassani, and Mohammad H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput., 10(1):18-31, 2019. doi: 10.1109/TAFFC.2017.2740923. URL https://doi.org/10.1109/TAFFC.2017.2740923.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaou Tang. From facial expression recognition to interpersonal relation prediction. CoRR, abs/1609.06426, 2016. URL http://arxiv.org/abs/1609.06426.
Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 814-818. ISCA, 2019. doi: 10.21437/Interspeech.2019-2396. URL https://doi.org/10.21437/Interspeech.2019-2396.
Zohar Jackson, César Souza, Jason Flaks, Yuxin Pan, Hereman Nicolas, and Adhish Thite. Jakobovski/free-spoken-digit-dataset: v1.0.8, August 2018. URL https://doi.org/10.5281/ zenodo.1342401.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807.03418, 2018. URL http://arxiv.org/abs/1807.03418.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics, 2011. URL https://aclanthology.org/P11-1015/.
Figure 6: We stratify ecosystem-level analysis on  $\mathrm{FER}+$  by instance-level annotator disagreement – which we take as a proxy for the ambiguity inherent to the input instance – and plot the difference between observed and baseline rates for each instance subset. We observe homogeneous outcomes for all subsets of data, regardless of the level of annotator disagreement on the instance.
A Data-centric Explanations for Homogeneous Outcomes
Prior work has explored model-centric explanations for homogeneous outcomes [Ajunwa, 2019, Engler, 2021, Bommasani et al., 2021, Creel and Hellman, 2022, Fishman and Hancox-Li, 2022, Bommasani et al., 2022, Wang and Russakovsky, 2023]. However, data-centric explanations are comparatively less explored. We posit that properties of the underlying data could contribute to the homogeneous outcomes that we observe in all of the datasets we examine. Intuitively, if we believe that some examples are 'hard' and others are 'easy', then we might expect to see models all fail for the 'hard' examples and all succeed for the 'easy' examples.
We test three variants of this hypothesis. In §A.1, we examine how the level of annotator disagreement in the ground truth label impacts ecosystem-level behavior. In §A.2, we test how the accuracy of human dermatologists in predicting the malignancy of a skin lesion image correlates with homogeneous outcomes. Finally, to build on these finer-grained empirical analyses, in §A.3, we introduce a more express theoretical model. Under this model, parameterized by two difficulty parameters, we compute a different baseline rate for ecosystem-level outcomes, showing it can better recover the observation distribution.
A.1 Annotator disagreement
To study the effects of annotator disagreement, we make use of the  $\mathrm{FER + }$  dataset. Each instance of the  $\mathrm{FER + }$  dataset, a facial emotion recognition dataset, contains emotion annotations from 10 human annotators; the emotion label is determined by majority vote [Barsoum et al., 2016]. Because each instance has been annotated by multiple annotators, we can calculate the annotator disagreement for each instance and use this as a proxy for the 'ambiguity' of the instance. For example, an instance where all 10 annotators agree that the label is 'sad' is less ambiguous than an instance where 6 annotators vote the label should be 'fear' and 4 vote that the label should be 'surprise'.
The test set of  $\mathrm{FER + }$  provided in HAPI contains instances with disagreement percentages ranging from  $0\%$  to  $50\%$ . We stratify on the disagreement percentage of the instances and compare baseline and observed ecosystem-level outcomes for each subset of instances.
Figure 7: Examples of homogeneous outcomes Instances that are sampled uniformly at random from "0 models correct" (top row) or "3 models correct" (bottom row) in  $\mathrm{FER + }$ . The systemic failures (top row) do not appear to be inherently harder for humans to classify; more extensive analysis appears in the supplement.
Figure 8: Interpretation of the relationship between instance-level annotator disagreement and homogeneous outcomes depends on if the strength of the effect is quantified as the ratio between observed and baseline rates (left plot) or as the difference between observed and baseline rates (right plot).