SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Examples. To build further intuition, we present several randomly sampled instances from the  $\mathrm{FER + }$  facial emotion recognition dataset in Figure 7. We emphasize that while systemic failures may share structure, we do not believe these instances are inherently harder than ones on which models perform well. The authors did not have difficulty labelling these examples, nor would other human labelers. We address the question of why systemic failures arise in §6.
Homogenous Outcomes manifest regardless of annotator disagreement. In Figure 6, we find that homogenous outcomes appear for all levels of annotator disagreement. While more ambiguous examples exhibit higher model error rates and systemic failure rates, the observed rate of homogeneous outcomes exceeds the baseline rate of homogeneous outcomes for all instance subsets. This suggests that instance-level ambiguity does not (fully) explain the existence of homogenous outcomes—at least in  $\mathrm{FER+}$ .
The intensity of the effect varies by disagreement level, but the direction of the relationship depends on how strength is quantified. In light of the observed existence of homogeneous
<sup>5</sup>We acknowledge the task of facial emotion recognition has been the subject of extensive critique [e.g. Barrett et al., 2019, Mau et al., 2021]). We provide examples for this task due to ease of visualization, but our claims also hold for examples from the text and speech modalities that are provided in the supplement.
(a) Subset of instances that both dermatologists classify correctly.
Figure 9: Homogeneous outcomes exists for the subset of instances where both dermatologists misclassify the image but doesn't exist for the subset where both dermatologists correctly classify the image.
(b) Subset of instances that precisely one dermatologist misclassifies.
(c) Subset of instances that both dermatologists misclassify.
outcomes across all levels of disagreement, we now examine how the intensity of this effect varies with annotator disagreement. The relationship between annotator disagreement and the intensity of homogeneous outcomes depends on the quantification method used to measure homogeneous outcomes intensity. When quantifying the intensity as the difference between observed and baseline rates, the effect becomes more pronounced as disagreement increases. However, when considering the intensity as the ratio between observed and baseline rates, as in the homogenization metric introduced by Bommasani et al. [2022], the effect is most pronounced at lower levels of disagreement.
A.2 Human accuracy
Each instance of the DDI dataset — a medical imaging skin lesion dataset — contains predictions from 2 models and 2 humans with the ground truth generated from an external process (in this case, a biopsy of the lesion). To understand how human-perceived difficulty relates to homogeneous outcomes, we stratify instances on the dermatologist accuracy of that instance (each instance has a dermatologist accuracy of  $0\%$ ,  $50\%$ , or  $100\%$ ) and examine homogeneous outcomes for each subset of instances.
Model outcomes are homogenous for instances on which dermatologists fail and heterogeneous for on which dermatologists succeed. In Figure 9, we find that model outcomes exhibit homogeneous outcomes for the subset of instances that both human dermatologists fail at to an even greater extent than observed at the population level. In contrast, the observed and baseline distributions match each other (meaning there is no homogeneous outcomes, heterogeneity or other form of deviation between the distributions) for the subset of instances that both dermatologists get right. Note that in the instances that both dermatologists fail at, both models systemically fail more than the baseline, but both also succeed more than the baseline. While the correlation between human systemic failures and model systemic failures seems intuitive, it's less clear why models jointly succeed more than the baseline on human systemic failures.
Given these surprising and unintuitive findings, we encourage future work to explain what features of these instances lead models to pattern together. We speculate that these instances lack sufficient informational cues, prompting the models to place excess emphasis on a narrow selection of features, consistent with the literature of machine learning models being susceptible to spurious correlations [e.g. Sagawa et al., 2020]. At face value, these results would suggest that human-level difficulty can be predictive of outcome homogeneity, but we emphasize two caveats: (i) the DDI dataset's sample size is limited (155 instances of dermatologists both failing and 356 instances of dermatologists both succeeding) and (ii) we rely on the judgments of just two human domain experts, meaning the findings may not generalize to larger annotator pools of non-experts (as are common in many NLP or computer vision datasets). In general, we caution against overgeneralizing this finding. We provide it as an initial foray into understanding example difficulty in a unique setting where we have data that supports an approximation of human-perceived difficulty.
A.3 More expressive theoretical models
Finally, while we consistently find that observed ecosystem-level behaviors yield more homogeneous outcomes than the instance-level independent model predictions would predict, we might intuit. there exists instance-level structure that should be encoded into the prior. Therefore, we consider a theoretical framework to encode richer priors on what we might expect of models. As a simple model, we will assume some instances are universally 'hard', meaning all models will perform worse on average across these instances, and others are conversely 'easy', meaning all models will perform better on average across these instances.
As before, we note that there is not a universal standard according to which examples can be considered 'hard.' Some examples are easy for humans but hard for models; other examples are easy for models but challenging for humans; and still other examples are challenging for some models but not others. The three hypotheses we consider in this section explore these observer-relative dimensions of hardness.
We use two parameters  $(\alpha, \Delta)$  to parameterize this model, thereby adjusting the baseline rate that we calculate for ecosystem-level outcomes.  $\alpha$  specifies the composition of 'hard' vs 'easy' instances in a dataset and  $\Delta$  controls how much harder or easier the hard or easy examples, respectively, are expected to be. Concretely,  $\alpha$  fraction of instances are categorized as difficult and the remaining  $1 - \alpha$  are easy. A model's failure rate  $\bar{f}_j$  over all examples scales to  $\bar{f}_j^{\mathrm{hard}} = (1 + \Delta)\bar{f}_j$  on hard examples and  $\bar{f}_j^{\mathrm{easy}} = \left(1 - \frac{\alpha\Delta}{1 - \alpha}\right)\bar{f}_j$  on easy examples.
The distribution of the baseline number of model failures  $t \in \{0, \dots, k\}$  follows a weighted sum of two Poisson-Binomial distributions parameterized by the scaled hard  $\bar{f}_j^{\text{hard}}$  and easy  $\bar{f}_j^{\text{easy}}$  model error rates.
$$
P _ {\text {b a s i l e n t}} ^ {\text {h a r d}} (t \text {f a i l u r e s}) = (\alpha) \text {P o i s s o n - B i n o m i a l} \left(\bar {f} _ {1} ^ {\text {h a r d}}, \dots , \bar {f} _ {k} ^ {\text {h a r d}}\right) [ t ] \tag {3}
$$
$$
P _ {\text {b a s i l e n t}} ^ {\text {e a s y}} (t \text {f a i l u r e s}) = (1 - \alpha) \text {P o i s s o n - B i n o m i a l} \left(\bar {f} _ {1} ^ {\text {e a s y}}, \dots , \bar {f} _ {k} ^ {\text {e a s y}}\right) [ t ] \tag {4}
$$
$$
P _ {\text {b a s i l e n}} (t \text {f a i l u r e s}) = P _ {\text {b a s i l e n}} ^ {\text {h a r d}} [ t ] + P _ {\text {b a s i l e n}} ^ {\text {e a s y}} [ t ] \tag {5}
$$
Identifying  $\alpha$  and  $\Delta$  values that recover the observed ecosystem-level outcomes in HAPI. We utilize this framework to identify which  $(\alpha, \Delta)$  combinations generate baseline distributions that recover the observed distributions in the HAPI datasets. We perform a grid search for  $\alpha \in [0.1, 0.5]$  with a step size of 0.1 and  $\Delta \in [0.2, 5]$  with a step size of 0.2. Note that certain  $(\alpha, \Delta)$  combinations can result in invalid error rates depending on the original error rates of models - i.e. when  $\left(1 - \frac{\alpha \Delta}{1 - \alpha}\right) \bar{f}_j < 0$ .
For each dataset, we identify the  $(\alpha, \Delta)$  combination that minimizes the L1 distance (equivalently the total variation distance) between the observed and baseline distributions. In Figure 10, we visualize the observed and baseline distributions for the distance-minimizing  $(\alpha, \Delta)$  pair for each dataset, and in Figure 11 we plot the L1 distance for each dataset as a function of  $\alpha$  and  $\Delta$ . The majority of the best  $\alpha$  values are 0.2 or 0.3 while the  $\Delta$  values can range from 1 to 4 – indicating that the  $\bar{f}_j^{\mathrm{hard}}$  can be up to  $5\mathrm{x}$  higher than  $\bar{f}_j$ .
High  $\Delta$  and low  $\alpha$  represents a small group of very difficult instances that all models struggle at. The combination of high  $\Delta$  and low  $\alpha$  values performing well suggests that we would need to expect that a relatively small fraction of the dataset contains instances that are significantly harder for all models to perform well at. Note that this framework assumes that all models consider the same instances to be 'hard': this agreement is a form of homogenization that could be caused about something inherent to the data or something about how the models are constructed.
B HAPI Experiments
In the main paper, we work extensively with the HAPI dataset of Chen et al. [2022a]. While we defer extensive details about the dataset to their work, we include additional relevant details here as well as any relevant decisions we made in using the HAPI dataset.
Figure 10: Observed and baseline distributions of ecosystem-level outcomes for the  $(\alpha, \Delta)$  combination that yields the lowest L1 distance for each dataset.
Figure 11: L1 distance between observed and baseline distributions for all datasets as a function of  $\alpha$  and  $\Delta$ .
Effect of conditioning on model  $h_i$  failing on distribution of other-model outcomes
Figure 12: Change in the probability of observing each outcome profile for the other models upon observing  $h_i$  fail.
We work with a subset of HAPI, a dataset introduced by Chen et al. [2022a] which contains predictions from commercial ML APIs on a variety of standard benchmark datasets from 2020-2022. HAPI contains benchmark datasets for three classification tasks and three structured prediction tasks. We only work with the classification tasks – spoken command recognition (SCR), sentiment analysis (SA), and facial emotion recognition (FER) – because they contain a single ground truth label where the notion of a ‘failure’ is clear (i.e. a misclassification).
The three classification tasks span 3 modalities (text, images, speech), and each task is associated with 4 datasets. However, we exclude one of these datasets: the COMMAND dataset has duplicate example IDs with differing predictions from the same ML API provider in the same time period. This makes calculating ecosystem-level outcomes impossible. After excluding the COMMAND dataset, there are 11 datasets that we conduct ecosystem-level analysis on: RAFDB [Li et al., 2017], AFNET [Mollahosseini et al., 2019], EXPW [Zhang et al., 2016], FER+ [Barsoum et al., 2016], FLUENT [Lugosch et al., 2019], DIGIT [Jackson et al., 2018], AMNIST [Becker et al., 2018], SHOP, YELP, IMDB [Maas et al., 2011] and WAIMAI.
Each dataset contains predictions from 3 commercial ML APIs in 2020, 2021, and 2022; however the HAPI API did not return predictions from the Face++ model on AFNET in 2022, so we use 2021 predictions for AFNET when conducting experiments that only use predictions from a single year.
The HAPI dataset is distributed at https://github.com/lchen001/HAPI under Apache License 2.0.
B.2 Leader Following Effects in Systemic Failure
One consequence of homogeneous outcomes is that it concentrates failures on the same users, so a user who interacts with a model and experiences a failure from that model is now more likely to experience a failure from every other model in the ecosystem. To quantify the strength of this effect, we examine how the probability of a user experiencing each outcome profile changes after that user experiences one failure from a model.
In Figure 12, we find that, consistent with the observed homogeneous outcomes in HAPI, observing a single model failure significantly increases the probability that the user will now experience failures from all other models in the ecosystem. However, we also find that the strength of this effect is strongly graded by the accuracy of the model for which we initially observe a failure. Upon observing the most accurate model in the ecosystem fail for a user, the probability of that user experiencing a systemic failure increases by  $37\%$  – whereas it only increases by around  $16\%$  when we observe the least accurate model fail.
$^{6}$ https://github.com/SophonPlus/ChineseNlpCorpus/tree/master/datasets/online Shopping_10 Cats  
 $^{7}$ https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset  
 $^{8}$ https://github.com/SophonPlus/ChineseNlpCorpus/tree/master/datasets/waimai_10k
Figure 13: We replicate the graph from Figure 3 but with various thresholds for how much a model must improve for us to include it in the analysis. The patterns discussed in §4 are consistent across choice of threshold.
This result suggests that instances that the most accurate model fails on are likely to be failed by the less accurate models as well. This 'leader following' phenomenon has implications for users in a model ecosystem: users failed by the most accurate model likely have few options for alternative models that could work for them.
B.3 Model improvement analysis is insensitive to threshold
In §4, we study how the improvement of a single model, in the sense that it becomes more impact, manifests at the ecosystem-level. To define improvement, we set the (slightly arbitrary) threshold that the model's accuracy improve by  $0.5\%$ , which we found to be large enough to be material to a model's performance while small enough to capture most model improvements. In Figure 13, we plot the outcome profile distribution in the observed improvements set against the distribution over the potential improvements (as in Figure 3b) for 6 different thresholds of change. Across all thresholds, the patterns we discuss in §4 hold: namely, models consistently under-improve on systemic failures. This confirms that the findings and qualitative understanding we present is not particularly sensitive to the exact value of this threshold.
B.4 Net Improvements
In §4, we define improvements as the specific instances that  $h_{\mathrm{imp}}$  misclassifies in the first year and correctly classifies in the second year. However, the size of this set — hereafter, gross improvements — is always larger than the number of net improvements the model makes because updates to the model tend to improve on some instances and regress on others. That is, there are two competing forces when models change: the instances the model flips from incorrect to correct, but also the instances the models from correct to incorrect. The difference of (i) and (ii) is the number of net improvements.
In Figure 14, we use net improvements instead of gross improvements, replicating the plot from in Figure 3. To calculate the outcome profile distribution over net improvements, we subtract the number of gross declines from the number of gross improvements for each outcome profile: the denominator is the number of gross improvements minus the number of gross declines across all outcome profiles.
(a) Outcome profile distribution using net improvements instead of gross improvements for (Baidu, Google) when Amazon improves on WAIMAI from 2020 to 2021.
(b) The distribution of outcome profiles for net improvements instead of gross improvements for all year-over-year model improvements across all datasets.
Figure 14: We replicate the graphs in Figure 3 but using 'net improvements' instead of 'gross improvements'. The overall trends are consistent between the two experiments: namely,  $h_{\mathrm{imp}}$  under-improves on systemic failures.
The trends are generally consistent with what we observed when using gross improvements:  $h_{\mathrm{imp}}$  tends to make little progress on systemic failures. In fact, when considering net improvements, models make even less improvement on systemic failures than before. We highlight WAIMAI as a striking case study: there is no net improvements on systemic failures, despite a  $2.5\%$  decrease in model error at the population level.
Overall, we emphasize that we recommend future research conducts analyses with both notions of improvements. While we expect in many cases, as we have seen here, that the qualitative trends will be similar, the interpretations may differ. For example, gross improvements more directly attend to the concern that there are some individuals who, year-over-year, continue to be failed by some or all models in the system. In contrast, net improvements more directly matches the sense in which models are improving.
B.5 When Models Get Worse
As a related question to what we examine in Figure 3b, we examine what happens to ecosystem-level outcomes when a model gets worse. We find that, when models get worse, they disproportionately introduce new systemic failures into the system by 'over-declining' on instances that other models were already failing for. This further highlights how single-model measurements often fails to align with ecosystem-level outcomes.
C Dermatology Experiments
We work with DDI (Diverse Dermatology Images), a dataset introduced by Daneshjou et al. [2022], which contains predictions from 3 models and 2 board-certified dermatologists on 656 skin lesion images; the task is to predict whether a lesion is malignant or benign. The ground truth label comes from an external-source: in this case, a biopsy of the lesion, which is considered the gold-standard labeling procedure in this domain.
The 3 evaluated models include ModelDerm [Han et al., 2020], a publicly available ML API, and two models from the academic literature - DeepDerm [Esteva et al., 2017] and HAM10k [Tschandl et al., 2018] - that were chosen by Daneshjou et al. [2022] on the basis of their "popularity, availability, and previous demonstrations of state-of-the-art performance." Note that, in this case, none of the models have been trained on any portion of the DDI dataset; the entire dataset serves as a test set.
Figure 15: The distribution of outcome profiles for all year-over- year model declines across all datasets
In addition, each image is annotated with skintone metadata using the Fitzpatrick scale according to one of three categories: Fitzpatrick I & II (light skin), Fitzpatrick III & IV (medium skin), and Fitzpatrick V & VI (dark skin). For all instances, the Fitzpatrick classification was determined using consensus review of two board-certified dermatologists. Additionally, a separate group of dermatologists rated the image quality of each image and discarded any low quality images; there was no significant difference in image quality ratings between images of different FST classifications.
Data on model and dermatologist predictions was graciously provided by Daneshjou et al. [2022], subject to the terms of their standard research use agreement described in https://ddi-dataset.github.io/index.html#access.
C.2 Analyses are insensitive to including/excluding HAM10k
In §5, we don't include predictions from Ham10K because the model predicts a negative on almost all instances: it has a precision of 0.99 but a recall of 0.06.
We decided to remove HAM10k because the pattern of near-universal negative predictions does not reflect model behavior we would expect of models deployed in clinical settings. Namely, even if deployed, the structure of the model errors are not particularly interesting and are largely predictable (in direction). Beyond these fundamental reasons for excluding the model, we also removed the model for reasons unique to our analysis. Including the model would have introduced an explicit class correlation in systemic failures (i.e. almost all systemic failures would be malignant instances and none would be benign instances) and would have complicated the comparison with humans since there would be three models but only two humans.
To confirm that our findings hold independent of this choice, in Figure 16, we replicate Figure 4 and Figure 5 but with outcomes from HAM10k included. We find that inclusion of HAM10k exacerbates the homogeneous outcomes of model outcomes and the racial disparities in model outcomes.
(a) Model outcomes when including HAM10K yield even more homogenous outcomes on DDI than in Figure 4
(b) The inclusion of HAM10K yields more pronounced racial disparities than in Figure 5.
Figure 16: We replicate the graphs in Figure 4 and Figure 5 but with the inclusion of HAM10K. Homogeneous outcomes and racial disparities in models are even more pronounced when including HAM10k.