SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Laura Gustafson, Megan Richards, Melissa Hall, Caner Hazirbas, Diane Bouchacourt, Mark Ibrahim
Fundamental AI Research (FAIR), Meta
Despite impressive advances in object-recognition, deep learning systems' performance degrades significantly across geographies and lower income levels—raising pressing concerns of inequity. Addressing such performance gaps remains a challenge, as little is understood about why performance degrades across incomes or geographies. We take a step in this direction by annotating images from Dollar Street, a popular benchmark of geographically and economically diverse images, labeling each image with factors such as color, shape, and background. These annotations unlock a new granular view into how objects differ across incomes/regions. We then use these object differences to pinpoint model vulnerabilities across incomes and regions. We study a range of modern vision models, finding that performance disparities are most associated with differences in texture, occlusion, and images with darker lighting. We illustrate how insights from our factor labels can surface mitigations to improve models' performance disparities. As an example, we show that mitigating a model's vulnerability to texture can improve performance on the lower income level. We release all the factor annotations along with an interactive dashboard to facilitate research into more equitable vision systems.
The widespread adoption of object-recognition systems afforded by advances in deep learning comes with a responsibility: systems should work equally well across groups of individuals. Previous work demonstrates object-recognition performance is far from equal across income levels and geographies [De Vries et al., 2019, Goyal et al., 2022, Rojas et al., 2022]. This disparity encompasses publicly available recognition systems, state-of-the-art supervised and self-supervised models. Most worrisome among these findings is that the performance degradation disproportionately affects lower income households. When Artificial Intelligence (AI) systems are deployed in applications such as medical imaging, their biases can lead to disproportional harm. For example, models diagnosing COVID-19 were found to rely on geographically-biased features such as the hospital's font to diagnose patients [Roberts et al., 2021].
While existing work measures performance disparities across incomes and geographies, addressing the performance gaps remains a challenge. Key to progress is understanding not just that, but why such disparities arise. One hypothesis raised in DeVries et al. [2019] is that objects as well as their environments can vary drastically across regions. When factors such as object shape or lighting in a region differ from those commonly seen during training, the shift can cause model performance to drop. However, no systematic study exists characterizing how such factors vary across regions and incomes. Identifying the factors associated with model disparities can shed light on research directions to improve performance degradation across incomes and geographies.
We take a step in this direction by annotating images from Dollar Street [Rojas et al., 2022], the most common benchmark for evaluating performance disparities in object recognition systems. Dollar Street contains 38k images of household objects spanning 54 countries across income levels. We annotate each image with factors to mark what makes each distinctive, such as color, pose, shape, and texture. We first analyze how images vary across incomes and regions using our factor labels in Section 6. We find images of some classes such as roofs differ considerably across regions (and incomes) while others (such as pens) hardly vary.
We then investigate how our factor labels can explain model mistakes. We find an overall correspondence between the distribution of factors per region (and income) and model performance. Even for the latest generation of foundation models, such as CLIP [Radford et al.], performance
degrades by as much as  $25.7\%$  (top-5 accuracy) across incomes. We also compare the performance of other popular models across learning paradigms (self-supervised, supervised), architectures (CNN, transformer-, MLP-based), as well as large scale pretraining [Goyal et al.]. We find remarkably similar vulnerabilities across these popular models.
Next, we precisely rank factors by examining how much more likely they are to appear among misclassifications. A factor much more likely to appear among misclassification suggests a model is vulnerable to the factor. In our analysis we find vulnerabilities in texture, occlusion, and darker lighting are most associated with models' performance degradation in lower incomes, Figure 1. We further study class-specific model vulnerabilities, finding strong associations between mistakes for particular classes and factors. For example, we find that for sofas, images labeled with texture are  $7.2\mathrm{x}$  more likely to appear among CLIP's mistakes than overall, suggesting texture bias is a vulnerability.
Finally, we study whether we can use robustness techniques to make fairness improvements. We show that mitigating the texture vulnerability surfaced by our analysis can improve performance disparities across incomes/regions in Section 5.2. We find a model trained to mitigate texture bias not only performs better overall  $+0.8\%$  (top-5 accuracy), but on the relevant subset of images (those marked with texture), improves accuracy by  $4.1\%$  for low incomes. This suggests factor labels not only explain mistakes, but can even reveal promising mitigations to combat the disparities we observe in vision models today. Along with our analysis, we release all the factor annotations with an interactive dashboard to enable research facilitating more responsible, equitable vision system.
To summarize, our contributions are 1) we annotate all of Dollar Street images with distinctive factor labels such as pose, background, and color, 2) we explain performance disparities in models (including CLIP) using our factor annotations to reveal vulnerabilities in texture, occlusion, and darker lighting, 3) we demonstrate mitigating the vulnerability to texture can improve performance disparities across incomes and geographies, 4) we release all our factor annotations with a dashboard (Figure 2) allowing researchers to interactively query and visualize image factor labels to spur research into equitable vision systems.
Model vulnerabilities explain performance drop
Figure 1: CLIP's vulnerability to texture, darker lighting, and occlusion are associated with performance disparities for lower incomes. We rank the most vulnerable factors based on how much more likely a factor is selected among misclassified images than overall. The example images are of dishbrushes from Dollar Street.
2 Annotating Dollar Street with factor labels
The Dollar Street dataset is the most common computer vision benchmark for classifying everyday objects (e.g armchairs, pens) across incomes and geographies. Households across the world upload images of the specified objects. These images are labeled with the object class, location, and income of the household. The income is standardized on an international scale by DollarStreet [DSI]. We use the procedure described in [Goyal et al., 2022] to aggregate household incomes into buckets: high, medium and low, and group countries into regions: Asia, Africa, Europe, The Americas. Table 3 in Appendix A.1.1 shows the number of images per income bucket/region pair.
2.1 Annotation Procedure
In order to explain the degradation in model performance across incomes or geographies, annotators labeled images in Dollar Street with the factors distinguishing each image. We select all 14k images overlapping with classes in the ImageNet-21k taxonomy from Ridnik et al. [2021], using the mapping from DollarStreet classes to ImageNet synsets from Goyal et al. [2022]. We follow the same annotation procedure as in Idrissi et al. [2022]. Since it's challenging to accurately label an image in isolation, we ask annotators to label how each image differs from a fixed set of three prototypical images chosen for each class. We define prototypical images for each class as those correctly classified by a ResNet-50 model with the highest confidence. We curate a list of sixteen potential factors that can distinguish an image from the prototypical images for its class. These factors include pose, various forms of occlusion, size, style, type or breed capturing common variations in images. Specifically, this set of sixteen factors has been shown to comprehensively cover most distinctive image factors via user studies comparing the sixteen factors against free-form text responses in Idrissi et al. [2022]. A full list is shown in Figure 2. Annotators select any number of factors they believe best distinguish each image. In addition, we ask annotators to provide text descriptions to account for factors outside the sixteen factors we provide, and ask if they agree with the original class label (see 2.3 for analysis of this). A more detailed description of the annotation setup and prototypical images is in Appendix A.1.
2.2 Factor label statistics
We first explore how frequently each factor was selected across income levels and regions. In Figure 2, we plot the distribution of factor labels across regions and income buckets. On average, annotators chose 3.2 factors per image (standard deviation 1.2). The two most correlated factors are color and pattern, with a correlation coefficient of 0.19.
Across all incomes, pose, background, and pattern are the most selected factors. For most factors, there is only a minor difference in the frequency that a factor was selected across income buckets. For texture, however, there's a noteworthy difference across income levels with  $10.2\%$  of images in the low income bucket labeled with texture compared to only  $4.2\%$  for medium and  $2.0\%$  for high income buckets. This implies texture is 5x more likely to be selected within the low income bucket (relative to the high income bucket), a stark difference.
The most commonly selected factors are consistent across regions and incomes. Similar to our observation across income buckets, the most striking difference across regions is for texture. Texture is selected for  $7.8\%$  of images in Africa but for only  $2\%$  of images in Europe—a 4x difference. The similarity of emerging patterns in factors across incomes to those across regions suggests that income bucket variation differences are also exhibited across geographies. This can in part be explained by the relative rates of co-occurrence of regions and income buckets in DollarStreet, see Table 3 in Appendix A.1.1.
2.3 Controlling for regional differences in raters' perceptions
Challenges naturally arise when running such a large annotation procedure. In our case, there can exist regional perceptions of the semantic meaning of every object label. Indeed, the Dollar Street object class labels were originally collected from the household members who took the image, rather than assigned retroactively, which means that regional perceptions could be a source of variance in the dataset.
Figure 2: Pose, background, and pattern are the most commonly selected factors. The left panel shows the percent of images by region and income that were labelled with each factor. Annotators labelled each image with the factors that most distinguished each image from the prototypical images of its class. On the right is a screenshot of the public interactive dashboard for the annotations
For annotating DollarStreet with factor labels, we sourced 12 annotators from Indonesia. We trained the annotators on this specific annotation task before gathering the final factor label annotations. Throughout the process, we had an auditor who reviewed the annotations and brought up ambiguities. However, when gathering the factor annotations, we found that these annotators had a number of times that they disagreed with the class label. We hypothesized this could be due to differences in regional perceptions of objects.
To control for the effect of such perceptual differences across regions, we created a second annotation task. To select the images for the task, we analyzed the results of the original factor annotations, which included an option for annotators to disagree with the object label. We selected the subset of images where the annotator disagreed with the label, which totaled 2,476 images, or  $18.1\%$  of the Dollar Street dataset. For the second task, we collected additional annotations for this disputed subset, asking a different set of annotators sourced from 6 countries (5 continents) whether they agreed or disagreed with the object label. In total, there were 79 annotators who provided 12,507 label annotations. See Appendix A.1.4 for examples of annotator disagreements, and classes with the highest and lowest levels of label disagreement.
We then compared the levels of disagreement between annotators from the region where the image was taken (source region) and annotators who were from other regions. If there were region-specific biases in the label, we would expect a much higher rate of disagreement for annotators not from the source region. For images in the second task, we found annotators from both the source and other regions disagreed with the original label at similar rates (an average of  $49.1\%$  and  $46.8\%$  respectively). This consistency suggests regional differences in label perceptions do not constitute a significant source of class variation in Dollar Street. Next, we study how today's best object recognition models perform across regions and incomes.
3 Modern models' performance degrades across incomes and geographies
Performance inequities are pervasive across architectures and training methods. We compose a study on DollarStreet encompassing models across architectures (convolutional, transformer, and feedforward), learning paradigms (self-supervised, supervised, contrastive), and pretraining datasets of various sizes (up to 1 billion images). We first study the popular foundation model CLIP, which has been shown to have strong zero-shot performance on several classification benchmarks [Radford et al.]. CLIP is trained on 400M text-image pairs using a text encoder and an image encoder enabling a user to perform zero-shot classification for any image. Here we prompt the model using the set of Dollar Street classes for each image to generate predictions. Our evaluation setup is described in detail in Appendix A.3. For the remaining models, to generate predictions on Dollar Street, the models are pretrained or finetuned on ImageNet21k and we use the same mapping from Goyal et al. [2022] and
Figure 3: Across architectures and learning procedures, model performance degrades similarly across regions (left) and incomes (right). Bars indicate top-5 accuracy.
detail our evaluation procedure in Appendix A.3. We compare the performance of the set of models across incomes and regions in Figure 3. We find most models have comparable drops in accuracy across incomes and regions despite model differences. BEiT performs worse overall, but still exhibits similar trends in performance gaps across incomes and geographies [Bao et al., 2021]. This suggests even large scale pretraining in models such as SEER [Goyal et al., 2021], modern architectures such as ViT [Ridnik et al., 2021] or MLP Mixer [Tolstikhin et al., 2021], and self-supervised learning still don't address performance inequities. Why do such consistent disparities arise? Next we study whether factors labels can explain the performance disparities in modern vision models. We focus our study to the CLIP ViT B/32 model as it has the highest performance on DollarStreet.
4 Explaining model performance disparities with factor labels
We now study how our factor labels can explain the model performance disparities we observed across regions/incomes. After ruling out variables such as image quality and class imbalance in training, we demonstrate how our factor labels can surface specific model vulnerabilities associated with degradation in performance across regions/incomes.
4.1 Controlling for factors not captured in our annotations
Performance disparities are not explained by image quality or training data class imbalance. As image quality has been shown to impact the performance of facial recognition models [Xu et al., 2014], we first investigate whether image resolutions differ across regions and effect model performance. Rojas et al. [2022] found very minor differences in average image quality across region in DollarStreet. We take this a step further and find no strong correlation ( $< 0.05$  Pearson's correlation coefficient) between image DPI and model performance (top-5 accuracy). Next, since class imbalance in training can skew model performance, we also investigate the extent to which class imbalance affects the disparities we observe. For ImageNet-21K pretrained or finetuned models (ViT, ResNet, MLPlexer, BEiT, and SEER), we calculate the Pearson correlation between number of images in each class for ImageNet-21K and model's top-5 accuracy. We found similarly weak correlations for all models, with coefficients less than 0.25 for top-5 accuracy (all values reported in A.4). These results suggest that variation in image quality and pretraining class imbalance explain very little of the variation model mistakes in Dollar Street. Next, we examine whether our factor labels can explain performance disparities.
4.2 Variation in factor labels are indicative of performance disparities
To assess whether our factor labels are indicative of model performance disparities, we measure whether larger differences in factor labels across incomes/regions correspond to larger degradations in model performance. Specifically, for each class we measure the Jensen Shannon Distances (JSD) between the factor label distributions of every pair of income buckets (and regions). This quantifies how images in a classes vary across income bucket (or region) pairs according to our factor labels.
Next, we calculate whether larger differences in images across incomes/regions correspond to larger disparities in model performance. We find as a class varies more across income pairs (according to the JSD of factor distributions), model performance gaps also increase, as shown in Figure 4. For example, classes that differ most across incomes (top quarter) suffer a  $3\mathrm{x}$  drop in accuracy compared to classes that differ the least across incomes (bottom quarter). We find a similar but less stark trend across regions shown in Figure 4. We acknowledge such an analysis of likelihoods is inherently not causal. Our results based on associations between factors and mistakes suggest our factor labels are indicative of performance disparities across regions/incomes.
4.3 Model performance disparities are most associated with texture, darker lighting, and occlusion
To more precisely assess which factors are most associated with mistakes, we use the same error ratio metric from Idrissi et al. [2022] to measure the association between each factor and model errors. Specifically, the error ratio for a factor quantifies how much more or less likely a factor is to appear among a model's misclassified samples as
$$
\frac {P (\text {f a c t o r X} \mid \text {m o d e l e r r o r s}) - P (\text {f a c t o r X})}{P (\text {f a c t o r X})} \tag {1}
$$
An error ratio greater than zero indicates how much more likely a factor is to appear among misclassified samples suggesting the factor is associated with model mistakes. For example, an error ratio of  $2\mathrm{x}$  indicates a factor is  $2\mathrm{x}$  more likely to be selected among misclassified samples than overall. An error ratio less than zero indicates a factor is less likely to appear among misclassified samples suggesting the model is robust to the factor. Since some factors are selected only for a few images, we exclude factors
selected for five or fewer images in our analysis. Doing so excludes style and brightness.
ss factor variation across
Figure 4: Differences across regions/incomes measured by our factor labels are indicative of performance disparities. The factor dissimilarity measures the distance (JSD) in factor label distributions across two regions/incomes for a specific class. The drop in accuracy measures the drop in performance for the given class across two regions/incomes.
Texture, occlusion, and darker lighting are most associated with model disparities across incomes and regions. We examine the five factors most associated with model mistakes (measured using error ratio). Overall, we find mistakes for CLIP with a ViT-B/32 encoder are most associated with texture, occlusion, or objects appearing too small as shown in Figure 5a. For example, texture appears  $+0.88\mathrm{x}$  more among CLIP's mistakes than overall. Similarly, occlusion appears  $+0.76\mathrm{x}$  and smaller  $+0.73\mathrm{x}$  more so among mistakes. We conduct  $\chi^2$ -test to verify such differences in factor prevalence are statistically significant (see Appendix A.4).
We find these vulnerabilities also explain the performance disparities we observe for low incomes and regions with lower performance (Africa). In Figure 5a we show the five factors most associated with model mistakes across incomes (and across regions in Figure 5b). We find in the low income bucket, texture has the largest error ratio with texture  $1.7\mathrm{x}$  more likely to be selected among misclassifications in the low income bucket. On the other hand, for the high income bucket misclassifications are associated with quite different factors. For example, smaller objects are most associated with mistakes in the high income bucket. We find a similar trend for across regions with texture, occlusion, and darker lighting most associated with mistakes in the region of Africa. We also detail model strengths by measuring factors that are much less likely to appear among mistakes in Appendix A.4. For example, we find CLIP is much less likely to misclassify images with partial views of objects.
4.3.1 Some model vulnerabilities are class-specific
Beyond explaining performance disparities across regions or income buckets overall, our factor labels enable us to explain vulnerabilities for specific classes. We show in Table 1 the factors most associated
Class	Region	Factors most associated with mistakes
shaving	Africa	shape (+5.8x), pattern (+0.2x), background (+0.1x)
sofas	Africa	texture (+7.2x), pattern (+0.3x), pose (+0.1x)
bathrooms	Africa	background (+1.1x), pose (+0.2x), color (+0.1x)
kitchen sinks	Africa	color (+0.6x), background (+0.2x), pose (+0.2x)
showers	Africa	background (+1.2x), pose (+0.4x), multiple objects (-1.0x)
Table 1: Class-specific vulnerabilities surfaced for the classes with lowest regional performance.
with mistakes for classes with the lowest performance across regions. We find the lowest performance is for images in the region of Africa with each class exhibiting class-specific vulnerabilities. For shaving, shape is most associated with the low performance as it's  $5.8\mathrm{x}$  more likely to appear among mistakes in Africa. For sofas, texture is  $7.2\mathrm{x}$  more likely to appear among mistakes. We find a similar pattern for classes with the low performance across incomes in Appendix A.4 and A.3. We also explore vulnerabilities at the more granular country level in Appendix A.4. These strong shifts in error ratios point to class-specific vulnerabilities.
Next we study vulnerabilities across a range of model architectures and learning procedures.
5 The effect of architecture and training procedure on mistake types
The factor labels also allow us to compare vulnerabilities across models. We first study vulnerabilities across a range of vision models then we show mitigating the most common vulnerability (to texture) can improve performance disparities across incomes and geographies.
5.1 Comparing vulnerabilities across architectures and training procedures
In Figure 6, we examine model vulnerabilities across architectures and training methodologies. We find that, for all models, texture, occlusion, and shape are con