SlowGuess/NeurIPS_2023_OCR huggingface dataset:
specially among the factors most associated with model mistakes. While texture is known to be a bias specifically for convolutional-models [Geirhos et al., Hermann et al., 2020], we find regardless of architecture or training procedure models have similar vulnerabilities.
5.2 Improving performance disparities by mitigating texture bias
Our analysis in Section 4.3 reveals texture is most associated with model's performance discrepancy across incomes and geographies. Can we improve this performance disparity by mitigating models' reliance on texture?
To assess this, we compare in Table 2 the performance of a standard ResNet-50 trained on ImageNet-1k compared to a ResNet-50 trained to mitigate texture bias [Geirhos et al.]. Since these are trained on ImageNet-1k (1 million images) rather than ImageNet-21k (14 million images), the overall performance is lower than other models we studied earlier (see Appendix A.5). Controlling for pretraining data, we observe a boost of  $+0.8\%$  in overall top-5 accuracy for the model mitigating texture bias. On the relevant
Top-5 Accuracy	Overall	Images Marked with Texture
low income	Africa	Asia
ResNet-50	32.2	25.2	18.6	24.4
Texture debiased	33.0	29.3	21.6	25.0
Table 2: Texture debiasing [Geirhos et al.] can improve performance across low income buckets and regions with lower performance for images marked with texture as a distinctive factor.
subset of images (those marked with texture as a distinctive factor), we find consistent improvements in accuracy for the low income bucket  $+4.1\%$  and in lower performing regions (Africa  $+3.0\%$  and Asia  $+0.6\%$ ). This suggests that factor labels do not only explain model mistakes, but can also reveal potential mitigations to combat performance disparities.
6 How do objects vary across incomes and geographies?
Leveraging the factors gathered in our annotation process, we can quantitatively assess how classes differ across incomes and regions. For example, we can identify classes that change the most across incomes (e.g. ceiling), as well as the corresponding factors that best differentiated high or low income ceilings (pose, subcategory, and texture).
To compare how classes differ across two regions (or incomes), we compute the distribution of factors for each class. Specifically, for every pair of regions (or incomes) we normalize the distributions per class then measure the Jensen-Shannon Distance (JSD) to characterize how images differ across regions (or incomes). Note the Jensen-Shannon Distance is the square root of Jensen-Shannon divergence between two distributions, and is a standard metric to compare discrete distributions [Endres and Schindelin, 2003]. A large JSD distance between two regions for a class indicates images differ across those regions.
Some classes' factors vary significantly across incomes and regions; others remain consistent. In Table 7a, we show the most starkly different classes across incomes/regions, along with their most distinguishing factors. Consistently, we find the largest differences are between low and high incomes, but don't find a consistent pattern with regions. For classes with the largest differences across regions/incomes, we find these differences to be significant with an average JSD more than twice the median JSD of all classes (across incomes/regions). Among classes differing most across regions are those relating to animals (chickens, pet foods); those differing most across incomes relate to building structures (roofs, ceiling, floors). We report full tables for the most similar and dissimilar classes across regions and incomes in Appendix A.2. In contrast, other classes don't vary across incomes/regions such as vegetable pots, phones, and pens as shown in Figure 7b. This suggests that while some classes can vary drastically across incomes or regions, others are quite similar.
Rojas et al. [2022], Singh et al. [2022], Goyal et al., 2022] have used Dollar Street to understand disparities in model performance between geographic and income groups, finding that many models
class	income bucket/region	differentiating factors
Most stark difference by income
roofs	low vs high	subcategory, pose, smaller
ceilings	low vs high	pose, subcategory, texture
diapers	low vs high	color, shape, texture
radios	low vs high	color, shape, subcategory
floors	low vs high	texture, pose, pattern
Most stark difference by region
chickens	Asia vs Europe	partial view, color, shape
chickens	Europe vs Africa	pose, partial view, color
diapers	Americas vs Africa	pose, color, partial view
pet foods	Asia vs Americas	color, texture, pattern
pet foods	Asia vs Europe	pattern, subcategory, color
(b)
Figure 7: Classes with most stark differences in factor distributions by Jensen-Shannon Distance (JSD) across incomes/regions are listed in 7a. Examples for roofs, the class with the largest JSD across incomes, and vegetable plots, the class with the smallest JSD across income are shown in 7b.
perform better on images from Europe and the Americas, as well as those from higher household incomes. While many of the aforementioned works focus on how model architectures affect disparity findings, additional studies [De Vries et al., 2019, Shankar et al., 2017] investigate the dataset itself to identify causes of variation in model performance, including the broader geographical distribution of images as compared to the model's training data. A number of datasets [Ramaswamy et al., 2023, Dubey et al., 2021, Kim et al., 2021] and dataset auditing tools [Wang et al., 2022] have since been developed with geographic diversity in mind. De Vries et al. [2019] investigated the use of English as a "base language" for data collection. Empirical studies have shown that the "concreteness" of English words can vary greatly where crowd-sourced annotators consider words like "human" and "bobsled" more concrete than words like "recreation" and "outage" [Brysbaert et al., 2014], and previous discussions of "class label perceptions" distinguish physical properties of a substance (such as orientation or texture) from a purpose relative to the specific being that interacts with the substance, such as being "sit-on-able" [Gibson, 1979]. The relationship between the true meaning of a concept versus its perceptible form remains contested for both models [Bender and Koller, 2020] and humans [Phillips, 2019]. Beyond Dollar Street, other works study variations in representations of concepts across visual factors including pose, background, occlusion, etc. within ImageNet [Idrissi et al., 2022] and collect supplementary multi-class labels [Yun et al., 2021, Beyer et al., 2020, Shankar et al., 2020]. Barbu et al. [2019] created a benchmark to measure a model's robustness to backgrounds, rotations, and viewpoints.
In this work, we take a step towards explaining why disparities in object-recognition systems arise. We annotate images from the Dollar Street dataset with distinguishing factors in order to explain how objects differ across incomes and geographies. Using these labels, we identify vulnerabilities in CLIP, a foundation model with impressive zero-shot classification performance. We find disparities in model performance are associated with texture, occlusion, and darker lighting. Finally, we surface initial promising mitigations such as texture debiasing that can improve performance disparities. This shines light on a promising research direction leveraging techniques in robustness for fairness gains. In future work, we plan to explore further targeted mitigations can improve performance disparities in vision systems. While our conclusions are limited by the number of samples and representative diversity of the Dollar Street dataset, we hope by releasing our factor annotations we spur further research into equitable vision systems.
8.1 Limitations
While grounding our new annotations in the widely used Dollar Street dataset allows for easy adopting and extensibility of existing analyses, we are also exposed to limitations of the underlying dataset. For example, Dollar Street contains imbalances across object classes and some subjectivity in class definitions across groups (e.g. most loved items, nicest shoes). We attempt to address these gaps by filtering out classes with few samples and using only classes with an ImageNet mapping.
In addition, annotators from different regions may have different perceptions of what constitutes each image factor and object class. We address this by providing annotators with examples of image factors and conducting an additional round of annotation focused on understanding regional variations in rater perceptions (see Section 2.3 and Appendix A.1.4). However, there may be lingering discrepancies left undiscovered.
Furthermore, the factors we annotate (e.g. color, shape, pose) are only a subset of factors that can relate to model performance. While these factors are relevant to standard computer vision benchmarks [Idrissi et al., 2022], additional factors may prove insightful, especially for domain specific-tasks. For example, annotations of foliage and shadows may be useful for auditing autonomous driving models.
Finally, our analyses are only a first attempt at explaining discrepancies and performing mitigations. We hope that our rich annotations and dashboard for dataset understanding enable future studies into evaluations and mitigations of geographic disparities.
Detailed income calculations for dollar street. https://drive.google.com/file/d/0B0HB08a-a9MbZFJZMTFEUkx0RWc/view?resourcekey=0-XFuTX8d2FyhmrdHMp35LDA. Accessed: 2023-01-24.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/97af07a14cacba681feacf3012730892-Paper.pdf.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.463. URL https://aclanthology.org/2020.acl-main.463.
Lucas Beyer, Olivier J. Henaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we done with imagenet? CoRR, abs/2006.07159, 2020. URL https://arxiv.org/abs/2006.07159.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911, 2014.
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 52-59, 2019.
Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recognition work for everyone?, 2019.
William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=qnfYsave0U4.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536, 2017.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on ImageNet. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8634-8644. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/shankar20c.html.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dólár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 804-814, June 2022.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021.
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision, pages 1-21, 2022.
Xiang Xu, Wan-Quan Liu, and Ling Li. Low resolution face recognition in surveillance systems. Journal of Computer and Communications, 2:70-77, 2014.
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. CoRR, abs/2101.05022, 2021. URL https://arxiv.org/abs/2101.05022.
A.1 Annotating Dollar Street with factor labels
A.1.1 DollarStreet Statistics
Region	Income Level
low	medium	high
Africa	2141	1443	280
Asia	1362	8673	1424
Europe	0	1443	1455
The Americas	339	2093	1223
Table 3: Number of images for each region, income level pair in Dollar Street.
Table 3 shows the number of images in Dollar Street for each income and region pairing. We observe the distribution across images and regions is far from uniform, implying region and income distributions skew of counts are entangled. Consequently, we present both region and income comparisons where appropriate in our analysis.
A.1.2 Prototypical Image Selection
We define prototypical images for each class as those correctly classified by ResNet-50 model with the highest confidence. We use a ResNet-50 model pre-trained on ImageNet21k from Ridnik et al. [2021]. We select the ImageNet classes that overlap with Dollar Street labels, using the mapping as defined in [Goyal et al., 2022]. We use a soft-max over the sub-section of ImageNet classes that are in the mapping. We take the top predictions and confidence for these ImageNet classes and use the defined mapping from IN21k to Dollar Street in order to make DollarStreet class predictions. Out of the box, the model does not perform well on DollarStreet. Running a full pass over the dataset with Batch Norm in train mode, without any updates to the model weights, helps with the distribution shift from ImageNet to DollarStreet images, meaning overall accuracy is higher.
We select the three images that the model predicts successfully with the highest confidence. If such images do not exist, prototypical images are hand-selected. Table 4 shows the prototypical images used for five classes.
A.1.3 Annotation Setup
Figure 8: Example annotation task.
Class	Prototypical Images
grains
plates
power outlets
cleaning floors
toothbrushes
Figure 8 shows an example of the annotation task. Annotators select the factors distinguishing each image among sixteen factors such as pose, various forms of occlusion, size, style, type or breed. Annotators can select any number of distinctive factors for each image. We source 10 annotators through a third party vendor from South East Asia. In addition, we ask annotators to provide text descriptions to account for factors outside the sixteen we provide. We trained annotators with examples so that they were familiar with the task before annotating the target images. We had intermediate QA from the third party vendor monitoring annotations for quality. We also ask annotators whether they agree with the original class label for each image.
A.1.4 Label Agreement Annotation Setup
Table 4: Prototypical images used for five classes.
Country	Number of annotators
India	8
Nigeria	9
Brazil	13
United Arab Emirates	6
United States	8
Table 5: Annotator breakdown for label agreement task.
For our follow up annotations about label agreement, we sourced 44 annotators from 5 different countries, with the full demographics shown in Table 5. We asked one annotator per country about each image in question. In Table 9, we show example images from the three most disputed classes, along with alternative labels suggested by annotators. In Table 6, we show the classes with the highest and lowest levels of disagreement among annotators.
Refrigerator
Cooler
Toilet Paper
Bucket
Toilet system
Water Tub
Faucet, bucket
Toothbrushes
Stick
Figure 9: Randomly sampled example images and alternative labels given for the three classes with most disagreement. The original class label is shown on the left, and the alternative label given by the annotator shown below each image.
Small container
class	% disagreement
toilet paper	88.4
refrigerators	83.5
toothbrushes	79.3
sofas	77.8
diapers	72.0
armchairs	70.6
showers	66.3
kitchen sinks	64.5
wall clocks	63.2
radios	60.4
class	% disagreement
medication	10.0
fruit trees	11.8
plates of food	12.3
trash	14.3
cleaning floors	14.5
ceilings	15.0
homes	15.0
books	15.0
cooking pots	19.8
wheel barrows	20.0
Table 6: Top ten classes with the highest percentage of annotators who disagreed (left) and agreed (right) with the original class label
A.2 How do objects vary across incomes and geographies?
We show the most dissimilar classes across incomes and regions by comparing the Jensen-Shannon Distance of the factor annotation distributions in Tables 7 and 8.
class	income bucket	differentiating factors
roofs	low vs. high	subcategory, pose, smaller
ceilings	low vs. high	pose, subcategory, texture
diapers	low vs. high	color, shape, texture
radios	low vs. high	color, shape, subcategory
floors	low vs. high	texture, pose, pattern
sofas	low vs. high	color, texture, multiple objects
kitchen sinks	low vs. high	shape, pose, background
toilet paper	low vs. high	color, pose, background
wardrobes	low vs. high	background, pattern, color
mosquito protections	low vs. high	color, subcategory, pattern
Table 7: Classes with most stark differences in factor distributions by Jensen-Shannon Distance (JSD) across incomes.
class	regions	distinctive factors
chickens	Asia vs. Europe	partial view, color, shape
chickens	Europe vs. Africa	pose, partial view, color
diapers	The Americas vs. Africa	pose, color, partial view
pet foods	Asia vs. The Americas	color, texture, pattern
pet foods	Asia vs. Europe	pattern, subcategory, color
ceilings	Europe vs. Africa	pose, subcategory, texture
roofs	Europe vs. Africa	subcategory, pose, texture
car keys	Asia vs. Europe	pattern, partial view, subcategory
make up	Europe vs. Africa	background, subcategory, pattern
goats	Asia vs. Africa	pattern, color, subcategory
We show the most similar classes across incomes and regions using the same procedure of comparing Jensen-Shannon Distance of the factor annotation distributions in Tables 9 and 10.
Table 8: Classes with most stark differences in factor distributions by Jensen-Shannon Distance (JSD) across regions
class	income buckets	distinctive factors
vegetable plots	low vs. high	multiple objects, background, color
phones	medium vs. high	background, pose, multiple objects
pens	medium vs. high	color, background, pattern
bikes	low vs. high	background, subcategory, smaller
armchairs	medium vs. high	color, background, pose
latest furniture bought	medium vs. high	subcategory, background, color
child rooms	medium vs. high	pose, pattern, color
wall clocks	medium vs. high	color, pose, shape
cooking utensils	medium vs. high	pose, shape, pattern
Table 9: Classes most similar in factor distributions by Jensen Shannon Distance across incomes
class	regions	distinctive factors
vegetable plots	The Americas vs. Africa	pose, background, pattern
phones	Asia vs. Europe	pose, background, color
pens	Europe vs. Africa	pose, color, pattern
wheel barrows	Europe vs. The Americas	color, pose, background
ceilings	Asia vs. Africa	subcategory, pattern, texture
pets	Asia vs. Europe	background, pattern, subcategory
stoves	Asia vs. Africa	subcategory, color, pattern
menstruation pads	Asia vs. The Americas	pose, subcategory, pattern
tvs	Europe vs. The Americas	partial view, subcategory, background
everyday shoes	Europe vs. The Americas	color, partial view, shape
Table 10: Classes most similar in factor distributions by Jensen Shannon Distance (JSD) across regions
A.3 Evaluation Setup
CLIP Prompt Engineering We use CLIP in a zero shot setting, where we prompt the model using the set of Dollar Street classes (e.g. medication, plates of food) for each image to generate predictions. We generate the text prompts for CLIP by combining the 80 prompt templates used in the original CLIP paper with each Dollar Street class name, substituting _ for spaces. We consider an image correctly predicted if the top 5 classes predicted by CLIP is associated with the photo. Note: Most photos in DollarStreet have only one label, but a small subset of (638) images containing multiple class labels (e.g. (cups, plates, dish racks) and (child rooms, kids bed, beds)).
ImageNet21k as a shared taxonomy For models outside of CLIP, we use ImageNet21k to ground our models in a shared taxonomy. Following Goyal et al. [2022], we map the ImageNet21k labels to DollarStreet classes. We consider the image correctly classified if any of the top 5 ImageNet21k classes predicted by the model are mapped to any of the DollarStreet classes associated with the photo. We note that the mapping is not 1:1, and multiple classes in DollarStreet have multiple classes in ImageNet 21k that map to the single class. All of the models used for evaluation excluding CLIP and SEER are trained on ImageNet 21k. SEER is pre-trained in a self-supervised manner, and the model is fine-tuned on the 108 classes in ImageNet 21k that overlap with DollarStreet prior to evaluation. For ImageNet-21k pretraining, we use models from Ridnik et al. [2021].
Figure 10: 10 classes with biggest performance discrepancy over regions (left) and income bucket (right).
Class level performance disparities Figure 10 shows the top 10 classes with the biggest performance disparity between groups for regions and incomes. We define the largest performance discrepancy as the maximum difference in accuracy between any two regions (or income buckets). At a class level, we find that the discrepancy in accuracy can be stark - over  $50\%$  for the classes with the widest gap. For both incomes and geographies, we find that the differences mostly pertain to
items in kitchens (dish racks, kitchen sinks) and items in bathrooms (showers, shaving, toilet paper, bathrooms).
A.4 Explaining model performance disparities with factor labels
As part of our analysis of model performance disparities, we investigate the impact of pretraining class balance and image quality. In Table 11, we show the Pearson correlation coefficients and p-values between each model's top-5 accuracy and the Image DPI, a measure of image resolution. In Table 12, we show the Pearson correlation coefficients and p-values between each model's top-5 accuracy and the ImageNet-21K class count. We excluded CLIP from this analysis as CLIP was trained on a proprietary dataset.
Model	Correlation, Top 5 Accuracy and Image DPI
ViT	-0.019 (p = 0.035)
ResNet50	-0.023 (p = 0.008)
MLPMixer	-0.026 (p = 0.002)
BeIT	-0.003 (p = 0.72)
SEER	-0.016 (p = 0.057)
CLIP	-0.035 (p = 0.00005)
Table 11: Pearson Correlation coefficients and p-values between each model's top-5 accuracy and image quality, as measured by DPI.
Model	Correlation, Top-5 Accuracy and Class Count
ViT	0.126 (p < 0.0001)
ResNet50	0.142 (p < 0.0001)
MLPMixer	0.135 (p < 0.0001)
BeIT	0.222 (p < 0.0001)
SEER	0.103 (p < 0.0001)
Table 12: Pearson Correlation coefficients and p-values between each model's top-5 accuracy and ImageNet-21K class counts. CLIP is not included, as it was trained on a proprietary dataset.