Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models
Authors: Joan Nwatu
Oana Ignat
Rada Mihalcea
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal
Income
Geodiversity
Evaluation
Analysis
TL;DR: We evaluate the performance of a SOTA VL model, CLIP, on a geo-diverse dataset containing household images associated with different income values, DollarStreet, and show that performance inequality exists among households of different income levels.

Abstract: Despite the impressive performance of current AI models reported across various tasks, performance reports often do not include evaluations of how these models perform on the specific groups that will be impacted by these technologies. Among the minority groups under-represented in AI, data from low-income households are often overlooked in data collection and model evaluation. We evaluate the performance of a state-of-the-art vision-language model (CLIP) on a geo-diverse dataset containing household images associated with different income values (DollarStreet) and show that performance inequality exists among households of different income levels. Our results indicate that performance for the poorer groups is consistently lower than the wealthier groups across various topics and countries. We highlight insights that can help mitigate these issues and propose actionable steps for economic-level inclusive AI development.

Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models
Joan Nwatu Oana Ignat Rada Mihalcea
University of Michigan Ann Arbor, USA
[jnwatu, oignat, mihalcea) @ @umich.edu
Abstract
0.35
Despite the impressive performance of current Al models reported across various tasks, per- formance reports often do not include evalua- tions of how these models perform the spe- cific groups that will be impacted by these tech- nologies. Among the minority groups under- represented in AI, data from low-income house- holds are often overlooked in data collection and model evaluation. We evaluate the per- formance of state-of-the-ar vision- Aision-language model (CLIP) on a geo-diverse dataset contain- ing household images associated with different income values (Dollar Street) and show that per- formance inequality exists among households of different income levels. Our results indicate that performance for the poorer groups con- sistently lower than the wealthier groups across various topies and countries. We highlight in- sights that can help mitigate these issues and propose actionable steps for economic-level in- clusive Al development. Code is available at Analysis for Bridging the Digital Divide.
0.30
CLIP score
260
205
2.32
(0, 1001
(160, 2001
(200, 400]
(400, 800)
(800, 160)
(1600, 32001
(3200, 6400]
(6400, 12800)
(12800, 25600)
Income ranges (S/onth)
Figure I: Median CLIP (Radford al., 2021) alignment scores across images in Dollar Street (Rojas et al., 2022) from different income ranges, together with average CLIP scores with confidence values for each range. We measure a trend of increasing CLIP scores as the income range increases.
for the global population - consequence of train- ing AT models with data that reflects a one-sided view of the world (Buolamwini and Gebru, 2018).
1 Introduction
The rising concern about the disparate impact of AI technologies different members of the gen- eral public has led to research investigating cases where Al models do not work well for different underrepresented groups (Bolukbasi et al., 2016 Gebru, 2020; Cirillo al., 2020; Ahn al., 2022: Shrestha and Das, 2022). While there is a sizable literature studying the disparate Al impact on peo- ple of different races and genders, less emphasis is placed on investigating the interaction between Al model performance and economic inequality in the world, even research inferring economic status from images suggests that the differences between the rich and the poor can be captured by AI models (Acharya et al., 2017: Gebru et al., 2017; Yeh et al., 2020a; Machicao al., 2022).
The impact of AT on the general public is rapidly growing, now getting within reach of people world- wide. More than ever, it is critical that these mod- els work well for everyone. Language and vision models are also expanding, with some already be- ing used as foundation models (Bommasani et al., 2022), as these models are trained on enormous datasets and have been shown to possess impres- sive capabilities on downstream tasks across vari- ous domains.
However, since most research that yielded foun- dation models comes from top companies in the Western tech industry, it is unsurprising that these models tend to be biased and perform unequally
'Jan Nwatu and Oana Ignat contributed equally to the manuscript. Rada Mihaleca initiated and guided the work, and provided overall supervision.
The consequences are prominent, as neglect- ing the impact on people of different socio-

---

economic levels, is further widening the "digital divide", by excluding low-income background peo- ple from benefiting from Al applications (Lutz, 2019; Carter et al., 2020; Kitsara, 2022; Khowaja et al., 2023). As technological progress threatens to widen the economic gap between the rich and the poor, it is essential to have a clear understanding of how state-of-the-arr models perform across all income levels, help these models achieve good performance across all economic levels (Miailhe and Hodes, 2017; Khowaja et al., 2023). To address the lack of research on evaluation across economic levels, we conduct an in-depth performance eval- uation of a state-of-the-art vision-language model on images from diverse household incomes. Based on our findings, we propose a series of actionable steps to "democratize" vision-language models and ensure everyone benefits from the upcoming Al revolution.
In computer vision (CV) research, classi- fiers trained on popular datasets such as Ima- geNet (Deng et al., 2009) or Openlmages (Krasin et al., 2017) yield frequent misclassifications due to geolocation (Shankar et al., 2017). Analyses of these datasets show that 60% of the data comes from only 6 (out of 195)* $ countries, all from North America and Europe (Shankar et al., 2017). Sim- ilarly, research by De Vries et al. (2019) investi- gates how CV models perform on diverse, cross- cultural images from the Dollar Street dataset (Ro- jas et al., 2022). This work is most similar to ours; however, their focus is on object-recognitinn mod- els, while ours is on a vision-language foundation model, CLIP (Radford et al., 2021). Their main finding, four years ago, is consistent with ours: "The systems perform relatively poorty on household items that commonly occur in countries with low household income," revealing that current CV mod- els still work significantly worse for people with low-income households.
We summarize our contributions as follows. First, we demonstrate the disparate performance of a vision-language foundation model across groups of different economic levels. By formulat- ing a series of research questions, we perform fine- grained analyses to identify topies and countries that require more research attention and highlight the possible issucs causing difficulty for vision- language models. Second, we uncover visual sim- ilarity analogies across countries and incomes from diverse household appearances. Lastly, based on insights from analyses, present six ac- tionable recommendations to improve equality in vision-language model performance across different socio-economic groups
Improving Cultural Representation in AL. While transfer learning techniques (Ruder et al., 2019; Rahimi et al., 2019; Conneau et al., 2020) shine ray of hope for increasing language diver- sity, Joshi et al. (2020) challenge the optimism towards transfer learning for multilingual NLP by highlighting that many low-resource languages con- tain typological features not adequately represented in richer resource languages like English. Other efforts revolve around data collection (Alahmadi et al., 2020; Koto et al.. 2020; Augustyniak et al.. 2022; Marreddy et al., 2022), which improve data collection techniques for low-resource languages and facilitate the participation of indigenous people in data collection for NLPP research.
2 Related Work
Using pre-trained models for vision tasks is com- mon practice (Donahue et al., 2014; Girshick et al., 2014; Wang and Russakovsky, 2023). However, Salman et al. (2022) indicate that bias from pre- trained models can be transferred over to down- stream tasks even if datasets for fine-tuning are explicitly debiased. At the same time, Wang and Russakovsky (2023) show that biases due to spu- rious correlations and under-represeentaioo can be counteracted through targeted dataset manipula- tions. In cases of under-represennation, they rec- ommend adjusting the proportion of positive labels belonging to the under-represented group and note that this process involves further collecting under-
Mcasuring the disparate impact of AI across cultures. Existing research the evaluation of vision and language models has led to the discov- ery of unequal performance across various factors like gender. language. and race. Further analy- ses of these models reveal that they are mainly trained on data collected from the Western world. For instance, the concentration of NLP research on English and handful of other languages has con- tributed to the incquality in developing language technologies for multiple NLP tasks across the world's languages (Blasi et al., 2022). Current NLP models yield lower performance on language tasks for non-V non-Wester Western languages (Hu et al., 2020; Khanuja et al., 2023).
hattps://waw. wor www. dowetoom.rr...... itt.oomteerrreo......o.oo..

---

represented data and carefully curating datasets for fine-tuning, Similarly, Ramaswamy et al. (2023) show that adding geo-diverse data to the training dataset increases model performance.
tinents. The images capture everyday household items (e.g.. "toothbrush", "toilet paper", "clothes"), which are called topics. While image resolution and size vary slightly across locations, relevant met- rics such as mean and median are similar; CV model will likely not be impacted by image resolution and size. All the images in the Dol- lar Street dataset have respective houschold socio- economic information, i.e., income and location.
Work toward increasing representation in CV datasets include GeoDE (Ramaswamy et al., 2023) one of the most geo-diverse datasets, GeoYFCC (Dubey et al., 2021) less diverse, with data mainly from Europe, Segment Anything (Kir- illov al., 2023) a large geo-diverse segmenta- tion dataset, and Dollar Street dataset (Rojas et al., 2022) which we use in this project because it con- tains income information and everyday human ac- tions and objects.
therefore,
Topic Representation. Each image is manually annotated with one or more related, textual top- ies: e.g., "adding spices to food while cooking", "spices". There are 291 unique topics, out of which we remove nineteen subjective topics following the work of De Vries et al. (2019) (e.g., "most loved item", "things wish I had"). All the subjective topics are found in Appendix A.
Using Images to Infer Economic Information. Research work making use of image data iden- tify the income of households or neighborhoods in the US includes Acharya et al. (2017) and Gebru et al. (2017), who identify low-income neighbor- hoods using Google Street View images. S They demonstrate the use of machine learning to predict the economic welfare of households as poten- tial alternative to traditional methods like wealth surveys. Xie et al. (2016) and Yeh al. (2020b) also use deep learning to detect predictive wealth features in the day and nighttime satellite imagery. Following previous work in the social sciences (Rosling et al., 2019) that demonstrated how in- come cuts across cultures/countries, the creators of the Dollar Street dataset point to how house- hold appearances seem to differ especially across income groups and not necessarily across countries, commonly believed. S
Income Representation. The Dollar Street dataset contains images from homes with monthly incomes ranging from 26.98 to 19, 671.08. The household income is calculated as consumption over an extended period (a year), expressed per adult equivalent, using the OECD (Organisation for Economic Co-operation and Development) modi- fied scale S, then further divided and displayed reflect monthly consumption. This number is de- rived from the houschold's self-reported consump- tion and income levels. The total consumption is measured in U.S. dollars, adjusted for purchasing power parity, to account for the varied cost of liv- ing among different countries. Further information regarding the calculations can be found on the Gap- minder website.
3 Methodology
We further group the income values into geomet- ric ranges and quartiles, as described in Rojas et al. (2022). The quartile binning method, in Table 1, divides the distribution of images into approxi- mately equal number of images per bin, allowing for fair comparisons between the bins.
We present the dataset, Dollar Street (Rojas et al., 2022) and the vision-language state-of-the- art model, CLIP (Radford et al., 2021), which we use for our experiments. By formulating and answer- ing series of research questions, we perform fine-grained performance analysis of CLIP on a socio-economically diverse dataset across topies, income levels, and countries. The resulting insights lead to actionable steps to improve vision-language models' performance across diverse incomes.
Location Representation. The dataset contains images from four continents: Africa, America, Asia, and Europe, and 63 countries that exist worldwide. The number of images for a given country ranges from 45 in Canada to 4, 704 in India, with a median of 407 images per country.
of the 195
3.1 Dollar Street Dataset
The Dollar Street dataset contains 38, 179 images collected from homes in 63 countries on four con-
OBCD-Noter wwt.://eddd..ooooo/ooooo/// 'https: http///wwpppp.. inde org/ dollar-stree
'http:/ htttp://wwer
httt:://www.occd.orrgg http://wwwoccccccoooo///ooooo
pdf
'https: :/tps:://wwwww///////// www.ggoollee le, com/ maps/
Shttps: pattps///ee //www./www//// wtw.w.apw........mmmmm/../m..... gaw.i gww.inder.r....m inderorg

---

3.2 State-of-the-art Vision-Language Model For our evaluation, we choose CLIP, as opposed to other language-vision models, due to its vast pop- ularity as foundation model (Bommasani al., 2022), i.e., its use in multitude of models and its impressive zero-shot performance across vari- ous tasks and datasets, e.g., text-to-image retrieval, image question answering. human action segmen- tation, image-sentence alignment- - (Cafagna et al., 2021). However, we observe these datasets contain mostly images from North America and Western Europe, and, to the best of our knowledge, we are the first to evaluate CLIP on more diverse data.
lowest income bin bin which currently represents around 20% of the world's population."
CLIP scores
Quartile name
Income range
VTT-B
VIT-L
ViT-G
32
14
14
poor
26.9 195.0
0.233
0.259
0.321
low-mid
195,4- 685.0
0.250
0.282
0.350
up-mid
694.0
1,998.0
0.257
0.295
0.363
rich
2,001.0 19.671.0
0.256
0.295
0.363
Table I: Average CLIP scores per income quartile for different visual encoders.
A similar trend can be observed when quartiles of the data distribution are used as income bins. In addition to CLIP ViT-B-32, we also conduct the same experiment for the following CLIP visual encoders: ViT-L-14 from OpenAl and ViT-G-14 (pre-trained on the LAION dataset). Table 1 shows the average CLIP scores aggregated across each quartile; the lowest income quartile has the lowest average CLIP score among all the other quartiles for the three CLIP model versions.
We use the CLIP model represent all the topics and their corresponding images. We use the pre- trained Vision Transformer model ViT-B/32 (Doso- vitskiy et al., 2021) to encode the images and the textual information. When computing CLIP tex- tual topic embeddings, we concatenate the topics with given prompts (e.g.. "This is photo of toi- let"), described in Radford et al. (2021). We then compute the CLIP alignment scores as the co- sine similarity between the representations of the topics and their respective images. We choose to use absolute CLIP scores in our experiments be- cause they have been shown to provide a strong signal on the relevance that an image has to a given topic (e.g., the creators of the widely used LAION dataset (Schuhmann et al., 2021) performed human evaluations to choose CLIP threshold of 0.30 to determine label-image relevance).
What other factors affeet CLIP performance? A closer inspection of the images reveals that in addition to income, there are other factors that af- fect CLIP's performance such diversity in topic appearance and topic subjectivity.
Figure 2 shows qualitative analysis of five ran- dom topics across income levels. We observe that diversity in topic appearance is more predomi- nant in lower-income households, probably due to a need to improvise "ag."nwwsppper" or "grass") and less standardization across object appearances.
4 Research Questions
RQ1. Does CLIP show varying performance based on different income levels associated with the images? Analyses of CLIP scores aggregated across income level groupings provide convincing evidence that CLIP's performance varies across different income levels.
The subjective topics (e.g., "idols", "things I wish I had", "most loved item", "next big pur- chasc") are also problematic for models to classify correctly as they are subject to different interpre- tations based on the individual's background and personal thoughts (e.g., one person's "next big pur- chase" is a "couch", while another's is a "cow" Figure in Appendix). Therefore, we chose to follow Ramaswamy et al. (2023) and not include these topics in our CLIP performance analysis.
We measure the association between CLIP scores and household images in our dataset. Us- ing Spearman's 's Rank Correlation coefficient, we find a correlation of 0.35, which indicates a mod- erate association between CLIP performance and income.
Note that the purpose of Figure 2 is not to high- light CLIP's failures, since it would be expected to perform poorly on these examples. Instead, we emphasize the existence of diversity in topic appear- ance and subjective topics as challenges to vision- language models and address how to mitigate them in Section 5.
Figure shows trend of increasing CLIP scores income range increases: performance is signif- icantly worse on images from poorer houscholds, while it peaks on images from upper-middle in- come class. Specifically, CLIP performs poorly (score < 0.25) on about 75% of images from the
www. wor dbank.org

---

0.23
0.14
CLIP
0.0
Topic: Toilet Paper
CUP score
0.2
0.25
0.21
0.24
0.0
Topic: Get Water
CLIP score
0.29
0.22
0.2
0.14
Topic: Light Source
CLI score
0.26
..29
0.2
0.19
Topic: Refrigerator
Income Range:
poor
low-mid
up-mid
rich
Figure 2: Qualitative analysis showing the data diversity across different income quartiles five random topics: "toilet paper", "get water", "light source", "refrigerator". The CLIP performance on the same topic is influenced the remarkably diverse appearance entities from the same topic, which often correlates with income. Our analysis draws attention to how diverse objects and actions appear in our everyday lives and calls for future work to consider this when building models and datasets. Best viewed in color.
When does CLIP perform surprisingly well for low-income images? When inspecting low- income images with high CLIP scores, we find potentially problematic reasons for CLIP's high performance.
This section provides a more fine-grained CLIP performance evaluation, where measure how many relevant images are retrieved by CLIP for each topic, country, and income level.
Specifically, we compute and analyze the recall score across topics, income levels, and countries. The recall score is computed as follows. First, for each topic, we compute the CLIP similarity score between all the images in the dataset and a given topic. Next, select the images with the top N scores, where N is the number of ground truth images corresponding the topic T. We then measure the percentage correct images (i.e., rep- resenting topic T) among these images.
Specifically, because CLIP performs like a "bag of words" model (Thrush et al., 2022; Castro et al., 2023), it has high performance when the image contains explicit visual representations of at least word in the topic (e.g., "get water" in Figure 2 has relatively high CLIP score if the "wattr" is visible in the image as "lake" "puddle"),
One approach to mitigate the occurrence this problem involves the use of tools that provide a framework for inspecting images for possible con- founders that might affect the performance of model (Wang et al., 2020).
Recall scores across income levels. We show the CLIP recall over all images, across different income quartiles in Figure 3. The true positives
RQ2. How many relevant images are retrieved by CLIP across countries and income levels?
'In Information Retrieval, this is referred to as R-Precision. It is the cross point where precision and recall are equal.

---

(TP) represent the images that were correctly "rec- ognized", while the false negatives (FN) repre- sent the images left out of the top N scored im- ages or "forgotten" images. As seen in Figure 3,
appears to perform well for topic, while further analysis reveals that it has dismal performance on data from the lowest income group, as our findings show (see more examples in Appendix Table 2).
10000
Somall >
Iran- Unitod Kingdom
45.5
54.5
#lmages
70.0
57.1
Burundd-
1545.
45.5
5000
86.5
Worst Recall Countries
55.35
Hati- 7:5
92:5
Best Recall Countries
2447 .5
45.7
54.3
NTP wM
42.9
42.6
51.75
8.3
8.5
91:7
91:5
Guatomala
3295, 0s
46.1
45,8
54.2
53.9
13.5
56.33
poor
low-mid
up-mid
rich
Phous New Gutnee-
90U9
Netheriancs
75
Late dT 10:1
09.9
Karachetan-
2ibbassoe 10.7
27.55
Lamia-
53.9
06.1
Income Quartiles
14.45
12:2
87.8
826 65
54.
45.9
Figure 3: CLIP Recall over all images: percentage of true-positive or "recognized" images and felse-negative or "forgotten" images for each income quartile. Increas- ingly more images forgotten in the lower-income bins, with 86.5% forgotten images in the poor quartile.
157.00 06
13:1
06.9
3360.05
62.5
37.5
50
100
50
100
Relevant Images
Relevant Images
Figure 5: Countries with high and low recall scores: 7/10 countries with the worst recall have low average incomes and are from Africa. The countries with the best recall scores have high average incomes and are from America, Europe, Asia. Countries with low recall also have low income, while most countries (apart from Guatemala) with high recall have high income.
CLIP has unequal performance in retrieving im- ages across different income groups and images collected from poor households are less likely to be retrieved correctly. We also find that out of the 199 topies that have image contributions from all four income groups, 137 (68.8%) topics show a simi- lar trend in unequal performance as displayed in Figure (see Appendix Table 2 for more details).
Disparate recall across countries. We compute the recall over all images, across countries. Fig- ure compares the top ten countries with the best and worst ratio of true-positives to false- negatives. The countries with the best recall have true-positive rates up to 40% higher than those with the worst recall. Note that most countries in the worst recall plot have low average incomes and are in Africa. The average income per country is computed as the mean income of the dataset houscholds in each country, which follows the real-world trend
Topics with highest recall deviations across in- come levels. Ideally, CLIP's performance for given topic should be relatively similar across all income levels. We calculate the variance between recall scores across income levels for each topic: Figure 4 displays the top ten topics with the highest recall variance between income levels.
200
RQ3. What is the diversity of topic appearances across countries and income levels? Analyzing the findings from the first two RQs, we observe that topics from lower-income households tend to have more diverse appearances. This is important be- cause, intuitively, topics with diverse appearances are more prone to vision-language model perfor- mance variations. For instance, the topic "toilet pa- per" appears in Figure 4 among those with the most variance in recall across income levels (poor: 7.3; low-mid: 36.7; up-mid: 69.9; rich: 86.5), and with at least five diverse appearances ("grass", "leaf", "newspaper", "water" and "toilet paper") in Fig- ure 2.
Recail acore
BR.
150
76.7
14.0
57.7
100
50
43.8
83.0
72.5
5.9
10.7
39.4
68.9
55.9
31.0
250
68.5
33.5
59.2
8.
$9.0
58.7
telee
teiet papar
Bn
Sp
ctove
Ad do
Lechere
26.90
cauch
28.90
Red.
clairing
shampote 8a
Topic
Figure 4: Top 10 topics with the largest standard devi- ation over the CLIP recall scores per income quartiles, recall varies the most across income on these topics.
As shown in the figure, topics like "toilet", "toi- let paper", and "wardrobe" that have high recall scores for the rich income level (> 80) also have extremely low recall scores for the poor income level (< 15). This is concerning because CLIP
'https: org/ wor I I deve opment- indi ind cadicatsss the filewwoldd d d- income- and-regi on html

---

To automatically discover which topics have di- verse appearances and identify the different forms that the images belonging to one topic could take e.g., "toilet paper" taking the form of "grass", "leaf" or "newspaper"- we find inspiration from word analogies (Bolukbasi et al., 2016), such as "man is to programmer as woman is to homemaker" and develop topic analogies for the visual repre- sentations of topies across countries and income levels: e.g., "toilet in poor Nigerian households as backyard is in rich Chinese households.
2. Bed is in poor Malawian households Wall is in rich French houscholds.
3. Washing Clothes is in poor Nigerian house- holds as Doing Dishes is in rich Nepalese houscholds.
pire
p80
0.95
Visual Similarity. First, visually represent topic for given country and income level (topic, country, income) as the average of all the CLIP visual representations of the corresponding images. Next, we compute the visual similarity between (topic, country, income) tuples as the cosine simi- larity between their visual representations.
backyard
backyard
0.90
backyard
bed
0.85
bed
beo
floor
floor
floor
tolict 555
0.75
toilt 555
toilet 555
backyard 555
0.70
backyard 511
0.00
floor
floor
Tollet, Nigeria,
Bathroom Privacy, Côte poor,
Rackyard Chins, rich, 0.88
Figure 7: Heatmap of the visual similarity scores be- tween images from different income levels (poor S, rich SSS), from three countries different continents (Nige- ria Colombia India 7 ), for four common topics (toilet, backyard, bed, floor). Best viewed in color.
Using Tollet, Tunisis poor, 0.90
Toilet Paper, Vietnam, vich, 0.93
Toilet, Nigetia, rich
In Figure 7, we compute heatmap of the visual similarity scores between images from different income levels, from three countries on different continents, for four common topics. Overall, we see how topics from poor households tend to be more similar to each other, in the green/ high sim- ilarity cluster, and not similar to topics from rich households, in the red/ low similarity cluster.
Bed, Malawwi, poor
Wall, Indla, poor, 0 9.95
Wail, France, rich, 0:80
Washing Clothes, Núgetia, poor
Washing poor,
Dishes, Nopal, rich.
We quantify appearance diversity for each topic across rich and poor income levels by counting how many of the visually similar topics are actu- ally similar, i..., textually similar - e.g., in Figure 6 "bed" and "wall" are visually similar, but not textu- ally similar. We compute the textual similarity as the cosine similarity between the sentence embed- dings (Reimers and Gurevych, 2019) of the query topic and the returned top ten most visually simi- lar topics. We count the topies as actually similar if their textual similarity is greater than thresh- old, 0.7. We find that topics from rich households have 33% actually dissimilar topics, while topics from poor households have 47%, demonstrating
Toothbrush, India, poor
Tooth paste, Myanmer, poor, 0.54
Zrinking Water Nigeria, rich, 9.88
Figure 6: For a given query-tuple (topic, country, in- come) on the left (e.g., "toilet paper in poor Nigeria") "), show the most visually similar tuples on the right. Best viewed in color.
In Figure 6, show examples of visual similar- ities between query-tuple and the returned tuples. We find compelling results, such as:
1. Toilet is in poor Nigerian houscholds as Back- yard is in rich Chinese households.

---

that topic diversity is higher in poor households than in rich houscholds.
expensive compared to web scraping methods, they have been proven improve overall model perfor- mance and mitigate bias (Ramaswamy et al., 2023; Kirillov et al., 2023). In contrast, datasets created using web scraping exclude data from households with limited/no access to the Internet, which re- flects a staggering 37% of the world's population." A solution is to supplement these datasets through crowd-sourcing to allow for broader reach. Simi- larly, shown in Figure 5, more attention should be given to data collection from underrepresented countries.
5 Lessons Learned and Actionable Steps
Our analyses yield several insights into the cur- rent state of vision-language models. We highlight lessons learned and propose actionable steps to en- sure future work builds more equitable models and datasets.
Invest effort to understand the extent of the digital divide in vision-language performance. Four years after De Vries et al. (2019) drew at- tention to the performance disparity across in- come levels for object recognition models, we show that popular vision-language model (CLIP) also yields poor performance on data from lower- income households. This new finding, together with the scarcity of research in this area, shows that the AI research community is not paying enough at- tention to poor socio-economic groups worldwide. The consequences can be devastating since, if ne- glected, AI will exponentially increase the already existing "digital divide".
Annotate diversity and subjectivity in datasets. For actions or entities that can be represented by remarkably different items that perform the same functionality, shown in Figure 2, it is essen- tial to reconcile these differences and inclusively categorize them. Furthermore, as highlighted by Ignat et al. (2019), people often use various names for the same action or object, and having just one label limits the diversity of our vocabulary. Ra- maswamy et al. (2023) also found this challenging when collecting data: "stove" was originally un- derrepresented until the definition of "stove" was clarified, workers did not always consider their cooking appliances to be "stove". One possible solution might involve the development of flexible labels, also considering the data provider.
Define evaluation metries that represent ev- eryone. We call for attention and scrutiny of broad performance metrics that give the illusion of high accuracy in AI models while performance on minority groups remains dangerously inadequate. One potential solution is to use the leo-location information of images when available to evaluate models by location and income, as income can of- ten be deduced from the location.
Furthermore, Chan et al. (2021) note that partic- ipation in data collection does not always equate to representation, as the research labs often dictate how data is labeled and categorized; models trained diverse datasets still make distortions through Western lens and not truly reflect the culture of the represented people. To ensure that data from across all income groups and countries are labeled to reflect the people's s authentic culture, representa- tives from these groups should also participate in the labeling process.
Match diversity standards to model purpose. Diversity standards for AI models and datasets must match the intended users. For example, an AI model built as a foundation baseline for vari- ous downstream tasks should be held to the highest standards regarding what training data is used, it should be able represent everyone equally.
Conclusion
Document training data. The origins of a model's s training dataset provide valuable informa- tion to prospective users and developers regard- ing the limitations and best use cases for a model. Model creators are responsible for documenting the data the model was trained and any known biases it may have (Mitchell et al., 2019; Gebru al., 2021; Wang and Russakovsky, 2023).
In this paper, evaluated the performance of state-of-the-art vision-language model (CLIP) on household images across different income groups, and different countries as found in the DollarStreet dataset. The results of our analyses demon- strated the existence of performance inequality and showed that data from lower-income groups are more likely to be represented inaccurately.
Invest in geo-diverse datasets. as much as collecting geographically diverse datasets can be
'https: / htwww//wumm un un. www.orggee//eeee http//ee/eellegeeeeee egate/ itu- billiiin wwwlooo-pppppleec wwwppe-still-oneeeeeeeeepppppepe still-offlilee

---

To mitigate these issues, we analyzed and quan-
pressed in this material are those of the authors and do not necessarily reflect the views of the Temple- ton Foundation or the Department of State.
tified the visual appearance diversity in topics a current challenge for vision-language models- and determined that there is higher topic visual appearance diversity in low-income images than higher income images. Furthermore, building on our findings, we shared six actionable steps for improving inclusion in Al development.
Ethical Considerations
Several ML models have been used to estimate the income bracket of neighborhoods. Since Dollar Street provides image data along with household income values, there is the potential for ML mod- els to be trained on the dataset to predict income (Acharya et al., 2017: Gebru et al., 2017: Yeh et al.. 2020a; Machicao et al., 2022).
We hope our work will contribute a better understanding of the limitations of current vision- language models and provide important directions to guide future research toward "democratizing" vision-language models and datasets.
We note that our analyses suggest that vision- language models could indirectly identify which images belong to lower-income groups. How- ever, our work does not contribute to encourage the disingenuous use of Al for tasks that lead discrimination or unethical applications. Instead, we create awareness of AT performance disparity across income groups and call the research commu- nity to action toward mitigating them.
Limitations
Out of all the publicly available image datasets, Dollar Street is the most diverse dataset that con- tains income information. However, some limita- tions exist in terms of country and income repre- sentation, since Dollar Street does not capture all representative households worldwide.
The dataset contains data from 63 countries, rep- resenting four continents (Africa, America, Asia, and Europe). Therefore, our analyses do not ac- count for 121 countries in the aforementioned conti- nents and the entire regions of Australia and Antarc- tica. Some countries have sparse contributions from only one or two households. For example. Canada has only 45 images, all contributed by one household. Since most European households in Dollar Street belong to the upper-middle and rich income group and about half of the African house- holds belong to the poor and lower-middle income group, our study does not capture the complete representation of the different economic groups in those countries.
References
Ambika Acharya, Helen Fang, and Shubha Raghvendra. 2017. Neighborhood watch: Using cnns to predict in- come brackets from google street view images. URL: http://ss23nn stanford. edu/reports/201 7/pdfs/556. pdf.
Jungyong Ahn, Jungwon Kim, and Yongjun Sung. 2022. The effect of gender stereotypes on artificial intel- ligence recommendations. Journal of Business Re- search, 141:50-59.
Dimah Alahmadi, Amal Babour, Kawther Saeedi, and Anna Visvizi. 2020. Ensuring inclusion and diver- sity in research and research output: A case for language-sensitive nip crowdsourcing platform. Ap- plied Sciences, 10(18).
Acknowledgement
Lukasz Augustyniak. Kamil Tagowski, Albert Sawezyn, Denis Janiak, Roman Bartusiak, Adrian Szymezak, Arkadiusz Janz, Piotr Szymański, Marcin Watroba, Mikolaj Morzy, et al. 2022. This is the way: design- ing and compiling lepiszcze, comprehensive nip benchmark for polish. Advances in Neural Informa- tion Processing Systems, 01-21805-21818
We thank the anonymous reviewers for their con- structive feedback. We are also grateful Artem Abzaliev, Ashkan Kazemi, and Longju Bai for proofreading the manuscript and providing several suggestions that helped shape the research ques- tions. Lastly, we thank the members of the Lan- guage and Information Technologies lab at the Uni- versity of Michigan for the insightful discussions during the early stage of the project. This project was partially funded by a grant from the Templeton Foundation (#62256) and grant from the Depart- Any opinions,
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic incqualities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 3486-5503 Dublin, Ireland. Association for Computational Linguistics.
ment of State (#STC10023GR001 findings, and conclusions or recommendations ex-
Tolga Bolukbasi, Kai-Wei Chang. James Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man

---

is to computer programmer as woman to home- maker? debiasing word embeddings. Advances in neural information processing systems, 29.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoff- man, Ning Zhang. Eric Tzeng, and Trevor Darrell. 2014. Decaf: A deep convolutional activation feature for generie visual recognition. In International con- ference on machine learning, pages 647-655. PMLR.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2022. On the opportuni- ties and risks of foundation models. arXiv preprint urXiv:2108.0720
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. ICLR.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Conference on fair- ness, accountability and transparency, pages 77-91. PMLR.
Abhimanyu Dubey, Vignesh Ramanathan, Alex 'Sandy' Pentland, and Dhruv Kumar Mahajan. 2021. Adap- tive methods for real-world domain generalization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14335-14344.
Michele Cafagna, Kees van Deemter, and Albert Gatt. 2021. What vision- lision-language modelssee' when they see scenes, arXiv preprint arXiv:2 109.07301.
Timnit Gebru, 2020. Race and gender. The Oxford handbook of ethics of al, pages 251-269.
Lemuria Carter. Dapeng Liu, and Caley Cantrell. 2020. Exploring the intersection of the digital divide and ar- tificial intelligence: A hermeneutic literature review. AIS Transactions on Human-Compal Interaction, 12(4):253-275.
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, and Li Fei- Fei. 2017. Using deep learning and google street view to estimate the demographic makeup of neigh- borhoods across the united states. Proceedings of the National Academy of Sciences, 114(50):13108- 13113.
Santiago Castro, Oana Ignat, and Rada Mihalcea. 2023. Scalable performance analysis for vision- language models. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics, Toronto, Canada. Association for Computational Linguisties.
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iïi, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86- 92.
Alan Chan, Chinasa T Okolo, Zachary Terner, and An- gelina Wang. 2021. The limits of global inclusion in ai development. Association for the Advancement of Artificial Intelligence.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Ji- tendra Malik. 2014. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587.
Davide Cirillo, Silvina Catuara-Solarz, Czuce Morey, Emre Guney, Laia Subirats, Simona Mellino, Annal- isa Gigante, Alfonso Valencia, María José Remente- ria, Antonella Santuccione Chadha, et al. 2020. Sex and gender differences and biases in artificial intel- ligence for biomedicine and healthcare. NPJ digital medicine, 3(1):81.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing. pages 4411 1-411. PMLR.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Oana Ignat, Laura Burdick, Jia Deng, and Rada Mi- halcea 2019. Identifying visible actions in lifestyle vlogs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6406 6417, Florence, Italy. Association for Compu- tational Linguistics.
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten, 2019. Does object recog- nition work for everyone? In Proceedings of the IEEF/CVF conference on computer vision and pat- tern recognition workshops, pages 52-59.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXi::2004.000 090955.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Simran Khanuja. Sebastian Ruder, and Partha Talukdar. 2023. Evaluating the diversity, equity, and inclusion of nlp technology: A case study for indian languages. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1718-1732.

---

Sunder Ali Khowaja, Parus Khuwaja, and Kapal Dev. 2023. Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review. arXiv preprint 1rXiv:2305.03/23
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In International Conference on Machine Learning.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Yen Lo, et al. 2023, Segment anything. arXiv preprint arXiv:2304.02643.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas- sively multilingual transfer for ner. arXiv preprint arXiv:1902.00/93
Irene Kitsara. 2022. Artificial intelligence and the digi- tal divide: From an innovation perspective. In Plat- forms and Artificial Intelligence: The Next Genera- tion of Competences, pages 265. Springer.
Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky, 2023. Be- yond web-scraping: Crowd-soureing a geograph- ically diverse image dataset. arXiv preprint arXiv:2301.0256/
Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020. Indolem and indobert: A benchmark dataset and pre-trained language model for indone- sian ulp. arXiv preprint arXiv:2011.0000
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija. Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy. 2017. Openimages: A public dataset for large-scale multi-label and multi- class image classification. Dataset available from
William A Gaviria Rojas, Sudnya Diamos, Keertan Ran- jan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The dollar street dataset: Im- ages representing the geographic and socioeconomic diversity of the world. Advances in Neural Informa- tion Processing Systems,
Christoph Lutz. 2019. Digital inequalities in the age of artificial intelligence and big data. Human Behavior and Emerging Technologies, 1(2):141-148.
Jeaneth Machicao, Alison Specht, Danton Vellenich, Leandro Meneguzzi, Romain David, Shelley Stall, Katia Ferraz, Laurence Mabile, Margaret O' O'brien, and Pedro Corrêa. 2022. A deep-learning method for the prediction of socio-economic indicators from street-view imagery using a case study from brazil. Data Science Journal, 21.
Han Rosling, Ola Rosling, and Anna Rosling Rönnlund. 2019. Factfulness. Lindhardt og Ringhof.
Sebastian Ruder. Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15-18. Minneapo- lis, Minnesota. Association for Computational Linguistics.
Mounika Marreddy, Subba Reddy Octa, Lakshmi Siree- sha Vakada, Venkata Charan Chinni, and Radhika Mamidi. 2022. Am resource-poor language? data sets, embeddings, models and analysis for four differ- ent nlp tasks in telugu language. ACM Transactions on Asian and Resorre Language Information Processing, 1--34.
Hadi Salman, Saachi Jain, Andrew Ilyas, Logan En gstrom, Eric Wong, and Aleksander Madry. 2022. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02M.
Nicolas Miailhe and Cyrus Hodes. 2017. Making the ai revolution work for everyone. The Fu- ture Society at the Harvard Kennedy School of Govermment, Cambridge, MA, http://aaiimttiaiiee.. ggggyppcotten/nlllud///77722-0.d At- ponouuaiitwoor--freeeeeeyee DOC MARCH- 2017 pdf.
Christoph Schuhmano, Richard Vencu, Romain Beau- mont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Ko- matsuzaki. 2021. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. In Pro- ceedings of the NeurlPS Data Centric A/ Workshop.
initiative.
Margaret Mitchell, Simone Wu. Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Iilena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Account- ability, and Transparency. ACM.
Shreya Shankar, Yoni Halpern, Eric Breck, James At- Wood, Jimbo Wilson, and D Sculley. 2017. No classi- fication without representation: Assessing geodiver- sity issues in open data sets for the developing world. arXiv preprint arxiv:171 171.00853.

---

Sunny Shrestha and Sanchari Das. 2022. Exploring gender biases in ml and ai academic research through systematic literature review. Frontiers in artificial intelligence, 5:976838.
A Subjective Topics
The 19 subjective topics that we remove: "favorite home decorations", "favourite item in kitchen", "favourite sports clubs", "how the most loved item is used", "icons", "idols", "latest furniture bought", "looking over the shoulder", "most loved item", "most loved toy", "most played songs the radio", "music idol", "next big thing you are planning buy". "playing with most loved toy", "thing dream about having". "things wish had", "using most loved item", "youth culture", "what wish could buy".
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality, In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 5238- 5248.
Angelina Wang, Arvind Narayanan, and Olga Rus- sakovsky. 2020. Revise: A tool for measuring and mitigating bias in visual datasets. International Jour- nal of Computer Vision, 130:1790 1810.
B CLIP performance other analyses
Angelina Wang and Olga Russakovsky, 2023. Overcom- ing blas in pretrained models by manipulating the finetuning dataset. arXiv preprint (XXx:22333.0116.
Figure 10 displays topics with trends that deviate from the plot shown Figure 3.
Michael Xie, Neal Jean, Marshall Burke, David Lo- bell, and Stefano Ermon. 2016. Transfer learning from deep features for remote sensing and poverty mapping. In Proceedings of the AAAI conference artificial intelligence, volume 30.
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. 2020a. Using publicly available satellite imagery and deep learning to understand eco- nomic ael-bbiig in africa. Nature communications, 11(1):2583.
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. 2020b. Using publicly available satellite imagery and deep learning understand eco- nomic well- bell-beig in africa. Nature Communications. 11.

---

score
0.20
0.19
0.18
200
400
600
1000
Income
Topic: Idols
p.24
0:23
0.20
0.19
D.16
0.14
200
400
600
800
1000
Income
Topic: Things Wish Had
CUP score
0.27
0.25
0.24
0.20
D..88
200
400
600
800
1000
Income
Topic: Most Loved Item
Cuje score
0.20
0.25
0.2
19
D.22
0.2
0.24
200
400
600
800
1000
Income
Topic: Next Purchase
Figure 8: Qualitative analysis showing the CLIP performance subjective data different incomes four random topies: "idols", "things wish had", "most loved item". "next big purchase".

---

Top 10 topics with highest F1 score
F1 Score
0.5
86.5
63.4
63.8
63.8
64.4
65.2
67.1
67.8
69.1
77.7
0.0
chickens
car keys
hand palm
lock on front door
bike
hair brush/comb moped/mo
tv
wall clock
glasses or lenses
Topics
(a)
Top 10 topics with lowest F1 score
F1 Score
0.05
7.7
6.1
6.3
4.4
0.00
0.9
10
2.1
2.6
2.7
smoke and steam exit
using toilet
dinner guests
house overview
photo guide images
cleaning after toilet
place where serving guests
bowls
waste dumps
surroundings
Topics
(b)
Figure 9: Top topies with highest (a) and lowest (b) F1 scores.

---

250
poor
51.0
low-mid
200
up-mid
61.9
Recall
150
24.1
rich
15.8
36.2
50.0
29.7
64.7
100
38.6
35.9
41.4
30.4
50
78.9
55.3
54.2
45.5
radio
cooking pots
phone
menstruation pads tampax
Topic
Figure Topies where recall for lowest income level is higher than recall for other income levels.
250
69.9
200
poor
66.4
Recall
low-mid
150
up-mid
rich
100
67.3
34.9
50
38.4
67.7
19.5
0
drainage
hair brushicomb
Topic
Figure 11: Topics where recall for is relatively high (hairbrush) and relatively low (drainage) across all income levels

---

Topic
Up-inid
Rich
Topic
Foor
Up-anid
addinng food while cooking
0.8
10.0
14.7
snap hands and body
11.0
20.9
42.1
37.7
spices
20.6
35.3
53.6
$6.5
shampoo
2.4
28.7
57.6
60.3
unswering the phone
0.0
0.0
10.0
13.4
kitchen sink
0.0
24.6
72.3
65.2
jewelry
0.0
armcchii
66.7
doing dishes
7.3
44.2
couch
4.1
33.3
59.1
75.3
social drank
20.7
26.2
26.2
29.0
sola
2,0
31.0
55.9
74,0
drying
7.7
33.3
42.0
40.3
pet
10.0
16.9
30.2
28.6
washing clothes/cleaning
5.6
22.6
59.0
65.7
place eating dinner
carings
sitring
0.0
19.0
33.8
35.7
nocklaces
40.0
42.9
63.6
68.4
backyard
2.6
17.5
42.3
50.0
piercings
11.1
40.9
52.9
65.0
drinking water
10.1
32.2
40.9
44,1
cating
0.0
7.4
37,5
27.3
street view
28.8
62.1
64.1
71.6
family cating
22.2
44.0
50.0
53.8
everyday
bathroom privacy
0.0
4.3
21.3
33.9
nicest shoes
27.4
39.1
68.3
mush/waste
10.2
37.2
45.6
57.3
family
10.7
40.7
50.9
42.6
kitchen
1.1
25.0
69.5
59.2
family snapshots
4,8
17.9
26.1
27.0
bathroom/toilet
1.0
31.0
60.7
45.3
decoration
4.9
21.8
33.7
38.7
washing
wedding photos
shower
0.0
18.8
56.6
64.9
leeth
41.7
49.4
76.7
86.4
bed
8.9
33.6
60.2
76.7
refrigerator
0.0
38.4
57.3
48.3
bed kids
3.8
14.7
41.4
33.3
nu
19.0
51.9
72.9
66.7
bedroom
4.2
27.5
45.8
59.2
freezer
20.0
35.1
38.7
50.0
bod
8.7
20.9
53.8
17.6
25.8
37.2
34.5
lock Front door
41.5
69.6
78.1
71.6
children room
1.6
19.2
38.0
39.4
mupell/nolooeycle
66.7
48.0
71.4
85.0
cleaning equipment
1.1
9.8
20.0
21.1
opening the front doer
0.0
8.0
34.8
22.9
play
living
53.4
61.7
fruits vegetables
6.2
54.9
bike
48.1
63.2
72.3
67.6
grains
28.2
29.4
30.0
47.8
books
21.9
42.5
65.5
78.2
vegetables
20.3
41.3
62.8
60.6
paper
3.8
18.2
24.0
29.1
shaving
0.0
15.1
29.2
30.8
hallway
37.5
plates
62.5
hand back
2.7
10.8
13.5
brushing hair
0.0
4.5
11.1
25.0
hand to closed
0.0
0.0
44.4
29.4
brushing teeth
4.6
26.1
35.7
50.0
washing hands
0.0
24.4
77.4
81.4
loothbrusk
17.0
37.3
65.5
64.4
playing
7.7
7.7
28.6
13.8
light
frentdoor keys
0.0
13.8
27.3
22.2
lightsourees by hed
0.0
26.7
31.0
28.6
wheell barrow
22.2
46.7
57.1
57.1
reading light
0.0
6.7
20.7
20.6
parking lor
0.0
10.7
20.0
45.8
tools
5.8
30.5
40.6
52.4
get water
2.0
21.9
13.7
plate Food
11.0
46.0
47.4
46.2
veiling
25.3
39.0
38.6
light in livingroom
1.3
21.8
44.3
44.0
pet foods
0.0
14.3
68.8
59.3
light sources
10.2
19.0
35.6
32.7
plugging into and out of power outlet
0.0
33.3
51.5
40.0
wall clock
36.0
74.1
89.7
89.4
power outlet
14.6
46.4
69.7
64.5
pouring drinking water
fish
pouring water
10.5
32.1
chopping ingredicats
0.0
27.8
40.0
43.5
switch onfoll
5.8
36.5
50.0
62.9
chopping fond
3.7
26.9
50.0
52.9
listening to the radio
0.0
8.3
15.4
11.1
loilet paper
7.3
36.7
69.9
56.5
reading
14.3
37.0
70.6
72.2
cleaning floos
reading
Blor
14.4
38.5
60.6
56.6
salt
19.0
31.5
40.2
45.1
washing detergent
0.0
9.8
21.5
24.6
laking leaspoon salt
0.0
13.0
33.3
26.3
closing the front door
0.0
5.0
28.6
29.0
shous
1.4
23.5
43.2
62.7
wardrobe
10.7
39.4
68.9
50.3
sleeping
0.0
50.0
61.5
52.2
and juckets
drinking
0.0
13.3
44.2
60.9
2.6
13.0
25.0
40.5
conking
13.8
19.6
39.1
32.6
street detail
4.9
17.5
29.0
43.5
stove/heb
5.9
41.6
72.5
76.7
source of light
0.0
17.6
34.5
30.6
knifes
31.0
51.1
77.6
70.7
worship places
0.0
11.8
37,5
20.6
prepuring
28.6
toothpaste toothbrush
0.0
cosmetics
0.0
43.8
61.1
55,3
the back book
0.0
0.0
27.3
13.6
cups/mugs/glasses
29.5
46.3
64.7
74,6
tuminng heater
0.0
0.9
44.4
16.7
tooth paste
21.0
52.5
66.4
59.8
walking towards front door
0.0
4.8
13.0
26.5
diapers baby-pants)
65.2
70.6
87.5
water outlet
1.2
19.2
39.7
dish washing brush/cloth
19.3
26.7
worshipping
0.0
4.0
36.4
dish washing soap
0.0
11.9
34.9
50.0
wall
9.0
18.8
33.3
36.0
hand palm
48.3
58.2
76.6
86.0
Table 2: Topics that have similar plot as Figure 3 and Recall values for Poor, Low-mid, Up-mid, Rich income levels: Where recall for rich and up-mid higher than recall for poor and low-mid.

---

man anded
Shouess wanes
Cote
big
eusbi
Usi
South Africa
Calombia
Togo
Kanya
Rwanda
Malaw:
Nepal
Cota
0.95
China
Myanmar
Indonesia
Halt
Bangladesh
Burkina Faso
Philppines
Zibbabwe
Cambodda
Papua
Nigeria
Camerson
0.90
Tunisia
Burundi
Liberia
Somalia
Thailand
Peru
Serbia $55
$55
Colombiaa $55
Brazi 555
0.85
Spain $55
Czech Republic $35
Switzerland aad
ILaly 555
Palistiinn
Echiopia $35
Nepal 555
Indenesia $55
Turkey $55
Sweden
Austria 555
United States
France $35
Mexico 555
Romania
South Karaa $35
Vielnam
Canada $35
Denmark 555
Figure 12: Heatmap of the visual similarity scores between images from different income levels (poor S, rich SSS), from sixty-four countries on different continents across all topics. Best viewed in color.


Reviews:
[{"review_id": "jtm59YPidI", "reviewer": "Reviewer_Vvi2", "summary": "", "questions": "Why not just use object classification accuracy but use the simple CLIP scores?\n\nFor visual similarity section, is there any way to extend the analysis to larger-scale?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Note of Thanks**   \n We appreciate your insightful review. We are glad you found the research question we address interesting, and the investigation comprehensive.\n\n**RC: Reviewer's comment**                      \n**AR: Author's response**\n\n\n_**Reasons to Reject**_\n\n**RC** Evaluation is not rigorous: When using CLIP score to identify if the text correctly describes the object in the image, it is very important to NORMALIZE the score. For example, we first need to calculate the score between the image and all the object categories. And then we normalize all the scores and pick up the maximal one. We can’t directly calculate the score, because even if “toilet paper” has high matching score with the image, there might be another object name that has higher score. The evaluation may hurt the soundness of the method a little bit.\n\n**AR**  _While this is an interesting point, note that in our paper, our emphasis is on evaluating CLIP for its foundational multimodal model's ability to recognize diversity in text-image associations.  As we mentioned in the paper, one topic/label can be represented by a variety of objects. Similarly, one object can be found in multiple topics.  If we were to normalize the scores, it means we would set our experiments to measure precision for object detection or classification, which is different from our objective. Instead, we chose to use the CLIP scores \"as is\" because we wanted to measure if the objects in the images can be identified/associated with the textual topics they are labeled as_.\n\n\n**RC** May lack more interesting findings: Although the authors provide very comprehensive analysis and in-depth insights, most of them are very similar to what previous works did. For example, in Liu et al., 2021 and Yin et al., 2021, the authors already find that there’s a large performance disparity across Western and non-Western countries. Yin et al., 2023 also quantify the performance gap on Dollar Street. Though it is not going down to the salary level, the basic observation is similar. What I expect is that, besides claiming that the models are biased, please also try to dig out the reason and the ways we can improve. That can make the paper more convincing and attract more people to follow and devote to make the models better. Visually Grounded Reasoning across Languages and Cultures. Liu et al., 2021 Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning. Yin et al., 2021 GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods. Yin et al., 2023\n\n**AR**  _Thank you for these pointers. While these papers investigate performance disparity along the cultural axis, our work provides insights into differences that exist across income levels irrespective of the country of origin.\nDespite having similar motivations, contributions from Liu et al., 2021, Yin et al., 2021, Yin et al., 2023  focus on cultural concept diversity, where language is a factor that can facilitate an understanding of the disparities. However, as we found in our work, there are also major performance disparities that exist within one culture along the income axis, which has not been accounted for in previous work._ \n\n_**Questions for Authors**_\n\n**RC** Why not just use object classification accuracy but use the simple CLIP scores?\n\n**AR** _Object recognition and its downstream applications are indeed apparent areas where equal representation is important. However, there is a danger for multimodal applications such as image captioning, video frame processing, and action recognition to suffer from a limited view of what textual concepts/topics can be associated with visual data.   Therefore, we chose to stick with CLIP similarity scores that measure basic text-image association (rather than focusing on the task of object recognition), to capture the disparities that exist at this fundamental level of associations in the multimodal space. Note also that the performance disparities specific to object recognition have been previously explored by De Vries et al. (2019)._\n\n \n**RC** For visual similarity section, is there any way to extend the analysis to larger-scale?\n\n**AR**  _Yes, following your suggestion we performed an analysis of the visual similarity scores for the 64 (country, income) pairs (where income belongs to the poor or rich quartile). We found that across all topics, the images from poor households are similar to each other, and those from rich households are similar to each other, regardless of the countries these images come from. This is consistent with our findings in RQ3. We plot these findings on a heatmap that you can access here [heatmap](https://anonymous.4open.science/r/Bridging_Digital_Divide-E41C/heatmap.pdf), which shows the visual similarity scores between images from different income levels (poor \\$,_ _rich \\$\\$\\$) across all topics, from 64 countries on different continents._\n\n\n1. De Vries, Terrance et al. (2019). “Does object recognition work for everyone?” In: Proceedings of the\nIEEE/CVF conference on computer vision and pattern recognition workshops, pp. 52–59.\n\n2. Liu, Fangyu et al. (Nov. 2021). “Visually Grounded Reasoning across Languages and Cultures”. In: Pro-\nceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and\nPunta Cana, Dominican Republic: Association for Computational Linguistics, pp. 10467–10485. DOI:\n10 . 18653 / v1 / 2021 . emnlp - main . 818. URL: https : / / aclanthology . org / 2021 .\nemnlp-main.818.\n\n3. Yin, Da, Liunian Harold Li, et al. (Nov. 2021). “Broaden the Vision: Geo-Diverse Visual Commonsense\nReasoning”. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.\nOnline and Punta Cana, Dominican Republic: Association for Computational Linguistics, pp. 2115–2129.\nDOI: 10.18653/v1/2021.emnlp-main.162. URL: https://aclanthology.org/2021.\nemnlp-main.162.\n4. Yin, Da, Feng Gao, et al. (2023). “GIVL: Improving Geographical Inclusivity of Vision-Language Models\nwith Pre-Training Methods”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern\nRecognition, pp. 10951–10961.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Great motivation and interesting research question: Studying the correlation between socio-economic factors and VL model performance is very important to let people aware of the potential bias and the severity of performance disparity for the applications. I’m impressed with Section 5 which revisits the problem and provide inspiring future steps. \n\nComprehensive investigation on the quality of generated instruction data: I'm glad to see that the authors utilize different metrics to evaluate the performance disparity across different income levels and countries. The comprehensive evaluation helps us better understand the effect of proposed methods. I’m especially impressed by multi-grained experiments that comprehensively support the authors’ arguments.", "reasons_to_reject": "Evaluation is not rigorous: When using CLIP score to identify if the text correctly describes the object in the image, it is very important to *NORMALIZE* the score. For example, we first need to calculate the score between the image and all the object categories. And then we normalize all the scores and pick up the maximal one. We can’t directly calculate the score, because even if “toilet paper” has high matching score with the image, there might be another object name that has higher score. The evaluation may hurt the soundness of the method a little bit.\n\nMay lack more interesting findings: Although the authors provide very comprehensive analysis and in-depth insights, most of them are very similar to what previous works did. For example, in Liu et al., 2021 and Yin et al., 2021, the authors already find that there’s a large performance disparity across Western and non-Western countries. Yin et al., 2023 also quantify the performance gap on Dollar Street. Though it is not going down to the salary level, the basic observation is similar. What I expect is that, besides claiming that the models are biased, please also try to dig out the reason and the ways we can improve. That can make the paper more convincing and attract more people to follow and devote to make the models better.\nVisually Grounded Reasoning across Languages and Cultures. Liu et al., 2021\nBroaden the Vision: Geo-Diverse Visual Commonsense Reasoning. Yin et al., 2021\nGIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods. Yin et al., 2023", "questions_to_authors": "Why not just use object classification accuracy but use the simple CLIP scores?\n\nFor visual similarity section, is there any way to extend the analysis to larger-scale?", "paper_topic_and_main_contributions": "The paper studies the gap in the evaluation of V&L models, specifically their performance on underrepresented groups. The researchers assess CLIP using a diverse dataset of household images linked to income levels, DollarStreet. The findings reveal performance disparities between different income groups, with consistently lower performance for poorer households compared to wealthier ones across topics and countries. The study emphasizes the need to address these inequalities and proposes actionable strategies for creating AI models that are more inclusive across economic levels.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Visually Grounded Reasoning across Languages and Cultures. Liu et al., 2021\nBroaden the Vision: Geo-Diverse Visual Commonsense Reasoning. Yin et al., 2021\nGIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods. Yin et al., 2023"}}, {"review_id": "rjvksMIet3", "reviewer": "Reviewer_hF5y", "summary": "", "questions": "Question A: How would you expect model performance to change based on model size?\n\nQuestion B: Would you expect multilingual variations of CLIP to have different findings?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Note of Thanks**   \n We are grateful for your positive comments on our paper, and we are excited to see that you find the paper to be very well-written and the evaluation important and novel.\n\n**RC: Reviewer's comment**   \n**AR: Author's response**\n\n_**Reasons to Reject**_\n\n**RC** The model only explores CLIP, and more specifically, they only use a single model variation of CLIP (ViT-B/32). Larger CLIP versions have better performance (ViT-L-14). Moreover, there are multilingual versions of CLIP that seem like a natural model to explore in these scenarios, even if all of the data is in English. Hence, there are natural unaddressed questions such as \"How does model size impact performance disparities?\" and \"How does multilingual training impact performance disparities?\" It is entirely possible that more accurate are more biased. Though, we can not know that without proper experiments.\n\n**AR** _We performed subsequent analyses, which show that a similar performance variation trend also exists for larger CLIP models as discussed in our answers to your questions (below)._\n\n_**Questions for Authors**_\n\n**RC** Question A: How would you expect model performance to change based on model size?\n\n\n\n**AR** _Before conducting the experiments, we expect that larger CLIP models would show a similar performance variation as CLIP ViT-B-32 which we used in our paper. We conducted the RQ1 experiment, which measures average CLIP scores across income bins, for the following CLIP versions: ViT-L-14 from OpenAI and ViT-G-14 pre-trained on LAION, and we found that the trend of decreasing average CLIP scores from high-income to lower-income household images still holds as shown in the table below._ \n\n\n| **Quartile name**  |**Income range**  | **Avg CLIP Score (ViT-B/32)**  | **Avg CLIP Score (ViT-L/14)**  | **Avg CLIP Score (ViT-G/14)** |\n|--------------------------|--------------------------|------------------------------------------|------------------------------------------|------------------------------------------|\n|poor                        | 26.9 - 195.0           |  0.233                                       | 0.259                                        | 0.321                                         |\n|low-mid                   | 195.4 - 685.0         | 0.250                                        | 0.282                                        | 0.350                                         |\n|up-mid                    | 694.0 - 1,998.0      | 0.257                                        | 0.295                                        | 0.363                                         |\n|rich                         | 2,001.0 - 19,671.0 | 0.256                                        | 0.295                                        | 0.363                                         |\n\n\n_To determine which models perform better on low-income data, we also intend to calculate R-precision scores following the RQ2 experiments for each model and include these results in the camera-ready version._ \n\n**RC** Question B: Would you expect multilingual variations of CLIP to have different findings?\n\n\n**AR**\n_This is a very interesting suggestion, thank you. We expect roughly similar performance but we plan to explore this in future work._\n\n_**Missing References**_\n\n**AR** We acknowledge that these papers are relevant to our work and we will add them to the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, this is a very well-written paper. The evaluation is important and novel. Moreover, the experiments are well done and showcase robust performance disparities across income levels.", "reasons_to_reject": "(Minor) The model only explores CLIP, and more specifically, thy only use a single model variation of CLIP (ViT-B/32). Larger CLIP versions have better performance (ViT-L-14). Moreover, there are multilingual versions of CLIP that seem like a natural model to explore in these scenarios, even if all of the data is in English. Hence, there are natural unaddressed questions such as \"How does model size impact performance disparities?\" and \"How does multilingual training impact performance disparities?\" It is entirely possible that more accurate are more biased. Though, we can not know that without proper experiments.\n\n\nREFERENCES:\nCarlsson, Fredrik, et al. \"Cross-lingual and multilingual clip.\" Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022.\n\nWang, Jialu, Yang Liu, and Xin Wang. \"Assessing Multilingual Fairness in Pre-trained Multimodal Representations.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\nDong, Xiaoyi, et al. \"Clip itself is a strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with vit-b and vit-l on imagenet.\" arXiv preprint arXiv:2212.06138 (2022).", "questions_to_authors": "Question A: How would you expect model performance to change based on model size?\n\nQuestion B: Would you expect multilingual variations of CLIP to have different findings?", "paper_topic_and_main_contributions": "This paper investigates the digital divide in terms of model performance of OpenAI's CLIP image-text model. The paper shows that CLIP scores or correlated with income level, R-Precision differs substantially across income levels (i.e., image retrieval quality given a topic), and also observes differences across countries where Africa-related images are observed to have the worst performance. The fine-grained analysis observes that the differences in performance occur because of larger image variations in low-income locations/settings. Finally, the paper introduces important lessons learned, e.g., introducing geographically-diverse datasets and ensuring that the data diversity standards/performance of models should match their intended use (e.g.,  general purpose models should have the highest standard).", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This paper may be relevant from a geographic analysis standpoint, though it only examines geographic differences within the US. I imagine the disparate differences would hold across income variations within a country.\n\nLwowski, Brandon, Paul Rad, and Anthony Rios. \"Measuring Geographic Performance Disparities of Offensive Language Classifiers.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022."}}, {"review_id": "SWMqTWhPGd", "reviewer": "Reviewer_bJzE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review, we appreciate your kind comments. We are very glad to hear that you found our paper to be clearly written and the findings to be meaningful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is a clearly written investigation of an important topic (economic / geographic bias in pre-trained models). The results show a meaningful difference in performance between the lowest income groups and other groups. And the discussion helps us situate and interpret that gap. Overall, the paper makes a convincing case that these sorts of geographic/demographic approaches to evaluation are important for ensuring that models generalize well.", "reasons_to_reject": "This a strong paper and there are no substantial reasons to reject it.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper uses an existing dataset of geo-coded images with category labels to investigate the economic (and geographic) performance distribution of a population Vision-Language model. The main findings are a gap between the poorest households and all other socio-economic levels. The analysis takes into account the social structure of various image topics (c.f. Figure 2) and overall provides a helpful discussion of the sources of this disparity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the paper investigates a well-motivated and interesting research question, using comprehensive evaluations. Reviewer Vvi2 raised a soundness concern related to unnormalized use of the CLIP score. However, I agree with the author response that the unnormalized score provides a measure of label-image relevance which is independent of the contrasting images/text chosen, as also used by previous work. I also agree that the paper does make a novel contribution, as previous work on this Dollar Street benchmark does not look at disparities related to income values, and while I agree with reviewer Vvi2 that this result might not be too surprising given prior work, this is still valuable to confirm.
Decision: Accept-Main