No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Authors: Angéline Pouget
Lucas Beyer
Emanuele Bugliarello
Xiao Wang
Andreas Peter Steiner
Xiaohua Zhai
Ibrahim Alabdulmohsin
Venue: NeurIPS 2024 poster (2024)
Keywords: cultural diversity
benchmarks
vision-language models
Primary area: human-AI_interaction
TL;DR: We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs).

Abstract: We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for developing VLMs that better represent global perspectives.

No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Angéline Pouget*t apouget@ethz.chl ch
Lucas Beyer 1beyer@google. com
Emanuele Bugliarello bugliarello@google com
Xiao Wang wangziao@google. com
Andreas Peter Steiner andstein@google..cm com
Xiaohua Zhai xzhai@google.co com
Ibrahim Alabdulmohsin ibomohsin@google. com
Google DeepMind
Abstract
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image- text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding, Notably, this performance gap is not captured by-and even at odds with-the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for developing VLMs that better represent global perspectives.
Introduction
Contrastive langunel models (VLMs) have emerged as powerful and versatile method to bridge the gap between visual and textual information in deep learning systems. They utilize dual-encoder architecture map both images and texts into a shared latent space. Representations in this latent space are learned leveraging large datasets of noisy image-text pairs from the web. Work including CLIP [46], ALIGN [31] and SigLIP [75] validates this approach scale with impressive zero-shot transfer results across wide range of downstream tasks.
However, due to the growing range of applications for contrastive VLMs, it imperative to evaluate them not only with respect to standard performance metrics, such as their classification accuracy on ImageNet-ILSRCC SLCC200 [17] or image-text retrieval performance on COCO [38], both of which are [53, 16]. but also in terms of "cultural diversity." We illustrate what we mean in Figure (a) and (b). When performing zero-shot classification on images from the Google Landmarks
Dataset (GL.vv)) [66], SigLIP models trained on only English-language image- text pairs (henceforth
Work performed while interning at Google DeepMind. CCorrespondig authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
em:
CN
Tower
(b) em: Notre-Dame Basil- ica of Montreal (Canada) globe: Catedral da Sé de São Paulo (Brazil)
(c) O-shot accuracy improvement [%] on the Dollar Street datasct when switching from English-only (en) to global data with English translation (globe-tl).
(Canada)
globe:
Milad
Tower
(Iran)
Figure 1: Models trained on English image-text pairs exhibit a lack of diversity when evaluated on images from other regions, sometimes confusing landmarks with similar ones located the West.
denoted en) tend to misclassify international landmarks as similar-looking landmarks located in English-speaking countries. Note that this is the currently prevalent pretraining approach in the literature. In contrast, SigLIP models trained on the full, global data (henceforth denoted globe) identify the correct landmarks.
similar observation can also be made in Figure (c), where we compare the zero-shot classification accuracy of model trained to English using the Google Translate API (henceforth denoted globe-tl). As can be seen, the en model seems to be biased towards data from Western regions. Switching globe-tl data lowers performance for images from North America and Europe but significantly improves performance for other regions such as South America and Africa that traditionally underrepresented in Al.
data that of model trained on globe data with its text translated
is worth clarifying that cultural diversity in our context is different from "fairness" and the mitigation of societal stereotypes. While recent work has shown that CLIP models perpetuate and amplify social biases and stereotypes present in the training data [27, 7, 2, 43], our emphasis is different. We focus improving the ability for VLM recognize and accurately interpret visual and textual data from a wide range of geographical, socioeconomic and cultural contexts, such as physical surroundings, traditions, customs and everyday goods. Evaluating cultural diversity is critical because it ensures that VLMs perform well across diverse environments and that they recognize and respect the varied perspectives that exist worldwide.
In this work, we present a comprehensive study of cultural and socioeconomic diversity, focusing on the impact of training data source, composition and processing (including translation), using the recently introduced SigLIP models [75] as a case study. We cover a wide range of benchmark datasets and evaluation metrics. Amongst our findings, English-only image-text pairs (en) achieve state of the art results on popular benchmarks (ImageNet, COCO), this filtering disproportionately hurts model performance for low-income households and regions, and negatively impacts cultural diversity. Crucially, these en models achieve demonstrably lower performance cultural diversity benchmarks global data. Conversely, models pretrained on the full, global data (globe, globe-tl) followed by brief English-only fine-tuning can match and even outperform the English-only baselines the popular Western-orientel benchmarks, while also performing demonstrably better in cultural diversity benchmarks. Hence, pretraining on global data yields better foundation models.
show that while SigLIP models trained on
after fine-tuning
more diverse and
In addition, we introduce the task of geo-localization based on datasets such as XM3600 [58], Dollar Street [49]. GeoDE [47] and GLDv2 [66]-as novel evaluation metric for cultural diversity, We show that, unlike, for example, XM3600 retrieval that evaluates multilinguality, geo-localization strongly dependent the global composition of the dataset used during pretraining.
In line with Alabdulmohsin et al. [2], we focus on contrastive learning for several reasons. These models have a wide range of applications, e.g. zero-shot classification and cross-modal retrieval, and are being increasingly adopted critical domains like healthcare [76, 52], and as backbone for other models [73, 72, 41]. We study SigLIP [75], the currently best-performing and widely used CLIP-

---

style model. SigLIP and CLIP operate on the same principle of aligning representaionss/mbeddigg for texts and images and the difference is only in the choice of the loss function.
2 Preliminaries
SigLIP Overview. Given mini-batch B- {I,, (1,, of image-text pairs, an image encoder f(.) and a text encoder g(-), SigI.IP aims to align the image embeddings x; = ((T)) in the given hatch with their corresponding text embeddings y; ((:T)//ll((( 2|2- The sigmoid-basee based loss (B)) processes every image-text pair independently with a positive label
24 1 for the matching pairs (4i, and a negative label žij = -1 for all other pairs (4,, Ti
B
B
L(B)
log (1 + exp (2i( txx y3 + b))),
where >> 0 and b are learnable temperature and bias parameters. We follow Zhai et al. [75] in most
aspects: use a Vision Transformer (ViT) [18] for images and a Transformer [60] for text, both of
size Base (B), with an embedding dimension of 768. Images are resized to 256 x 256 resolution and we use 16 x 16 patch size. Text imput is tokenized using model trained on mC4 [69], Common Crawl-based dataset covering 101 languages, with a vocabulary size of 250 We keep a maximum of 64 tokens. We use the modified Adafactor optimizer [54, 74] for all our experiments with an initial learning rate of 10 reverse square root decay, weight decay of 10 44 and 50 k warmup and cooldown steps. Training batch size is 16k. Models using Tensor Processing Units (TPUs) [32]. Each model is trained on 10B image-text pairs (roughly 610k steps) for about k0 TPUv2 core-hours, so they are compared
developed in the Big Vision codebase [6]
mamphle-matched regime.
Pretraining Data. We base our analysis on a range of models trained on different subsets of the WebLI dataset, a high-volume image-text dataset collected from the public web [15]. Each example in our filtered subsets contains a caption in the original language as well as an English translation if the caption not in English. Hence, we can distinguish between three different dataset variants: (1) globe, (2) en, and (3) globe-tl. Here, globe denotes the raw, multilingual data with minimal filtering applied (e.g., removing sensitive and personally identifiable information [15]. We denote its subset that contains only English captions by en. This mirrors the filtering that is currently applied in several influential papers, including CL.P, ALIGN and SigL.IP, as well as the common way [22, 21, 67, 56, 57] of using LAION [51] and DataComp [22]. The third and last variant is globe-tl, which consists of the same images as globe but with an added prepprccessing step which any non-English text is machine-translane the impact of multilinguality, such as low-resource languages being severely underrepresented [8], and cultural diversity of the trained models. Since it is plausible that VLMs may not leverage their full multilingual potential when prompted in non- English languages, as has been observed for LLMs [20], we report results for both globe and globe-tl in all experiments and highlight any notable differences. To determine statistical significance of our results, we train 3 models each for en, globe and globe-tl with different random seeds. We perform two-sample t-tests and report the 95% confidence intervals.
to English. We use this variant to differentiate between
Evaluation Data. To evaluate cultural diversity, we use five datasets: Dollar Street [49]. GeoDE [47], GLDv2 [66], XM3600 [58] and MaRVL [39]. These datasets satisfy the criteria of being of sufficiently high quality and collected with geographical diversity in mind, see Figure 2. In addition, they can readily be used for evaluating contrastive VLMs without decoder, by supporting either zero- or few-shot classification or cross-modal retrieval.
The Dollar Street (DS) [49] dataset encompasses 38 479 images depicting 289 household items com- monly found in everyday settings across 63 countries. Each image is tagged with object descriptors and demographic data such region, country, and monthly income. Following Rojas et al. [49], we map 96 topics to ImageNet classes, resulting in a subset of 21 536 images. This subset is split into training and testing sets (17 228 and 307 images, respectively). Notably, object tags (including the fuzzy matching to ImageNet labels) are not mutually exclusive. We use this dataset for zero-shot object classification and geo-localization DS has been released under the CC BY-SA 4.0 license.
We also evaluate our models on a geographically diverse subset of the Google Landmarks Dataset v2 (GLDv2) [66], featuring 1 542 images representing 884 landmarks from 84 countries [35]. These
3

---

Dollar Street
GeoDE
GLDv2
10
8
6
10
8
10
XM3600
MaRVL
ImageNet/ COCO
Figure 2: Data distribution [%] for each of the evaluation datasets, only approximate in MaRVL [39] based on the 5 languages collected in the dataset. Dollar Street [49], GeoDE [47], GLDv2 [66] and XM3600 [20] are geographically diverse. MaRVL is included because it focuses on underrepresented regions, such as Asia and East Africa. By comparison, ImageNet examples are mostly from a few Western countries (see for instance [53]). COCO has nearly identical distribution to ImageNet [16].
images serve as the public and private test datasets for the 2021 Google Landmark Retricval Challenge, providing benchmark for evaluating landmark recognition algorithms [34]. We use it for zero-shot landmark classification using the English names of all 884 landmarks as class labels. GLDv2 images have CC-0 Public Domain licenses. Annotations are licensed by Google LLC under CC BY 4.0.
GeoDE [47] is geographically diverse dataset comprising 61 940 manually annotated images categorized into 40 classes. The dataset emphasizes object classes across six world regions: Europe, Americas, West Asia, East Asia, Africa, and Southeast Asia. Images were collected through crowd- sourcing, with rigorous manual verification procedures implemented to ensure data integrity. We use GeoDE for zero-shot object classification as well as for geo-localization. Since GeoDE does not have train and test splits, shuffle the data and take the first 000 images for training and the rest for testing in the few-shot cco-localization evaluations. The dataset is licensed under CC BY 4.0.
The Multicultural Reasoning over Vision and Language (MaRVL) dataset [39] presents an ImageNet- style concept hierarchy designed to encompass a broader linguistic and cultural spectrum. The dataset incorporates five languages: Indonesian, Mandarin Chinese, Swahili, Tamil, and Turkish. Both the conceptual categories and associated images are curated exclusively by native speakers of each respective language. While the primary task on MaRVL involves validating statements concerning image pairs, the dataset holds potential for broader applications, including single-image evaluation metrics. The MaRVI. texts and features are distributed under the CC BY 4.0 license. Image access is provided only for (non-commercial) research purposes.
Finally, Crossmodal-3600 (XM3600) [58] is multilingual evaluation dataset that comprises 3 600 images accompanied by 261 375 human-generated reference captions spanning 36 languages. The dataset is sourced from the Open Images Dataset [36], with 100 images per language. Quality assurance measures, including post-annotation verification process, attest to the overall high quality of the captions. We report both image-caption retrieval and lco-localization results. In the few-shot looalocaliz evaluation, since XM3600 does not have train and test splits, we randomly shuffle the data and use the first 1 800 images for training and the second I 800 images for testing. The annotations are licensed under the CC BY 4.0 license.
Summary of Findings. Before digging into the detailed results, we summarize our key findings:
The currently predominant paradigm of directly or indirectly filtering the training data to English image-text pairs negatively impacts cultural diversity and disproportionately hurts communities of lower socioeconomic status, exacerbating existing disparities. Its impact is demonstrably captured by zero-shot classification accuracy on Dollar Street, GLDv2, GeoDE, and MaRVL (Section 3.1, Table 1, Figure 3). This has been known for vision datasets, such as ImageNet and OpenImages [53], but is relatively less explored in image-text pretraining data scraped from the Web.
2. The image features learned by models trained on such filtered data are less culturally diverse. We demonstrate and quantify this by introducing the few-shot geo-localization task. A

---

Table 1: Filtering training data to English image text pairs negatively impacts cultural diversity but improves performance on standard benchmarks. Asterisk (") denotes statistical significance at the 95% confidence level. No statistically significant differences are observed for XM3600 retrieval.
en
globe
globe-tl
vs. globe-tl
Culturally diverse zero-shot evaluations
Dollar Street
48.52
10.53%
48.82 10.34%
49.96 10.71%
+1.44%
GLDv2
43.84
50.52%
46.18 =1.30%
49.46 #1.17%
+5.62%
GeoDE
91.82 +0.39%
92.00 =0. 10%
92.84 =0.05%
+1.02%
MaRVI. Concepts
68.30 =0.50%
69.09 28.2
69.96 +0.35%
+1.66%
Crossmodal (XM3600) retrieval top-1 recall
English captions
Image - Text
50.60 11.54%
49.10 28.88
49.74 11.0%%
-0.86%
Text + Image
47.74 23.33%
45.01 = 21.21%
44.48 11883
33.26%
Native captions translated to English
Image Text
62.73
+1.88%
60.70 -.38%
62.02 +1.1%
-0.71%
Text - Image
56.49
+1.28%
$2.60 112%
54.13 +1.28%
-2.36%
Prevalent Western-oriented benchmarks
0-shot ImageNet
70.36 1..28%
66.81 10.18%
68.23 10.19%
2.13%
COCO I-I R@!
59.28
=0.90%
55.81 59%
54.00 11.39%
-5.28%
COCO R@
42.91 50.56%
38.09 50.58%
37.78 23.23%
-5.13%
linear probe on the image encoder shows en trained image encoders to be significantly less "world-knowledgee" than globe or globe-tl trained image encoders (Section 3.2, Table 2).
3. These performance disparities are not reflected by, and even at odds with, the currently most popular (and often sole reported) benchmarks based on ImageNet [17] and COCO [38]. In addition, we present evidence that benchmarks used to evaluate multilinguality such as XM3600 [58], are insufficient to evaluate models' cultural diversity (Section 3.3, Table 1).
4. As a potential way out of this conundrum, we find that pretraining on unfiltered (global) data followed by fine-tuning on English-only data improves cultural diversity without sacrificing performance on the popular Western-centric benchmarks. This allows practitioners and researchers to strike a balance between these otherwise competing metrics. These perfor- mance improvements across benchmarks are further enhanced by translating training data to English (Section 3.4, Figure 4, Figure 5).
In summary, we call for stop of pretraining on directly or indirectly English-filtere data.
3 Detailed Results
3.1 No filter for improved cultural diversity
To assess models' cultural diversity, we report zero-shot classification accuracy on Dollar Street, GLDv2, GeoDE and MaRVL. This evaluation includes tasks such as recognizing common house- hold items across different countries and income brackets (Dollar Street and GeoDE), identifying significant landmarks and places of worship (GLDv2) and categorizing image concepts (MaRVI.). Detailed results are provided in Table Pretraining on globally diverse data yields substantial enhancements across all zero-shot classification metrics related cultural diversity. However, these improvements stand in contrast to the popular benchmarks of ImageNet zero-shot accuracy and COCO retrieval scores. Given their prominence, it is not surprising that filtering training data to English image- text pairs, directly or indirectly, has quickly established itself as the preferred choice [46, 31, 75, 22, 21, 67, 56, 57]. When considering the cultural and geographical diversity of the resulting models however, a very different picture presents itself with models trained on globe-tl across all four benchmarks by a significant margin.
and globe outperforming those trained
Indeed, a more fine-gran tne-grrined analysis zero-shot classification on Dollar Street confirms that filtering to English-only training data disproportionately hurts low-income and non- Western communities;
5

---

Dollar Street
MaRVL Concepts
54
73
Accuracy
70
48
70
63 44
75
40
75 64
Figure 3: Filtering English-only data further exacerbates existing performance disparities across socioeconomic subgroups. LEFT: Zero-shot classification results for Dollar Street, disaggregated by income level (z-axiss). The performance difference between en and globe-tl is larger for lower-income households. Also, the performance disparity between the lowest and highest income groups is 32.5% en (from 29. 29.9% in $0-200 income group 62.4% in $1998 income group), but this gap reduced (improved) to 37.4% in globe-tl. RIGHT: MaRVI. Concepts classification accuracy disaggregated by each of the five languages/regions: Pretraining on globe-tl improves performance for Indonesian, Turkish and Mandarin Chinese and yields similar performance to en for Tamil and Swahili.
Figure (right) and Figure 3 (left). In MaRVI, performance for all regions (except TA) tend to benefit from using globally diverse training data, as shown in Figure 3 (right). The difference in classification accuracy for different income groups and geographic regions a worrying signal the inherent biases of filtering to English-only data. Removing the English language filter unsurprisingly leads to a drop in performance for images from Western countries, as exemplified by ImageNet and COCO benchmarks, but it significantly improves performance for the rest of the world.
3.2 Few-shot geo-localization
The disparities between models' cultural diversity become more pronounced when considering the image encoder's few-shot geo-localization performance, which we introduce in this work. This task involves learning predict the geographical origin of an image, be it at the country regional level, with only a limited number of training samples per location. We do this using a linear classification probe on the image representation, employing squared loss and 1.2 regularization problem that admits closed-form solution. Our training dataset comprises a constrained number of images per location, with results reported for sample sizes of 5, 10, or 25 instances. Should the available examples for a given location in the training set fall below this threshold, we utilize all accessible samples. Our results (Table 2) suggest that few-shot reo-localization holds promise as novel metric to assess cultural diversity in VLMs.
Independent of the prediction target being more fine- fine-grained (country) (region), both globe and globe-tl have a significant edge over en, as shown in Table 2. These results suggest that models that that capture country- or region-specific information. Another point that is noteworthy, especially when comparing Table Table 2, that the difference in performance between the globe and
coarse-grain grained
trained sufficiently diverse and global data fail to learn features
Table 2: Global data improves few-shot geo-localization performance significantly. Performance differences are statistically significant for all reported results at 95% confidence level (*).
Task
Shots
en
globe
globe-tl
en vs. globe-tl
5
11.33 10.22%
14.16 10.23%
12.81 10.60%
+1.18%
Dollar Street (country)
10
17.45 10.33%
21.51 10.56%
20.42 +0.61%
+2.97%
25
24.40 +0.38%
30.11 +0.30%
29.17 +0.27%
+4.77%
5
14.35 +0.36%
19.04 +0.32%
19.24 40.46%
+4.89%*
XM3600
10
18.56 10.17%
25.83 +0.50%
25,98 +0.94%
+7.42%
(country)
25
25.96 +0.33%
34.76 +0.90%
33.85 +0.28%
+7.89%*
5
12.66 +0.37%
19.59 +0.63%
19,91 +1.37%
+7.25%
GeoDE (country)
10
16.41 1..566t
26.29 10.38%
26.23 10.81%
+9.82%
25
23.24 10.26%
37.13 10.51%
36.54 10.27%
+13.30%
5
28.18 1..22%%
33.32 10.37%
34.51 11.90%
+6.33%
GeoDE (region)
10
32.03 +1.01%
40.48 +0.3/%
41.09 +1.33%
+9.06%
25
38.86 +0.75%
49.61 40.73%
50.38 11.3%%
+11.53%

---

ImageNett Fine-Tuning Impact
GLDv2 Fine-Tuning Impact
globe-d
0-shet 0-shot Accuracy
0.68
globe d
globe d
en
0.46
0.66
0.42
0.64
0.38
0.62
0.34
500k
540K
580k
620k
500k
540k
580k
620k
Training stop
Training stop
Figure 4: Fine-tuning globe-tl on en quickly catches up with en for ImageNet zero-shot evaluation while also performing better on GLDv2. Conversely, fine-tuning en on globe-tl does not suffice to close the gap in performance on culturally diverse benchmarks.
globe-tl models is notably smaller in the few-shot setting. This is not surprising since the standard zero-shot classification prompt templates and object landmark names are all in English and, hence, this more closely resembles the globe-tl training data version and does not leverage the multilingual capabilities of the globe model. In the few-shot setting, by contrast, we only use the image encoder's representations and hence the impact of the text tower on the evaluation results is reduced.
3.3 Decoupling multilinguality and cultural diversity
As shown in Table 1, models trained on English-only data (en) perform best on Western-oriented benchmarks. New benchmarks such as XM3600 have recently been introduced to evaluate multilin- guality in VLMs. To recall, XM3600 contains 100 images each from 36 different linguistic regions, captioned by native speakers all 36 languages. At first, it might seem that performing image-text retrieval based on the English captions the English translation of the native captions for all 3 600 images could serve as a viable signal for cultural diversity.
However, when comparing our models' performance on these two tasks, we do not find any statistically significant differences between the three models. Unsurprisingly, the globe model performs best when performing retrieval non-English captions since the other two models have only seen English texts during training. However, when evaluating all models on the Iaglish-language captions or the English translations of captions in other languages, there are statistically significant differences between the three model variants. We hypothesize that this because XM3600 is derived from the Open Images dataset, which contains primarily Western images [53] or, images quite similar to what already available in English domains. Closer inspection most images from non- Western countries are likely taken by tourists. For example, among the 100 images from the Arab world, 12% are images of cars. These images do not adequately reflect cultural differences. Moreover, since the original caption language is often English, similar images likely to have been included the en training data, explaining why there are no statistically significant differences between the three models. When comparing retrieval scores between English captions and English translations of captions in other languages across all three models, we observe significant difference. Upon further investigation, detailed descriptions of the target image. leading to higher retrieval accuracy. This discrepancy might not be associated with cultural diversity or multilinguality but rather reflects variance in annotators.
the XM3600 images confirms that
found that nonglihlish captions tend provide more
Based on these findings, we argue that datasets originally created for evaluating multilinguality, such as XM3600, might not be sufficient for evaluating cultural diversity in multimodal systems.
3.4 Bridging the gap
Fine-tuning. As shown in Table 1, improving diversity generally results loss of performance on standard benchmarks. In general, we found that the two objectives typically compete with each other: Figure 4, where we take two models pretrained on either en or globe-tl data and fine-tune them on the other dataset for a short duration. Clearly, improving culturally diverse metrics is accompanied by a loss in performance on Wester--oriette benchmarks and vice versa. This even more clear when looking at the correlation coefficients across metrics of over 40 models, shown in Figure 5 (b).
7

---

Data Ratio Trade-off
Correlation Matrix of Evaluation Metrics
1.00
C1 globod
(3) Oeshot INlt
data mniking
globo on
COCO
0.75
obbed!
(3) COCO 121
0.50
0-shet Imagetet
Dolar
Monn Couffiden
70
(5) D-shot GLDv2
O-shot GeoDE
0.00
(7) 0-shot MeRVI
50%
-0.25
[8] Gec-loc Dollar Street
68
(9) XN1300
(10) Goo loc GooDE tcountry)
67
Goo loc GooDE (ragion)
O-shor Gi Dv2
-1.00
(11)
Figure 5: LEFT: Fine-tuning allows for controlled trade-off between cultural diversity and perfor- mance standard benchmarks. Fine-tuning globe-tl on en is strictly better than fine-tuning en on globe-tl, but mixing training data different proportions achieves better trade-off overall. Values in percentages [%] correspond to the fraction of time training is restricted endata. RIGHT: Correlation coefficients the evaluation metrics computed based on over 40 fully trained models.
In Figure 5 (a), this trade-off further visualized. We first pretrain models on either en or globe-tl for set number of steps (varying between 100k and 600k) and then switch to the other dataset for the remainder of the training duration. Each model trained for approximately 610k steps in total. We plot the resulting performance the end of training for each of these switch-over points. In line with what we see in Figure 4, we find that very little fine-tuning is sufficient to significantly impact model performance and improve performance either for standard benchmarks (green squares) or for the cultural diversity evaluations (red circles), depending on which dataset was used at the end. Hence, we fine-tune for more than 100k steps, final model performance is mostly determined by the dataset that the model is fine-tuned on.
Notably, it is possible to improve performance cultural understanding (e.g. improving O-shot accuracy in GLDv2 from 44% in en to over 45.5%) without impacting ImageNet O-shot accuracy by pretraining on globe-tl and fine-tuning that model English-only pretrained models achieve lower performance on culturally diverse benchmarks even after fine-tur fne-tuning global data with the tradeoff curve being clearly suboptimal to the observed when fine-tume line-tunin culturally diverse models on en data. Therefore, pretraining on globally diverse data before fine-tuning on English-only image-text pairs allows for nuanced trade-off between catering to cultural diversity and achieving strong performance on well-established benchmarks.
data. However, the opposite does not hold.
Data mixing. Besides fine-tuning, we also study the impact of mixing the two versions of data during pretraining. Because en is subset of globe-tl, mixing is equivalent to assigning more weight to English data. Figure 5 highlights that data mixing is as good as fine-tuning (if not better) in achieving balance between Western-oriee orienntd and culturally diverse benchmarks. Moreover, it is applicable in settings where we may have more than 2 different data splits. However, data mixing entails training new models ab initio, thus incurring higher computational cost compared to fine- tuning our setting, we observe that fine-tuning for as few as 50k steps often sufficient. By contrast, training new model from scratch is more than 12 times expensive. Table provides detailed comparison between data mixing and fine-tuning for similar mixing ratios.
To conclude, both fine-tuning models pretrained globe-tl well as choosing appropriate data mix during training can be viable approaches to navigate the trade-off between cultural diversity and optimizing performance Western-oriented but well-studied benchmarks, such as ImageNet.
3.5 Quality Filters
We apply quality filtering using an internal model trained on image-text pairs to calculate the image- text similarity score, in order to assess if our empirical findings continue to hold this context. The model was trained on global data to make sure it accurately assesses quality for global data, of which English data a substantial fraction, and threshold was tuned balance quality and quantity. We filter out about 60% of the data in experiments. Appendix B shows that our main

---

findings continue to hold even when quality filters are applied. For instance, quality-filtert filtered globe-tl performs better than quality-filteree en on O-shot Dollar Street, GLDv2, and GeoDE but performs worse on Western-oriented henchmarks, such as ImageNet and COCO retrieval. In addition, the improvement in quality-filtered globe-tl over quality-filtered en is particularly significant for few-shot geo-localization tasks.
4
Related Work
range of prior work has studied biases in zero-shot classifiers related to a range of sensitive attributes including gender, race and age [1, 25, 27, 12, 24] and found that CLIP models perpetuate biases present in the training data [7, 23, 2]. More recent work extends this analysis to zero-shot performance across groups of different income levels and geographic regions [43, 78]. While several of these papers highlight the central role of training data and the potentially large impact of design choices such the data source filtering techniques [22] or translation of English captions for cross-modal multilingual encoders [10, 11, 45), to our knowledge, we are the first to study the impact of image-text pair filtering and text translation on cultural understanding in contrastive VLMs.
Contrastive models are usually evaluated on a range of benchmark datasets including ImageNet [50] (and variations [29,5]] 5)), COCO [38] and Flickr30K [71]. These datasets have been shown to reflect heavy Western bias [53, 16, 55]. Over the last few years, a range of alternative benchmarks have been proposed, including DollarStreet [49]. GeoDE [47], Crossmodal-33 -300 [58], GLDv2 [66], GeoNet
[33]. YFCC [19], XGQA [44], MaXM [13]. Ego4D [26] and ED-VCR [70]. While we have used some of these in our experiments, we decided against using others for the following reasons. GeoNet uses only images from North America and Asia and is hence not sufficiently diverse. Geo contains images from 62 different countries, but Europe-centric and based images with noisy tags [47]. xGQA mainly evaluates multilinguality; the starting point for its creation is a monolingual English dataset that was then translated. MaXM an adaptation of XM3600 (which we already use) to multilingual VQA, which contrastive VLMs do not support. Ego4D contains images from only 9 countries with majority of the images from English-speakingg countries (US, UK). GD-VCR is geo-diverse commonsense benchmark, but is VQA task unsuited for contrastive VLMs.
The closest work to ours is [48], which argues that progress in global data (Dollar Street and GeoDE) has been much slower than the progress on ImageNet. Unlike their work, however, we study the impact of the training data mixture, study the impact of translation, suggest setup where both types of metrics can be improved, and propose geo-localization as an evaluation metric. We also consider a broader set of datasets in our study, such as XM3600, GLDv2, and MaRVL.
5 Limitations and Future Work
While our work highlights the importance of incorporating cultural and socioeconomic diversity considerations into contrastive VLMs, several limitations should acknowledged. Firstly, our experimental results are based on recently popular contrastive, encoder-only SigLIP models. While our analysis offers valuable insights, it should be extended to generative VLMs [40, 37, 77, 3, 14, 59, 61, 63, 4, 68, 65, 64, 62]. Secondly, our work primarily highlights the importance of utilizing all available data when pretraining foundation models, but there is potential for additional measures to further improve cultural diversity, such via regularization, data balancing, weight averaging [30].
Table 3: A comparison between data mixing and fine-tuning using identical ratios of en vs. globe-tl examples; e.g., 5 globe-tl for - globe-tl means that the model pretrained data for 508k steps and then fine-tuned on globe-tl data for approximately 102k steps.
Osshol uccuraccy on GLDv2
O-shol accurucy on ImageNet
Proportions
Data mixing
en -> globe-il
globe-tl ->
Data mixing
en - globe-tl globe-tl ->
5 globe-tl
46.43%
46.24%
44.81%
70.28%
68.45%
70.87%
en 2
globe-tl
46.04%
49.22%
44.29%
70.18%
68.36%
70.43%
globe-tl
48.31%
48.23%
15.27%
69.67%
68.12%
70.63%
globe-t1
48.70%
19.22%
15.33%
69.02%
68.18%
70.46%
globe-t!
49.68%
49.81%
45.91%
68.92%
67.92%
70.33%

---

Thirdly, although our study consciously separates cultural diversity and multilinguality. investigating their intersection presents an intriguing subject for future research. Fourthly, acknowledging the vagueness of the notion of culture and cultural diversity, we recognize that our experiments are mainly comparing model performance across different countries, regions, or income groups. We do not offer a precise definition of cultural diversity in the context of VLMs and do not claim to cover all aspects of cultures in our analysis. Lastly, we do not address the relationship between cultural diversity and social biases that have been shown to be perpetuated by VLMs. Exploring these connections presents an opportunity to develop more inclusive Al systems.
6 Conclusion
This work highlights the importance of considering cultural diversity when training contrastive VLMs. We recommend that researchers and practitioners move away from training models English-only image-text pairs. While this approach may seem beneficial when considering performance on popular benchmarks, such as ImageNet and COCO, it discards vast amount of valuable and culturally diverse training information and disproportionately hurts communities of lower socioeconomic status. Our findings suggest that (i) pretraining on the full dataset followed by short fine-tuning on English-only data, or (ii) pretraining on a mixture of data, achieve good performance on standard benchmarks while also promoting cultural awareness. When doing so, is important to acknowledge that there seems to be trade-off between optimized performance on standard benchmarks and maintaining cultural diversity. Practitioners should, hence, carefully consider the intended use case and the importance of cultural understanding of the resulting model when deciding their pretraining data mixture, also taking into account unintended biases possibly manifesting downstream models and applications. In specific scenarios, where downstream use is limited to English and multilingualism is not required, we have found that translating the training data into English a viable option. However, the latter approach should be applied judiciously to avoid sacrificing valuable cultural context within the data.
Acknowledgement
The authors would like to thank Michael Tschannen and Jeremiah Harmsen from Google DeepMind for their feedback on earlier drafts of this manuscript, as well as Tobias Weyand and Maribeth Rauh from Google DeepMind for the helpful discussions.
References
[1] Agarwal, S., Krueger, G., Clark, J., Radford, A., Kim, J. W., and Brundage, M. (2021). Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.028/1
[2] Alabdulmohsin, I., Wang, X., Steiner, A. P., Goyal, P., D' Amour, A., and Zhai, X. (2023). Clip the bias: How useful is balancing data in multimodal learning? In The Twelfth International Conference on Learning Representations. 2,
[3] Alayrac J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. 9
[4] Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond. 9
[5] Beyer, L., Hénaff, J., Kolesnikov, A., Zhai, X., and Oord, A. (2020). Are we done with imagenet? arXiv preprint arXiv:2006.07159,
[6] Beyer, I.., Zhai, X., and Kolesnikov, A. (2022). Big vision. https://giii com/ roogle-research/big..... vision. 3, 16, 19
10

---

[7] Birhane, A., Prabhu, V. U., and Kahembwe, E. (2021). Multimodal datasets: misogyny, pornog raphy, and malignant stereotypes. arXiv preprint arXiv:2110.01963. 2,
[8] Blasi, D., Anastasopoulos, A., and Neubig, G. (2021). Systematic inequalities in language technology performance across the world's languages. arXiv preprint arXiv:2110.06755 3
[9] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., et al. (2018). Jax: composable transformations of python+ numpy programs. 16
[10] Bugliarello, E., Cotterell, R., Okazaki, N., and Elliott, D. (2021). Multimodal pretraining unmasked: A meta-analysis and a unified framework of usion-and-llanuree berts. Transactions of the Association for Computational Linguistics, 9:978 994. 9
[11] Bugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., and Vuli'e, (2022). IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, page 2370-2392, Balitmore, MA. PMLR. 9
[12] Cabello, L., Bugliarello, E., Brandl, S., and Elliott, D. (2023). Evaluating bias and fairesss in gender-neutral pretrained vision-and lan-lan models. In Bouamor, H., Pino, J., and Bali. K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8465 8483,, Singapore. Association for Computational Linguistics.
[13] Changpinyo, S., Xue, L., Szpektor, I., Thapliyal, A. V., Amelot, J., Chen, X., and Sorieut, R. (2022). Towards multi-lingual visual question answering. arXiv preprini 22Xiv:2209005400) 9
[14] Chen, X., Wang. X., Beyer, L., Kolesnikov, A., Wu, J.. Voigtlaender, P., Mustafa, B., Goodman, S., Alabdulmohsin, I., Padlewski, P., Salz, D., Xiong, X., Vlasic, D., Pavetic, F., Rong, K., Yu, T., Keysers, D., Zhai, X., and Soricut, R. (2023). Pali-3 vision language models: Smaller, faster, stronger.
[15] Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., al. (2022). Pali: A jointly-scaled multilingual language- image model. arXiv preprini arXiv:2209.06794 3, 16
[16] De Vries, T. Misra, I., Wang, C., and Van der Maaten, L. (2019). Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 32-59. 1,4,9
[17] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR. 1,5, 16
[18] Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., De- hghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 3, 16
[19] Dubey, A., Ramanathan, V., Pentland, A., and Mahajan, D. (2021). Adaptive methods for real-world domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14340-144 14349.
[20] Etxaniz, J., Azkune, G., Soroa, A., de Lacalle, L., and Artetxe, M. (2023). Do multilingual language models think better in english? arXiv preprini urXiv:2308.01221 01223. 3,4
[21] Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. (2023). Data filtering networks. CoRR, abs/2309.17425. 5,
[22] Gadre, S. Y., Ilharco, G., Fang. A., Hayase. J., Smyrnis, G., Nguyen. T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. (2024). Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36. 5.9
[23] Garcia, N., Hirota, Y., Wu, Y., and Nakashima, Y. (2023). Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6957-6966.

---

[24] Goyal, P., Duval, Q. Seessel, I., Caron, M. Misra, I., Sagun, L.. Joulin, A., and Bojanowski, P. (2022a). Vision models are more robust and fair when pretrained on uncurated images without supervision. 9
[25] Goyal, P., Soriano, A. R., Hazirbas, C., Sagun, L.,, and Usunier, N. (2022b). Fairness indica- tors for systematic assessments of visual feature extractors. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 70-88. 9
[26] Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022). Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18995-19012. 9
[27] Hall, M., Gustafson, I.., Adcock, A., Misra, I., and Ross, C. (2023). Vision-language models performing zero-shot tasks exhibit gender-based disparities. arXiv preprint arXiv:2301.11100 2,
[28] Heek, J., Levskaya, A.. Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. (2020). Flax: A neural network library and ecosystem for jax. Version 0.3, 3:14-26. 16
[29] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. (2021). Natural adversarial ex- amples. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15262-15271. 9
[30] Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Korblith, S., Farhadi, A., and Schmidt, L.. (2022). Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262-29277.
[31] Jia, C., Yang. Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916, PMLR. 1,5
[32] Jouppi, N. P., Young, C., Patil, N., Patterson, D. A., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P., Chao, C.,, Clark, C., Coriell, J., Daley, M., Dau, M. Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho. C. R., Hogberg. D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Koch, A., Kumnr, N., Lacy, S., Laudon, J., Law, Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M. Thorson. G.. Tian, B.. Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang. W., Wilcox, E., and Yoon, D. H. (2017). In-datacenter performance analysis of a tensor processing unit. CoRR,
[33] Kalluri, T., Xu, W., and Chandraker, M. (2023). Geonet: Benchmarking unsupervised adaptation across geographies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15368-15379. 9
[34] Kim, Z., Araujo, A., Cao, B., Askew, C., Sim, J., Green, M., Yilla, N., and Weyand, T. (2021). Towards a fairer landmark recognition dataset. arXiv preprint arXiv:2108.008 08874.
[35] Kim, Z., Araujo, A., Cao, B., Askew, C., Sim, J., Green, M., Yilla, N., and Weyand, T. (2022). Improving faimness in scage-scale object recognition by crowdsourced demographic information. arXiv preprint 2rXiv:2206.01320
[36] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, L., Pont- Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2020). The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision,
[37] Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models.
12

---

[38] Lin. T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P.. Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects context. In ECCV. 1, 9, 16
[39] Liu, F., Bugliarello, E., Ponti, E. M., Reddy, S., Collier, N., and Elliott, D. (2021). Visually reasoning across languages and cultures. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 3, 3,4.10 16
[40] Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. (2023). Unified-io Scaling autoregressive multimodal models with vision, language, audio, and action. 9
[41] Minderer, M., Gritsenko, A., Stone, A., Neumann, M.. Weissenborn, D., Dosovitskiy, A., Mahendran, A., Amnb, A., Dehghani, M., Shen, Z., et al. (2022). Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728-755 Springer. 2
[42] Mitchell, M., Wu, S., Zaldivar, A., Barnes, Vasserman, L., Hutchinson, B.. Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model cards for model reporting. In Proceedings of the conference on fairness, accountabillty, and transparency, pages 220-229.11 16
[43] Nwatu, J., Ignat, O., and Mihalcea, R. (2023). Bridging the digital divide: Performance variation across socio-economic factors in vision-language models. arXiv preprint arXiv:2311.05574. 2, 9
[44] Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M. O.,, Roth, S., Vulié, I., and Gurevych, I. (2021). xgqa: Cross-lingual visual question answering. arXiv preprint arXiv:2109906000 9
[45] Qiu, C., Oneatä, D., Bugliarello, E.,, Frank, S., and Elliott, D. (2022). Multilingual multimodal learning with machine translated text. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Findings of the Association for Computational Linguisties: EMNLP 2022, pages 4178-4192 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. 9
[46] Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8763. PMLR. 1,5
[47] Ramaswamy, V. V., Lin, S. Y., Zhao, D., Adcock, A., van der Maaten, L., Ghadiyaram. D., and Russakovsky, O. (2024). Geode: a geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems, 36. 2, 3, 4, 9, 16
[48] Richards, M., Kirichenko, P., Bouchacourt, D., and Ibrahim, M. (2024). Does progress on object recognition benchmarks improve real-world generalization? In ICLR.
[49] Rojas, W. A. G., Diamos, S., Kini, K. R., Kanter, D., Reddi, V. J., and Coleman, C. (2022). The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2, 3, 4, 9, 16
[50] Russakovsky, O., Deng, J., Su, H., Krause, J.. Satheesh, S., Ma, S., Huang, Z.. Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International jounhiaaaloouuaall of computer vision, 115:211-253
[51] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. (2022). LAION-5B: an open large-scale dataset for training next generation image-text models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022. 3
[52] Sellergren, A. B., Chen, C., Nabulsi, Z., Li, Y., Maschinot, A., Sama,, A., Huang, J., Lau, C., Kalidindi, R., Etemadi, M., et (2022). Simplified transfer learning for chest radiography models using less data. Radiology, 305(2):454-46) 2
13

---

[53] Shankar, S., Halpem, Y.. Breck, E., Atwood, J., Wilson, J., and Sculley, D. (2017). No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv: 1, 7,9
[54] Shazeer, N. and Stemn, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. 3
[55] Smith, B., Farinha, M., Hall, M., Kirk, R., Shtedritski, A., and Bain, M. (2023). Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets. arXiv preprint arXiv:2305.15400
[56] Sun, Q. Fang. Y., Wu, L.,, Wang, X., and Cao, Y. (2023). EVA-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303. 15389. 3,5
[57] Sun, Q. Wang, J.. Yu, Q., Cui, Y., Zhang, F., Zhang, X., and Wang, X. (2024), EVA-CLIP-18B scaling CLIP to 18 billion parameters. CoRR, 2ld222222242.. 3,5
[58] Thapliyal, A. V., Pont-Tuset, J., Chen, X., and Soricut, R. (2022). Crossmodal-3600; A massively multilingual multimodal evaluation dataset. arXiv preprint 220.: 2,3, 4, 9, 16
[59] Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., and Beyer, L., (2023). Image captioners are scalable vision learners too. In NeurlPS. 9
[60] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N.. Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing
systems, 30. 3, 16
[61] Wan, B., Tschannen, M., Xian, Y., Pavetic, F., Alabdulmohsin, I., Wang, X., Pinto, A. S., Steiner, A., Beyer, L.,, and Zhai, X. (2024). Locea: Visual pretraining with location-aware captioners. 9
[62] Wang, J., Chen, D., Wu, Z., Luo, C., Zhou, L., Zhao, Y., Xie, Y., Liu, C., Jiang, Y.-G., and Yuan, L. (2022a). Omnivhone foundation model for image-language and video-language tasks. 9
[63] Wang, J., Yang, Z., Hu, X., Li, I.,, Lin, K., Gan, Z., Liu, 7., Liu, C., and Wang, I.. (2022b). Git: generative image-to-text transformer for vision and language.
[64] Wang. P., Yang. A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022c). Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequence learning framework.
[65] Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. (2022d). Simvlm: Simple visual language model pretraining with weak supervision. 9
[66] Weyand, T., Araujo, A., Cao, B., and Sim, J. (2020). Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575-2588 1, 2, 3, 4, 9, 16
[67] Xu, H., Xie, S., Tan, X. F., Huang, P., Howes, R., Sharma, V., Li, S., Ghosh, G., Zettlemoyer, I., and Feichtenhofer, C. (2023a). Demystifying CLIP data. CoRR, abs/2309. 16671. 3, 5
[68] Xu, H., Ye, Q. Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., Wang, W., Xu, G., Zhang, J., Huang, S., Huang, F., and Zhou, J. (2023b). mplug-2: A modularized multi-modal foundation model across text, image and video. 9
[69] Xue, L.,, Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934
[70] Yin, D., Li, L. H., Hu, Z., Peng, N., and Chang, K.-W. (2021). Broaden the vision: Geo-diverse visual commonsense reasoning. arXiv preprint arXiv:2109.06860 9
[71] Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78. 9
14

---

[72] Yu, J., Wang, Z., Vasudevan, V., Yeung. L.. Seyedhosseini, M., and Wu, Y. (2022). Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0199
[73] Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021). Florence: A new foundation model for computer vision. arXiv preprint
arXiv2211.11132.2
[74] Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022). Scaling vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12104-12113.
[75] Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023). Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303. 1, 2, 3, 16
[76] Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al. (2023). Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303. 00915, 2()):6. 2
[77] Zou, X., Dou, Z.-Y., Yang. J., Gan, Z., Li, Li, C.,, Dai, X., Behl, H., Wang. J., Yuan, L., Peng, N., Wang, L., Lee, Y. J., and Gao, J. (2022). Generalized decoding for pixel, image, and language.
9
[78] Zunair, H., Khan, S., and Hamza, A. B. (2024). Rsud20k: A dataset for road scene understanding in autonomous driving. arXiv preprint 2Xiv:2401100322.
15

---

A Appendix
A.1 Model Card
Model details following Mitchell et al. [42].
Model Architecture: The model architecture contains two towers, vision transformer
encoder [18] and language transformer encoder [60], both of size B. Models are trained using contrastive pretraining technique with sigmoid loss [75].
Inputs:
The vision encoder takes image reshaped to 256 256 input. The text encoder takes tokenized text cropped to the first 64 tokens as input.
Outputs: The vision and text encoders both output a d-dimensional feature vector where d 768.
Intended Use: The primary use is to conduct research on multimodal applications, such as zero-shot classification and retrieval. We use the models study the impact of training data filtering cultural diversity.
As noted in several prior works, multimodal systems can pick up societal biases. While we demonstrate some of those issues this work, our analysis necessarily limited in scope.
Known Caveats:
System Description: Models are analyzed stand-alone setting and not used as part of a larger system.
Upstream Dependencies: None
Downstream Dependencies: None
Hardware Software: Models are developed using JAX [9] and Flax [28] in the Big Vision [6] codebase. They trained on Google Cloud TPUs.
Compute Requirements: Each model is trained on 16 x 16 TPU chips on 10B seen image-text pairs. typical training run takes 3.3 days.
Model Initialization: The model is trained from random initialization.
Model Size: Each SigLIP model has ViT B/16 image encoder and size B text encoder.
Training Dataset: We use different subsets of WebLI [15], which consists of images with alt-texts from the public web.
Evaluation Datasets: We evaluate the models ImageNNt--ILSRC22012 [17], MS COCO [38]. Dollar Street [49], Google Landmarks Dataset v2 [66], GeoDE [47].
MaRVL [39] and Crossmodal-3600 [58].
16

---

B Impact of Quality Filters
To assess the impact of quality filters on our findings, we train models, and globe-tl, on IB image-text pairs, following the same training setup used in the paper. Table 4 shows summary of these results. We observe that our empirical findings also hold in this setting. For instance, quality-filten filttred globe-tl performs better than quality- filttredffeerddd en on O-shot Dollar Street, GLDv2, and GeoDE but performs worse on Western-orientee benchmarks, such as ImageNet and COCO retrieval. In addition, the improvement in few-shot geo-localization for quality-filtere globe-tl over quality-filtered en particularly significant.
Table 4: Applying quality filters to SigLIPs does not change the primary conclusions. Filtering training data to English image-text pairs continues to negatively impact cultural diversity even though it improves performance on standard benchmarks.
globe-tl
en vs. globe-1l
Culturally diverse zero-shot evaluations
Dollar Street
46.05%
48 28.28%
+2.23%
GL.v22
28.21% %
30.67%
+2.46%
GeoDE
90.53%
90.57%
+0.04%
Prevalent Western-oriented benchmarks
0-shot ImageNet
66.96%
66. 36%
-0.64%
COCO I-+T R@]
56.72%
52.80%
-3.92%
COCO T R@
37.33%
31.46%
-2.87%
10-shot geo-localization
Dollar Street (country)
9.40%
9.82%
+0.42%
GeoDE (country) GeoDE (region)
10%
14.73%
+4.63%
21.97%
28.22%
+6.25%
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
18

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: Yes)
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We use the SigLIP trainer as well as range of pre-trained SigLIP models that have been released as part of the Big Vision codebase [6]. While we are unable provide
access to the training data, all evaluation datasets are publicly available.
19

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.ca
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips.cc.a wttp:cccpppccccceeeeeeee..cceeee gwiiiii/g/////////eeeeeiea httpgguuudd//eoeeuubiiiiiiiiiiii
CodeSubmisss wwwsssssssslieeeeeeeoe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support th main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
20

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All experiments are executed on TPUs. A single model is trained for approxi- mately 40K TPUv2 core-hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerppp.eeeee cc cctppblic//eteeeecceeceeeceeeeee wechlicc//hiitiiiteeee csGuidelieees
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper advocates for the evaluation of VLMs on diverse datasets, aiming to potentially yield positive societal impacts by promoting more inclusivity in Al research and development.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
21

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
We cite the original owners or datasets, models, and codebase used in our experiments.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcome com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
22

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "Qt3FSDQZ7B", "reviewer": "Reviewer_dZWv", "summary": "The paper compares performance of VLMs trained on just English data, multilingual data and the English translations of the multilingual data, on a range of benchmarks including Western-centric ones as well tasks that involve geographically diverse inputs. The paper finds that (i) training on just English data negatively impacts cultural diversity, (ii) existing multilingual evaluations may not work as well as intended due to their images still being Western-centric, (iii) it is possible to improve cultural understanding without sacrificing performance on the popular Western-centric benchmarks", "questions": "1. Since **en** is a subset of **globe** (and thus assumably containing fewer training samples), are all models just trained until convergence? How are the different training set sizes controlled for in the experiments?\n\n2. In Conclusion, the authors claim that \"there seems to be a trade-off between optimized performance on standard benchmarks and maintaining cultural diversity\" - I thought the paper already shows in Section 3.4 that this trade-off could be avoided?", "limitations": "The authors have sufficiently address the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The evaluations are done throughly, and a new test is proposed, which addresses limitations of existing evaluations (e.g. XM3600 images are still Western-centric).\n\n2. Despite some concerns about the setup (see Weaknesses below), I think the paper raises an important message and some of the findings probably would hold for other training setups\n\n3. The paper proposes a new training pipeline to improve cultural understanding without impacting performance on Western benchmarks (ImageNet). This probably has practical value for a lot of ML practitioners.", "weaknesses": "1. In the \"Summary of Findings\" section, the claims should be made more specific and grounded in evidence, e.g. \n\n(i) \"exacerbating existing disparities\" --> the authors should show how much the performance *gap* between best and worst groups increase by training on **en** versus **globe**, \n\n(ii) instead of saying \"world knowledgeable\", quote specific numbers and findings.\n\n2. In figure 1 (and other parts of the paper), the authors claim that models trained on only English data do poorly on geographically benchmarks, e.g. by confusing landmarks with similar ones in Western countries. However, this result is expected given that the **en** data is a subset of **globe**, and the landmarks test may be in-distribution for models trained on **globe** and out-of-distribution for those trained on **en**. I think the paper's claim could be made stronger by better controlling for these confounding factors (e.g. by removing test-set knowledge contamination).\n\n3. Besides, I find the dataset construction used in the experiments unrealistic: the authors state that \"globe denotes the raw, multilingual data with minimal filtering applied\" - many VLMs are trained on highly filtered multilingual datasets (e.g. LAION)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the thoroughness, importance and practicality of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**Q1: How much does training on en instead of globe increase the performance gap between groups of high and low socioeconomic status?**\n\nWe show examples of this in Figure 1.c and Figure 3 (left). For instance, zero-shot classification accuracy on Dollar Street for low-income groups ($0-200) drops from 35% with *globe-tl* to less than 29.9% with *en*. Similarly, accuracy on examples from the African continent drops from 38.4% using *globe-tl* to less than 35.8% using *en*. Because of this, pretraining on English-only data exacerbates existing disparities because there is already a large gap in performance between low-income and high-income groups (as shown in the figures) and training on English-only data increases these performance gaps.\n\n**Q2: How does additional, quality-based data filtering affect the findings presented in this work?**\n\nThank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.\n\n**Q3: Shouldn’t it be expected that the globe models outperform the en model on benchmarks measuring global understanding?**\n\nNot necessarily. Please refer to the global response for our answer to this question.\n\n**Q4: How and for how long are the models trained?**\n\nAs is detailed in Section 2 of our paper, all models are trained on 10B image-text pairs (roughly 610k training steps). Hence, all models are compared on a *compute-matched* setup: *en* models are trained for ~3 epochs, while *globe* models are trained for a single epoch. Please refer to the global response for more details.\n\n**Q5: Can the trade-off between good performance on standard benchmarks and maintaining cultural diversity be avoided?**\n\nWhile pre-training on global data and fine-tuning on English data allows to balance performance on standard and culturally diverse evaluation metrics, neither fine-tuning nor data mixing allow for complete avoidance of the tradeoff between these metrics (as can be seen in Figure 5 (left)).  It is however possible that other approaches (such as different training paradigms or model weight merging) might lead to improved tradeoffs and maybe eventually even a complete avoidance thereof. We are looking forward to future work in this area!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BApnDaWCu", "reviewer": "Reviewer_mxHk", "summary": "For the field of training contrastively learned-based VLM, this paper firstly displays that training from English data would lead to worse cultural diversity in zero-classification evaluation. By discarding the influence of languages in evaluation, this paper proposed a geo-localization task, which could observe a trained model underperforms globe and globe-tl data. Besides, the paper also explores the possible strategies for achieving a better trade-off between standard performance and cultural diversity. According to the observations, the authors found that pre-train on the globe and fine-tuning in English is the optimal strategy with less cost.", "questions": "- Could you please restate the definition of the concept of Socioeconomic Diversity for me? Additionally, how can it be measured? What level of diversity do we expect the model to achieve?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper points out that cultural diversity is important but neglected in standard benchmarks.\n- The paper discussed the practice of development strategies in pretrain and fine-tuning the constrastive VLM for the trade-off between standard performance and cultural diversity, which makes the practice benefit from this research.", "weaknesses": "- The writing is not clear enough. The structure and relation between different parts of this paper is not clear.\n- The necessity of the proposed geo-localization task remains unclear in this paper. We don’t know the necessity of geo-localization compared with prior metrics that could reflect the cultural diversity of constrastive VLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We provide our answers to the reviewer’s raised questions and concerns below, are looking forward to their response, and hope for a favorable reassessment of our scores.\n\n**Q1: How is the paper structured?**\n\nWe are sorry to hear that the reviewer found the structure of our paper to be unclear. We motivate and present a brief overview of our work in Section 1, before moving on to discussing the chosen model architecture as well as training and evaluation datasets in Section 2. We also briefly summarize our key findings at the end of Section 2 and explain them in more detail in Section 3. We conclude with a discussion of related work, limitations and future work. If the reviewer has any specific suggestions for improving clarity, we would be happy to hear them and incorporate them in the revised version of the paper.   \n\n**Q2: How does the proposed geo-localization task compare to alternative metrics of cultural diversity in contrastive VLMs?**\n\nAs can be seen in Figure 5 (right), the proposed geo-localization task is well correlated with other metrics of cultural diversity in contrastive VLMs (such as zero-shot classification on Dollar Street, GLDv2, GeoDE and MaRVL). One important distinction, however, is that the few-shot geo-localization task only evaluates the learned image embeddings while discarding the text tower, which can provide insightful additional information. In addition, it provides a very strong signal as shown in Table 2, where differences in accuracy can reach up to 10%.\n\n**Q3: How is socioeconomic diversity defined and measured in the context of this work?**\n\nAs noted in Section 5, we do not claim to offer a precise definition of cultural or socioeconomic diversity in the context of VLMs. Instead, we use model performance across different countries, regions, and income groups as a useful proxy for these concepts. We acknowledge in Section 5 (limitations) that this may not cover all aspects of cultural and socioeconomic differences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDNbyBYVC9", "reviewer": "Reviewer_JWWG", "summary": "The paper shows that vision-language models trained solely on English filtered data displays a bias towards western-centric benchmarks. They then present a simple solution for training models that perform well on globally diverse datasets while not sacrificing performance on gold standard datasets such as ImageNet. They accomplish this by training on the unfiltered dataset then fine-tuning on the relevant subset for the downstream task.", "questions": "Minor Comments: \n- It would be useful to add the size of WebLI for comparison to LAION since most CLIP-like models have been trained on LAION. \n- I think the commas in the paper have been messed up for the numbers. \n\n\n\nI think this statement in the contributions is a bit strong and overclaiming. The logical leap from zero-shot classification on Dollar Street to harming low socioeconomic groups is a bit far:\n\"filtering the training data to English image–text pairs negatively impacts cultural diversity and disproportionately hurts communities of lower socioeconomic status, exacerbating existing disparities. Its impact is demonstrably captured by zero-shot classification accuracy on Dollar Street.\"", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality:** \n\nThe paper presents a relevant and unaddressed problem of cultural bias. The remedy of only filtering for english data after pretraining is simple and effective. \n\n**Quality:**\n\nThe experiments are thorough and properly support the claims of the paper. \n\n**Significance:**\n\nThe paper could impact how researchers train foundational models and help to mitigate cultural bias in future models. \n\n**Clarity:** \nThe paper is reasonably clear to understand. There are small issues such as with figure 1, but nothing glaring.Overall the weaknesses are minor.", "weaknesses": "Overall the weaknesses are minor.\n\n- One potential confounding factor is CLIP-filtering which most text-image models use to improve accuracy. It’s unclear how this approach would interact with this and other filtering techniques. For example, it’s known that CLIP models [1] tend to filter out non-english captions. \n\n- One other potential weakness is that the technique could be computationally expensive since CLIP needs to be retrained on a potentially large English subset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the originality, quality, significance, and clarity of our work. We hope that our rebuttal below addresses all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.\n\n**Q1: How does additional, quality-based data filtering affect the findings presented in this work?**\n\nThank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.\n\n**Q2: Does the proposed removal of the English-only data filter increase computational cost?**\n\nWe believe our findings are orthogonal to having additional, quality-based data filters that can be applied in practice to limit computational cost incurred during training. In particular, while we do not challenge the use of quality filters, our work warns against using filters that are based on English-only datasets or favor English-only captions. Please refer to the global response for a more detailed response.\n\n**Q3: Size of WebLI**\n\nAs is detailed in Section 2 of our paper, all models are trained on 10B image–text pairs (roughly 610k training steps). Hence, all models are compared on a *compute-matched* setup: *en* models are trained for ~3 epochs, while *globe* models are trained for a single epoch. Please refer to the global response for more details.\n\n**Q4: What evidence supports the claim that English-only data filtering exacerbates existing disparities?**\n\nWe show examples of this in Figure 1.c and Figure 3 (left). For instance, accuracy on Dollar Street for low-income groups ($0-200) drops from 35% using *globe-tl* to less than 29.9% when trained on English-only data. Similarly, accuracy on examples from the African continent drops from 38.4% using *globe-tl* to less than 35.8% using English-only data. Because of this, pretraining on English-only exacerbates existing disparities because there is already a large gap in performance between low-income and high-income regions (as shown in the figures) and training on English-only data increases those gaps.\n\nWe also thank the reviewer for the comment regarding the commas and will make sure to carefully check this for future revisions of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GfLjPVGhqY", "reviewer": "Reviewer_nvDK", "summary": "This paper discusses the impact of the common practice of filtering for English-only data on training vision-language models, showing that this negatively impacts performance on tasks covering diverse cultural regions and backgrounds. The work proposes foregoing this filtering step to improve this representation, as well as potentially using a short English-only fine-tuning stage to achieve a more acceptable balance between standard task performance and diversity.", "questions": "Why is only SigLIP tested? Even restricted to CLIP-style models, it seems like the same methodology could be applied to any vision-language dual-encoder model, and to initialization schemes other than the random initialization used (L577) which presumably could have a significant effect on global representation.\n\nCan you elaborate on how filtering hurts performance for “low-income households” (L50)? The datasets used (L98 on) are explicitly chosen to cover diverse geographical regions, which includes developing countries, but it is not clear whether socioeconomic status has a clear effect on results when controlling for geographic region and/or language. This also relates to the paper title which explicitly mentions socioeconomic diversity, though the paper seems to demonstrate the effect of geographic diversity specifically.", "limitations": "Overall the presentation of limitations is thoughtful and fairly exhaustive. There lacks a discussion of resource and compute limitations, which affect the ability to train on larger, less-filtered datasets.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Bias in vision-language models, and generally in machine learning, is an important problem. The proposed mitigation method (training on global data without English-only filtering) makes sense and the results seem to support this practice.", "weaknesses": "The contribution of the paper seems limited, as it is unsurprising that training on global data improves performance on benchmarks measuring global data understanding. The findings are not contextualized with prior works on the effect of training CLIP-style models on large, weakly-curated datasets or for multilingual understanding [1-3]. Results are only shown for one particular type of dual-encoder VLM (SigLIP). There is also no consideration of the effect of limited resources for training, which may make training on a larger less-curated dataset less feasible and use more computational resources, which may disproportionately impact global research communities, calling into question the paper’s categorical call to stop filtering such datasets for English-language data (L161).\n\n\n[1] Carlsson et al. Cross-lingual and multilingual CLIP. LREC 2022\n\n[2] Chen et al. mclip: Multilingual clip via cross-lingual transfer. ACL 2023\n\n[3] Cherti et al. Reproducible scaling laws for contrastive language-image learning. CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful and insightful feedback. We especially appreciate their positive assessment of the importance and presentation of our work. We provide our answers to the reviewer’s raised questions and concerns below, are looking forward to their response, and hope for a favorable reassessment of our scores.\n\n**Q1: Shouldn’t it be expected that the *globe* models outperform the *en* model on benchmarks measuring global understanding?**\n\nWhile we agree that this might seem intuitive at first sight, it is not necessarily true. There are English websites published in most countries and pictures of landmarks are frequently captured by tourists, so it is possible that the English subset of the web is sufficient to capture cultural diversity. Our contribution is to provide a comprehensive evaluation of cultural diversity in SOTA contrastive VLMs, and we hope that these results suffice to clear any lingering doubts about the necessity of training on global data when building foundation models. Please refer to our global response for more detailed information.\n\n**Q2: Are the findings related to prior work on multilingual understanding?**\n\nWe thank the reviewer for the additional pointers to prior work on multilingual understanding and will make sure to include these references in the revised version of our paper. In the context of this work, however, we purposely disentangled multilinguality from cultural diversity because we believe these measure different aspects. As noted in Section 5, we do, however, see the investigation of their intersection as an intriguing subject for future research and are looking forward to new insights in that area!\n\n**Q3: Do the findings hold for other contrastive VLMs beyond SigLIP (e.g., CLIP)?**\n\nSigLIP and CLIP operate on the same principle of aligning representations/embeddings for texts and images and the difference is only in the choice of the loss function. We expect the same results to hold for CLIP as well. The reason we use SigLIP is because of its superior performance and widespread use (>1M downloads/month in Huggingface).\n\n\n\n**Q4: How does additional, quality-based data filtering affect the findings presented in this work?**\n\nThank you for raising this question. We have conducted new experiments to verify this. Please refer to the global response for a summary of our findings. Generally, we observe that the same conclusions hold even after applying quality filters based on image-text similarity. We will add more details about this in the revised version of the paper.\n\n**Q5: Are the findings limited to geographic diversity?**\n\nWhile geographic and socioeconomic diversity are inherently linked due to different income levels and economic circumstances in countries and regions around the world, Dollar Street does contain data from different income groups for every region. Kindly refer to Figure 3 (left), in which we specifically address these differences.\n\n**Q6: Does the proposed removal of the English-only data filter increase computational cost?**\n\nWe believe our findings are orthogonal to having additional, quality-based data filters that can be applied in practice to limit computational cost incurred during training. In particular, while we do not challenge the use of quality filters, our work warns against using those filters that are based on English-only datasets or favor English-only captions. Please refer to the global response for a more detailed response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for the detailed and constructive questions and comments. We especially appreciate the positive feedback on the thoroughness of our experiments, significance of findings, and clarity of work. In the below, we answer the concerns shared by multiple reviewers. We hope that these, in addition to our responses to individual reviewers, help clarify any open questions.

**Q1: How does additional, quality-based data filtering affect the findings presented in this work?** 

This is an excellent point and we plan to include this in the revised version of the paper. We have pretrained two additional SigLIP models with quality-based filtering on 1B image–text pairs each: *en* and *globe-tl*. We evaluate these on Dollar Street, GeoDE, ImageNet and COCO. We observe that the same qualitative conclusions continue to hold even after applying a quality filter based on image-text similarity. For instance, *globe-tl* performs better than *en* on 0-shot Dollar Street, GLDv2, and GeoDE but performs worse on Western-oriented benchmarks, such as ImageNet and COCO retrieval. In addition, the improvement in geolocalization is particularly significant. 

| Task | en | globe-tl |
| ---- | -- | -------- |
|Dollar Street | 46.1% | 48.3% |
|GLDv2 | 20.7% | 28.2% |
|GeoDE | 90.5% | 90.5% |
|0-shot ImageNet | 67.0% | 66.3% | 
|COCO I2T R@1 | 56.7% | 52.8% |
|COCO T2I R@1 | 37.3% | 34.5% |
|Dollar Street 10-shot | 9.4% | 9.8% | 
|GeoDE 10-shot (country) | 10.1% | 14.7% | 
|GeoDE 10-shot (region) | 22.0% | 28.2% | 

We plan to additionally train all three variants *en*, *globe*, and *globe-tl* for 10B quality-filtered image–text pairs as was done in the paper, and include the results in the supplementary material of the paper. But as shown above, the same conclusions hold even after applying quality filters.





**Q2: Shouldn’t it be expected that the globe models outperform the en model on benchmarks measuring global understanding?**

We thank the reviewers that have raised this important question. While this might seem intuitive at first sight, it is not necessarily true. There are English websites published in most countries and pictures of landmarks are frequently captured by tourists, so it is possible that the English subset of the web is sufficient to capture global diversity. In addition, we have found that certain seemingly global datasets (such as XM3600) seem to fail in capturing cultural nuances. Hence, a key contribution of our work is identifying four zero-shot classification datasets actually capturing global understanding, as well as introducing the few-shot geo-localization task. Given that this is the first comprehensive evaluation of cultural diversity in SOTA contrastive VLMs as far as we know, we hope that these results suffice to clear any lingering doubts about the necessity of training on global data when building foundation models.

**Q3: How and for how long are the models trained?**

As is detailed in Section 2 of our paper, all models are trained on 10B image–text pairs (roughly 610k training steps). Hence, all models are compared on a *compute-matched* setup: *en* models are trained for ~3 epochs, while *globe* models are trained for a single epoch. We have observed that performance gaps between *en* and *globe* models continue to persist even for significantly longer training durations. We will add more details about this last point in the revised version of the paper.

**Q4: Does the proposed removal of the English-only data filter increase computational cost?**

All models presented in our paper incur the same computational cost due to being trained for the same number of total examples seen (albeit different numbers of unique examples). As can be seen in our response to Q1, our findings are orthogonal to having additional, quality-based data filters that are often applied in practice to limit computational cost incurred during training. In particular, while we do not challenge the use of quality filters, our work warns against using filters that are based on English-only datasets or favor English-only captions.


We are grateful again for the reviewers' detailed and constructive feedback. If we have satisfactorily answered your questions, we hope you would consider revising your scores. Otherwise, please let us know if there are any other questions or concerns, so we can respond to them during the discussion period.

Meta-review: The paper presents a timely and valuable contribution to addressing cultural bias in vision-language models (VLMs), demonstrating that training on global data improves cultural understanding—without significantly sacrificing performance on standard benchmarks. All the reviewers comments have been addressed. All the scores recommend acceptance. Congrats to the authors.

Pre-training on global data and fine-tuning with English provides a practical and simple solution for balancing performance across culturally diverse datasets. The paper introduces new evaluation tasks, such as geo-localization, which strengthen its contributions.

While some concerns remain regarding the novelty of certain findings and the clarity of the structure, the reviewers generally agree that the work raises important issues on dataset biases and will benefit the broader machine learning community. The paper’s thorough evaluations and promising results support its relevance and potential impact.

Despite minor reservations about computational costs and dataset filtering, these concerns are orthogonal to the current contribution and ripe areas for future research.
Decision: Accept (poster)