Multilingual Diversity Improves Vision-Language Representations
Authors: Thao Nguyen
Matthew Wallingford
Sebastin Santy
Wei-Chiu Ma
Sewoong Oh
Ludwig Schmidt
Pang Wei Koh
Ranjay Krishna
Venue: NeurIPS 2024 spotlight (2024)
Keywords: multilingual data
vision-language models
data curation
data filtering
CLIP
Primary area: machine_vision
TL;DR: On standard English vision benchmarks (e.g. ImageNet), translating multilingual data into English and allowing more of such data to be included in the training set boost performance substantially, even if some of the benchmarks are English-centric.

Abstract: Massive web-crawled image-text datasets lay the foundation for recent progress in multimodal learning. These datasets are designed with the goal of training a model to do well on standard computer vision benchmarks, many of which, however, have been shown to be English-centric (e.g., ImageNet). Consequently, existing data curation techniques gravitate towards using predominantly English image-text pairs and discard many potentially useful non-English samples. Our work questions this practice. Multilingual data is inherently enriching not only because it provides a gateway to learn about culturally salient concepts, but also because it depicts common concepts differently from monolingual data. We thus conduct a systematic study to explore the performance benefits of using more samples of non-English origins with respect to English vision tasks. By translating all multilingual image-text pairs from a raw web crawl to English and re-filtering them, we increase the prevalence of (translated) multilingual data in the resulting training set. Pre-training on this dataset outperforms using English-only or English-dominated datasets on ImageNet, ImageNet distribution shifts, image-English-text retrieval and on average across 38 tasks from the DataComp benchmark. On a geographically diverse task like GeoDE, we also observe improvements across all regions, with the biggest gain coming from Africa. In addition, we quantitatively show that English and non-English data are significantly different in both image and (translated) text space. We hope that our findings motivate future work to be more intentional about including multicultural and multilingual data, not just when non-English or geographically diverse tasks are involved, but to enhance model capabilities at large.

Multilingual Diversity Improves Vision-Language Representations
Thao Nguyen'*
Matthew Wallingford
Sebastin Santy'
Wei-Chiu Ma
Sewoong Oh
Ludwig Schmidt
Pang Wei Koh1.2
Ranjay Krishna
Univeessiyy of Washington
Allen Institute for Artificial Intelligence
Abstract
Massive web-crawled image-text datasets lay the foundation for recent progress in multimodal learning. These datasets are designed with the goal of training a model to do well standard computer vision benchmarks, many of which, however, have been shown to English-centrie (e.g., ImageNet). Consequently, existing data curation techniques gravitate towards using predominantly English image-text pairs and discard many potentially useful non-English samples. Our work questions this practice. Multilingual data is inherently enriching not only because it provides a gateway to learn about culturally salient concepts, but also because it depicts common concepts differently from monolingual data. We thus conduct a systematic study to explore the performance benefits of using more samples of non-English origins with respect to English vision tasks. By translating all multilingual image-text pairs from a raw web crawl to English and re-filterin them, we increase the prevalence of (translated) multilingual data in the resulting training set. Pre-training on this dataset outperforms using English-only or English- dominated datasets on ImageNet, ImageNet distribution shifts, mmage-English-tex retrieval and on average across 38 tasks from the DataComp benchmark. On a geographically diverse task like GeoDE, also observe improvements across all regions, with the biggest gain coming from Africa. In addition, we quantitatively show that English and -on-lsh data are significantly different in both image and (translated) text space. We hope that our findings motivate future work to be more intentional about including multicultural and muitilingual data, not just when non-English or geographically diverse tasks are involved, but to enhance model capabilities large.
Introduction
Today, the predominant pre-training paradigm for vision-language models relies on large quantities of image-text pairs scraped from the web [5] 24 As raw web data contains significant amount of noise, automatic data filtering approaches are designed to curate a high-quality subset and maximize the performance of model trained on this subset standard computer vision benchmarks (e.g., ImageNet). However, these benchmarks typically only evaluate in English, and many of them have been shown to be geographically biased: for instance, ImageNet images mostly sourced from North America and Western Europe [43. Consequently, it is possible that we are designing data curation algorithms that propagate a monolingual bias, i.c., filtered datasets are increasingly dominated by English image-text pairs. In fact, a lot of highly cited work-including CLIP [35], ALIGN (1) and BASIC 333 relies exclusively on English data. Using more multilingual data for training is often only a deliberate design decision when non-Eni -on-lihishh tasks are involved [45 26
"Corresonndnnce to thaottnêcs. washington.co edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
Concepts in Hon-Engglsh Distribution
English Dinribution of "Stove" Imagen
Non English Distribution of "Stove" Imagus
ME M <- 3
Napolson GiiidD Ipn Cast fron Stove Iody Vent From Deoa
Gercuseerd bensan rus Stock Fotograte
A common BII walking lhrough the flod plain
Kapeleon GwiesD Ion Cast fron Stowe Body Vent Free Gos
Rossted baninna the stoue Stock proto
Gestoomcee broozaii courgente bamboe steemer griz M
Whthen Inlend smazing tahhara kbchee idese ceedgn cabiict
versatioeagyy inprancce abMB: KTM odntoor women
Freah ateamed
kitohen island stove amauing ic sitchens kbch idless closign with cabinete
How to brickt metal soove in house how to brick huading baleee ron stove
broccol and cour gellaron on lanniasii iteamet grey
Polich
meiiiiiloo Yos Restorayynna Klask Cturns Odeer Crnek Minatl Restcrioyon
modemn lights kitchen decar uncercabinct stowo hood in white for
Turnk: losay benzyniwww kuccennce Obrer Royalta free
Bandelisan Nall reetboatioon of the clazsi lving room
moderm lights decorr uncercelmet stove hood white for
Turkish ooffoe on gaszline stowe Ficture of Royalty From
Figure 1: Multilingual image-text data adds diversity to the English data distribution in various, significant ways (a) We show some examples of culturally salient concepts that would exist "high-quality" English data (as determined by CLIP score), such as "bamboo steamer", "kiji" (the national bird Japan) and yalı (a traditional architecture style for Turkish waterside houses) (b) Even for a common everyday object (" ("stove"), non-English and English images portray very different visual representations.
Multilingual data enriches any monolingual data distribution; multilingual data brings attention to culturally salient concepts and introduces new perspectives and annotations for the same visual category [48]. As illustrated in Figure I there are certain native concepts, e.g. 'kiji" (the national bird of Japan), that are more likely be conveyed Japanese (non-English) captions compared to English ones. Even in the case of a common everyday object ('stove'), the non-Englishh and English images look very different. Despite the diversity present in multilingual data, is disproportionately excluded from existing scaee-ccal pre- pre-training corpora.
In this paper, we investigate the counterfactual: can we improve on English vision lasks by diversifying the cultural and linguistic backgrounds of the training data?
Our investigation is motivated by dichotomy: English image-text pairs constitute minority of any random web crawl (in our estimate, one-third); yet, they form a majority in popular pre-training datasets such LAION-5B 22 DataComp
[14 and DFN [3].
common for web-scraped corpora to remove "low-quality" data by using a model (e.g., OpenAl CLIP) to compute image-text alignment and rank the raw data samples. However, this process often disproportionately favors English data if the filtering model also has an English bias [14 20 In addition to discarding many potentially useful non-English image-text pairs, this can also negatively impact the geographical and cultural representation of the resulting dataset, and consequently, the model's performance on certain underrepresented populations [38 36 10].
Our key observation is that the diversity present in multilingual data can be confounded by the language the data is in, making it difficult to observe the empirical benefits of using such data in model training. To offer more systematic study of the effectiveness of multilingual data contrast to English-only or English-dominated datasets we fix the language medium, translate all captions from DataComp's 's 228M-sample web crawl [14 to English with an advanced translation model. We then re- re-filter the data pool and train a CLIP model on this translated multilingual data. We focus on two types of evaluations:
(i) on standard English vision tasks including ImageNet, MSCOCO and Flickr retrieval, and (ii) on geographically diverse images, e.g. from GeoDE (36). which contains images of common objects across different geographical locations. We acknowledge that translation sometimes be too literal, subject to losing the intent and richness of the original

---

phrasing. Nevertheless, we hope findings from our work provide starting point for studying how to leverage the diversity of multilingual data more effectively.
Our contributions are as follows:
We demonstrate that with translation, non-English data does benefit English vision tasks. In particular, training on more samples of non- woo--English origins leads to better performance on ImageNet, ImageNet distribution shifts and Image-English-ter retrieval. On the DataComp benchmark with fixed compute budget, our best-performing approach that leverages translated multilingual captions outperforms training on just filtered raw captions by 2.0% on ImageNet and 1.1 percentage points on average across 38 tasks. When training for longer (which mimics the number epochs large- scale multimodal models are often trained for), these performance gaps increase 4.2% and 2.1 percentage points respectively.
On geographically diverse task such as GeoDE, training on translated multilingual data leads to 4.2% boost in accuracy on average compared training on filtered raw data, with performance improvement observed for all regions, especially for Africa where the increase is 5.5%.
We analyze in detail the differences between English and (translated) non-English image-text pairs. We quantitatively show that they capture distinct distributions, both in text and image space, even after they are converted to the same language medium. Consequently, it is beneficial to combine high-quality data from both sources as much as possible, since they are inherently complementary.
In summary, despite the abundance of "sufficiently useful" English data, existing data curation techniques can always do better in the data diversity axis by being more deliberate about including data from other language and cultural backgrounds. This way of enhancing diversity in turn leads to a better vision-language model in general, offering performance benefits beyond non-English vision tasks tasks involving geographically diverse images. We will release the raw captions and the corresponding English translations for the 128M image-text pairs used in our experiments.
2 Related Work
Existing data collection and filtering approaches induce Western bias in downstream datasets and models; benchmarks that seek to capture this bias still receive relatively little attention. Consequently, despite evidence showing cultural and geographical limitations in popular vision datasets, the use of multilingual data is mostly intended for pre-training and fine-tuning multimodal models to do well on non-English tasks. Our work seeks include more image-text pairs of non-English origins in the pre-training dataset, and shows that this process can improve performance, even on English-centric vision tasks.
Western bias of existing models and datasets Several papers have studied biases in popular datasets, especially biases that correlate with culture and geographic locations. Notably, Shankar et al. 43 find that NeageNet and OpenImages exhibit substantial US-centric and eurocentric representation bias. In NLP, Santy et al. [40] find that existing datasets align predominantly with Western and White populations. It is not only the data collection process that leads to Western bias, but also the data preprocessing pipeline. For instance, automated data filtering with scores output by a model, e.g., OpenAl CLIP, has been commonly adopted as way to discard low-quality web-crawled samples. Little is known about the potential biases induced by this approach. Hong et al. (20) recently show that CLIP score filter is more likely to include data related to Western countries compared to that of non- Non-Western countries. In [4] (Figure 24), the authors offer evidence that CLIP filtering implicitly performs some English filtering, as the top CLIP score examples are increasingly dominated by English image-text pairs. Consequently, all these dataset biases translate to performance disparity, as demonstrated by existing work showing that the accuracy of vision systems drops significantly on non-Western inputs [10 49 38 36), or low-resource languages [16]
Improving the availability of non-English data in multimodal datasets Translation has been popular technique to address the limited availability of large-scale and high-quality -nglishh
data in training and evaluation [45] 47 32 2 [6]. In addition to translating English captions into the language of interest, previous work also uses curated list of common words in the native
language to scrape image-text pairs from the web 266 17.. COCO-CN
25 extends the MSCOCO
dataset [7] with manually written Chinese captions.
3

---

Most closely related to our setup is the LAION- Translatad dataset 229, which translates 3B samples of LAION-5B from many languages into English using the M2M100 model [12]. Compared this dataset construction, we (i) use more advanced translation model, NLLB (9, that covers twice as many languages, (ii) work with mostly raw data while LAION was heavily filtered and thus could contain biased representation of multilingual data. To the best of our knowledge, no existing work has experimented with the LAION-Translan dataset.
Adapting CLIP post-training for multilingual tasks Geigle et al. [15] translate high-quality English data into 95 languages, and use these translated samples to re-align an image encoder previously trained on English data to a multilingual language model. Similarly, Chen et al. [ propose re-training OpenAI CLIP on a mix of Chinese and English data to enhance its multilingual representation. In [4]. the authors explore adaptation without any image data, solely fine-tuning the text encoder with English captions from MSCOCO, Google Conceptual Captions, and VizWiz translated to other languages. Visheratin [45 replace the text encoder of OpenAl CLIP with the text encoder from the NLLB model, and fine-tune the new model on multilingual image-text pairs obtained from translating LAION-COCO' 's English captions [41] into 200 languages. In contrast to these papers that employ multilingual data for the purpose of adapting to non- -an-llllshhh tasks, we focus on using multilingual data to do better on common vision tasks that are in English.
Using multilingual data significantly enhances data diversity Our study is partly inspired by findings from Ye et al. [48], who show that multilingual synthetic captions obtained from existing image captioning systems provide higher semantic coverage than monolingual ones, over 3.6K images. Our experiments instead use raw web-crawled data and explore the performance benefits of embracing cultural and linguistic diversity in (mostly) human-generated captions scale.
3 Experimental Setup
Given a starting pool of raw image-text pairs scraped from the web, many of which contain non- English captions, we experiment with ways to preprocess and filter this pool into high-quality dataset. The quality of the dataset is measured by the zero-shot performance of a CLIP model trained on it from scratch.
Data We experiment with the medium pool of the DataComp benchmark [1]. which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered corpora such as LAION [42), DataComp applies minimal data preprocessing, involving only NSFW filtering, deduplication of evaluation sets, and face blurring. This allows the candidate pool to stay close the natural distribution of the raw web data as much as possible, in addition to enabling maximum flexibility in dataset design.
Translation model To detect language and translate the raw captions from DataComp into English, we use the No Language Left Behind (NLLB) translation model [9, which is considered of the-art. NLLB is the first to translate across 200 languages, including low-resource ones that are not currently supported by common translation tools. We use the 600M-parameter model publicly available on HuggingFace to allow for fast inference on our large data corpus. All 128M captions from DataComp are translated to English; examples could be found Appendix A We provide some quantitative analysis of the translation quality in Appendix C
Training After translating the captions of all samples in the raw data pool, we filter them based on cosine similarity between image and text embeddings. We experiment with using OpenAI CLIP- VIT-L/14 (35) and the public Data Filtering Network (DFN) from [3] to obtain the embeddings, and subsequently, the cosine similarities. The DFN, specifically designed to filter data for subsequent model training, was trained on three public datasets deemed as
-Conaltt-CCoc Caption 12M [ Conceptual Captions 3M 44 and Shutterstock 15M [30]. We find that indeed the public DFN is better at data filtering compared to OpenAl CLIP, measured by the performance of CLIP trained on the corresponding filtered datasets (see Appendices E and G).
We pretrain CLIP model 35 each filtered subset with ViT-B/32 as the image encoder, and follow DataComp's 's hyperparameters; details can be found in Appendix [B] Unless specified otherwise, all models are trained for the same compute budget (128M steps) determined by DataComp. For some

---

Table 1: On the DataComp benchmark, training on translated captions outperforms training on raw captions across a range of metrics; using both types of captions yields even more performance gains. We report the performance of select baselines on the DataComp benchmark 14]; all baselines are trained for the same number of steps specified. Here the filtering threshold (and thus the resulting dataset size) has been tuned for each baseline and we only show the filtered subset that yields the highest average accuracy. We find that with the same filtering method (i.e., using DPN score), training on translated captions ( ("Filtered translated captions") )) is more effective than training on raw captions ("Filtered raw captions") ")) as seen from higher performance on ImageNet, ImageNet distribution shifts, retrieval, GeoDE (worst-region accuracy) and on average across 38 tasks. Combining both sources of captions leads to the best performance. Appendix G contains the full results.
Baseline name
Dataset
ImageNet
ImageNet
Retrieval GeoDE
Average over 38 tasks
size
shifts
Training with DataComp setup ( (128M steps)
Filtered raw captions
25.6M
0.316
0.260
0.282
0.688
0.350
Filtered raw captions, replaced with translated
25.6M
0.304
0.252
0.268
0.668
0.331
cuptions
Filtered translated cuptions
25.6M
0.329
0.275
0.296
0.709
0.359
Filtered English-only captions
16.0M
0.283
0.236
0.278
0.666
0.327
Filtered raw captions U Filtered translated cup-
34.2M
0.329
0.271
0.298
0.720
0.364
tions
Filtered raw captions Filtered translated cap-
51.2M
0.336
0.280
0.301
0.725
0.361
tions
Training for 10x longer (1.28B steps)
Filtered raw captions
38.4M
0.414
0.340
0.344
0.742
0.414
Filtered translated captions
38.4M
0.427
0,347
0,352
0.771
0,414
Filtered raw captions U Filtered translated cap-
34.2M
0.441
0.359
0.353
0.775
0.427
tioms
Filtered raw captions Filtered translated cap- tions
51.2M
0.456
0.369
0.371
0.776
0.435
select baselines, we also experiment with training for 10x longer. The fixed architecture, compute and hyperparameter setup allow us isolate data quality as the main factor influencing performance.
Evaluation We perform zero-shot evaluation of trained CLIP models using the 38 tasks from DataComp. These tasks involve recognition and classification of wide range of domains (e.g., texture, scene, metastatic tissue, etc.) in addition to image-text retrieval and commonsense association. Among them, we pay particular attention to commonly cited metrics such ImageNet accuracy, ImageNet distribution shift accuracy a proxy for natural robustness, and retrieval performance. ImageNet shifts include ImageNet-V2 V2 (37, ImageNet Sketch [46), ImageNet-A [1), ImageNet-O 19.. ImageNet-R (18) and ObjectNet M Retrieval score is the average of the performance on
Flickr30K [50], MSCOCO and WinoGAViL [3). Throughout the paper we also highlight GeoDE
worst-region performance 336 task that involves geographically diverse images to demonstrate the added benefits of geographical inclusivity that training on more (translated) multilingual captions
offers.
4 Impacts of using (translated) multilingual captions on standard vision tasks
We explore training each caption distribution separately, as well as combining them. Below describe the baselines from Table more detail:
Filtered raw captions: As mentioned in Section 3 we use the public DFN from 13 by default to filter the starting pool (128M samples). Given the images and the corresponding web-crawled captions, we experiment with varying the filtering threshold to keep top x% of the pool based on DFN score. In Table I we only report the best average performance obtainable after the filtering threshold has been tuned, and the resulting dataset size. Refer to Appendix G for the full results.
Filtered translated captions: Similar the approach above, we tune the filtering threshold, but using DFN score between an image and the English translation of the original web-crawled caption. Filtered English-only captions: Similar to "Filtered raw captions" baseline, here we also tune the filtering threshold to keep only subset of the pool with the highest DFN scores, but with an additional constraint of only filtering from samples with web-crawled captions already in English.
5

---

15.0
Figure 2: Filtering with translated captions allows substantially more (translated) English samples be included in the final training set. While English data only makes up about one-third of the raw web crawl, it dom- inates the top 20% of the pool, selected based on DFN score between image and caption. With translation, English-transe transssaadd oon-English captions now make up the majority of the "high- quality" data and thus are more likely to be se- lected for training.
Number of datapoints (M)
12.5
10.0
7.5
Eng
2.5
Non-Eng
0.0 Filtered raw captions
Filtered translated captions
Filtered raw captions, replaced with translated captions: Given samples from "Filtered raw captions" (i.e. again, based on the cosine similarity score between image and original web text), we keep the images selected and replace the raw captions with the corresponding English translations. Filtered raw captions U Filtered translated captions: We combine image-text pairs from "Filtered raw captions" and "Filtered translated captions" subsets uncovered above. However, these subsets have about two-thirds of their images in common (see Appendix D). For such images, we only include one copy in the final training set and use English-translate rest of the images in "Filtered raw captions" that do not appear in "Filtered translated captions", we include them in the training set with the corresponding original captions (which could be non-English).
caption by default. For the
Filtered raw captions Filtered translated captions: We combine "Filtered raw captions" and "Filtered translated captions"; the overlapping images between these two subsets would appear twice in the final training set one copy with the original web caption and one copy with the English-translated caption.
4.1 Overall performance trends
Combining high-quality raw and translated captions offers the best performance We find that using both sources of captions, and image data data-since top (image, raw text) pairs and top (image, translated text) pairs only have two-thirds of the images in common-leads to the best performance (bolded entries of Table Dp. This approach surpasses training only high-quality raw data by 2% on ImageNet, ImageNet shifts and retrieval, and improves GeoDE worst-region performance by 3.7%. We note that this is not simply due having more unique image- text samples, filtered subsets of similar sizes but using a single source captions (e.g., top 40% raw captions totalling 51.2M samples) yields significantly lower performance (Appendix G
Using only translated multilingual captions still better than using only raw captions Zooming in "Filtered translated captions" and "Filtered raw captions" baselines, we find that the former outperforms the latter on many standard metrics (ImageNet, ImageNet distribution shifts, image- text retrieval). This is unexpected in light of prior work showing that ImageNet exhibits strong amerocentric and eurocentric bias [33 with images from America and Great Britain taking up 53% of the dataset.
4.2 Ablations
We perform more ablation studies to disentangle the reasons for the performance gains from using high-quality translated captions, as well verify that the gains robust.
The performance gain from using translated captions not simply due to converting all text data to common language medium Given image-text pairs from the "Filtered raw captions" subset, we replace the web-crawled captions with the corresponding English translations ("Filtered raw captions, replaced with translated captions"). This intervention on only the captions leads to performance drop across the board. We hypothesize that this is due to noise in the translation process, (i) many web captions formed by stringing together short, ungrammatical phrases and thus are "outtof-ddstributte languages in the sentence, thereby leading noisy language detection and translation.
for the NLLB translation model, (ii) web captions may contain multiple
Re-filtering data after translation is also necessary due to significant changes in the data ranking Besides noisy artifacts introduced by translation, the process also changes the image-text cosine

---

102
5.5
4.4.3
3.6
Accurasy dif
2.34
1.5 16
0.7 070057700
C.1
elevr oloxast objed distncc
motel
powee clevr mount
all
cars
Seter Rower
Imageniet
Good for
country
(lart
welibi ceiisisaeerlycr 17
lovc, wecaal
vea 2007
tel
Inagenatt
sunco
Imaggn da
Imngens Reets
en airoava lbg @ilicnl
Ininwwet
petss
Bisr Ik troll
tet ologed ventile distonce
eurond
Figure 3: With the same degree of filtering, training with (image, translated caption) pairs improves performance on 28 out of 38 tasks compared training with (image, raw caption) pairs, including on ImageNet distribution shifts, retrieval, and tasks with geographically diverse inputs. We compare performance on each task of the DataComp benchmark between training with raw captions and training with translated captions. Both datasets have been filtered with image-text cosine similarities output by the public DFN [X] select the top 30% examples. We find that using translated captions leads to 1.5 percentage points improvement on average across tasks. We highlight the performance changes on ImageNet distribution shifts (red), retrieval (blue) and faimess-related tasks (dark yellow).
similarity score and thus the quality ranking of the data samples in the pool. More specifically, find that while "Filtered raw captions" is dominated by English samples, (translated) non- non-English samples make up the majority of "Filtered translated captions" (Figure 2). These two filtered subsets only share about two-thirds of the images in common, see Appendix Dor more details. Therefore, by changing the caption distribution, we are also inducing changes to the image distribution that the best-perforning model would see.
The benefits of training with translated multilingual captions are consistent across data filtering networks As alluded to in Section 3 we also explore using cosine similarities output by OpenAl CLIP-VIT-L/14 (35) for data filtering. The full results for this ablation be found in Appendix Similar to the previous observations, we find that using filtered translated captions yields better performance than using filtered raw captions.
The performance benefits of using translated data persists with much longer training duration We also experiment with training for 10x more steps (i.e., 1.28B samples seen) as this is more in line with the number of epochs typical vision-language models are often trained on (e.g., OpenAl CLIP models were trained on 400M datapoints for 32 epochs (35). When using either the caption or the translated caption distribution, setting the filtering threshold to top 30% of the pool works best. Even though the two filtered datasets now yield the same average accuracy, training high- quality translated captions still offers significant advantages when it comes to ImageNet, ImageNet shifts, retrieval and GeoDE. Combining high-quality data from both sources of captions continues to be the best performing approach, giving 4.2% improvement on ImageNet and 2.1 percentage points improvement average, compared to just training on filtered raw captions. Results for more baselines can be found in Appendix H
4.3 Individual task analysis
After observing improvement across different metrics from using more (translated) multilingual captions, in this section we break down the performance changes for each of the 38 tasks in the DataComp benchmark. The base model for comparison is CLIP trained on top 30% image-text pairs filtered from the raw data pool, and the new improved model is the one trained on top 30% image-text pairs after the same pool has been all translated to English. Both models are trained for 128M steps. Averaged across 38 evaluation tasks, the latter yields a 1.5 percentage points improvement. The biggest gains come from Flickr retrieval, fairness (GeoDE, Dollar Street) and remote sensing (EuroSAT, RESISC45) tasks (Figure 3)
7

---

80
Figure 4: On GeoDE, using filtered trans- lated captions leads to improvements across all regions compared to using tered raw captions, with Africa observ- ing the biggest gain. We break down the GeoDE performance by region and compare training on top 30% translated captions training on top 30% raw captions. On av- erage, classification accuracy improves by 4.2%, and the improvement applies all re- gions the dataset, especially Africa where accuracy gain the biggest 5.5%.
Filtered raw captions
Filtered translated captions
75
Accuracy (%)
60
Africa
East Asia
Southeast Asia Americas West Asia
Europe
English data
Non-English data
Figure 5: Visualizations of what an SVM deems typical of images with En- glish captions and those with non-English captions. We show examples of easy- to-classify images in English versus non- English data classification task. Be- sides the product logo and text in some images that are suggestive of the language distribution, the image con- tent mostly depicts common scenes and objects.
In particular, on GeoDE (3).. which consists of images common objects crowd-sourced from six different regions across the world, we find that using translated multilingual captions makes CLIP perform better on all regions, with the biggest gain coming from Africa images (5.5%) and the second biggest gain coming from the Europe region. This is unexpected given that African languages only make up small fraction of the training set, and after translation, more European (compared to African) language samples make it the resulting filtered subset (Appendix [D). Besides, is worth noting that on GeoDE, our best baseline from Table [[ ("Filtered raw captions & Filtered translated captions") outperforms the current best baseline on DataComp's medium scale ("HypeSampler" (22) by 1.5%, measured in terms of worst-region accuracy (Africa).
Understanding the differences between English and (translated) non-English data
Given that using more image-ter pairs of non-Engln nog-Eng origins in the training set offers significant benefits on most vision tasks, including those that are English-centric, we seek to further understand the various ways that non-English data complements and improves the diversity of English data, in both image and text space.
5.1 Image distribution
As a proxy for capturing image distribution differences,, we train simple classifiers - Support Vector Machine (SVM) CLIP embeddings and ResNet-50-to distinguish images with English captions from those with non-English captions. We randomly sample 100K images from each distribution for training and 10K for testing. We only images from the top 20% of the candidate pool (based on DFN cosine similarity score), to ensure that (i) these are the samples that the best- perf-renfonning CLIP models eventually trained on, (ii) images of sufficient quality, to the extent that they have fitting captions accompanying them.
We note that this classification task is non-trivial for number of reasons:

---

Many images are duplicated across the web, i.e., after DataComp performs image deduplication it is possible for these images to appear with either English captions or non-English captions in our data pool.
The language detection model is not perfect and web-crawled captions may contain more than one language in the same sentence.
Images with non- -on-English captions contain many sub sub-distibbttions of images, some of which may overlap with the distribution of images with English captions (c.g. curocentric data).
Despite these challenges, our simple classifiers achieve 67% accuracy on the binary classification task, significantly better than random chance per- formance. In Figure []] show some examples of what the SVM deems easy to classify. Overall this experiment suggests that the distribution of images with non- oon-English captions is sufficiently distinct from that of images with English captions. There- fore, not training on more of the former means we are missing out on a considerable amount of visual information that can only be found in separate part of the web.
Text distributions
MAUVE score
Translated non- English vs. Translated non English
0.964 = 0.005
Translated English vs.
0.957 1 0.005
Translated English
English vs.
0.890 - 0.004
Translated English
Translated English vs. Translated non- non-English
0.616 + 0.010
English vs.
0.449 + 0.008
Translated non-English
Table 2: There exists substantial gap between the distribution of English captions and that of non- English captions, even when we apply translation to both, suggesting that they capture different con- tents. We use MAUVE score (44 to measure the difference between English captions and (translated) non-Englissh captions the training set. We find that (i) translation indeed introduces some artifacts and changes what "English" texts may look like, (ii) the English text distribution is remarkably different from the non- non-English one, even after they are converted to the same medium with translation. All scores are aver- aged over 3 randomly sampled sets of 10K captions.
5.2 Text distribution
In the text space, we leverage MAUVE score [34] to quantify the differences between English cap- tions and non- non-English captions that have been translated to English. MAUVE was originally de- signed to measure the gap between machine- and texts. The metric computes KL divergences in a quantized, low-dimensional space after embedding text samples from each distribu- tion with a language model (by default, GPT-2). The output score ranges between 0 and and the higher it is, the more similar the text distributions
are. Similar our analysis in the image space, we only use caption samples from the top 20% of the candidate pool (based on DFN score).
In Table [ sanity check, we randomly sample two disjoint sets of 10K captions from the same text distribution (e.g., non-English captions having been translated to English with the NLLB translation model, or English captions having been passed through the same model). We find that the two sets indeed exhibit high MAUVE scores (above 0.95). When comparing raw English captions to English captions that have been passed through the NLLB model (i.e., "translated English"), we find that the MAUVE score decreases slightly (0.890), indicating that the translation process introduces some artifacts making English- transsaaed English text look somewhat different from raw English text.
When comparing raw English texts and non-English texts, both having been passed through the translation model and thus undergone the same "preprocessing" (i.e., English translation), the resulting MAUVE score is relatively low (0.616). This signals that independent of differences in language, what is discussed in English captions and non- English captions differs in many ways. We should therefore leverage both sources of text information as much as possible for training.
6 Discussion
Limitations We fix the data filtering method to be based on image-text cosine similarity output by trained model, and study the impact of selecting training data based on different caption distributions. We show that the advantages of using translated multilingual data are robust to the choice of the filtering network. However, our best-performing baseline is currently not state-of-thr the- on the DataComp benchmark [1]]. remains an open question whether the performance benefits of our method persist with other score metries, e.g. hyperbolic entailment [22] currently the best method for DataComp's medium scale, or other filtering methods that are used jointly with CLIP score, e.g. T-MARS 27 which also removes text-spotting images with limited visual information.

---

Besides, we acknowledge that translation can introduce artifacts and reduce the richness of expressions in the original languages. Prior work has shown that translated sentences are less effective compared to manually-writer sentences as sources of training data for a vision task, e.g. image captioning 28 23 Our work mainly leverages translation as a way to convert all image-text pairs to the same medium, and remove confounding impacts of language in data selection and model training.
Conclusion In this work, we bring all web-crawled image-text pairs into a common language medium via English translation, and systematically study the empirical benefits of using non- non-English data with respect to standard computer vision tasks (that are in English). By including significantly more (translated) multilingual data in the filtered training set, the improved cultural and linguistic diversity in turn lcads to substantial gains across all major metrics- metrics-ImageNee robustness, retrieval capabilities and average performance across 38 tasks- even if some of these metrics have been shown to overfit to English. We also find that despite being translated to the same language, English and non-Englishh data distributions still distinct from each other.
distribution shift
Future work This work motivates future studies into data curation techniques that directly improves the diversity of data origins. Another interesting direction of exploration is adapting trained CLIP models from this paper for multilingual benchmarks, such as by re-training the text encoder (that has only been trained on English and English- translattt captions) with the technique proposed in [4. We hypothesize that text adaptation alone is sufficient for our models to perform competitively on non-English tasks, owing the presence of significantly more multilingual and multicultural images in our pre-training dataset.
Broader impact While most studies have looked into non-English data with the goal of increasing societal representation and subsequently improving performance on under-served populations or tasks, we observe that non-English data can actually help enhance model capabilities as a whole (including on standard English benchmarks). This suggests that diverse representation in training data, e.g. as measured by cultural and linguistic backgrounds, should be a deliberate design decision in the data curation process, instead of existing only as byproduct of the preprocessing pipeline or out of societal considerations.
Acknowledgments and Disclosure of Funding
We thank Stability AI for the assistance with compute resources. We grateful to Rachel Hong, Jonathan Hayase, Cheng-Yo Yu Hsieh, Anshul Nasery and Amita Kamath for offering feedback on the manuscript and other helpful discussions. TN is supported by the UW-Meta Al Mentorship Program. This work is supported in part by the Sony Faculty Innovation grant, NSF awards 2019844, 2134012, and 21471,, NSF AI Institute for Foundations of Machine Learning (IFML), Microsoft Grant for Customer Experience Innovation, Open Philanthropy, and the Allen Institute for AI. PWK supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Innovation under the AI Visiting Professorship Programme (award number AIVP-2024-001).
References
[1] A. Barbu, D. Mayo, Alverio, W. Luo, C. Wang, D. Gutfreund, Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[2] L. Barrault, F. Bougares, L. Specia, C. Lala, D. Elliott, and S. Frank. Findings of the third shared task on multimodal machine translation. In Third Conference on Machine Translation (WMT18), volume 2, pages 308-327, 2018
[3] Y. Bitton, N. Bitton Guetta, R. Yosef, Y. Elovici, M. Bansal, G. Stanovsky, and R. Schwartz. Winogavil: Gamified association benchmark to challenge vision-and lnd--angdd models. Ad- vances in Neural Information Processing Systems, 35:26549-26564, 2022.
[4] F. Carlsson, P. Eisen, F. Rekathati, and M. Sahlgren. Cross-lingual and multilingual clip. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6848- 6854, 2022.
10

---

[5] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-texi pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[6] S. Changpinyo, L. Xue, I. Szpektor, A. V. Thapliyal. J. Amelot, M. Yarom, X. Chen, and R. Sori- cut. Maxm: Towards multilingual visual question answering. arXiv preprint arXiv:2209.05400 2022.
[7] x. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504. 503.5,0,,,, 2015.
[8] Z. Chen, G. Liu, B.-W. Zhang, F. Ye, Q. Yang, and L. Wu. Altclip: Altering the language encoder in clip for extended language capabilities. arXiv preprint urXiv:2211.06677 2022.
[9] M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, al. No language left behind: Scaling human-centered machine 2022.
translation. arXiv preprint arXiv:2207,0466
[10] T. De Vries, I. Misra, C. Wang, and L. Van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 52-59, 2019.
[11] D. Elliott, S. Frank, K. Sima' an, and L. Specia. Multi30k: Multilingual ingish--germa image descriptions. arXiv preprint 1oXXiv:6050.00
2016.
[12] A. Fan, S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, N. Goyal, T. Birch, V. Liptchinsky, S. Edunov, E. Grave, M. Auli, and A. Joulin. Beyond english-centrie multilingual machine translation, 2020.
[13] A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V. Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17425, 2023.
[14] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[15] G. Geigle, A. Jain, R. Timofte, and G. Glavaš, mblip: Efficient bootstrapping of multilingual vision-llms. arXiv preprint arXiv:23307 1.Xi::2307.06300, 2023.
[16] G. Geigle, R. Timofte, and G. Glavaš. Babel-imagenee Massively multilingual evaluation of vision-and- languuage representations. arXiv preprint arXiv:2306.0866 2023.
[17] Gu, X. Meng. G. Lu, L. Hou. N. Minzhe, X. Liang, L. Yao, Huang. W. Zhang, X. Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems, 15:26418-26433 2022.
(18) D. Hendryeks, S. Basart, N. Mu, S. Kadavath, F. Wang, F. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of outtt--distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8349, 2021.
[19] D. Hendrycks, K. Zhao. S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[20] R. Hong, W. Agnew, T. Kohno, and Morgenstern. Who's in and who's 's out? a case study of multimodal clip-filtering in datacomp. arXiv preprint arXiv:2405.0880 08209, 2024.
[21] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[22] W. Kim, S. Chun, T. Kim, D. Han, and S. Yun. Hype: Hyperbolic entailment filtering for underspecified images and texts. arXiv preprint arXiv:2404.17500 2024.
[23] W. Lan, X. Li, and J. Dong. Fluency-guided cross-lingual image captioning. In Proceedings of the 25th ACM international conference on Multimedia, pages 1549-1557, 2017.
[24] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023,

---

[25] X. Li, C. Xu, X. Wang, W. Lan, Z. Jia, G. Yang, and J. Xu. Coco-en for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia, 21(9):2347-236)) 2019.
[26] Y. Liu, G. Zhu, B. Zhu, Q. Song, G. Ge, H. Chen, G. Qiao, R. Peng, Wu, and J. Wang. Taisu: A 166m large-scale high-quality dataset for chinese vision- lisionlanguaae pre-training. Advances in Neural Information Processing Systems, 35:16705-167111 2022.
[27] Maini, S. Goyal, Z. C. Lipton, Z. Kolter, and A. Raghunathan. T-mars: Improving visual representations by circumventing text feature learning. arXiv preprint arXiv:2307.03132 2023,
[28] T. Miyazaki and N. Shimizu. Cross-lingual image caption generation. In Proceedings of the 54th Annual Meeting of the Association for Compitational Linguistics (Volume I: Long Papers),
pages 1780-1790, 2016.
[29] M. Nezhurina, R. Beaumont, R. Vencu, and C. Schuhmann. Laion translated: 3b captions translated to english from laion5b. URL httpttt/ts/:/oooonnn.
[30] T. Nguyen, G. Ilharco, M. Wortsman, S. Oh, and L.. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455-21144 2022.
[31] T. Nguyen, S. Y. Gadre, G. Ilharco, S. Oh, and L. Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[32] J. Pfeiffer, G. Geigle, A. Kamath, J.-M. O. Steitz, S. Roth, I. Vulić, and I. Gurevych. xgqa: Cross-lingual visual question answering arXiv preprint arXiv:2109.000 06002, 2021.
[33] H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning, 2021. URL http:://rxxii org/ubs/21 10000..
[34] K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers.
Advances in Neural Information Processing Systems, 34:4816-4825 2021.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision.
In International conference on machine learning, pages 3748-8763. PMLR, 2021.
[36] V. V. Ramaswamy, Y. Lin, D. Zhao, A. Adcock, L. van der Maaten, D. Ghadiyaram, and O. Russakovsky. Geode: geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems, 36, 2024.
[37] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[38] M. Richards, P. Kirichenko, D. Bouchacourt, and M. Ibrahim. Does progress on object recognition benchmarks improve real-world generalization? arXiv preprint arXiv:2307. 2023.
[39] W. A. G. Rojas, S. Diamos, K. R. Kini, D. Kanter, V. Reddi, and C. Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In sixtt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[40] S. Santy, J. T. Liang, R. L. Bras, K. Reinecke, and M. Sap. NIpositionality: Characterizing design biases of datasets and models. arXiv preprint arXiv:2306.01943, 2023.
[41] C. Schuhmann, A. Köpf, R. Vencu, T. Coombes, and R. Beaumont. Laion coco: 600m synthetic captions from laion2b-en. URL httpt:://ainoonn.ooo. attpa////a///oaaoccooooo////////
[42] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 2022.
[43] S. Shankar, Y. Halpern, E. Breck, J. Atwood, J. Wilson, and D. Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. 2017.
arXiv preprint arXiv:
[44] Sharma, N. Ding, S. Goodman, and R. Sorieut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
12

---

[45] A. Visheratin. Nilb-clip-tre train performant multilingual image retrieval model on budget. arXiv preprint arXiv:2309.01859, 2023.
[46] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing
local predictive power: Advances in Neural Information Processing Systems, 32, 2019.
[47] A. Yang. J. Pan, Lin, R. Men, Y. Zhang, J. Zhou, and C. Zhou. Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01333 2022.
[48] A. Ye, S. Santy, J. D. Hwang, A. X. Zhang, and R. Krishna. Cultural and linguistic diversity improves visual representations. arXiv preprint
00.v:3300.14356. 2023.
[49] D. Yin, L. H. Li, Z. Hu, N. Peng, and K.-W. Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. arXiv preprini urXiv:2209.10 06860, 2021.
[50] Young. A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
13

---

A Examples of translated data (no cherry picking)
Master
Raw caption: Lovely 2nd Floor Plans Part 4: Floor Plan Language detected: eng_ Latn
Trunslation: Lovely 2nd Floor Plans Part 4; 2nd Floor Plan
Bed
Bod
Second Floor Plan
Raw caption: ビン 7 3 7 & 1 ラクンシ : Language detected: apn_Jpan Translation: Pink: The bright pink lounge of the interior: cell phone wallpaper.
Raw caption: CW&#8217:s The Originals Joseph Morgan Jacket Language detected: eng Latn
Translation: CW The Originals Joseph Morgan Jacket
Raw caption: PURPLE AZALEAS UP CLOSE Language detected: yue Hant Translation: Purple Azleas up close.
Raw caption: Paso paso: Cómo el permiso para circular auto en vacaciones verano Garantia Plus
Language detected: spa Latn Translation: Step by step: How to get a driving permit on summer vacation
Raw caption: Een hangende decoratie van Vivi Gade papieren diamantvormen
Language detected: nid_Lath Translation: pending decoration of Vivi Gade paper diamond shapes
14

---

Raw caption: Neymar n plus joué en compétition depuis le 28 novembre 2021, à Saint-Etienne. Icon Sport Language detected: fra Latn
Translation: Neymar has not played in competitive match since 28 November 2021,
10
LIGUZ
Raw caption: Ring Alarm 5-piece kit (2nd Gen) home security system with optional 24/7 professional monitoring Works with Alexa
Language detected: eng Latn
Translation: Ring Alarm S-piece kit (2nd Gen) home security sys- tem with optional 24/7 professional monitoring Works with Alexa
Raw caption: Alcatras Hapisanesi Language detected: tur Latn Translation: The Alcatras Prison
WIN
caption: Coque iPhone XS Max Olaf Reine des neiges bon- homme neige
Language defected: fra Latn
Translation: Iphone XS Max Olaf Snow Queen Snowman
Raw coption: Multiracial Thumbs Up Against Blue Sky Language detected: eng. Latn Translation: Multiracial Thumbs Up Against Blue Sky
Raw caption: #4## waiheke island E M We * Sri
Language detected: LLann
Translotion: Last year school camp went to Waiheke island
15

---

B More training details
We follow the training and evaluation protocols the DataComp benchmark [14 refer Appendices M and N of this previous work for more details. To summarize, we use ViT-B/32 as the image encoder for CLIP, and fix the hyperparameters used for training: learning rate 5e-4, 500 warmup steps, batch size 4096, AdamW optimizer B3 0.98.
Since the compute budget is fixed, for the DataComp setting (128M training steps), each of our baseline takes about hours with 8 A40 GPUs and 40 CPUs. With the same amount of resources, for experiments involving training for longer (1.28B steps), each baseline takes about 80 hours. We report all baselines that we ran in Appendices
and
C Translation quality
Here provide quantitative assessment of the quality of caption translation offered by the NLLB model [2]. We sample 100K captions from the raw data pool and backtranslate the English-translated caption into the original (detected) language (e.g.. Chinese text English translation Chinese translation of the English-translated text). To evaluate the translation quality, we compute the cosine similarity between the initial web-scraped text and the backtranslated text using embeddings from the multilingual Sentence- Bettence-BERTTT BERT model |? J. We find that on average the cosine similarity (and thus, translation quality) remains relatively high (0.63). In the table below, we report the top 5 and bottom 5 languages that observe the highest and lowest translation quality as captured by our metric, computed over least 30 text samples per language.
Table 3: Top and bottom languages where web-scraped captions observe the highest and lowest translation quality by the No Language Left Behind model [9]. out all languages detected in our raw data pool. Translation quality is measured by how much the semantic meaning is preserved after the caption is translated into English and subsequently backtranslated into the original language.
Language
Text cosine similarity after backtranslation (f)
English
0.886
Norwegian Nynorsk
0,883
Bengali
0,883
Russian
0.860
Norwegian Bokmål
0.839
Marathi
0.271
Irish
0.240
Standard Latvian
0.233
Chechen
0.0595
Karachay- Barak
0.00280
Changes in data composition due to translation
D.1 Differences in image uids between "Filtered raw captions" and "Filtered translated captions"
In this section, we provide some statistics of the differences in image-text pairs selected for "Filtered raw captions" and "Filtered translated captions", when both caption distributions are filtered similar extent with the public DFN from [13.. We find that at either 20% or 30% selectivity threshold, both filtered subsets have about two-thirds of their images common. addition, filtering the initial pool using (image, translated caption) cosine similarity score always leads to translated multilingual captions taking up the majority of the resulting training set. This is not the case when filtering with (image, raw caption) cosine similarity score.
16

---

Table Analysis number samples of English and non-English origins "Filtered raw captions". "Filtered translated captions" and their intersection.
Data subset
Total (M)
size
Number of English
Number of English captions (M)
captions (M)
Top 20% Raw captions
25.6
14.9
10.7
Top 20% Translated captions
25.6
11.0
14.6
Top 20% Raw captions Top 20%
17.1
10.7
6.4
Translated captions
Top 30% Raw captions
38.4
20.4
18.0
Top 30% Translated captions
38.4
15.4
23.0
Top 30% Raw captions n Top 30% Translated captions
25.7
15.0
10.7
D.2 Language composition of the filtered subsets
Below we show the most common languages top 20% raw captions and top 20% translated captions.
nob_Latn
Latn
Latn
bod bod_
Tibt
swe Lat_Latn
Arab
hun Lat_Latn
hun_Latn
dan_Lati
ind_Latn
Let_Latn
Cetn
cat_Lat
tur_
Lat_Latn
Lat_Latn
ind_
nit_Latn
Lat_
Lat_
mid_Latn
ita Latn
Lat_Latn
Vie_Latn
spt_Latn
jpn_Jpan
ita_Latn
spt_Latn
fra Latn
fra. Latn
LatnLatn
rus_Cyrl
dat_Latn
yue Hant
yue Hant
det_Latn
kor Hang
kor Hang
LatnLatn
Latn
20
40
60
0
10
20
30
40
Composition (%)
Composition (%)
Figure 6: Top 20 languages that are most common in top 20% raw captions (left) and top 20% translated
multilingual captions (right) (both filtered with the public DFN model).
17

---

D.3 Changes in language composition
Comparing image-text pairs selected in top 20% translated captions those selected in top 20% raw captions, we show below the languages that observe the biggest change in their representation in the resulting training set:
Lat_Latm
yan_Hant
kor Hang
Lag_Latn
zho_Hans
pes_Arab
Lat_Latn
ces_Lath
ukr_Cyr
bod__iby
arb Arab
Lat_Latn
nid_Latm
ita Latn
Por_Latn
Latn
fra Latn
Lat_Latn
jpn_Jpan
rus Cyrl
-550 -12.5
-7.5
-5.0
2.5
Composition change (%)
Figure 7: Top languages that see the biggest change (in absolute percentage) in their representation in the final training set when we filter with translated multilingual captions versus with raw web-scraped captions.
18

---

E Experiments with OpenAl CLIP score filtering
Table shows results of our experiments with data filtering using OpenAl CLIP-ViT-L//
35
Besides the baselines described in Section 4 the table also contains "Top 30% raw captions U top 30% translated captions, using translated caption for all",, where we take all the images uncovered from "Top 30% raw captions" and "Top 30% translated captions", deduplicate them, and use the corresponding translated captions all these images.
We find that with OpenAl CLIP as the filtering network, some of our observations from Section continue hold true: (i) using translated multilingual captions is better than using captions, and (ii) the performance gain from training with translated captions requires re-filtering the entire data pool after translation (as seen from comparing the first two baselines of the table).
Table 5: The benefits of using translated multilingual captions still hold when using cosine similarity score from OpenAl CLIP for filtering. This table shows performance of all basclines we experiment with for filtering with OpenAl L/T-L4441 Again, the compute budget is fixed and all basclines trained for 128M steps. We find that training on filtered translated captions also outperforms training on filtered raw captions this case.
Baseline name
Dataset size
ImageNet
ImageNet Retrieval GeoDE shifts
Average over 38 tasks
Top 30% raw captions
38.4M
0.273
0.230
0.251
0.683
0.328
Top 30% raw captions. replaced with trans-
38.4M
0.260
0.224
0.248
0.660
0.322
lated cuptions
Top 30% translated captions
38 38.4M
0.292
0.250
0.267
0.695
0.342
Top 50% captions
64. 1M
0.254
0.218
0.262
0.670
0.315
Top 50% translated captions
64.1M
0.265
0.230
0.276
0.704
0.320
Top 30% raw captions top 30% translated captions, using translated caption for
47.7M
0.275
0.234
0.261
0.683
0.326
Top 30% raw captions U top 30% translated captions
47.7M
0.284
0.247
0.260
0.696
0.340
Top 30% raw captions & top 30% translated captions
76.8M
0.289
0.250
0.262
0.696
0.335
19

---

Comparison to training with synthetic captions
Table 6: When fixing training images replacing translated English captions with synthetic captions generated by BLIP2, we find that performance decreases in general. Since filtering from translated captions exposes CLIP to both new images and new text distributions, we seek to disentangle the impact of these two factors model performance. Our results suggest that having access more diverse images alone (without the corresponding translated multilingual captions) may be insufficient achieving performance gains.
Baseline name
Dataset size
ImageNet
ImageNet shifts
Retrieval GeoDE
Average over 38 tasks
Top 20% translated captions
25.6M
0.329
0.275
0.296
0.709
0.359
Top 20% translated captions, replaced with synthetic captions
25.6M
0,283
0.255
0.350
0.696
0.336
Top 30% translated captions
38.4M
0.311
0.265
0.305
0.718
0.351
Top 30% translated captions, replaced with
38.4M
0.282
0.253
0.371
0.703
0.341
synthetic captions
As observed in Section 4.1, training on filtered translated captions outperforms training on filtered raw captions across all major metries. This could be attributed both changes in captions (from original web-crawled texts to English-translated texts) as well as changes in images (since "Filtered raw captions" and "Filtered translated captions" only share some of the images in common, see Appendix D). Here we attempt to disentangle the contribution to performance gain from these two changes.
Given the images selected by filtering based on (image, translation caption) cosine similarity (i.c., "Top 20% translated captions", "Top 30% translated captions"), we generate synthetic captions for each image using BL.P2 model (24) and the generation hyperparameters from [31. Training on the new (image, synthetic caption) pairs leads to lower performance overall compared to training on the original (image, translated caption) pairs (Table 6). This suggests that having access to more diverse (non-English) images in the training is sufficient boost performance; the diversity coming from translated multilingual captions is also necessary for obtaining the accuracy gains.
We acknowledge that since BLIP2 possible that the captioning model finds it difficult to caption non- non-English images. This ablation study experiment is thus mostly exploratory, and more experiments needed to assess the performance benefits coming from seeing more nng-Esglish images, versus seeing more diverse linguistic concepts in (translated) -nngllissh captions.
pre-trained relatively few multilingual samples, it is
20

---

G All DEN filtering baselines
Table Here we report baselines we experiment with using the public DFN LB] filtering; all
models are trained for 128M steps as set by the DataComp benchmark. For each caption distribution (i.e., raw/ translated/ English-only), only the filtering threshold that yields the best average performance across 38 tasks is shown in Table
Baseline name
Dataset size
ImageNet
ImageNet shifts
Retrieval GeoDE
Average over 38 tasks
Top 20% captions
25.6M
0.316
0.260
0.282
0.688
0.350
Top 20% raw captions, replaced with trans-
25.6M
0.304
0.252
0.268
0.668
0.331
lated cuptions
Top 30% raw captions
38.4M
0.297
0.246
0.280
0.663
0.337
Top 40% raw captions
51.2M
0.267
0.222
0.274
0.669
0.320
Top 20% translated captions
25.6M
0.329
0.275
0.296
0.709
0.359
Top 30% translated captions
38.4M
0.311
0.265
0.305
0.718
0.352
Top 40% translated captions
51.2M
0.289
0.248
0.288
0.709
0.332
Top 20% raw English-only captions
8.0M
0.260
0.218
0.234
0.603
0.303
Top 30% raw English-only captions
12.0M
0,280
0.238
0.259
0.630
0.326
Top 40% raw English-only captions
16.0M
0.283
0.236
0.278
0.666
0.327
Top 50% raw English-only captions
20.0M
0.277
0.236
0.280
0.668
0.321
Top 20% raw captions top 20% translated captions, using translated caption for
34.2M
0.316
0.265
0.289
0.716
0.353
Top 20% raw captions U top 20% translated captions
34.2M
0.329
0.271
0.298
0.720
0.364
Top 20% raw captions & top 20% translated captions
51.2M
0.336
0.280
0.301
0.725
0.361
Top 30% raw captions & top 30% translated captions
76.8M
0.295
0.248
0.282
0.673
0.340
21

---

H
Training for longer
Here we show the results of all the baselines that we train for 1.28B steps (i.e., 10x the number of steps set by DataComp). In Table we find that when using either raw web-crawled captions or translateddddd captions, filtering for top 30% of the pool does best, and translated multilingual captions continue to yield better performance on standard metrics compared to raw captions. We also provide a breakdown of performance differences between these two baselines across 38 tasks from DataComp in Figure 8
Table 8: When the training duration increased by 10x compared to the DataComp setting, training on translated multilingual captions continues to outperform training on raw captions across range of metrics; using both sources of captions continues to yield the best performance. We show performance of all the baselines that are trained for 1.28B steps. Even though using filtered raw captions and using filtered translated captions yield similar average performance (0.414 percentage points), the latter still surpasses the former ImageNet, ImageNet distribution shifts, retrieval and GeoDE (worst-region accuracy). We also note that these performance gaps widen with training duration (see Table for a comparison).
Baseline name
Datuset size
ImageNet
ImageNet Retrieval GeoDE shifts
Average over 38 tasks
Top 20% raw captions
25.6M
0.423
0.345
0.331
0.751
0.407
Top 30% raw captions
38.4M
0.414
0.340
0.344
0.742
0.414
Top 40% raw captions
51.2M
0.417
0.344
0.358
0.746
0.410
Top 20% translated captions
25. 25.6M
0.421
0.348
0.346
0.754
0.412
Top 30% translated captions
38.4M
0.427
0.347
0.352
0.771
0.414
Top 40% translated captions
51.2M
0.421
0.348
0.346
0.754
0.412
Top 20% raw captions U top 20% translated captions
34.2M
0.441
0.359
0.353
0.775
0.427
Top 20% captions top 20% translated captions
51.2M
0.456
0.369
0.371
0.776
0.435
Top 30% raw captions top 30% translated captions
76.8M
0.419
0.347
0.345
0.771
0.429
2.9
3.1
Accuracy diffrennee (%)
0.30:
13
0.3.-.22-0.111
-6
344
-10
-18
wilds/camelyent 7 aa
-20
Mi Glosust vhhicle dialanca
disee odfect distana
Howars
arho
cxteachfg (01
pots
witistinndcam dcam
did
sund577
Imageniss
pram
bou101
fove airzatt
Cla 100
fnndendaa12
morb
objechel
Baiaar sireet
insegnnrtty
dlevr count oil
Emaryonetta
Bingeneea
entis
avin
filrr. 1k
Figure 8: With the same degree of filtering, training with (image, translated caption) pairs improves per- formance on 23 out of 38 tasks compared to training with (image, raw caption) pairs, including ImageNet distribution shifts, retrieval, and tasks with geographically diverse inputs. We compare performance each task of DataComp benchmark between training with raw captions and training with translated captions. when both are trained for 1.28B steps. Both datasets have also heen filtered with image-text cosine similarities output by the public DFN [3] to select the top 30% examples. We find that when we increase training duration be x0x longer than DataComp's setting, using translated multilingual captions and using captions yield similar average performance across 38 tasks. However, the former still outperforms the latter most of the ImageNet distribution shifts (red), retrieval (blue) and fairness-related tasks (dark yellow).
22

---

I More performance analysis
In addition to the analysis we provide further breakdown of performance changes the income group and class levels, on Dollar Street (Figure and ImageNet (Figure (0) respectively, when we swap the training distribution from top 30% raw captions to top 30% translated captions.
80
Figure 9: On Dollar Street, using trans- lated multilingual captions leads to per- formance improvement across all income groups. Dollar Street (39 another fairness- related task that involves classifying images of everyday items collected from households around the world with different socineco- nomic hackgrounds. We break down the per- formance on this dataset by income groups and find that training on top-quality trans- lated captions improves the classification ac- curacy across all groups, compared to train- ing on top-quality raw captions.
Filtered raw captions
75
Filtered translated captions
Top aacurary
$27-200
S200-684
$685-198 Monthly income ($USD)
$1998-19671
40
Per-classs accuracy diference (%)
20
-20
40
80
Intertainment cartor
Entisbudhe Sonmarbund
chambered alvo salmon Qelsntal
Maltess
oplory
Bullokk crrrt
aegore dide rade dael siggner
harp pescora
coo tul soorr
Aatonn Broer
Fyvan
tant
Kery Blue Ternr
cato
rock brautty lish
caron
common
Figure 10: 40 ImageNet classes that observe the largest changes in classification performance when we train on top translated multilingual captions compared to top raw captions. We show 40 categories from ImageNet that see the biggest change in accuracy when (translated) multilingual data is included in the training set.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We enumerate the contributions in the introduction and describe the contribu- tions in more detail in subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This work does not contain any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We describe the experimental setup in Section 3 and full training hyperparam- eters in Appendix B The translation model publicly available on HuggingFace and we plan to release the translated captions well.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our experiments mostly use the training code and data from the DataComp benchmark [14,, which are publicly available. We did not open-source the translation code, but we describe in detail in Section how we apply translation to all samples in the pool and re-filter them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /pipss.cc/pullc//guu guwdes/ddeeeeee CodeSubmeeeeeeeeeeeeee ss ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We follow the hyperparameter setup of the DataComp benchmark [14 and describe the hyperparameters in Section 3 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to compute constraints and the many ablation experiments that are involved, we did not run multiple random seeds for each baseline. The only experiment with multiple runs for computing MAUVE score. We plan to run more random seeds for select basclines to accommodate the review process.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide compute details in Appendix In general, our training runs follow the hyperparameters and infrastructure of the DataComp benchmark [14), readers could refer to this previous work for more details regarding the compute design.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics htttpp////ourrpp..cc/p//////cccc
Answer: [Yes]
Justification:
We have read and followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts of our work in the Discussion section.
27

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We have not yet released the translated caption data but plan follow the guidelines from DataComp's dataset release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and follow the license of DataComp and NLLB model. We also cite the three public data sources that the DFN was trained on.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.papersstt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The data preprocessing and data filtering pipelines are described in detail in Section 3] We plan to release 128M English- the review period.
English-translated translated captions on HuggingFace after
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our experiments do not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments do not involve human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "M6vPoYzk1M", "reviewer": "Reviewer_xoEn", "summary": "In this paper, the authors conduct a thorough exploration of how multilingual image-text pairs benefit English vision tasks. They first present how to effectively utilize translated data to improve performance on standard vision tasks and derive valuable conclusions through detailed ablation studies. Second, they illustrate the differences in image and text distributions between English and non-English image-text pairs. These findings highlight the potential of leveraging multilingual datasets to enhance the robustness and accuracy of vision models.", "questions": "No", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents effective strategies for utilizing translated data to enhance performance on standard vision tasks\n2. Detailed ablation studies are carried out to derive valuable conclusions, which help the community understand the impact of various factors.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. No quantitative assessment for the quality of translation.\n2. Only evaluation on representation task. Does such data strategy also work for the training of Multimodal LLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback!\n\n**No quantitative assessment for the quality of translation.** Given your concern, we have performed additional analysis of the translation quality. We sampled 100K multilingual captions from our raw data pool and backtranslated the English-translated caption into the original language (e.g. Chinese text -> English translation -> Chinese translation of the English-translated text). Then we computed the cosine similarity between the initial web caption and the backtranslated caption using embeddings from the multilingual Sentence-BERT model (Reimers et al., 2019), to assess how much semantic meaning is preserved after translation. We find that on average the cosine similarity (thus, translation quality) remains relatively high (0.63). Below we report the top 5 and bottom 5 languages that observe the highest and lowest translation quality as captured by our metric (each has at least 30 samples). We will add the full analysis in the next version of the paper.\n\n| Language | Text cosine similarity after backtranslation |\n| -------- | ------- |\n| English | 0.886 |\n| Norwegian Nynorsk | 0.883 |\n| Bengali | 0.883 |\n| Russian | 0.860 |\n| Norwegian Bokmål | 0.839 |\n| | |\n| Marathi | 0.271 |\n| Irish | 0.240 |\n| Standard Latvian | 0.233 |\n| Chechen | 0.0595 |\n| Karachay-Balkar | 0.00280 |\n\n**Only evaluation on representation task. Does such data strategy also work for the training of multimodal LLM?** We note that our evaluations already contain 38 datasets involving recognition and classification of a wide range of domains (e.g. texture/ traffic sign/ scene/ metastatic tissue/ geolocation/ animal/ etc.), in addition to image-text retrieval and commonsense association.\n\nExtending the findings from this work study to multimodal LLM training would be an interesting direction for future work, but is orthogonal to our contribution; we will add this to the Discussion section. Our results demonstrate that there is so much diversity and visual information to be leveraged from multilingual data, to the extent that training on predominantly non-English samples can improve CLIP’s performance on standard English benchmarks. We hypothesize that leveraging this diversity will also be beneficial for training multimodal LLMs, especially since many of which still rely on English-dominated image-text datasets (e.g. LAION) and still use CLIP as the image encoder (e.g. LLaVA-1.5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLqvhQ2peH", "reviewer": "Reviewer_9sUx", "summary": "The authors investigate whether *multilingual* vision-language data improves the *English-only* performance on a model in vision-language tasks.\n\nThey translate captions from DataComp from English to other languages and train a CLIP model on these multilingual captions.\n\nThey find that this action boosts performance on both English-only tasks like imagenet matching and geographically diverse tasks like GeoDE.", "questions": "Figure 1: where are these examples from? Are they actually in the datacomp datasset? Doesn't translating datacomp captions into other languages keep the learned representations in the \"English concept space\" of sorts?", "limitations": "Yes, limitations are sufficiently covered.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a super efficient way to gain a performance boost on DataComp (for revs who might not be aware, this is a cool benchmark for assessing efficient training of vision-language representations evaluated over a bunch of downstream datasets within a fixed compute budget). Any method that boosts DataComp performance across so many tasks for free (just translation instead of collecting new captions) is awesome in my opinion.\n\nPerformance boost on GeoDE (geographically sorted img classification) is improved considerably in all regions, justifying the purpose of the intervention.", "weaknesses": "Unclear how the proposed automated translation pipeline ties in to the original motivation (culturally specific items not being captured in English data)\n\nCould benefit from a bit deeper description of what the GeoDE task is for unfamiliar readers to improve reach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and for recognizing the strengths of our work!\n\n**How the proposed automated translation pipeline ties in to the original motivation (culturally specific items not being captured in English data).** We discuss the presence of culturally salient concepts in the Introduction mainly to motivate how multilingual data is inherently enriching and complements English data. Improving the availability of culturally specific items in the final training set is not what our method specifically optimizes for, but is likely to happen with the inclusion of substantially more samples of non-English origins after filtering (see Figure 2 of main paper). \n\n**Deeper description of what the GeoDE task is for unfamiliar readers.** Thank you for raising this point. GeoDE is a geographically diverse dataset containing 61K images spanning 40 common object categories, collected from 6 different world regions via crowd-sourcing. We will add more details of this task to the paper.\n\n**Where are the examples in Figure 1 from? Are they actually in the datacomp dataset?** Yes, these examples are taken from our training data, i.e. DataComp.\n\n**Doesn't translating datacomp captions into other languages keep the learned representations in the English concept space?** We interpret this question as whether translation would reduce the richness of the concept space (especially when it comes to translating culturally salient concepts). We empirically observe that after translating multilingual captions to English, some culturally salient concepts are still preserved - refer to Figure 1 (left) of the main paper for examples. Our work has also acknowledged this potential limitation, that “translation can sometimes be too literal, subject to losing the intent and richness of the original phrasing”. Nevertheless, we hope that our findings inspire future work to (i) be more inclusive of data of non-English origins during training, (ii) investigate other ways to effectively leverage the diversity of multilingual data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4OlJocIrwy", "reviewer": "Reviewer_7AUA", "summary": "This paper points out an important issue in current training of CLIP models---the push for including more english-centric / english-only data  in the pretraining dataset. The paper points out that this is mainly driven by the downstream evaluation test-beds primarily being english-focused and hence the need to include multi-lingual data has not been found so far. The paper conducts a thorough study of using multilingual data into the pretraining datasets of CLIP models by using caption translation models. The paper evaluates and shows improvements on several downstream datasets from the 38-Datacomp benchmark evaluation suite, and provides analysis on the key differences between multilingual and english-only pretraining datasets.", "questions": "I have a few additional questions/comments which I include below:\n\n- How do you estimate the english part of the web-crawl? You mention it is in 1/3rd in the introduction.\n\n- Is there an explanation for why filtered translated captions might improve english-centric performance over the filtered raw captions? This seems counter-intuitive given as you say in the paper, that ImageNet is mostly western-centric. This also seems to contradicts the results of the No-Filter paper [1].\n\n- Could the model trained on this filtered raw subset itself be used as a DFN, and potentially further improve the performance when training on the raw multilingual pool? The idea being that you might have to do less tuning on the filtering threshold since the model is more robust to noise from the translated captions.\n\n- In fig 3, are the results from the \"filtered raw captions\" and \"filtered translated captions\" methods from tab 1? Doesn’t seem so since the performance improvements on the 38 tasks mentioned in the caption of fig 3 is 1.5% whereas the performance improvements from tab 1 is only 0.9%.\n\n- Why does performance on Food101 go down? Is it because all the food classes included in Food101 are primarily English-centric? I am certain that is not the case. I would be very interested in a more detailed analysis of why the performance on Food101 goes down when we include more multilingual data since that is one aspect that I would have thought improves quite a lot. \n\n[1] Pouget et al, No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models", "limitations": "The authors are explicit about their limitations. Any further limitations I think are mentioned in the weaknesses section above.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and easy to follow\n- All the claims in the paper are verified with substantial empirical evidence \n- The paper's main message is adequately represented throughout the paper without any distracting claims, hence all the experiments done in the paper flow very smoothly\n- The paper points out a very important issue in current pretraining datasets of CLIP models, and showcases a simple intuitive fix for future models and pretraining datasets", "weaknesses": "- The paper could benefit from some additional analysis. I note down some analayses that I think might strengthen the paper below.\n\n- For a particular concept is it possible to showcase the diversity afforded by looking at only English data and multilingual data? For instance, take the concept of “stove”. You could perhaps manually take 10-30K images of a stove (assuming that many exist), by filtering the texts for this particular concept, and do the same for both English only and multilingual data. Then, quantify the diversity of images, by perhaps taking a strong Dino-v2 pretrained encoder and then clustering them, and perhaps looking at some intra- and inter-cluster distance metrics. I think this would nicely validate the qualitative visualisation in figure 1 with some empirical evidence that multilingual data indeed boosts the diversity of visual concepts in pretraining datasets. It would be great to see this analysis on perhaps one or two concepts.\nWhy is this an important analysis? My reasoning would be that while it is totally plausible that multilingual concepts add diversity, it is unclear to me how much of this is actually true? Since it is plausible that English webpages also contain such diverse visual concepts, I think it would be nice to see if there are “new visual concepts” being included in the pretraining corpus, or is it mostly that these “culturally-diverse” visual concepts were present in the english-only pretraining datasets, but we just boost up their frequency in the pretraining dataset? I think either of the two cases is a valid justification for the claim that multilingual data increases diversity, but it would be nice to have a precise answer for what exactly out of the two is the case.\n\n- The main claims of the paper that multilingual data improves VL representations is sufficiently backed up, but benchmarking on culturally diverse / multilingual-sourced data is still limited. The paper only considers GeoDE and DollarStreet for evaluation. I would recommend to also benchmark and report the performance on some more culturally and geospatially diverse datasets like [1,2,3,4]. Some of these datasets might need to be ported into a classification / retrieval format, but in general I think this is an important experiment to do to further validate the important of multilingual-sourced pretraining data.\n\n- By filtering on top of the translated image-text pairs, it seems likely that the training data diversity increases, and potentially train-test similarity [5] also increases. This could be an added confounder in the takeaways of the paper. Could the authors comment/discuss this a bit more in the paper?\n\n- The analysis in the paper would further be improved by checking how the distribution of CLIPScores look before and after the translation. Potentially there would be a shift to the right in the similarity distribution, but it would be a good analysis to include for better intuitions on what might be good filtering thresholds if such a model were to be used in the future for vector arithmetic / data filtering itself.\n\n[1] Kalluri et al, GeoNet: Benchmarking Unsupervised Adaptation Across Geographies\n[2] Weyand et al, Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval\n[3] Yin et al, Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning\n[4] Romero et al, CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark\n[5] Mayilvahanan et al, Does CLIP’s generalization performance mainly stem from high train-test similarity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the valuable suggestions!\n\n**Showcase the diversity for certain concepts.** Per your suggestion, we ran additional analysis to compare the diversity of images for the same concept. We sampled 1K images from each data distribution for which the corresponding (translated) captions mention “stove”. With DINOv2, we obtained embeddings of these images and used them to cluster each data distribution into 20, 50 and 100 clusters. Figure 2 in our global response shows the average inter-cluster distance of “stove” images in English data versus non-English data. Overall the latter group of images yields higher inter-cluster distance, suggesting that the clusters are more well-separated and the non-English “stove” images are more diverse. We will add this experiment to the paper.\n\n**Benchmarking on culturally diverse data is still limited.** Our work does not explicitly seek to improve performance on culturally diverse tasks. We explore whether increasing the diversity of the data origins, and consequently allowing the training set to be dominated by non-English samples, can improve performance on vision tasks *in general*, especially on those that dictate the field’s state-of-the-art like ImageNet. Improvement on geographically diverse benchmarks is an added benefit in this process. However, given the reviewer’s feedback, we have added a new evaluation on Google Landmarks dataset (Weyand et al., 2020). When training for 10x longer, using filtered raw captions yields 16.9% classification accuracy, while our best baseline (mixing filtered raw captions and translated captions) gets 18.5% accuracy.\n\n**By filtering on top of the translated image-text pairs, potentially train-test similarity also increases.** Changes in train-test similarity may happen as a result of using translated captions, but that should not be a confounder in the paper takeaways because (i) the proposed translation method does not directly optimize for train-test similarity, (ii) we apply the same filtering process (i.e. based on DFN score) to both raw and translated captions, (iii) we measure performance on a large number of vision tasks (38).\n\n**Checking how the distribution of CLIPscore look before and after the translation.** We have added this analysis in our global response (see Figure 1 of the attached PDF). We sampled 10K images with English captions and 10K with non-English captions from the initial pool, and compared how the DFN scores change with translation. Unsurprisingly, DFN scores for non-English samples generally increase after the captions are translated into English. This in turn leads to a right shift in the distribution of image-text similarity scores when looking at the score histograms before and after translation. As image-text alignment (measured by DFN score) tends to help with empirical performance, this shift suggests that translation increases the availability of good training data.\n\n**Estimate that English takes up ⅓ of any web crawl.** This estimate is computed from using the NLLB model to detect languages in the DataComp dataset. Since DataComp performed minimal preprocessing (i.e. NSFW removal and test set deduplication), we hypothesize that the initial DataComp pool is representative of the natural distribution of raw data on the web that is appropriate for training.\n\n**Improvement on ImageNet contradicts the results of the No-Filter paper.** We note some differences between our experiment setup and that of concurrent work:\n\n(1) We compare training on filtered translated captions (which is *dominated* by non-English data) to training on filtered raw captions (which is *dominated* by English data). In contrast, the No-Filter paper compares multilingual data (globe) to its strict subset containing *only* English data (en).\n\n(2) We perform filtering and retain only a small fraction of the original pool (20-30%), whereas the No-Filter paper trains on web data with “minimal filtering”. We posit that filtering is important to obtain competitive performance with translated captions, especially given the noise introduced by the translation process.\n\nWe believe our setup better mimics how pre-training data curation is commonly done in practice (e.g. LAION, which was also heavily filtered and contains a mix of English and non-English data).\n\n**Why filtered translated captions might help improve English-centric performance.** While the fraction of images of English origins is lower when we use filtered translated captions instead of filtered raw captions (60% → 40%, Figure 2 of our paper), the number of such images in the final training set is still significant. As a result, the model can still learn English-centric concepts to a certain extent. We hypothesize that the performance on English vision tasks is further reinforced by training on high-quality, diverse multilingual data, which helps induce better visual features and robustness in general.\n\n**Average performance improvement in Figure 3.** The analysis in Figure 3 of our paper is performed on models trained with the top 30% of the initial pool (see description in Lines 251-254), whereas the first section of Table 1 involves training on the top 20% of the pool. We pick the 30% threshold for further analyses because (i) it is the baseline that yields the best performance when training until convergence, (ii) the average performance gap between filtered translated captions and filtered raw captions is larger at this threshold, allowing us to observe the differences on individual tasks better.   \n\n**Why does performance on Food101 go down?** We note that (1) the accuracy change for the Food101 task between using filtered translated caption and using filtered raw captions is relatively small (-0.2%), and (2) the Food101 class names are still dominated by English-centric concepts. Besides, the noise in the translation process may affect whether the class mentions in the original languages are still preserved after translation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lD3QcwZhrf", "reviewer": "Reviewer_2aum", "summary": "The paper conducts a systematic study to explore the performance benefits of using translated non-English data for English vision tasks. By translating multilingual image-text pairs from a raw web crawl to English and re-filtering them, the authors show that continual pre-training on this data increases the performance of the English model on a diverse set of tasks with a single model.", "questions": "Possible experiments to make the paper better:\n- Add more multicultural evaluation datasets\n- Add more evaluations to show the benefits of using diverse training data could benefits multiple models sizes and types\n- Perform similar training and evaluation in other languages, to show all languages would benefits from diverse training images and captions", "limitations": "While this research may be interesting for those working primarily on monolingual research (i.e., English), its framing should be more sensitive to the many efforts related to multilingual and multicultural NLP. Personally, I think the benefits of using diverse images and captions can be achieved by collecting diverse images and paraphrasing captions or through better sampling of the training data, rather than just using multilingual or multicultural datasets and translating them. As a community, we already lack people working on multicultural and multilingual models, yet this work uses data and research from other cultures to improve **English-only** models (although with good intentions). Perhaps there could be some discussions on this.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors show that by incorporating translated multilingual and multicultural VL dataset can improve English model's results empirically on English only tasks.\n* The paper provides interesting analysis showing that using diverse training data improves benchmark results that require geographical or cultural diversity.", "weaknesses": "- The improvement from using translated data diminished as the training converged, which is not surprising. Similarly, using more training data to improve the evaluation results on downstream tasks is also expected. Hence, the novelty of insights is limited in this paper.\n- The paper only evaluated on one LLM, which more evaluations are required to validate the generality of the claim of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback.\n\nIt is very important that we clarify a misconception: our work deals with vision-language models and vision benchmarks. We do not train any LLMs or make claims about the state of multilingual NLP research. \n\n**Framing should be more sensitive to multilingual efforts in NLP/ This work uses data to improve English-only models while we already lack people working on multilingual models.** To clarify,\n\n(i) We support multilingual and multicultural efforts. While benchmarking and improving performance on multilingual tasks is not the focus of our work, as one step in this direction, we enable English-language models to gain exposure to more samples of diverse origins and consequently, do better on geographically diverse benchmarks (e.g. GeoDE and DollarStreet). We discuss how to adapt models from our paper to further enhance their multilingual capabilities under the “Future work” section. Comparing the multilingual performance of these post-training adaptation methods to directly training on non-English captions merits a separate investigation. \n\n(ii) Instead, the goal of our work is to challenge the status quo: many ML practitioners seek to improve state-of-the-art performance on popular benchmarks, which happen to be English vision tasks like Imagenet. In the process of overfitting to these benchmarks, previous work discards a lot (if not all) of non-English samples due to the belief that non-English data does not benefit English evaluations. We question this practice. But in order to *incentivize change*, we need to show that it is possible to improve the English-centric “state-of-the-art” by using more data of non-English origins. We hope that our positive findings could help multilingual and multicultural efforts become the default design choice for data curation, instead of existing as a second priority or as a separate (societal) consideration.\n\n**The improvement from using translated data diminished as the training converged, which is not surprising.** We want to clarify that the results we reported show the opposite. From Table 1 of the main paper, our best baseline, “Filtered raw captions & Filtered translated captions”, outperform “Filtered raw captions” by 2.0% on ImageNet and 1.1 percentage points on average across 38 tasks. When training for 10x more steps, the performance gaps increase to 4.2% and 2.1 percentage points respectively.  \n\n**Using more training data to improve the evaluation results on downstream tasks is also expected.** For our baselines, we fix the training budget and the amount of raw data available (i.e. initial pool). With each caption distribution, we tune the filtering threshold, and consequently the size of the resulting filtered subset used for training (this was reported in Table 1 of the main paper). We note that with this setup, increasing *filtered data quantity* is *not* guaranteed to lead to better performance on downstream tasks, as larger subset (with less strict filtering) means fewer passes through each datapoint. In Table 1 we only report the *best performance obtainable* after filtering each data distribution. Table 6 (Appendix) contains the full results, accounting for the number of samples used. While “Top 20% raw captions & Top 20% translated captions” indeed uses more training data than “Top 20% raw captions” (51.2M versus 25.6M), when controlling for data quantity taken from the latter distribution, “Top 20% raw captions & Top 20% translated captions” still significantly outperforms “Top 40% raw captions” (both having 51.2M samples).\n\n**Add more multicultural evaluation datasets.** Given the reviewer’s concern, we have added a new evaluation on Google Landmarks dataset (Weyand et al., 2020). When training for 10x longer, using filtered raw captions yields 16.9% classification accuracy, while our best baseline (using a mix of filtered raw captions and translated captions) gets 18.5% accuracy. \n\n**Show the benefits of using diverse training data could benefit multiple models sizes and types, and other languages.** Due to compute constraints, we did not experiment with more model types and translating to other languages, but we will add these points to our Discussion section. For context, obtaining the baseline numbers for our paper (see Appendix Tables 6 & 7) took about 850 GPU hours with 8 A40 GPUs. Besides, we find that extending our approach to other languages (e.g. translating all captions to Chinese and measuring performance on Chinese tasks) is sufficient scope for a separate paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers again for providing thoughtful reviews of our work. Here we highlight several new results/ analyses taking your feedback into consideration:
\
&nbsp;
1. Reviewer xoEn wondered about the translation quality. In response, we sampled 100K captions from our raw data pool and backtranslated the English-translated caption into the original language (e.g. Chinese text -> English translation -> Chinese translation of the English-translated text). To assess the translation quality, we computed the cosine similarity between the initial web text and the backtranslated text using embeddings from the multilingual Sentence-BERT model (Reimers et al., 2019). We find that on average the cosine similarity (and thus, translation quality) remains relatively high (0.63). Below we report the top 5 and bottom 5 languages that observe the highest and lowest translation quality measured by our metric:
| Language | Text cosine similarity after backtranslation |
| -------- | ------- |
| English | 0.886 |
| Norwegian Nynorsk | 0.883 |
| Bengali | 0.883 |
| Russian | 0.860 |
| Norwegian Bokmål | 0.839 |
| | |
| Marathi | 0.271 |
| Irish | 0.240 |
| Standard Latvian | 0.233 |
| Chechen | 0.0595 |
| Karachay-Balkar | 0.00280 |

2. Reviewer 7AUA asked about DFN score changes before and after translation. We analyzed the score differences for 10K English and 10K non-English samples, and found that translation improves the image-text similarity score of non-English samples, as well as the overall score distribution of the data pool. Refer to Figure 1 of the attached PDF for more details.
\
&nbsp;
3. Reviewer 7AUA suggested adding another metric for quantifying the diversity of multilingual data by measuring clustering distances. In Figure 2 of the response PDF, we showed that for a specific concept such as “stove”, non-English images form more distinct clusters than English ones. We randomly sampled 1K images with English captions and 1K with non-English captions, such that the (translated) captions mentioned “stove”, and embedded them with the DINOv2 model. Across different numbers of clusters uncovered, non-English data generally yields higher inter-cluster distance, suggesting that the “stove” images with multilingual captions are more heterogeneous compared to those with English captions.
\
&nbsp;
4. Since Reviewers 2aum and 7AUA requested more multicultural evaluations, we ran a new evaluation on Google Landmarks dataset (Weyand et al., 2020). When training for 10x longer, our best baseline (using a mix of filtered raw captions and translated captions) yields 18.5% classification accuracy, outperforming the baseline trained on just filtered raw captions by 1.6%. We note that the performance on this task is limited given that our best data mix only uses ~51M samples.
\
&nbsp;
5. Last but not least, Reviewer 2aum had concerns about our investigation not being sensitive to the many efforts related to multilingual and multicultural NLP. **Here we would like to restate the goal of our work:**

We advocate for increasing the cultural and linguistic diversity of the training data for vision-language models (VLMs), and view our findings as complementary to other multilingual and multicultural efforts. While these efforts have led to new geographically diverse benchmarks as well as new ways to increase the availability of high-quality non-English data, their adoption is still relatively limited. This is evident from the fact that many popular models that are considered state-of-the-art (e.g. MetaCLIP, SigLIP, ALIGN) were still trained on entirely English image-text pairs. In order to change this status quo, it is important to demonstrate that training on predominantly data of non-English origins can do better than training on predominantly English data, especially on standard vision tasks that define the field’s state-of-the-art (e.g. ImageNet). Our work provides such evidence. We hope this can lead to more active exploration and adoption of multilingual and multicultural data in mainstream VLM training, instead of using this data only when under-served populations or tasks are involved.

Meta-review: This paper presents a conceptually simple but extensive studies of how translated multilingual (non-English) image-text pairs benefit English vision tasks. They also provide an intriguing analysis of the differences in image and text distributions between English and non-English image-text pairs, supporting the claim that multilingual data enhance diversity.

Overall, all reviewers were favorable to the unique focus and findings of the paper, and were satisfied with the authors' rebuttal. As a result, their final scores are unanimously toward acceptance, with strong supports from three reviewers. The AC would like to support their judgement.
Decision: Accept (spotlight)