Multilingual Diversity Improves Vision-Language Representations

Authors: thao  nguyen
matthew  wallingford
sebastin  santy
wei-chiu  ma
sewoong  oh
ludwig  schmidt
pang wei koh
ranjay  krishna
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:14:14

Abstract: Massive web-crawled image-text datasets lay the foundation for recent progress in multimodal learning. These datasets are designed with the goal of training a model to do well on standard computer vision benchmarks, many of which, however, have been shown to be English-centric (e.g., ImageNet). Consequently, existing data curation techniques gravitate towards using predominantly English image-text pairs and discard many potentially useful non-English samples. Our work questions this practice. Multilingual data is inherently enriching not only because it provides a gateway to learn about culturally salient concepts, but also because it depicts common concepts differently from monolingual data. We thus conduct a systematic study to explore the performance benefits of using more samples of non-English origins with respect to English vision tasks. By translating all multilingual image-text pairs from a raw web crawl to English and re-filtering them, we increase the prevalence of (translated) multilingual data in the resulting training set. Pre-training on this dataset outperforms using English-only or English-dominated datasets on ImageNet, ImageNet distribution shifts, image-English-text retrieval and on average across 38 tasks from the DataComp benchmark. On a geographically diverse task like GeoDE, we also observe improvements across all regions, with the biggest gain coming from Africa. In addition, we quantitatively show that English and non-English data are significantly different in both image and (translated) text space. We hope that our findings motivate future work to be more intentional about including multicultural and multilingual data, not just when non-English or geographically diverse tasks are involved, but to enhance model capabilities at large.

Main paper: 
 introduction: 
 Today, the predominant pre-training paradigm for vision-language models relies on large quantities of image-text pairs scraped from the web [35, 24] . As raw web data contains a significant amount of noise, automatic data filtering approaches are designed to curate a high-quality subset and maximize the performance of a model trained on this subset on standard computer vision benchmarks (e.g., ImageNet). However, these benchmarks typically only evaluate in English, and many of them have been shown to be geographically biased: for instance, ImageNet images are mostly sourced from North America and Western Europe [43] . Consequently, it is possible that we are designing data curation algorithms that propagate a monolingual bias, i.e., filtered datasets are increasingly dominated by English image-text pairs. In fact, a lot of highly cited work-including CLIP [35] , ALIGN [21] and BASIC [33] -relies exclusively on English data. Using more multilingual data for training is often only a deliberate design decision when non-English tasks are involved [45, 15, 26] .Figure 1 : Multilingual image-text data adds diversity to the English data distribution in various, significant ways (a) We show some examples of culturally salient concepts that would not exist in "high-quality" English data (as determined by CLIP score), such as "bamboo steamer", "kiji" (the national bird of Japan) and yalı (a traditional architecture style for Turkish waterside houses) (b) Even for a common everyday object ("stove"), non-English and English images portray very different visual representations.Multilingual data enriches any monolingual data distribution; multilingual data brings attention to culturally salient concepts and introduces new perspectives and annotations for the same visual category [48] . As illustrated in Figure 1 , there are certain native concepts, e.g. 'kiji' (the national bird of Japan), that are more likely to be conveyed in Japanese (non-English) captions compared to English ones. Even in the case of a common everyday object ('stove'), the non-English and English images look very different. Despite the diversity present in multilingual data, it is disproportionately excluded from existing large-scale pre-training corpora.In this paper, we investigate the counterfactual: can we improve on English vision tasks by diversifying the cultural and linguistic backgrounds of the training data?Our investigation is motivated by a dichotomy: English image-text pairs constitute a minority of any random web crawl (in our estimate, one-third); yet, they form a majority in popular pre-training datasets such as LAION-5B [42] , DataComp [14] , and DFN [13] .It is common for web-scraped corpora to remove "low-quality" data by using a high-performing model (e.g., OpenAI CLIP) to compute image-text alignment and rank the raw data samples. However, this process often disproportionately favors English data if the filtering model also has an English bias [14, 20] . In addition to discarding many potentially useful non-English image-text pairs, this can also negatively impact the geographical and cultural representation of the resulting dataset, and consequently, the model's performance on certain underrepresented populations [38, 36, 10] .Our key observation is that the diversity present in multilingual data can be confounded by the language the data is in, making it difficult to observe the empirical benefits of using such data in model training. To offer a more systematic study of the effectiveness of multilingual data-in contrast to English-only or English-dominated datasets-we fix the language medium, translate all captions from DataComp's 128M-sample web crawl [14] to English with an advanced translation model. We then re-filter the data pool and train a CLIP model on this translated multilingual data. We focus on two types of evaluations: (i) on standard English vision tasks including ImageNet, MSCOCO and Flickr retrieval, and (ii) on geographically diverse images, e.g. from GeoDE [36] , which contains images of common objects across different geographical locations. We acknowledge that translation can sometimes be too literal, subject to losing the intent and richness of the original phrasing. Nevertheless, we hope findings from our work provide a starting point for studying how to leverage the diversity of multilingual data more effectively.Our contributions are as follows:• We demonstrate that with translation, non-English data does benefit English vision tasks. In particular, training on more samples of non-English origins leads to better performance on ImageNet, ImageNet distribution shifts and image-English-text retrieval. On the DataComp benchmark with a fixed compute budget, our best-performing approach that leverages translated multilingual captions outperforms training on just filtered raw captions by 2.0% on ImageNet and 1.1 percentage points on average across 38 tasks. When training for longer (which mimics the number of epochs largescale multimodal models are often trained for), these performance gaps increase to 4.2% and 2.1 percentage points respectively. • On a geographically diverse task such as GeoDE, training on translated multilingual data leads to 4.2% boost in accuracy on average compared to training on filtered raw data, with performance improvement observed for all regions, especially for Africa where the increase is 5.5%. • We analyze in detail the differences between English and (translated) non-English image-text pairs.We quantitatively show that they capture distinct distributions, both in text and image space, even after they are converted to the same language medium. Consequently, it is beneficial to combine high-quality data from both sources as much as possible, since they are inherently complementary.In summary, despite the abundance of "sufficiently useful" English data, existing data curation techniques can always do better in the data diversity axis by being more deliberate about including data from other language and cultural backgrounds. This way of enhancing diversity in turn leads to a better vision-language model in general, offering performance benefits beyond non-English vision tasks or tasks involving geographically diverse images. We will release the raw captions and the corresponding English translations for the 128M image-text pairs used in our experiments. 
related work: 
 Existing data collection and filtering approaches induce Western bias in downstream datasets and models; benchmarks that seek to capture this bias still receive relatively little attention. Consequently, despite evidence showing cultural and geographical limitations in popular vision datasets, the use of multilingual data is mostly intended for pre-training and fine-tuning multimodal models to do well on non-English tasks. Our work seeks to include more image-text pairs of non-English origins in the pre-training dataset, and shows that this process can improve performance, even on English-centric vision tasks.Western bias of existing models and datasets Several papers have studied biases in popular datasets, especially biases that correlate with culture and geographic locations. Notably, Shankar et al. [43] find that ImageNet and OpenImages exhibit substantial US-centric and eurocentric representation bias. In NLP, Santy et al. [40] find that existing datasets align predominantly with Western and White populations. It is not only the data collection process that leads to a Western bias, but also the data preprocessing pipeline. For instance, automated data filtering with scores output by a model, e.g., OpenAI CLIP, has been commonly adopted as a way to discard low-quality web-crawled samples.Little is known about the potential biases induced by this approach. Hong et al. [20] recently show that CLIP score filter is more likely to include data related to Western countries compared to that of non-Western countries. In [14] (Figure 24 ), the authors offer evidence that CLIP filtering implicitly performs some English filtering, as the top CLIP score examples are increasingly dominated by English image-text pairs. Consequently, all these dataset biases translate to performance disparity, as demonstrated by existing work showing that the accuracy of vision systems drops significantly on non-Western inputs [10, 49, 38, 36] , or low-resource languages [16] .Improving the availability of non-English data in multimodal datasets Translation has been a popular technique to address the limited availability of large-scale and high-quality non-English data in training and evaluation [45, 47, 6, 32, 11, 2, 16] . In addition to translating English captions into the language of interest, previous work also uses a curated list of common words in the native language to scrape image-text pairs from the web [26, 17] . COCO-CN [25] extends the MSCOCO dataset [7] with manually written Chinese captions.Most closely related to our setup is the LAION-Translated dataset [29] , which translates 3B samples of LAION-5B from many languages into English using the M2M100 model [12] . Compared to this dataset construction, we (i) use a more advanced translation model, NLLB [9] , that covers twice as many languages, (ii) work with mostly raw data while LAION was heavily filtered and thus could contain a biased representation of multilingual data. To the best of our knowledge, no existing work has experimented with the LAION-Translated dataset.Adapting CLIP post-training for multilingual tasks Geigle et al. [15] translate high-quality English data into 95 languages, and use these translated samples to re-align an image encoder previously trained on English data to a multilingual language model. Similarly, Chen et al. [8] propose re-training OpenAI CLIP on a mix of Chinese and English data to enhance its multilingual representation. In [4] , the authors explore adaptation without any image data, solely fine-tuning the text encoder with English captions from MSCOCO, Google Conceptual Captions, and VizWiz translated to other languages. Visheratin [45] replace the text encoder of OpenAI CLIP with the text encoder from the NLLB model, and fine-tune the new model on multilingual image-text pairs obtained from translating LAION-COCO's English captions [41] into 200 languages. In contrast to these papers that employ multilingual data for the purpose of adapting to non-English tasks, we focus on using multilingual data to do better on common vision tasks that are in English.Using multilingual data significantly enhances data diversity Our study is partly inspired by findings from Ye et al. [48] , who show that multilingual synthetic captions obtained from existing image captioning systems provide higher semantic coverage than monolingual ones, over 3.6K images. Our experiments instead use raw web-crawled data and explore the performance benefits of embracing cultural and linguistic diversity in (mostly) human-generated captions at scale. 
experimental setup: 
 Given a starting pool of raw image-text pairs scraped from the web, many of which contain non-English captions, we experiment with ways to preprocess and filter this pool into a high-quality dataset. The quality of the dataset is measured by the zero-shot performance of a CLIP model trained on it from scratch.Data We experiment with the medium pool of the DataComp benchmark [14] , which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered corpora such as LAION [42] , DataComp applies minimal data preprocessing, involving only NSFW filtering, deduplication of evaluation sets, and face blurring. This allows the candidate pool to stay close to the natural distribution of the raw web data as much as possible, in addition to enabling maximum flexibility in dataset design.Translation model To detect language and translate the raw captions from DataComp into English, we use the No Language Left Behind (NLLB) translation model [9] , which is considered state-ofthe-art. NLLB is the first to translate across 200 languages, including low-resource ones that are not currently supported by common translation tools. We use the 600M-parameter model publicly available on HuggingFace to allow for fast inference on our large data corpus. All 128M captions from DataComp are translated to English; examples could be found in Appendix A. We provide some quantitative analysis of the translation quality in Appendix C.Training After translating the captions of all samples in the raw data pool, we filter them based on cosine similarity between image and text embeddings. We experiment with using OpenAI CLIP-ViT-L/14 [35] and the public Data Filtering Network (DFN) from [13] to obtain the embeddings, and subsequently, the cosine similarities. The DFN, specifically designed to filter data for subsequent model training, was trained on three public datasets deemed as high-quality-Conceptual Caption 12M [5] , Conceptual Captions 3M [44] , and Shutterstock 15M [30] . We find that indeed the public DFN is better at data filtering compared to OpenAI CLIP, as measured by the performance of CLIP trained on the corresponding filtered datasets (see Appendices E and G).We pretrain a CLIP model [35] on each filtered subset with ViT-B/32 as the image encoder, and follow DataComp's hyperparameters; details can be found in Appendix B. Unless specified otherwise, all models are trained for the same compute budget (128M steps) as determined by DataComp. For some captions across a range of metrics; using both types of captions yields even more performance gains. We report the performance of select baselines on the DataComp benchmark [14] ; all baselines are trained for the same number of steps as specified. Here the filtering threshold (and thus the resulting dataset size) has been tuned for each baseline and we only show the filtered subset that yields the highest average accuracy. We find that with the same filtering method (i.e., using DFN score), training on translated captions ("Filtered translated captions") is more effective than training on raw captions ("Filtered raw captions") as seen from higher performance on ImageNet, ImageNet distribution shifts, retrieval, GeoDE (worst-region accuracy) and on average across 38 tasks. Combining both sources of captions leads to the best performance. Appendix G contains the full results. select baselines, we also experiment with training for 10× longer. The fixed architecture, compute and hyperparameter setup allow us to isolate data quality as the main factor influencing performance.Evaluation We perform zero-shot evaluation of trained CLIP models using the 38 tasks from DataComp. These tasks involve recognition and classification of a wide range of domains (e.g., texture, scene, metastatic tissue, etc.) in addition to image-text retrieval and commonsense association. Among them, we pay particular attention to commonly cited metrics such as ImageNet accuracy, ImageNet distribution shift accuracy -a proxy for natural robustness, and retrieval performance. ImageNet shifts include ImageNet-V2 [37] , ImageNet Sketch [46] , ImageNet-A [19] , ImageNet-O [19] , ImageNet-R [18] and ObjectNet [1] . Retrieval score is the average of the performance on Flickr30K [50] , MSCOCO [7] and WinoGAViL [3] . Throughout the paper we also highlight GeoDE worst-region performance [36] -a task that involves geographically diverse images-to demonstrate the added benefits of geographical inclusivity that training on more (translated) multilingual captions offers.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Our experiments mostly use the training code and data from the DataComp benchmark [14] , which are publicly available. We did not open-source the translation code, but we describe in detail in Section 3 how we apply translation to all samples in the pool and re-filter them.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
impacts of using (translated) multilingual captions on standard vision tasks: 
 We explore training on each caption distribution separately, as well as combining them. Below we describe the baselines from Table 1 in more detail:• Filtered raw captions: As mentioned in Section 3, we use the public DFN from [13] by default to filter the starting pool (128M samples). Given the images and the corresponding web-crawled captions, we experiment with varying the filtering threshold to keep top x% of the pool based on DFN score. In Table 1 , we only report the best average performance obtainable after the filtering threshold has been tuned, and the resulting dataset size. Refer to Appendix G for the full results. • Filtered translated captions: Similar to the approach above, we tune the filtering threshold, but using DFN score between an image and the English translation of the original web-crawled caption. • Filtered English-only captions: Similar to "Filtered raw captions" baseline, here we also tune the filtering threshold to keep only a subset of the pool with the highest DFN scores, but with an additional constraint of only filtering from samples with web-crawled captions already in English. With translation, English-translated non-English captions now make up the majority of the "highquality" data and thus are more likely to be selected for training.• Filtered raw captions, replaced with translated captions: Given samples from "Filtered raw captions" (i.e. again, based on the cosine similarity score between image and original web text), we keep the images selected and replace the raw captions with the corresponding English translations. • Filtered raw captions ∪ Filtered translated captions: We combine image-text pairs from "Filtered raw captions" and "Filtered translated captions" subsets uncovered above. However, these subsets have about two-thirds of their images in common (see Appendix D). For such images, we only include one copy in the final training set and use English-translated caption by default. For the rest of the images in "Filtered raw captions" that do not appear in "Filtered translated captions", we include them in the training set with the corresponding original captions (which could be non-English). • Filtered raw captions & Filtered translated captions: We combine "Filtered raw captions" and "Filtered translated captions"; the overlapping images between these two subsets would appear twice in the final training set -one copy with the original web caption and one copy with the English-translated caption.Combining high-quality raw and translated captions offers the best performance We find that using both sources of captions, and image data-since top (image, raw text) pairs and top (image, translated text) pairs only have two-thirds of the images in common-leads to the best performance (bolded entries of Table 1 ). This approach surpasses training on only high-quality raw data by 2% on ImageNet, ImageNet shifts and retrieval, and improves GeoDE worst-region performance by 3.7%. We note that this is not simply due to having more unique image-text samples, as filtered subsets of similar sizes but using a single source of captions (e.g., top 40% raw captions totalling 51.2M samples) yields significantly lower performance (Appendix G).Using only translated multilingual captions is still better than using only raw captions Zooming in on "Filtered translated captions" and "Filtered raw captions" baselines, we find that the former outperforms the latter on many standard metrics (ImageNet, ImageNet distribution shifts, imagetext retrieval). This is unexpected in light of prior work showing that ImageNet exhibits strong amerocentric and eurocentric bias [43] , with images from America and Great Britain taking up 53% of the dataset.We perform more ablation studies to disentangle the reasons for the performance gains from using high-quality translated captions, as well as to verify that the gains are robust.The performance gain from using translated captions is not simply due to converting all text data to a common language medium Given image-text pairs from the "Filtered raw captions" subset, we replace the web-crawled captions with the corresponding English translations ("Filtered raw captions, replaced with translated captions"). This intervention on only the captions leads to performance drop across the board. We hypothesize that this is due to noise in the translation process, as (i) many web captions are formed by stringing together short, ungrammatical phrases and thus are "out-of-distribution" for the NLLB translation model, (ii) web captions may contain multiple languages in the same sentence, thereby leading to noisy language detection and translation.Re-filtering data after translation is also necessary due to significant changes in the data ranking Besides noisy artifacts introduced by translation, the process also changes the image-text cosine [13] to select the top 30% examples. We find that using translated captions leads to 1.5 percentage points improvement on average across 38 tasks. We highlight the performance changes on ImageNet distribution shifts (red), retrieval (blue) and fairness-related tasks (dark yellow).similarity score and thus the quality ranking of the data samples in the pool. More specifically, we find that while "Filtered raw captions" is dominated by English samples, (translated) non-English samples make up the majority of "Filtered translated captions" (Figure 2 ). These two filtered subsets only share about two-thirds of the images in common, see Appendix D for more details. Therefore, by changing the caption distribution, we are also inducing changes to the image distribution that the best-performing model would see.The benefits of training with translated multilingual captions are consistent across data filtering networks As alluded to in Section 3, we also explore using cosine similarities output by OpenAI CLIP-ViT-L/14 [35] for data filtering. The full results for this ablation can be found in Appendix E. Similar to the previous observations, we find that using filtered translated captions yields better performance than using filtered raw captions.The performance benefits of using translated data persists with much longer training duration We also experiment with training for 10× more steps (i.e., 1.28B samples seen) as this is more in line with the number of epochs typical vision-language models are often trained on (e.g., OpenAI CLIP models were trained on 400M datapoints for 32 epochs [35] ). When using either the raw caption or the translated caption distribution, setting the filtering threshold to top 30% of the pool works best. Even though the two filtered datasets now yield the same average accuracy, training on highquality translated captions still offers significant advantages when it comes to ImageNet, ImageNet shifts, retrieval and GeoDE. Combining high-quality data from both sources of captions continues to be the best performing approach, giving 4.2% improvement on ImageNet and 2.1 percentage points improvement on average, compared to just training on filtered raw captions. Results for more baselines can be found in Appendix H.After observing improvement across different metrics from using more (translated) multilingual captions, in this section we break down the performance changes for each of the 38 tasks in the DataComp benchmark. The base model for comparison is CLIP trained on top 30% image-text pairs filtered from the raw data pool, and the new improved model is the one trained on top 30% image-text pairs after the same pool has been all translated to English. Both models are trained for 128M steps. Averaged across 38 evaluation tasks, the latter yields a 1.5 percentage points improvement. The biggest gains come from Flickr retrieval, fairness (GeoDE, Dollar Street) and remote sensing (EuroSAT, RESISC45) tasks (Figure 3 ). We show examples of easyto-classify images in our English versus non-English data classification task. Besides the product logo and text in some images that are suggestive of the language distribution, the image content mostly depicts common scenes and objects.In particular, on GeoDE [36] , which consists of images of common objects crowd-sourced from six different regions across the world, we find that using translated multilingual captions makes CLIP perform better on all regions, with the biggest gain coming from Africa images (5.5%) and the second biggest gain coming from the Europe region. This is unexpected given that African languages only make up a small fraction of the training set, and after translation, more European (compared to African) language samples make it to the resulting filtered subset (Appendix D]). Besides, it is worth noting that on GeoDE, our best baseline from Table 1 ("Filtered raw captions & Filtered translated captions") outperforms the current best baseline on DataComp's medium scale ("HypeSampler" [22] ) by 1.5%, measured in terms of worst-region accuracy (Africa).5 Understanding the differences between English and (translated) non-English dataGiven that using more image-text pairs of non-English origins in the training set offers significant benefits on most vision tasks, including those that are English-centric, we seek to further understand the various ways that non-English data complements and improves the diversity of English data, in both image and text space. 
image distribution: 
 As a proxy for capturing image distribution differences, we train simple classifiers-a Support Vector Machine (SVM) on CLIP embeddings and a ResNet-50-to distinguish images with English captions from those with non-English captions. We randomly sample 100K images from each distribution for training and 10K for testing. We only use images from the top 20% of the candidate pool (based on DFN cosine similarity score), to ensure that (i) these are the samples that the best-performing CLIP models are eventually trained on, (ii) images are of sufficient quality, to the extent that they have fitting captions accompanying them.We note that this classification task is non-trivial for a number of reasons:• Many images are duplicated across the web, i.e., after DataComp performs image deduplication it is possible for these images to appear with either English captions or non-English captions in our data pool. • The language detection model is not perfect and web-crawled captions may contain more than one language in the same sentence. • Images with non-English captions contain many sub-distributions of images, some of which may overlap with the distribution of images with English captions (e.g., eurocentric data). There exists a substantial gap between the distribution of English captions and that of non-English captions, even when we apply translation to both, suggesting that they capture different contents. We use MAUVE score [34] to measure the difference between English captions and (translated) non-English captions in the training set. We find that (i) translation indeed introduces some artifacts and changes what "English" texts may look like, (ii) the English text distribution is remarkably different from the non-English one, even after they are converted to the same medium with translation. All scores are averaged over 3 randomly sampled sets of 10K captions.Despite these challenges, our simple classifiers achieve 67% accuracy on the binary classification task, significantly better than random chance performance. In Figure 5 , we show some examples of what the SVM deems easy to classify. Overall this experiment suggests that the distribution of images with non-English captions is sufficiently distinct from that of images with English captions. Therefore, not training on more of the former means we are missing out on a considerable amount of visual information that can only be found in a separate part of the web.In the text space, we leverage MAUVE score [34] to quantify the differences between English captions and non-English captions that have been translated to English. MAUVE was originally designed to measure the gap between machine-and human-generated texts. The metric computes KL divergences in a quantized, low-dimensional space after embedding text samples from each distribution with a language model (by default, GPT-2). The output score ranges between 0 and 1 and the higher it is, the more similar the text distributions are. Similar to our analysis in the image space, we only use caption samples from the top 20% of the candidate pool (based on DFN score).In Table 2 , as a sanity check, we randomly sample two disjoint sets of 10K captions from the same text distribution (e.g., non-English captions having been translated to English with the NLLB translation model, or English captions having been passed through the same model). We find that the two sets indeed exhibit high MAUVE scores (above 0.95). When comparing raw English captions to English captions that have been passed through the NLLB model (i.e., "translated English"), we find that the MAUVE score decreases slightly (0.890), indicating that the translation process introduces some artifacts making English-translated English text look somewhat different from raw English text.When comparing raw English texts and non-English texts, both having been passed through the translation model and thus undergone the same "preprocessing" (i.e., English translation), the resulting MAUVE score is relatively low (0.616). This signals that independent of differences in language, what is discussed in English captions and non-English captions differs in many ways. We should therefore leverage both sources of text information as much as possible for training. 
discussion: 
 Limitations We fix the data filtering method to be based on image-text cosine similarity output by a trained model, and study the impact of selecting training data based on different caption distributions. We show that the advantages of using translated multilingual data are robust to the choice of the filtering network. However, our best-performing baseline is currently not state-of-the-art on the DataComp benchmark [14] . It remains an open question whether the performance benefits of our method persist with other score metrics, e.g. hyperbolic entailment [22] -currently the best method for DataComp's medium scale, or other filtering methods that are used jointly with CLIP score, e.g. T-MARS [27] which also removes text-spotting images with limited visual information.Besides, we acknowledge that translation can introduce artifacts and reduce the richness of expressions in the original languages. Prior work has shown that translated sentences are less effective compared to manually-written sentences as sources of training data for a vision task, e.g. image captioning [28, 23] . Our work mainly leverages translation as a way to convert all image-text pairs to the same medium, and remove confounding impacts of language in data selection and model training.In this work, we bring all web-crawled image-text pairs into a common language medium via English translation, and systematically study the empirical benefits of using non-English data with respect to standard computer vision tasks (that are in English). By including significantly more (translated) multilingual data in the filtered training set, the improved cultural and linguistic diversity in turn leads to substantial gains across all major metrics-ImageNet, distribution shift robustness, retrieval capabilities and average performance across 38 tasks-even if some of these metrics have been shown to overfit to English. We also find that despite being translated to the same language, English and non-English data distributions are still distinct from each other.Future work This work motivates future studies into data curation techniques that directly improves the diversity of data origins. Another interesting direction of exploration is adapting trained CLIP models from this paper for multilingual benchmarks, such as by re-training the text encoder (that has only been trained on English and English-translated captions) with the technique proposed in [4] . We hypothesize that text adaptation alone is sufficient for our models to perform competitively on non-English tasks, owing to the presence of significantly more multilingual and multicultural images in our pre-training dataset.Broader impact While most studies have looked into non-English data with the goal of increasing societal representation and subsequently improving performance on under-served populations or tasks, we observe that non-English data can actually help enhance model capabilities as a whole (including on standard English benchmarks). This suggests that diverse representation in training data, e.g. as measured by cultural and linguistic backgrounds, should be a deliberate design decision in the data curation process, instead of existing only as a byproduct of the preprocessing pipeline or out of societal considerations. We follow the training and evaluation protocols of the DataComp benchmark [14] ; refer to Appendices M and N of this previous work for more details. To summarize, we use ViT-B/32 as the image encoder for CLIP, and fix the hyperparameters used for training: learning rate 5e-4, 500 warmup steps, batch size 4096, AdamW optimizer β 2 = 0.98.Since the compute budget is fixed, for the DataComp setting (128M training steps), each of our baseline takes about 8 hours with 8 A40 GPUs and 40 CPUs. With the same amount of resources, for experiments involving training for longer (1.28B steps), each baseline takes about 80 hours. We report all baselines that we ran in Appendices E, G and H.Here we provide a quantitative assessment of the quality of caption translation offered by the NLLB model [9] . We sample 100K captions from the raw data pool and backtranslate the English-translated caption into the original (detected) language (e.g., Chinese text → English translation → Chinese translation of the English-translated text). To evaluate the translation quality, we compute the cosine similarity between the initial web-scraped text and the backtranslated text using embeddings from the multilingual Sentence-BERT model [? ]. We find that on average the cosine similarity (and thus, translation quality) remains relatively high (0.63). In the table below, we report the top 5 and bottom 5 languages that observe the highest and lowest translation quality as captured by our metric, computed over at least 30 text samples per language. Table 3 : Top 5 and bottom 5 languages where web-scraped captions observe the highest and lowest translation quality by the No Language Left Behind model [9] , out of all the languages detected in our raw data pool. Translation quality is measured by how much the semantic meaning is preserved after the caption is translated into English and subsequently backtranslated into the original language.Text In this section, we provide some statistics of the differences in image-text pairs selected for "Filtered raw captions" and "Filtered translated captions", when both caption distributions are filtered to a similar extent with the public DFN from [13] . We find that at either 20% or 30% selectivity threshold, both filtered subsets have about two-thirds of their images in common. In addition, filtering the initial pool using (image, translated caption) cosine similarity score always leads to translated multilingual captions taking up the majority of the resulting training set. This is not the case when filtering with (image, raw caption) cosine similarity score. Comparing image-text pairs selected in top 20% translated captions to those selected in top 20% raw captions, we show below the languages that observe the biggest change in their representation in the resulting training set: Table 5 shows results of our experiments with data filtering using OpenAI CLIP-ViT-L/14 [35] .Besides the baselines described in Section 4, the table also contains "Top 30% raw captions ∪ top 30% translated captions, using translated caption for all", where we take all the images uncovered from "Top 30% raw captions" and "Top 30% translated captions", deduplicate them, and use the corresponding translated captions for all these images.We find that with OpenAI CLIP as the filtering network, some of our observations from Section 4 continue to hold true: (i) using translated multilingual captions is better than using raw captions, and (ii) the performance gain from training with translated captions requires re-filtering the entire data pool after translation (as seen from comparing the first two baselines of the table ). As observed in Section 4.1, training on filtered translated captions outperforms training on filtered raw captions across all major metrics. This could be attributed to both changes in captions (from original web-crawled texts to English-translated texts) as well as changes in images (since "Filtered raw captions" and "Filtered translated captions" only share some of the images in common, see Appendix D). Here we attempt to disentangle the contribution to performance gain from these two changes.Given the images selected by filtering based on (image, translation caption) cosine similarity (i.e., "Top 20% translated captions", "Top 30% translated captions"), we generate synthetic captions for each image using BLIP2 model [24] and the generation hyperparameters from [31] . Training on the new (image, synthetic caption) pairs leads to lower performance overall compared to training on the original (image, translated caption) pairs (Table 6 ). This suggests that having access to more diverse (non-English) images in the training set is not sufficient to boost performance; the diversity coming from translated multilingual captions is also necessary for obtaining the accuracy gains.We acknowledge that since BLIP2 was pre-trained on relatively few multilingual samples, it is possible that the captioning model finds it difficult to caption non-English images. This ablation study experiment is thus mostly exploratory, and more experiments are needed to assess the performance benefits coming from seeing more non-English images, versus seeing more diverse linguistic concepts in (translated) non-English captions.Table 7 : Here we report all the baselines we experiment with using the public DFN from [13] for filtering; all models are trained for 128M steps as set by the DataComp benchmark. For each caption distribution (i.e., raw/ translated/ English-only), only the filtering threshold that yields the best average performance across 38 tasks is shown in Table 1 . Here we show the results of all the baselines that we train for 1.28B steps (i.e., 10× the number of steps set by DataComp). In Table 8 , we find that when using either raw web-crawled captions or English-translated captions, filtering for top 30% of the pool does best, and translated multilingual captions continue to yield better performance on standard metrics compared to raw captions. We also provide a breakdown of performance differences between these two baselines across 38 tasks from DataComp in Figure 8 .Table 8 : When the training duration is increased by 10× compared to the DataComp setting, training on translated multilingual captions continues to outperform training on raw captions across a range of metrics; using both sources of captions continues to yield the best performance. We show performance of all the baselines that are trained for 1.28B steps. Even though using filtered raw captions and using filtered translated captions yield similar average performance (0.414 percentage points), the latter still surpasses the former on ImageNet, ImageNet distribution shifts, retrieval and GeoDE (worst-region accuracy). We also note that these performance gaps widen with training duration (see We compare performance on each task of the DataComp benchmark between training with raw captions and training with translated captions, when both are trained for 1.28B steps. Both datasets have also been filtered with image-text cosine similarities output by the public DFN [13] to select the top 30% examples. We find that when we increase training duration to be 10× longer than DataComp's setting, using translated multilingual captions and using raw captions yield similar average performance across 38 tasks. However, the former still outperforms the latter on most of the ImageNet distribution shifts (red), retrieval (blue) and fairness-related tasks (dark yellow).In addition to the analysis in 4.3, we provide further breakdown of performance changes at the income group and class levels, on Dollar Street (Figure 9 ) and ImageNet (Figure 10 ) respectively, when we swap the training distribution from top 30% raw captions to top 30% translated captions. [39] is another fairnessrelated task that involves classifying images of everyday items collected from households around the world with different socioeconomic backgrounds. We break down the performance on this dataset by income groups and find that training on top-quality translated captions improves the classification accuracy across all groups, compared to training on top-quality raw captions. NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We follow the hyperparameter setup of the DataComp benchmark [14] and describe the hyperparameters in Section 3 and Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to compute constraints and the many ablation experiments that are involved, we did not run multiple random seeds for each baseline. The only experiment with multiple runs is for computing MAUVE score. We plan to run more random seeds for select baselines to accommodate the review process.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss broader impacts of our work in the Discussion section.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The data preprocessing and data filtering pipelines are described in detail in Section 3. We plan to release all 128M English-translated captions on HuggingFace after the review period.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our experiments do not involve human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our experiments do not involve human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper presents a conceptually simple but extensive studies of how translated multilingual (non-English) image-text pairs benefit English vision tasks. They also provide an intriguing analysis of the differences in image and text distributions between English and non-English image-text pairs, supporting the claim that multilingual data enhance diversity.

Overall, all reviewers were favorable to the unique focus and findings of the paper, and were satisfied with the authors' rebuttal. As a result, their final scores are unanimously toward acceptance, with strong supports from three reviewers. The AC would like to support their judgement.

Reviews: review: paper_summary: paper_summary: In this paper, the authors conduct a thorough exploration of how multilingual image-text pairs benefit English vision tasks. They first present how to effectively utilize translated data to improve performance on standard vision tasks and derive valuable conclusions through detailed ablation studies. Second, they illustrate the differences in image and text distributions between English and non-English image-text pairs. These findings highlight the potential of leveraging multilingual datasets to enhance the robustness and accuracy of vision models.
strength_weakness: strength_weakness: 1. No quantitative assessment for the quality of translation.
2. Only evaluation on representation task. Does such data strategy also work for the training of Multimodal LLM?
questions: questions: No
limitations: limitations: Yes
paper_summary: paper_summary: The authors investigate whether *multilingual* vision-language data improves the *English-only* performance on a model in vision-language tasks.

They translate captions from DataComp from English to other languages and train a CLIP model on these multilingual captions.

They find that this action boosts performance on both English-only tasks like imagenet matching and geographically diverse tasks like GeoDE.
strength_weakness: strength_weakness: Unclear how the proposed automated translation pipeline ties in to the original motivation (culturally specific items not being captured in English data)

Could benefit from a bit deeper description of what the GeoDE task is for unfamiliar readers to improve reach.
questions: questions: Figure 1: where are these examples from? Are they actually in the datacomp datasset? Doesn't translating datacomp captions into other languages keep the learned representations in the "English concept space" of sorts?
limitations: limitations: Yes, limitations are sufficiently covered.
paper_summary: paper_summary: This paper points out an important issue in current training of CLIP models---the push for including more english-centric / english-only data  in the pretraining dataset. The paper points out that this is mainly driven by the downstream evaluation test-beds primarily being english-focused and hence the need to include multi-lingual data has not been found so far. The paper conducts a thorough study of using multilingual data into the pretraining datasets of CLIP models by using caption translation models. The paper evaluates and shows improvements on several downstream datasets from the 38-Datacomp benchmark evaluation suite, and provides analysis on the key differences between multilingual and english-only pretraining datasets.
strength_weakness: strength_weakness: - The paper could benefit from some additional analysis. I note down some analayses that I think might strengthen the paper below.

- For a particular concept is it possible to showcase the diversity afforded by looking at only English data and multilingual data? For instance, take the concept of “stove”. You could perhaps manually take 10-30K images of a stove (assuming that many exist), by filtering the texts for this particular concept, and do the same for both English only and multilingual data. Then, quantify the diversity of images, by perhaps taking a strong Dino-v2 pretrained encoder and then clustering them, and perhaps looking at some intra- and inter-cluster distance metrics. I think this would nicely validate the qualitative visualisation in figure 1 with some empirical evidence that multilingual data indeed boosts the diversity of visual concepts in pretraining datasets. It would be great to see this analysis on perhaps one or two concepts.
Why is this an important analysis? My reasoning would be that while it is totally plausible that multilingual concepts add diversity, it is unclear to me how much of this is actually true? Since it is plausible that English webpages also contain such diverse visual concepts, I think it would be nice to see if there are “new visual concepts” being included in the pretraining corpus, or is it mostly that these “culturally-diverse” visual concepts were present in the english-only pretraining datasets, but we just boost up their frequency in the pretraining dataset? I think either of the two cases is a valid justification for the claim that multilingual data increases diversity, but it would be nice to have a precise answer for what exactly out of the two is the case.

- The main claims of the paper that multilingual data improves VL representations is sufficiently backed up, but benchmarking on culturally diverse / multilingual-sourced data is still limited. The paper only considers GeoDE and DollarStreet for evaluation. I would recommend to also benchmark and report the performance on some more culturally and geospatially diverse datasets like [1,2,3,4]. Some of these datasets might need to be ported into a classification / retrieval format, but in general I think this is an important experiment to do to further validate the important of multilingual-sourced pretraining data.

- By filtering on top of the translated image-text pairs, it seems likely that the training data diversity increases, and potentially train-test similarity [5] also increases. This could be an added confounder in the takeaways of the paper. Could the authors comment/discuss this a bit more in the paper?

- The analysis in the paper would further be improved by checking how the distribution of CLIPScores look before and after the translation. Potentially there would be a shift to the right in the similarity distribution, but it would be a good analysis to include for better intuitions on what might be good filtering thresholds if such a model were to be used in the future for vector arithmetic / data filtering itself.

[1] Kalluri et al, GeoNet: Benchmarking Unsupervised Adaptation Across Geographies
[2] Weyand et al, Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
[3] Yin et al, Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning
[4] Romero et al, CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
[5] Mayilvahanan et al, Does CLIP’s generalization performance mainly stem from high train-test similarity?
questions: questions: I have a few additional questions/comments which I include below:

- How do you estimate the english part of the web-crawl? You mention it is in 1/3rd in the introduction.

- Is there an explanation for why filtered translated captions might improve english-centric performance over the filtered raw captions? This seems counter-intuitive given as you say in the paper, that ImageNet is mostly western-centric. This also seems to contradicts the results of the No-Filter paper [1].

- Could the model trained on this filtered raw subset itself be used as a DFN, and potentially further improve the performance when training on the raw multilingual pool? The idea being that you might have to do less tuning on the filtering threshold since the model is more robust to noise from the translated captions.

- In fig 3, are the results from the "filtered raw captions" and "filtered translated captions" methods from tab 1? Doesn’t seem so since the performance improvements on the 38 tasks mentioned in the caption of fig 3 is 1.5% whereas the performance improvements from tab 1 is only 0.9%.

- Why does performance on Food101 go down? Is it because all the food classes included in Food101 are primarily English-centric? I am certain that is not the case. I would be very interested in a more detailed analysis of why the performance on Food101 goes down when we include more multilingual data since that is one aspect that I would have thought improves quite a lot. 

[1] Pouget et al, No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
limitations: limitations: The authors are explicit about their limitations. Any further limitations I think are mentioned in the weaknesses section above.
paper_summary: paper_summary: The paper conducts a systematic study to explore the performance benefits of using translated non-English data for English vision tasks. By translating multilingual image-text pairs from a raw web crawl to English and re-filtering them, the authors show that continual pre-training on this data increases the performance of the English model on a diverse set of tasks with a single model.
strength_weakness: strength_weakness: - The improvement from using translated data diminished as the training converged, which is not surprising. Similarly, using more training data to improve the evaluation results on downstream tasks is also expected. Hence, the novelty of insights is limited in this paper.
- The paper only evaluated on one LLM, which more evaluations are required to validate the generality of the claim of this paper.
questions: questions: Possible experiments to make the paper better:
- Add more multicultural evaluation datasets
- Add more evaluations to show the benefits of using diverse training data could benefits multiple models sizes and types
- Perform similar training and evaluation in other languages, to show all languages would benefits from diverse training images and captions
limitations: limitations: While this research may be interesting for those working primarily on monolingual research (i.e., English), its framing should be more sensitive to the many efforts related to multilingual and multicultural NLP. Personally, I think the benefits of using diverse images and captions can be achieved by collecting diverse images and paraphrasing captions or through better sampling of the training data, rather than just using multilingual or multicultural datasets and translating them. As a community, we already lack people working on multicultural and multilingual models, yet this work uses data and research from other cultures to improve **English-only** models (although with good intentions). Perhaps there could be some discussions on this.

Comments: title: Author response
Author Response
Follow up
Re
Rebuttal follow-up
Response to author rebuttal
comment: Thank you for acknowledging our rebuttal and for adjusting the rating! We are glad to hear that our response has provided clarification to your earlier questions.
Thank you for acknowledging our rebuttal. As mentioned in our comment, we are happy to engage in further discussion at any point in the future as well.
**Motivation of translating from English**

This explanation is reasonable. I'd like to see this clarified in the intro of the CR.

**Other answers**

Reasonable, thank you for clarifying. I forgot that datacomp pre-filtering does indeed contain multilingual data.

**Translating datacomp captions**

After the clarification, I understand. You are indeed augmenting the concept space by translating into English here.

Thank you for your responses, I will update my soundness score.
Thank you for acknowledging our additional results and for adjusting the rating! We will make sure to include the new analyses as well as a discussion of the No Filter paper results in the next version of our work.
Thank you very much for the additional clarifications. I have revised the score to reflect my current assessment of the paper.
We thank reviewers again for their time. As the discussion period is drawing to a close, we would like to ensure we have addressed all of the concerns. Let us know if you you have any further questions/ thoughts after our rebuttal, we would be happy to engage in further discussion.

-- Paper authors
I thank the authors for their thorough analysis and further experiments. 

- The clustering experiment now adds a very strong demonstration that the non-English data improves the diversity of visual samples even for concepts that are already present in the English pool. This is a great result!

- GLD experiments look great and provide additional validation for the significance of including non-english data.

- The points discussing the differences between your and the No-Filter paper are quite valid and important. I would encourage the authors to add this as an explicit discussion section either in the related works or the appendix.

After reading all the other reviewer comments and the author's responses to them, I am inclined to strong accept this paper as it makes a significant contribution to the data-centric VL community and provides relevant, practical insights. I am increasing my score to 8.