Meta CLIP 2: A Worldwide Scaling Recipe
Authors: Yung-Sung Chuang
Yang Li
Dong Wang
Ching-Feng Yeh
Kehan Lyu
Ramya Raghavendra
James R. Glass
LIFEI HUANG
Jason E Weston
Luke Zettlemoyer
Xinlei Chen
Zhuang Liu
Saining Xie
Wen-tau Yih
Shang-Wen Li
Hu Xu
Venue: NeurIPS 2025 spotlight ()
Keywords: CLIP
curation
multimodal
web
multilingual
data curation
pretraining
visual-language
contrastive learning
long-tail distribution
scaling
Primary area: applications
TL;DR: We generalize CLIP training to worldwide web-scale, with +0.8% better than English only counterpart on zero-shot ImageNet classification (no compromise), SoTA on zero-shot multilingual: 57.4% on CVQA and 50.2% on Babel-ImageNet.

Abstract: Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary to address the above challenges and present a recipe enabling mutual benefits from English and non-English world data.
In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-of-the-art without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with 64.3% on image-to-text retrieval. Code and model are available at https://github.com/facebookresearch/MetaCLIP.

Meta CLIP 2: A Worldwide Scaling Recipe
Yung-Sung Chuang! 122 Yang Li', Dong Wang', Ching-Feng Yeh', Kehan Lyu',
Ramya Raghavendra', James Glass?, Lifei Huang', Jason Weston',
Luke Zettlemoyer',
Xinlei Chen' Zhuang Liu?,
Saining Xie*, Wen-tau Yih', Shang-Went Li Hu Xu
'FAIR, Meta
MIT
Princeton University
*New York University
Abstract
Contrastive Language- Image Pretraining (CLIP) is popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further learning from the worldwide web data is still challenging: (1) curation method is available to handle data points from non- non- -nnglnh world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CL.IP from scratch worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary address the above challenges and present a recipe enabling mutual benefits from English and non-English world data. In zero-shot ImageNet classification, Meta CLIP VIT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-oftthe-a without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-Imager with 50.2% and XM3600 with 64.3% on image- to-text retrieval. Code and model are //gi h/tpps://ge thub. commfffcebookrreeercceeeeeaaaaa
available at https:t:///.
1 Introduction
Contrastive Language-Image Pre-training (CLIP) [1 has become an essential building block of modern vision and multimodal models, from zero-shot image classification and retrieval serving as vision encoders in multimodal large language models (MLLMs) 2 3 5 CLIP and its majority variants 66 7 adopt an English-only setting, and Meta CLIP introduces a scalable data curation algorithm to meticulously extract billion-scale English dataset that exhausts long-tailed concepts in Common Crawl. The algorithm transforms the distribution of the raw Internet into controllable and balanced training distribution defined by metadata (e.g., visual concepts composed by human experts) and training distribution known as one key contributor to performance. In contrast, popular CLIP reproductions outsource such key contributor to external resources, e.g., OpenCLIP [ trained on LAION g and DFN I0 rely on pretrained CLIP models for black-box filtering to keep only high-confidence data. Such approaches resemble distillation of an existing CLIP teacher model and produce intractable distributions owned by an outsourcing party.
1 Work done during an internship at Meta.
Work done when working Meta. 6 Project leads.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Tra ined on English data
Trained on non English data
Trained worldwide (English non non-English) data
Break curse
English helps non-English
51
X Curse of muitilinguality
50
Zero-shtt Accuracy
80-
69
Average Accuracy
58
68
78-
Average
47
10
Scaling
Pres SoTA
Prev. SelA
VIT-L/14
VIT+
VIT-H/14
VIT-H/44
ImageNet (English)
Babel-ImageNet (Muitilingual)
CVQA (Multilingual)
Figure 1: (Left) CLIP training suffers from the curse of multilinguaaiityy that the English performance of CLIP model trained on worldwide (i.e., English + non-English), billion-scale data is worse than its English-only counterpart, even when applying our recipe on ViT-L/14; scaling to ViT-H/14 enables non-English data helps English-only CLIP. (Right) English data also helps non-English CLIP.
Although being the most widely used "foundation" models, most CLIP variants, including the scalable Meta CLIP, rely on English-only curation and thus discard the other, e.g., 50.9% [II] of non-English, worldwide web data. To extend CLIP training and data to the worldwide web for the next level of scaling, we inevitably have handle these non-English image-text pairs-a barrier refer to the worldwide scaling challenges, which are issues not yet being solved after years of attempts to train CLIP on multilingual data:
Challenge #1: Lack of' a fundamental data curation method handle non-English data at scale. Existing attempts either conduct no curation the raw, non-English image-text pair data all (e.g., [4]), ), or rely on proprietary and private data sources (e.g., WebLI [13 that drives mSigLIP and SigLIP [16 is built from Google Image Search 18)..
distilling from English CLIP 122 or machine translation [13
Challenge #2: Worse English performance than English-only CLIP. This is also known curse of multilinguality text-only large language models (LLMss). For instance, mSigI.IP 1.5% worse than its English-onb counterpart, SigLIP, on ImageNet [16), while SigLIP 2 [17] prioritizes English performance at the cost of even worse multilingual results than mSigLIP. Hence, disparate models have to be used to optimize English and non- -on-English performance the same time.
This work. We present Meta CLIP 2, the first ever recipe developing CLIP with training from scratch on native worldwide image-text pairs, without relying on outsourced resources, such as any private data, machine translation, or distillation. We empirically show that the curse of multilinguality CLIP is the consequence insufficient scaling due the lack of proper recipe for worldwide data curation and model training. When metadata, data curation, model capacity. and training are carefully designed and scaled jointly, we show that only the performance trade-offs between English and non- non-Enge Ennglislssh data disappear, but the two become mutually beneficial. Achieving such worldwide scaling is highly desirable, especially when English Internet data is exhausted soon [19
Our Meta CLIP recipe is built on top of English Meta CLIP, where overlapping with OpenAI CLIP's vanilla architecture deliberately maximized. The overlap makes findings generalizable to CLIP and its variants, compared to system works (cf.. [16] 17 (00) aiming at state-of-the-arr (SoTA) performance with combination all available techniques. Such combination involves confounding factors or comparison on outsourced resources instead of CLIP itself. The Meta CLIP 2 recipe introduces three principled innovations for scaling worldwide. 1) Metadata. We scale the English Meta CLIP metadata to 300+ languages on Wikipedia and multilingual WordNet. 2) Curation algorithm. We build per-language substring matching and balancing curate concept distribution for non- non-Enge Engliih data similar to the English counterpart. 3) Training framework. We design the first worldwide CLIP training framework, including an increase of seen image-text pairs during training proportional to the increased data size from the added non-English data examples, and study on minimal viable model capacity to learn from worldwide scale data. shown Fig. [T] although ViT-L/14 (the largest model size used by OpenAl) still suffers the curse of multilinguality. VIT-H/14

---

breaks the curse. English accuracy rises from 80.5% to 81.3% on ImageNet and surprssingyy new SoTA is set with minimal CLIP architecture changes for multilingual image-to-ter toxtext retrieval (XM3600 64.3%, Bmbel-ImageNet 50.2%, and CVQA 57 3...
Together, Meta CLIP 2 enables the following desirable results by nature. Mutual benefits from the English and non-English worlds. Non-English data now can better support an English- only model and vice versa, which is critical in the era when English data is depleting. 2) Full multilingual support. Meta CLIP 2 never drops image-text pairs simply by languages and yields models outperforming all the previous multilingual systems, such as mS [16] and SigLIP [T7]. 3) Native-language supervision. Models learn directly from alt-texts written by native speakers rather than synthetic machine translations [21 14]. 4) Cultural diversity. Meta CLIP retains the entire global distribution of images and thus inherits the comprehensive cultural and socioeconomic coverage advocated by [1]] Such coverage improves lco-localization and region-specifer recognition. 5) No-filter philosophy. With the curation algorithm designed towards worldwide data, Meta CLIP? 2 removes the last filter (i.e., whether the alt-text is in English) in pipeline, achieving better diversity and minimizing biases introduced by filters [21]. 6) Broader impacts foundation data. This work provides a foundational data algorithm designed for worldwide scale, and benefits not only CLIP, but also efforts using CLIP data such as MLLM 22 22]. SSL (Web-DINO (3) and image
generation (DALL-E 24
and diffusion models 25
2 Related Work
2.1 Evolution of CLIP and its Data Processing
CLIP (1] and its variants 26 6 [6 learn versatile image and text representations that are generally
useful for downstream tasks 2 27 4]. Such multimodal contrastive learning and transformer archi- tectures become standard components in vision and multimodal research. Data is key contributor to CLIP's performance [28 77. Two major processing approaches for CLIP data emerge: curation from scratch, and distillation from external resources. One key difference is that the former yields more controllable distribution and the latter has intractable distribution owned by an outsourcing party.
Curation from scratch. OpenAl CLIP []] curates a training dataset of 400M image-text pairs from scratch and publicizes high-level curation guidance. Meta CLIP [7] makes OpenAl's guidance formal curation algorithm and scales the curation to 2.5B pairs. The algorithm is model-free, no blackbox filtering, and fully transparent to enable training entirely from scratch on public data source, where the data distribution is curated to align with metadata composed by human experts (e.g., WordNet and Wikipedia).
Distillation from external resources. bistillation-based methods usually have good performance and save compute by learning from teacher model's knowledge [30]. However, in the context of CLIP training the teacher is usually external blackbox system, which introduces intractable bias. For example, LAION-400M/5B [ 31) (used by OpenCLIP [6] relies on OpenAl CLIP-filter and
DFN [10] using filter model trained on high-quality private data [32 Recently, SigLIP [16] and
SigLIP 2 17 learn from data source WebLI [15) which is derived from Google Image Search [18]
2.2 Vision Encoding
CLIP-style models are widely used as vision encoders in MLLM, where language supervision in CLIP training helps learn compact and semantic-rich visual representations. In contrast, traditional visual representation learning is based on self-supervised learning (SSL) methods like SimCLR [3] DINOv2 34 and purely relies on the full visual signal without language bias. There are variants that take advantage of both. SLIP 35 combines language and SSL. supervision: LiT (36) trains a vision encoder first and conducts language alignment later; Perception Encoder (2) shows early layers of CLIP representation yields vision-driven features with less semantic alignment. Recently, Web- DINO 23 shows SSL has better scalability on Meta CLIP curated large-scale data. In summary, CLIP focuses on human-aligned representations optimized for compact models and efficient downstream uses; SSL models aim to preserve all visual information as a general pretraining approach. We envision more synergy from the two research lines due to complementarity.
'Here, "curation" refers to select and align training data distribution with human from raw data source, excluding data filtering that is also referred to curation in many works like DataComp 28 29
and DEN [10]
3

---

2.3 Multilingual CLIP Models
Due the lack of open source curation for public worldwide data, initial attempts multilingual CLIP models are mainly distillation approaches. M-CLIP [13] and mCLIP [12] simply leverage existing English-only CLIP as the vision encoder and trains multilingual text encoder with low-quality multilingual pairs. To incorporate non-English data, subsequent works 37
14 21] leverage machine translation techniques, either translating -an-English captions into English or vice versa. These distillation-ban basson models carry existing English CLIP bias or translation bias on nonhuman-captioned data. mSigL.P [66] substantially advanced multilingual performance by leveraging multilingual data from WebLI [SS which is an undisclosed dataset built with private data processing pipeline instead of publicly available worldwide data curation algorithm.
However, mSigLIP and other multilingual CLIP models suffer from the curse of multilinguality. e.g.. mSigLIP is 1.5% worse ImageNet accuracy than its English-only counterpart SigLIP. Recently, SigLIP 2 adopts notably English-centie centric design of having 90% of its data in English, which is much higher than mSigLIP. Mixed results are also observed [38] on English benchmarks when scaling SigLIP from WebLl's 10B to 100B raw data, suggesting the challenges of scaling W beyond.
3 The Meta CLIP 2 Recipe
Our recipe of scaling CLIP to native worldwide data and training comprises three steps shown in Fig. 2 (1) constructing worldwide metadata, (2) implementing worldwide curation algorithm, and (3) building training framework for worldwide model. For generalizable recipe and findings, Meta CLIP 2 is designed to maximize overlapping with OpenAl CLIP and Meta CLIP, and only adopts necessary changes to learn from worldwide data.
OpenAI CLIP
Wilpoeda O
English Metadata
English Curation
English Training
0 MetaCLIP
English
Non-English
Scaling
Scaling
All
Data Pool
Wilspedia
Worldwide Metadata
O MetaCLIP2
Worldwide Curation
Worldwide ide Training
Murlingus Whendhow
329 langs. (Sec. 3.2)
language fiiter (Sec. 3.3)
2.3x seen pairs (Sec. 3.4)
Figure 2: Overview of Meta CLIP recipe: scaling CLIP data and training to worldwide scope.
3.1 Revisit of Meta CLIP Algorithm
We revisit the original Meta CLIP algorithm to illustrate how English-based CLIP data is curated with metadata constructed from human knowledge. The algorithm first constructs metadata M, list of high-quality visual concepts, from corpora written by human experts. .M contains 500k entries, a combination and deduplication of entities from four high-quality sources: 1) all English WordNet Synsets, 2) Wikipedia English unigrams, and 3) bigrams, and 4) Wikipedia page titles. Then, the algorithm performs substring matching on each alt-text (from a given image-text pair in the data pool D) using metadata M obtain list matched_ aeccchdd_etrr___eeeeeee entry_ids. Global counting is conducted to calculate the number of matches over DD for each entry in M as entry_count Finally, the algorithm applies balancing to transform the raw image-text pair distribution into a distribution that is balanced for head and tail concepts and ready for training, by associating each pair with sampling probability. Specifically, the count per entry is first converted into a probability of sampling each entry, entry-prob, where tail entries (defined as entry_count < t) have probability set to as sampling probabilities. Each pair is then
and all the other head entries have l/entry_comme sampled based on probabilities of matched entries in its alt-text. Here, / is a threshold to decide head vs. tail entries and set to 20k in OpenAl CLIP; Meta CLIP raised i to 170k for scaling to billion English pairs.

---

3.2 Worldwide Metadata
We address the first challenge for worldwide scaling by constructing the missing metadata to cover the non-English world. We maintain independent metadata per language since such design is intuitive (c.g., the same word "mit" has different meaning in English and Germany), has better performance (see ablation in Sec. 4.2.2). and is flexible for adding and curating a new set of languages in future.
Our metadata is from the same four sources as OpenAl CLIP and Meta CLIP, but beyond English. Key changes are highlighted as follows. 1) Multilingual WordNet: include all synsets from 31 languages. 2) Wikipedia Unigrams and 3) Bigrams: we process unigram and bigram from Wikipedia dumps dated on May 2024, which include corpora in 329 languages. We clean the corpora into plain text with WikiExxtactor [39]. For most languages, we use space and punctuation to tokenize text into words, and then count unigrams and bigrams (with PMI scoring described in Appendix A.2). For languages without space separation (c.g., some Asian languages), we use open-source tokenizers (see Table in Appendix A.I)) developed by local communities properly split text into words and meanwhile maintain the semantic integrity. 4) Wikipedia Titles: we use page titles from 40 random dates of Wikipedia snapshots and rank these titles by click-through traffic for each language.
3.3 Curation Algorithm
Next, we scale curation to worldwide data tanguage-by-languee The curation algorithm is detailed below and summarized in pseudo-code as Algorithm | First, we conduct language identification (LID) 40 to classify the language of the alt-text from an image-text pair, and choose language- specific metadata to match concepts. The sets of languages covered by LID and metadata sources (e.g.. Wikipedia) are usually different, so we first establish a mapping between one language in LID to unique set of languages in metadata entries. The languages in the metadata mapped to the same language in LID are merged into one group. This ends with a dictionary representation of metadata, M, where the keys are each language in LID and the values are the combined metadata of each group of languages. We also include a key "other" for metadata of languages that cannot be associated with any language in LID. Each alt-text (text) in D is applied with LID for predicting its language (text. (text.lang lang). After that, as in the Meta CLIP algorithm summarized in Sec. Sec.B.I] we run substring matching with metadata corresponding to predicted languages: mat chec entry- _ids substr_ substr_match ca (text, teteex.langg). ) and aggregate global count, the number of matches of each entry, in entry_counts.
With counts calculated, we balance occurrence of concepts across pairs. In data curation for English CLIP described above, threshold t is designed to limit the matches per metadata entry, where entries with matches fewer than 1 are defined as tail entries (or concepts) and otherwise head. Image-text pairs from head concepts are downsampled by sampling probability derived from / to balance training data distribution. Thus, depends on the size of raw data pool (e.g., a larger pool has higher counts for the same entry). OpenAI CLIP sets to 20k for 400M pairs; Meta CLIP [7] tunes to 170k for scaling the training dataset to 2.5B and keeping the same ratio, 6% of matches from tail concepts, that OpenAl CLIP leverages to obtain the 400M pairs. For worldwide data, the data size and the counts of matches differ greatly across languages, so t should be language-dependent. Applying a single threshold t to all languages yields suboptimal performance, e.g., a larger for a language with fewer pairs may yield too many pairs of head concepts and dilutes tail concepts in the curated data (see Sec. 4.2.2).
To derive for each language, we leverage the invariance assumption adopted Meta CLIP algorithm design, the percentage of tail matches (i.e., 6%)., and apply it across languages. With this assumption, we determine t in two steps. (1) From fen to p. we calculate the global tail proportion for all languages, based matches of English tail entries decided by f-n- (2) From p to fhang: for each non-English language, we reversely find the language-speccffc threshold tiang based on the calculated p to ensure the same tail proportion across all languages. Detailed implementation of these two steps is shown as the t_to_p() and p_to_ _tto_(( () functions in Algorithm| T With flang entry_counts is converted to entry-prole similarly as in Meta CLIP but for each language.
Putting everything together, Algorithm | ttakes raw image-text pairs D, metadata M, and an arbitrary threshold for English len input, and outputs a curated dataset of balanced and diverse training pairs, D*, with three stages. Stage 1 performs language-specifie substring matching for each alt-text, text, based on LID results and corresponding metadata, and obtains match counts, entry_ counts, for
5

---

each language and entry. Stage 2 computes thresholds Flan from fen- Stage 3 samples image-texi pairs based on matched entries in text with probabilities entry-probs. Pairs matched with tail entries are always selected (i.e., probability 1.0); pairs with head entries have sampling probabilities Mang / eeetrry_coo _ounts [lang]. Sampled pairs compose DD (see efficient implementation details in Appendix A.3)
Algorithm 1: Pscudo-code Meta CLIP Curation Algorithm in Python/NumPy.
Input:
p(ist) raw (image, text) pairs: each text is assigned with language ttext.lang" by LID; M(dict) worldwide metadata: key->language code; value (1ist) >omttdatt for that language; t_en(int) English threshold on counta of head/tail entry cutoff: OpenAI CLIP->20K, Meta CLIP ->170k;
Output:
D_atar D_star(list): curated image-text pairs;
# helper functions to compute t for each language.
def t_ t_to_p(t, entry_count): return entry_count [entry_count tl tt](mmm) entry_count.
ppt entry_ entry_count):
sorted_count count (ertry count)
cumsum_count np. cameuussortd_comm cumsum_prob cumeum_count count / sorted_count.
return sorted corted_ccont[iie count (np.abas (cumsum_pree prob p)) ar]
Stage 1: sub-string matching, entry_counte [lang: np. ze.zero(len (p.xerr((ee((((ang)) for lang in M) for image, text in
call substr_mates which returns matched entry ids.
mextheatched, entrr_iie substr_match(ter. mer.mmattchtttee (etext.lang]] entry_counts [text. [text.matched, aatcced__nttry.te
# Stage 2:
compute t for each langauge.
t_to_p(t_enn entry_counts (conts[""en]]) t ( for lang in entzy_counts:
t +[lang] = pp, entry_counts [lang])
Stage 3: balancing via indepenent sampling per language.
entry-probs {
lang in entzy_counte:
entry_counta [lang] [entry_comm county [lang] [1ang]] [1ang]
entry_probs [lang] t[lang] entry_counts [lang]
D_star - []
for image, text in D:
for entry_id id text htt.mmaaccee_eeryyy
random. sandom.random(( randon() < entry-probb [text.lang] [entry_id]:
D_star, append appeand(inage,, (image, text))
3.4 Training Framework
Adopting data prepared with worldwide curation current CLIP training framework addresses first challenge, but curse of multilinguality still exists as shown in Fig. [] Thus, we further design the worldwide CLIP training framework. To make our framework and findings generalizable to CLIP and its variants, framework follows OpenAI/Meta CLIP's training setting and model architecture with three additions: (I) multilingual text tokenizer, (2) scaling seen training pairs, and (3) study of minimal viable model capacity. The first is required to support worldwide languages and discussed in Sec. 4.2.2 for various choices; details of the latter two are described below.
Scaling seen pairs. Expanding from an English-only dataset and distribution to worldwide naturally increases the number of available image-text pairs. Training CL.PP for worldwide distribution with the same number of seen pairs as English CLIP downsamples English training pairs and harms English

---

performance. Hence, we scale seen pairs proportionally to the growth of data size from non- non-Engliish pairs, to ensure the amount English seen pairs unchanged during the worldwide CLIP training. This is achieved by increasing the global training batch size, which encourages cross-lingual learning, and meanwhile keeping the other training hyperparameters unchanged. We choose a 2.3x scaling of global batch to reflect that English pairs constitute 44% of our training data. We ablate other choices of global batch size in Sec. 4.2.1
Minimal viable model capacity. Lastly, we study the minimal model expressivity to enable learning on extra seen pairs and break the curse of multilinguality. As in Fig. T we find that even a ViT-L/14 (largest model provided by OpenAl) suffers from the curse due deficient capacity, and ViT-H/14 is the inflection point to break the curse (strong performance improvement in both English and non-English tasks).
4 Experiment
4.1 Dataset and Training Setup
Following Meta CLIP pipeline, we collect 5B image-text pairs sourced from the Internet that are publicly available. After LID, there are about 44% of alt-texts are in English, which are par with the scale of English-only data from Meta CLIP 77. For generalizable recipe and findings, we base training setup OpenAI CLIP's ViT-L/14 and Meta CLIP ViT-H/14, except changes necessary for enabling worldwide capability, as described in Sec. 3.4 and ablated in later subsections. The full
details can be found in Table and Appendix B
4.2 Evaluation
We first present the main ablations of Meta CLIP 2 on wide range of English and multilingual zero- shot transfer benchmarks, along with other multilingual CLIP baselines for comparison (Sec. 4.2. then we conduct a comprehensive ablation study on the variants of metadata, curation and tokenizer (Sec. 4.2.2). Lastly, evaluate the embedding quality of Meta CLIP 2 downstream tasks for culture diversity (Sec. 4.3) and building MLLM (Sec. 4.4).. Additionally, we conduct analysis on embedding alignment and uniformity [1 (Sec. 4.5) and distillation (Sec.(4.6).
English Benchmarks
Multilngual Benchmarks Flicker36.
Model
VIT Size (Mes.)
Data
Seca Pairs
val
IN
SLIP 26
DC
Babel -IN
XML5600 T 1-
CVQA EN LOC
T-EI--T
-200
T-a11-1
XTD-10
T- 1-
XTD-200
XEXECU 0
T2/4(224)
LAION 30
32B (2.53)
77.0
80:4
65.5
34.0
50:47 60.5
36,17482 48.2
43.2/46.2
B7.17 735.4
#2.5 22.574552
B/16/250)
WebLir128)
(3.0x)
48.5
51,87 45.7
34,01 36,3
80.8
37.8 40.6
mS
5048062256)
WebLiit2B)
(3.0x)
69.1
46.4
50.0
36,87
39.97
85.67
42.5
5048095256)
Wahl.ir
48.2
Meta
1/14(224)
H/14124))
English12 50)
Engissr2.58B
(1.0x)
Mesa CLIP 2
1/14224)
Workówide
Eaglish
(2.3x)
(1.0x)
193.5
63.5
45.3/583
59.2755.1
41.9/45.8
85.0
41./448
English
138 (1.0%)
20.4
72.6
68.7
Mesa CLIP
H/14(224)
Workswiike
Nop- Eing.
1788 (1.3x)
198 (1.0%)
71.4
79.5
71.1
63.1
61.7
67.2
47.1
49.9
49 62.6
46.9/599
59.8/56.3
59.91 56.0
49.1 52.1
47.5 50.5
23.2 85.7
85 8
47.0/497
46.6 499.
Wirkkwiske
208 (2.3%)
$1.3
74.5
69.6
50.2
14.3
61.5/5744
50.97.5.2
16.1
15.751.0
Table 1: Main ablation: Meta CLIP breaks the curse of multilinguality when adopting ViT-H/14,
with seen pairs scaled (2.3x) proportional to the added non- -nnlish data. Meta CLIP 2 outperforms mSi with fewer seen pairs (72%), lower resolution (224px vs. 256px), and comparable architec- tures (H/14 vs. SO400M). We grey out baselines those are So So'TA-aiming systems with confounding factors. Here, numbers of seen pairs are rounded to the nearest integer (e.g., 12.8B->13B..
4.2.1 Main Ablation
We first ablate the effects of scaling seen training pairs and minimal viable model capacity that break the curse of multilinguality, with the following two groups of 6 training runs. Two trainings are in ViT-L/14 on worldwide curated data and its English portion, where global batch size and seen pairs are set 2.3x and 1.0x compared to OpenAl CLIP and Meta CLIP setting (i.e.. 1.0x has 12.8B seen pairs, or 400M for 32 epoches as in OpenAT CLIP). Four runs are on ViT-H/14 with different subsets of curated data demonstrate the effects of English data helping multilingual performance and vice versa. We denote each run based on subsets trained with and corresponding seen pairs: 1) Worldwide
7

---

(2.3x) with the full-fledged worldwide curated data; 2) Worldwide (1.0x) with 1) downsampled; 3) English (1.0x) with English portion of 1); 4) Non- Non-English (1.3x) with the non-English portion.
We adopt the following two groups of zero-shot transfer benchmarks and discuss the limitations in Appendix G 1) English-only benchmarks ImageNet (IN val) [42, SLIP 26 tasks (SLIP 26 avg.) [35]. and DataComp 37 tasks (DC 37 avg.) [28]; ; 2) multilingual benchmarks on Babel- ImageNet (Babel-IN) [3] (averaged zero-shot classification on IN with classes and prompts trans- lated into 280 languages), XM3600 4 (multilingual text-to-image, T-+, and image-lo-text, I-T,, retrieval with an averaged recall on 36 languages), CVQA [55 (multilingual multi-choice vi- sual question answering with English and local averaged answer accuracy), Flickr30k-200 [46] (Flickr30k test translated into 200 languages), XTD-10 [4 (multilingual image-text retrieval on MSCOCO [8] averaged Recall@) over 7 languages), and XTD-200 [46] (XTD10 translated into 200 languages). In Table [ we observe that Meta CLIP 2 on ViT-H/14 with worldwide data and scaled seen pairs consistently outperforms its counterparts English (1 (1.0x) and Non-English (1.3x), on both English and multilingual tasks, effectively breaking the "curse of multilinguality". The curse still exists in non-scaled seen pairs, Worldwide (1.0x) or smaller ViT-L/14 model even with Worldwide (2.3x)). We further provide gradient conflict analysis to help understand the root of the curse in Appendix
Although SoTA is non-goal for Meta CLIP 2. its full recipe demonstrates strong performance with fewer seen pairs (72% of SigLIP series) and lower resolution (224px vs mSigLIP's 256). Meta CLIP surpasses mSigLIP on IN, SLIP 26, and DC 37, and the recent SigLIP 2 on the latter two. More significantly, Meta CLIP 2 sets many SoTA multilingual benchmarks, e.g., Babel-IN (43.8%), XM3600 (+1.1 11%+1.1/%%/1 CVQA (+3%/+7.66 ). Flicker-30k-24 -00 (+7.7% and XTD- 200 (+6.4% SigLIP 2 prioritizes English (90% of its training data in English), while it is worse than IS on multilingual tasks and Meta CLIP 2 on most English benchmarks except IN.
We also provide per-language analysis in Appendix D gross-lingual translation in Appendix F
XM3600 T 11II T
CVQA EN LOCAL
Ablation Steps
Metadata
Alt-texts
Babel-IN
1: English CLJP
English
English
67.5
2: remove English filter
English
all, in set
66.9
3: no language isolation
all. in set
all, set
62.1
31.2
37.8
49.7
49.8
45.8
4: language isolation with fang fen
all, by lang-
all, by lang-
61.1
31.5
37.9
49.4
49.0
46.5
5: language specific frug
all, by lang.
all, by lang.
64.7
31.5
38.1
50.0
50.3
46.6
Table 2: Ablation study of metadata and alt-texts combination on ViT-B/32 using English 1.0x and Worldwide 1.0x with mT5 tokenizer. flang/ten are the count thresholds for each language/English.
4.2.2 Ablation on Metadata, Curation, and Tokenizer
We further ablate the transition from metadata and curation focuses solely on English to their worldwide equivalents using the VYT-B/32 encoder for efficiency. We evaluate zero-shot transfer on IN for English and Babel-IN, XM3600 and CVQA for multilingual. As in Table starting from English-only CLIP, we first remove the English filter on alt-texts so that all alt-texts are curated by English metadata, resulting in 0.6% drop on IN, indicating English isolation separating text or metadata by LID before matching important. Then, we replace English metadata using all metadata merged without separation, yielding even worse English performance but start building up multilingual capability. Next, we isolate substring matching and curate alt-text with the same len over all languages. This further lowers English performance since 4en is too high for non-Englise -on-English and let head data dominate curation. Lastly, we compute tang. to keep the same ratio of head-to- toill concepts for each language. This improves English and non-English performance, while curse of multilinguality remains unresolved in ViT-B/32 until the main ablation described above.
XM3600
CVQA
Tokenizer
Vocab. Size
IN val
Babel-IN avg.
T~III>>
LOCAL
mT5 (mSigLIP) [49
250k
64.7
31.5
38.1
50.0
50.3
46.6
Gemma (SigLIP 2) 500
256k
63.7
26.1
36.1
47.8
48.3
44.0
XLM-Roberts [51]
250k
64.0
31.1
38.0
49.8
49,8
46.1
XLM-V 52
900k
64.7
32.7
40.0
51.4
50.4
47.4
Table 3: Ablation study of various multilingual tokenizers with ViT-B/32 and Worldwide 1.0x.
8

---

To minimize changes in model architecture, we only swap the English tokenizer for a multilingual one. Four popular tokenizers are studied on our zero-shot benchmarks. As shown in Table the XLMMV vocabulary yields the strongest performance in both the English and non-English world.
Dollar Street
Model
Data
Seen Pairs
GLDv2
GeoDE
Top-1
Top-5
16
WebLI 15
40B (3.0x)
36.0
62.5
45.3
94.5
SigLIP 2
17
WebLI(I
15
40B (3.0x)
36.7
61.9
48.5
95.2
English
138 (1.0x)
37.2
63.3
52.8
93.4
Non-Hnglish
178 (1.3x)
35.7
61.3
68.6
91.7
Meta CLIP
Worldwide
13B (1.0x)
37.2
63.7
65.8
94.3
Worldwide
29B (2.3x)
37.9
64.0
69.0
93.4
Table 4: Zero-shot classification accuracy cultural diversity benchmarks. Meta CLIP 2 models
are in ViT-H/14 and LSigLIP/S 2 are in ViT-S0400M. mSigLIP/SSgLII 2 are SoTA-aiming systems with many factors changed and thus greyed out.
4.3 Cultural Diversity
Following protocols in and 38 we perform zero-shot classification and few-shot geo- localization on range of geographically diverse benchmarks. Specifically, we include zero-shot classification with Dollar Street [53). GeoDE [34], and GLDv2 [55] in Table[4] and few-shot geo- localization (2 on Dollar Street. GeoDE and XM3600 in Fig. 3 We find that only changing the training data distribution, from 13B English to 13B worldwide pairs, yields significantly better perfor- mance, and scaling to 29B worldwide pairs improves further, except for the on-par, probably saturated performance in GeoDE. Fig. shows similar trend for evaluating on few-shot geo-localization.
Dollar Street
GeoDE (country)
GeaDE tregian)
X643600
50-
53-
165
Accuracy 190
Accurace 190
400
30 1
22-
5
20
25
30
Shots
Shats
Shotss
Shota
mS5ppP (Web 405)
MotaCUP2 Engi sh (L38)
MutacUPP2 2 Worldalde (138)
SIJUP 2 (Aib 408)
MutuCUP Nun Engl (278)
MotaCUP 2 Worddale (296)
Figure 3: Few-shot geo-localization accuracy cultural diversity benchmarks.
4.4 Building Multi-modal LLM with Meta CLIP
We evaluate the efficacy of Meta CLIP being used as a vision encoder in downstream multilingual MLLMs with frozen-encoder setup (56). with details in Appendix F Table 5 shows that switching the frozen vision encoder from mS to Meta CLIP 2 consistently improves MILM performance over the wide range of evaluation, including both English and multilingual tasks. Scaling Meta CLIP 2 from 13B to 29B seen pairs shows better results. These results show that curating worldwide data not only enhances retrieval classification but also transfers to MLLMs.
Culture Understonding Captioning
Short VOA
Multi subject Reasoning
Model (VTT Size)
Data
Seen Pairs
CVQA mul
MaRVI. ea
XM100 ea
xGQA en
MaXM mul
XMMMU en
M33xaa
en
mul
mul
mul
en
mul
en
mul
mSigLIP [6] (S0400M)
WebLI [B
40B (3.0x)
63.2
55.8
86.8
82.9
30.5
16.4
63.5
59.5
51.4
52.1
45.4
44.7
57.6
49.1
English
13B (1.0x)
46.0 55.9
88.1
83.7
30.1
16.6
64.2
60.2
54.5
53,5
43.4
43.4
59.3
48.5
Meta CLIP (H/14)
Non-Eng.
17B (1.3x)
52.3
57.7
86.5
82.8
30.0
16.4
64.3
60.5
53.3
50.4
45.7
45.0
57.6
49.1
Worldwide
13B (1.0x)
67.1
59.4
87.7
83.5
30.3
16.3
64.1
60.2
52,9
52,9
47.2
45.4
59.6
47.5
Worldwide
29B (2.3x)
67.5
59.9
88.1
83.8
30.3
16.8
64.3
60.3
53.3
50.3
46.4
45.9
58.9
Table 5: Multilingual MLLM tasks from PangeaBench [56]
We also observed the English-only Meta CLIP performs much worse on CVQA (translated) English benchmark, indicating the importance of training on non-Englist data. Interestingly, while some tasks like CVQA and M3Exam show clear improvement trends after adding non-English data, some other tasks, e.g. XM100, XGQA and MaXM, exhibit similar performances after switching from English-only to multilingual models. This indicates these benchmarks can be insensitive to the improvement on culturally diverse visual features, but rely more language ability of MLLMs.

---

4.5 Aligmment and Uniformity
Following [41, we further measure the embeddings quality across different CLIP models. To avoid various unknown biases from different benchmarks, we use 5k holdout image-text pairs not used in our training and report alignment and uniformity scores, where alignment measures the relevance of an image and a text and uniformity measures how images distributed in vision encoder's embedding space. Note that we have control on whether these 5k pairs are leaked in other baselines. From Fig. 4 we can see that Meta CLIP exhibits good scores in both alignment and uniformity (lower is better), whereas mSigL.I SigL.P may have non-trivial bias on our collected holdout data.
Alignment vs. Uniformity Across Models
Sighi
SSLLP
-04
McSHLPP modks
(lower is better)
flower better)
Sist
1.-
50420M/14
-08
3.4
NIA (Nen-Eng 3x|
MctaCUP2
H/14 (Ergg 1x]
MetsCL
H2L4 (W/V) 2.3x)
Ma 2.3H
MotaCUP2 H114 (Ww
MetaCLIF2 IF2
U14 (Eng 1x]
14
Lnttern (lower is better)
Figure 4: Alignment and uniformity scores 41 indicates worldwide data.
calculated on our collected 5k holdout data, WW
4.6 Distilling ViT-H/14 into Smaller Models
To reduce the inference cost while maximizing performance. we distill the ViT-H/14 model into the smaller models such as ViT-L/14. Our results in Table demoonsrttttttttee that although the teacher model, ViT-H/14, is considerably large, its knowledge can be effectively compressed into smaller ViT-L/14 student through distillation. The distilled model trained on worldwide data performs better than the from scratch worldwide model on all benchmarks and the English-only model in most cases, while still suffering the curse of multilinguality on ImageNet. More models, including ViT-S/16, ViT-M/16, ViT-B/32, ViT-B/16, ViT-bigG/14, and their corresponding text encoders using the smaller mT5 tokenizer, are available on the official website.
English Benchmarks
Maltllngual Benehmarks
Training
VIT Size (Res.)
Data
Pairs
val
ang.
DC:37
ag.
Bahel -IN
T-1
CVQA EN
T-l1-T
Blickerkk -200
XTD-10 T-11-T
XTD-200 T-II-T
From Sensch
1/14(224)
Worldwids
Eng ish
29B (2.3x)
J38 (1.0%)
76.8
79.5
60.3
67.2
66.0
63.5
442
453/58.
55./ 59.8/563
41.9/45.8
$2.8/850
41.9/448 45.0/43.
Disilled
L/14/234)
Worldwide
29B (2.3xg
79.2
70.9
67.4
45.7
47./6022
46.8/499.
$3.91 96.0
Table 6: Distillation into smaller models: we show that the distilled ViT-L/14 can be close to the
performance of its English counterpart.
Conclusion
We present Meta CLIP 2, the first CLIP trained with worldwide image-text pairs from scratch. Existing CLIP training pipelines, designed primarily for English, cannot straightforwardly generalize worldwide setting without incurring an English performance degradation due lack of curation for worldwide data or the "curse of multilinguality" Our careful study suggests that the curse can be broken by scaling metadata, curation, and training capacity, where English and non-Enge non-English world benefit each other. Specifically, Meta CLIP 2 (VIT-H/14) surpasses its English-only counterpart on zero-shot IN (80.5% - 81.3%) and sets new SOTA on multilingual benchmarks such as XM3600, Babel-IN and CVQA with one single model. We envision our findings along with the fully open- sourced metadata, curation and training code encourage the community to move beyond English- centric CLIP and embrace the worldwide multimodal web.
10

---

Acknowledgments
We thank Thao Nguyen and Bernie Huang for insightful discussions, Wei Cheng and Guan Pang for evaluation on downstream use cases. At MIT, Yung-Sung was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-194 19-2- 1000. The views and conclusions contained in this document are those of the authors and should not interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[2] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 20x7v:240..21783 2024.
[3] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- -an--Bptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai. Anja Hauth, Katie Millican, al. Gemini: family highly 2023.
capable multimodal models. arXiv preprint arXiv:2312.
[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in nearal information processing systems, 36:34892-34916, 2023.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[6] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
[7] Hu Xu, Saining Xie, Xiaoqing Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024.
[8] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs arXiv preprint 1tx.x:2111.022111 2021.
[9] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25294, 2022.
[10] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. /7425, 2023.
[11] Wikipedia. Languages used the internet. https://en weten.wwkkin www.wwiiiiiia.... org/woo///................... wiki/ Languages_usae _ue___uee___o on_ pon_thh the_ She_Innernet] 2025. It reports 50.9% of web content is non- son-English
by 2025. Accessed: 2025-05-15.
[12] Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. melip: Multilingual clip via cross-lingual transfer. In Proceedings of the 6/ Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2023.
pages

---

[13] Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingus and multilingual clip. In Proceedings of the thirteenth language resources and evaluation conference, pages 6848-6854, 2022.
[14] Thao Nguyen, Matthew Wallingford, Sebastin Santy, Wei-Chiu Ma. Sewoong Oh, Ludwig Schmidt, Pang Wei W Koh, and Ranjay Krishna. Multilingual diversity improves vision- language representations. Advances in Neural Information Processing Systems, 37:91430. 91459, 2024.
[15] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-iman -aagae model. In The Eleventh International Conference on Learning Representations, 2023.
[16] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11955-1986, 2023.
[17] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprini arXiv:2502. 14786, 2025.
2:
[18] Da-Cheng Juan, Chun-Ta Lu, Zhen Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. Graph-rise: Graph-regularizeed image semantic embedding. arXiv preprint arXiv: /902. 10814, 2019.
[19] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of I1m scaling based on human-generated data. arXiv 2022.
preprint arXiv:2211.044:
[20] Daniel Bolya, Po- Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
[21] Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Steiner, Xiaohua Zhai, and Ibrahim M Alabdulmohsin. No filter: Cultural and socioeconomic diversity in contrastive vasion-language models. Advances Neural Information Processing Systems, 37: 37:106474-106496 2024.
[22] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09878. 2024.
[23] David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. arXiv preprint mrXXi:2504.01017, 2025.
[24] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint 12092 2021.
[25] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[26] Chao Jia, Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. International conference on machine learning, pages 4904 4916. PMLR, 2021.
12

---

[27] Wenliang Dai. Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirly-sevnnth Conference on Neural Information Processing Systems, 2023.
[28] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.
[29] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp- In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200-14282, 2024.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
[31] Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, al. Laion-5b: An open large-scale dataset for training next generation image-text models. 2022.
[32] Kanehana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfci Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5571 15554,, 2023.
[33] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1,59--6077. PmLR, 2020.
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pages 1-31, 2024.
[35] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint 1rXiv:2112.127500 2021.
[36] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[37] Gabriel Oliveira dos Santos, Diego AB Moreira, Alef Iury Ferreira, Jhessica Silva, Luiz Pereira, Pedro Bueno, Thiago Sousa, Helena Maia, Nádia Da Silva, Esther Colombini, et al. Capivara: Cost-efficient approach for improving multilingual clip performance on low-resource languages. arXiv preprint arXiv:23 13683, 2023.
[38] Xiao Wang, Ibrahim Alabdulmohsin. Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai Scaling pre-training to one hundred billion data for vision language models. arXiv preprint 0rXiv:2502.06617. 2025.
[39] Giusepppe Attardi. Wikiextractor. http:////gthacooo//tttt/t///attt
2015. 2015
[40] Édouard Grave, Piotr Bojanowski, Prakhar Gupta, armand and Tomáš Mikolov. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
13

---

[41] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages 9929-9939, PMLR, 2020.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-20 12222, 2015.
[43] Gregor Geigle, Radu Timofte, and Goran Glavaš. Babel-imagenet: Massively multilingual evaluation of vision-and-langee and-language representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5064-5084, 2024.
[44] Ashish V Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600; A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715-729, 2022.
[45] David Orlando Romero Mogrovejo, Chenyang Lyu, Haryo Akbarianto Wibowo, Santiago Gón- gora, Aishik Mandal, Sukamnya Purkayastha, Jesus-German Brizz-Baaajas, Emilio Villa Cueva, Jinheon Back, Soyeong Jeong, Injy Hamed, Zheng Xin Yong. Zheng Wei Lim, Paula Mónica Silva, Jocelyn Dunstan, Mélanie Jouitteau, David LE MEUR, Joan Nwatu, Ganzorig Batnasan, Munkh-Erdene Otgonbold, Munkhjargal Gochoo, Guido Ivetta, Luciana Benotti, Laura Alonso Alemany, Hernán Maina, Jiahui Geng. Tiago Timponi Torrent, Frederico Belcavello, Marcelo Viridiano, Jan Christian Blaise Cruz, Dan John Velasco, Oana Ignat, Zara Burzo, Chenxi White- house, Artem Abzaliev, Teresa Clifford, Gráinne Caulfield, Teresa Lynn, Christian Salamea- Palacios, Vladimir Araujo, Yova Kementchedjhieva, Mihail Minkov Mihaylov, Israel Abebe Azime, Henok Biadglign Ademtew, Bontu Fufa Balcha, Naome A Etori, David Ifeoluwa Ade- lani, Rada Mihalcea, Atnafu Lambebo Tonja, Maria Camila Buitrago Cabrera, Gisela Vallejo, Holy Lovenia, Ruochen Zhang, Marcos Estecha- Garitha-Gaitagiiiiia, Mario poarriuuz-Caantelar Togeer Ehsan, Rendi Chevi, Muhammad Farid Adilazuarda, Ryandito Diandaru, Samuel Cahyawijaya, Fajri Koto, Tatsuki Kuribayashi, Haiyue Song, Aditya Nanda Kishore Khandavally, Thanmay Jayakumar, Raj Dabre, Mohamed Fazli Mohamed Imam, Kumaranage Ravindu Yasas Nagas- inghe, Alina Dragonetti, Luis Fernando D' D'Haro, Olivier NIYOMUGISHA, Jay Gala, Pranjal A Chitale, Fauzan Farooqui, Thamar Solorio, and Alham Fikri Aji. CVQA: Culturally-divers multilingual visual question answering benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[46] Alexander Visheratin. Nilb-clip-trann performant multilingual image retrieval model on a budget. arXiv preprint arXiv:2309.014 01859, 2023.
[47] Pranav Aggarwal and Ajinkya Kale. Towards zero-shot cross- cross-lingus lingual image retrieval. arXiv preprint arXiv:20/2.05107 2020.
[48] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1:Xi::5504.0030 2015.
[49] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former. arXiv preprint 211:20000.11944 2020.
[50] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822 2024.
[51] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at seale. arXiv preprint 191.01111. 2019.
[52] Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zeutlemoyer, and Madian Khabsa. Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models. arXiv preprint 2xXx:23301..0722 2023.
14

---

[53] William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. Advances in Neural Information Processing Systems, 35:12979--2990, 2022.
[54] Vikram Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recogmition. Advances in Neural Information Processing Systems, 36:66127-661333 2023.
[55] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pages 2575-2584 2020.
[56] Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neu- big. Pangea: A fully open multilingual multimodal Ilm for languages. arXiv preprint arXiv:2410. 16153, 2024.
[57] Wikipedia. Scriptio continua.
[58] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824-5838 2020.
[59] Zirui Wang. Zachary C Lipton, and Yulia Tsvetkow. On negative interference in multilingual models: Findings and meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, 2020.
[60] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467- 10485, 2021.
[61] Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Man-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. In Findings of the Association for Computaaional Linguistics: ACI. 2022, pages 2497-2511, Dublin, Ireland, 2022. Association for Computational Linguistics.
[62] Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. Maxm: Towards multilingual visual question answering. ArXiv preprint, 0ass/209.004000 2022.
[63] Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484 -655544-55005 2023.
[64] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
Icce, 2009.
[65] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint
2017.
[66] Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 52 59, 2019.
[67] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 19567): 1981, 2020.
15

---

A Implementation Details for Metadata and Curation
A.1 Unigram and Bigram Tokenizer for Special Languages
Most modern languages around the world adopt writing systems that use "spaces" to separate words, except for some of the Asian languages, known as "scriptio continua" [[7].. We find several open source tokenizers for many of these languages developed by local communities, as shown Table 7 in order to properly split text into words while preserving semantic integrity. Note these tokenizers are only used to process Wikipedia dump labeled with the listed wiki codes (e.g., not on alt-texts' substring matching).
Wiki Code
Tokenizer Name
URL
bo,dz ja.ryu km
Tibetan Tokenizer
Japanese Tokenizer
Khmer Tokenizer
https://github http://hhaao.ooooo//////o///e/ep
khmer
Lao Tokenizer
https://Rithub https///ittuub..cco/aaaapppoppop .cw.://aanaaghhong//all......
my
Myanmar Tokenizer
https://eithub. htapii.ooooouuuueae com/Thurakuugg1600 httpa//oooooo//aaaaaooooo/ooo///
th
Thai Tokenizer
zh,zh__laan classical,zh__yn
Chinese Tokenizer
Table 7: Tokenizers for special languages.
A.2 Fix Ranking Bigram with Raw PMI
Although Meta CLIP follows OpenAl's description on ranking bigrams by point-wise mutual infor- mation (PMI), we observed that raw PMI for bigrams overemphasizes extremely rare pairs (e.g., bigram appearing only once typo), yielding unintuitive high scores. For example "AAAAAB CCCCCB" appears high. mitigate this, we (i) temper rarity by multiplying PMI with a sublinear count factor and (ii) subtract a baseline using lower-percentile PMI threshold that roughly marks the onset of meaningfulness.
Let o(W1, W2) be the bigram count, i.e., the times that W1 and wy co-exist adjacently in the corpus, c(w) the unigram count, and the total token count in the corpus. We define
p(w1, w2) PMI(wI: w2) = log p p(w1) ). p(w2) = log c(w1, W2)
c(w) )c(wa)
Let PM730% denote the 30th percentile of the empirical PMI distribution over observed bigrams in a language (a baseline for "starts-to-be- -e-aiiig-u-- associations). Our final bigram score is
Score(w1, w22) [c(w], w2) +1)0 x (PMI(w1, PMT30%)
This new formulation weig-weights spurious high-PMI, low-count bigrams while preserving genuine high-frequency associations; the percentile shift suppresses background noise from weakly associated pairs. After replacing bigram ranking with the new scoring metric, we got the following top-5 bigrams: "United States", "of the", "New York", "such as", "has been".
A.3 Scaling Curation
Worldwide scaling of data curation significantly increases time and space complexity due storing metadata across hundreds of languages. To efficiently handle this complexity, we leverage several efficient implementations:
Efficient String Matching: We adopt the Aho-Corasick algorithm V which utilizes prefix trees (tries), for rapid substring matching. The matching speed is about 2k times faster than Meta CLIP's brute-force implementation, enabling matching with million-scale metadata.
16

---

Lazy Metadata Loading: We pre- re-build and store the metadata into an Aho-Corasick automaton for each language separately, loading these automaton dynamically and only when encountering new language for alt-text during processing, thereby minimizing the total number languages encountered for each shard of data and saving re-compiling time for automation new shard.
Memory Management for Probabilities: To address memory constraints during sampling for balancing, utilize memory memory-mapped file loading (mmap) to efficiently access counts per entry across all languages, preventing out-of-memory errors caused by loading all the counts from different languages.
Mitigation and Benchmark Deduplication We run a state-of-the-art safety classifier remove NSFW contents (e.g., adult, sexual, violence) from training data. We also apply face detector to remove human biometries and personally identifiable information from data. To avoid benchmark leakage, we remove any overlap with ImageNet evaluation sets by performing deduplication using 64-bit hashes. These hashes are generated by applying random projection to feature embeddings from a similarity search model, reducing them to 64 dimensions followed by sign-based quantization.
B Training Setup
To remove confounding factors and generalize findings, follow OpenAI CLIP and Meta CLIP training setup with changes for worldwide scaling, detailed Table
Typerparameter
OpenAl CLIP / Meta CLIP
Meta CLIP
Activation Function
QuickGELU
QuickGELUU
Seen Pairs
12.8B
29B (2.3x)
Batch Size
32768
75366 (2.3x)
Leurning Rate
4.0c-4 (L/14, H/14)
4.0c-4 (H//14)
Warm-up
2k
2k
Table 8: Hyperparameters of OpenAl CLIP/ Meta CLIP vs Meta CLIP 2.
C Analysis on Cross-Lingual Gradient Conflicts
We hypothesize that the primary cause of the "curse of multilinguality" is insufficient model capacity to acquire new capabilities (e.g., concepts, domains, and languages) without harming existing ones. practical indicator of this phenomenon is language interference inside the model. Inspired by PCGrad [58], originally proposed for multitask learning and later extended to multilingual XLM settings [59] we design a gradient confticr analysis to diagnose interference. Concretely, using XM3600 (36 languages), we compute gradients from model checkpoints and measure cross-lingual interference via cosine similarity between gradients from English examples and those from each non-English language, then average across all non-English languages. All checkpoints are pretrained on the Worldwide 29B schedule; we report the midway (epoch 16) and final (epoch 32) checkpoints.
Worldwide (29B)
Gradient Similarities
VIT-L/14
VIT-H/14
Midway (Epoch 16)
0.508
0.688
Final (Epoch 32)
0.546
0.697
Table 9: Average cosine similarity of gradients (English vs. each non-English language, then averaged) on XM3600. Higher is better (fewer gradient conflicts).
We observe that smaller models (L/14) exhibit lower similarities-i.c. stronger interference and gradient conflicts- larger ones (H/14) throughout training. With more conflicts, L/14 spends valuable optimization steps mitigating cross-lingual disagreement rather than learning semanties, leading to degraded English performance when trained on multilingual data versus English-only In contrast, H/14 shows consistently higher similarities even early in training, suggesting reduced conflict that allows the model to jointly learn from English and non-English data. This alleviates interference, enables positive transfer, and could be the potential reason why it breaky the curse of multilinguality.
17

---

D Correlation Between Training Data Volume and Performance Among Languages
We examine XM3600 zero-shot retrieval for the top-10 languages by training volume versus the remaining languages. We have the following observations: (1) Volume effect exists. Top-10 languages average 62.6/75.6 (T-+/I1-)) versus 47 47.2/59.9 for others, indicating clear volume effect on average. (2) English not best. Despite the largest volume, English lags behind German (the best among all listed; 69. (9.2/83.6). (3) Strong tail performers exist. 18 non-top-10 10 languages (e.g..
it, hu, ro, uk) exceed 50% on both directions, showing that factors beyond raw volume matter.
What is beyond the volume effect? We hypothesize two additional drivers: (i) linguistic/cultural proximity (benefiting from transfer with closely related or culturally overlapping languages), and (ii) structural saaracterssics/expeessveeee of the language (e.g., morphology, tokenization efficiency, domain overlap with pretraining corpora). These factors can amplify or dampen the benefit of volume during multilingual pretraining.
Language
Text Image
Image Text
47.4
60.8
39.4
cs
51.0
66.1
da
61.0
75.1
el
52.1
68.4
56.9
73.7
Language
Text >Image
Image- Text
fil
24.8
36.7
51.6
62.2
hi
26.1
41.8
57.2
72.5
hr
57.3
72.9
67.1
78.5
63.9
zh
61.1
72.6
it
64.0
78.2
67.8
79,9
he
60.8
76,2
65.1
79.9
ko
54.8
70.1
65.8
78.3
0.5
1.2
pt
60.4
72.6
53.2
66.9
de
69.2
83.6
no
57.7
73.2
61.1
76.2
pl
61.4
75.9
Avg (Top-10)
62.6
75.6
quz
2.5
6.5
ro
64.8
SV
57.6
73.8
Sw
10,0
16.6
tc
26.1
37.1
th
57.7
71.4
tr
55.7
68.4
uk Avg (Non-Top-10)
60.0
74.7
47.2
59.9
Table 10: XM3600 Recall (higher is better). Left: top-10 languages by training volume. Right:
languages outside the top-10.
E
Setup and Details of MLLM Evaluation
While zero-shot classification and retrieval benchmarks demonstrate the standalone capabilities of
Meta CLIP 2, real- world applications often require grounding in generative models. Thus, we conduct the experiment of using Meta CLIP 2 model vision encoder in MLLM in Sec. 4.4 Here, provide more details about the settings and task details.
F.1 Training Setup
For evaluation, we leverage the open-sourced MILM 56 implementation and apply exact the same model setup except that we vary the vision backbone with each vision encoder be evaluated. The MLLM is trained as the following. First, vision- connector is trained to align the vision
18

---

encoder features to the language backbone. Then, we tune the MLLM with 6M samples spanning 39 languages. We followed the same training recipe, including learning rate 1e-3 and batch size 128 for vision-language connector training, and learning rate 2e-5 and batch size 512 for finetuning. Both stages are coupled with cosine learning rate scheduler with warmup ratio of 0.03. For evaluating the quality of embeddings from vision encoder, we make one change in the training that during the finetuning stage, we freeze the vision backbone, while all weights are finetuned the original setting.
E.2
Task Details
The trained MILMM models with varying vision encoders are then evaluated on PangeaBench [56]. which includes following tasks:
Culture Understanding: CVQA evaluates model's capability in cultural reasoning using visual questions with diverse global contexts across 31 languages and 13 scripts [5]. Unlike our embedding- only experiment in Table L here we follow the generative setting to select answers based on the
MLLM's output probabilities. MaRVL tests cross-lingual visual reasoning with culturally grounded entailment tasks in multiple non- non-English languages [60].
Captioning: XM100 is compact multilingual captioning benchmark with 100 diverse images selected from XM3600 across 36 languages for efficient and diverse evaluation
Short VQA: xGQA extends the GQA dataset to multilingual settings to measure tross-lingual VQA performance [61] MaXM offers multilingual VQA tasks covering different scripts and question types to test model understanding beyond English [2]
Multi-subject Reasoning: XMMMU is a translated subset of MMMU validation questions into six languages to evaluate academic reasoning in a multilingual setup. M3Exam poses real-world multimodal exam questions across subjects, requiring both visual and textual comprehension (3
F
Cross-Lingual Translation Capability
We probe whether the model acquires cross moosl translation behavior without explicit supervision. Given an image that visually depicts the Chinese character "Fij" ("dog"), we compute cosine simi- larities between the image embedding and candidate text prompts across languages, then rank the candidates. As expected, the exact Chinese character "Vaj" yields the highest score. Notably, the Japanese word "t 182" (dog) ranks highest within Japanese candidates and achieves substantially higher similarity than English "dog." suggesting stronger cross-lingual coupling between Chinese and Japanese scripts.
Word
Description
Cosine Sim.
"dog" in Chinese (exactly visualized on image)
0.54325
*
"dog" in Chinese, literary/ancient usage
0.04636
"cat" in Chinese
0.00025
$1
"jackal" / 'wild dog" in Chinese
0.03427
R
"wolf" in Chinese
0.01405
dog
English "dog"
0,08239
diagram
Unrelated word
0.00143
cat
English "cat" English "puppy"
0.00005
puppy
0.02826
hound
English "hound"
0,05586
1
"dog" in Japanese "cat" in Japanese
0. 0.19320
ta
0.00064
Table 11: Cosine similarities between the image of the character "Xa" and multilingual text prompts. Higher is better.
We make the following observations: (1) Cross-modal alignment reflects cross-lingual relations. The strong scores for "Yig" and 382"" indicate the model maps visual text to semantically equivalent words across languages. (2) Script proximity matters. Japanese scores exceed English for this example, plausibly due to closer linguistic/seript overlap with Chinese. (3) Robustness amid noise. Despite inevitable Internet-scale noise and partial misalignment between OCR-like visual tokens and
19

---

alt-text, the model still exhibits emergent cross- lingull translation behavior supporting the premise that remaining faithful natural data distributions can mitigate noise effects.
G Limitation on Benchmark
High-quality benchmarks are essential for researchers to understand the efficacy of proposed changes. After decades meticulous efforts, the community has established reliable and diverse datasets to enable research advancement in vision and multimodal areas [64 42 []. However, these datasets consist mainly of content scraped from North America and Western Europe (NA+EU) and focus on English 66 66]. It is long and resource- resource-intensive intensive endeavor to build similar benchmarks for unbiased and comprehensive evaluation of worldwide data and resulting representations, for the world outside NA+EU or English-speaking community, due the complexity of covering diverse concepts across geo-locations, cultures, and languages. X.36600 44 aims to build geographically diverse datasets by selecting images from Open Images Dataset 667 based metadata of GPS coordinates, but later research [21 suggests Open Images Dataset is biased towards Western images or specific activities (c.g.. tourism). GeoDE [54] recruits human workers on crowdsourcing platform collect geographically diverse images for predefined object classes. Crowdsourcing is an economic way to collect human annotations, but the demographic background and proficiency the workers not guaranteed, nor is the quality of the collected data. Few efforts such as CVQA 45] attempt to scale annotation and control quality simultaneously by utilizing experts in machine learning community or existing materials as seeds. These efforts offer relatively unbiased evaluation with reasonable coverage capabilities (e.g.. cultural diversity, multimodal problem solving for exam questions across countries) of interests. We believe benchmarks of similar quality but built for evaluating more general and comprehensive capabilities will reveal true potential of worldwide data and resulting representations developed in this work.
H Licenses for Existing Assets
Below we list the licenses for all existing assets used in this work, including code, models, and datasets.
Meta CLIP Code:
-License: CC-BY-NC
-URL: htwp://pw/ggtttoooo/////eeeeerer
Wikkipedia Dumps:
-
-License: CC BY-SA 4.0
-
-URL: htll:httts//dddmm wikimedia org/
WordNet:
-License: WordNet License 3.0
-URL: wtwp:httts///ooooo princeton. edu/
Multilingual WordNet:
-License: WordNet License 3.0
-URL: TRLL:httss://mmeee org
Models Used for Evaluation:
XLM-CLIP
License: MIT
* .URL: Srrbrtaa--argee-lall---------
-SiiLLPP-O040OM
* License: Apache 2.0 * URL:/https:/ /
-mSigLIP httppp//hhhgggagacc/o//o///////d
* License: Apache 2.0
20

---

* URL: https//s/.ggggggo......o///////a SigLIP 2-S0400M
License:
Apache 2.0
Code and Data for Evaluation:
CLIP benchmark, including Flickr30k-200, XTD-11, XTDD-000
*
License: MIT
* URL: /ttps:://.ttu...... https////g.aaa..ccc//AAAAAAAAAC1 benchaark
DataComp evaluation
* License: MIT
*
URR:
SLIP
evaluation
License:
MIT
URL: htt.:/ttts//ggtah..cooo//ccooert
Pangea, including the MaRVL, XMI00, XMMMU, M3Exam datasets.
License: Apache 2.0
XM3600
* License: CC 4.0
CVQA
* License: CC BY-SA 4.0
*
org/
GLDv2
* License: BC
* Uttp:ttppe//gghahoooooc////o/ooo
Babel-Image-Ve
* License: BabelNet Non-Commerrial License and MIT license
*
ImageNett
ImageNet
License: non-commercial research and educational purposes, detailed in https
wwwp..imggennet.rr//oooo
* TRL::https:///// imw.e- net. org
Dollar Street
* License: CC BY-SA 4.0 URR:
*
tthe-dollar-street--ae
GeoDE
* License: CC 4.00
*
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Sec. B we introduce the metadata construction and scalable curation algo- rithm that enable our models learn from 300+ languages without relying on translation or private data. In Sec. 4 we demonstrate that our approach breaks the curse of multilinguality by scaling model capacity and seen training pairs, showing mutual benefits between English and non- Ennginh performance. We also provide comprehensive ablations (Sec. 4.2.2
21

---

metadata, curation, multilingual tokenizers, and additional downstream evaluations on cul- tural diversity tasks (Sec. (..). Taken together, the contributions and scope outlined in the abstract and introduction are fully supported by the methodology and analysis throughout the paper and appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Sec. we discussed explicitly that the goal of this paper is to offer generalizable recipes and comparable results to mainstream CLIP architectures. Pushing SoTA performance is not the direct goal and we encourage the community to adopt our recipe for their own CLIP system. More details are discussed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
22

---

Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the experiment details in Sec. 4.2 and Appendix A We further provide the source code in supplemental material to make the results reproducible and verifiable.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Answer: [Yes]
Justification: We provide the source code on httth::s/gtiiiiii com/facabookresearchh MetaCLIP with descriptions and instructions. We use open-access public data from the Internet and the open benchmarks for evaluation. We disclose all the experiment details in for reproducing the experiment results.
See. and Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //ipps.ccpublic/guuidee httpuuulllgggudse//eeeeee CodeeSbmii issiiissionnn ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose all the experiment details Sec. Appendix A (data curation details), and Appendix B (training details) for reproducing the experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In experiment,
the training of VIT-B/32 model for three times report the standard deviation for ImageNet accuracy in Appendix [B] For larger models that take weeks to train, i.e. VIT-L/14 and ViT-H/14, we only the experiment because is difficult and expensive to repeat the large-scale pretraining.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Discussed Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Yes, this research conforms, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include the discussion of broader impacts in the end of introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As discussed in Appendix A.3 we use safety classifier to remove NSFW, CSAM and human face content when processing the training data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite assets, such as code, data, and models, by their papers or repositories and follow licenses and terms properly. More details about the licenses of used assets are discussed Appendix H
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, check http:p///ggtttt..ccooooooo/reeer tails.
for de-
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only fine-tune LLMMs and test them on the standard MLLM benchmarks to evaluate the feature quality of our visual encoders for solving standard VQA tasks, as described Sec. and Appendix Otherwise, we do not use LLM developing any other components in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy /att:e///morrigsec/////oofrrrrrs for what should or should not be described.
28


Reviews:
[{"review_id": "CTcnei5dTT", "reviewer": "Reviewer_pL9G", "summary": "This paper proposes MetaCLIP-2, which is extended from English to multiple languages compared to the previous version of MetaCLIP. This work proposes a dynamic threshold t_lang adjustment method based on English data to balance the concept distribution. It achieves good results on non-English multilingual language benchmarks, while maintaining comparable performace to previous methods on English language tasks. This paper conducts detailed ablation experiments to prove that English data can help improve non-English, and the model performs well in cultural diversity tasks.", "questions": "- The 6% tail concept ratio t_en is based on the experience of English data. Has the optimality of this ratio been verified for low-resource languages (such as African languages)? How to alleviate the filtering bias caused by the difference in data volume between languages?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- MetaCLIP-2 is currently the best multilingual image-text alignment model, which will benefit many domains, such as image generation or cross-regional cultural integration research.\n\n- It provides a complete solution from metadata construction, multilingual training, and dynamic filtering of training data based on the different languages.\n\n- Detailed experimental evaluation and ablation experiments prove its effectiveness.\n\n**Weakness**\n- Although the MetaCLIP-2 shows good performance, it is more due to engineering optimization and massive resources, such as data and computing resources, which makes this paper more like a technical report than an academic paper.\n\n- The 6% tail concept ratio is based on the experience of English data and may not be optimal for low-resource languages.", "was_revised": true, "final_justification": "After thoroughly reviewing the rebuttal and the manuscript, I believe all my concerns have been properly addressed, and I don't have any further questions.", "rebuttal": "We sincerely appreciate your comprehensive review and positive comments, including recognizing our work as 'the best multilingual image-text alignment model', 'a complete solution', and highlighting our 'detailed experimental evaluation'. Below, we address the reflective questions you raised and hope our responses clarify your concerns:\n> “Although the MetaCLIP-2 shows good performance, it is more due to engineering optimization and massive resources, such as data and computing resources, which makes this paper more like a technical report than an academic paper.”  \n\nThis work studies the scaling of inter-language data (which poses brand new challenges not exist in intra-language, e.g., English only, scaling) and the curse of multilinguality that emerges in inter-language scaling. We present a proof-of-concept recipe, with only necessary changes and minimal viable model capacity, showing that ViT-H/14 is the first critical point that breaks the curse. All these mark the 0 to 1 change as the spirit of academic paper, which enables further science discovery to further understand the curse and improve solutions. We totally agree that training a worldwide CLIP model involves non-trivial engineering challenges. We provide a solid solution to these challenges, which have largely remained unaddressed in the community, as the foundation for future academic research and lower the implementation barrier. Specifically, we deliberately minimize CLIP model architecture changes to reduce engineering complexity and concentrate solely on data research. We believe our recipe will pave the way for future studies to continue improving the model performance.  \n\nAlso, we strive to share comprehensive implementation details on purpose to reduce the barrier of scientific exploration and encourage the application of open research principles. We believe our emphasis on details and transparency will empower the community to accelerate innovation. Our work stands in contrast to a growing trend where large labs with substantial resources become opaque and private—particularly regarding their data. Our openness is not just a practice, but a scientific contribution in itself.\n\n> “The 6% tail concept ratio t_en is based on the experience of English data. Has the optimality of this ratio been verified for low-resource languages (such as African languages)? How to alleviate the filtering bias caused by the difference in data volume between languages?”\n\nThis is a very insightful question. Being agnostic to data volume is also our goal (line 191-195 in the main paper) and that’s why we didn’t use t_en for all languages but use p (6%, ratio of tail-to-all concepts) since it is volume independent. The assumption behind is that there is a “sweet spot” that balances new and common concepts that humans perceive everyday to make most people feel comfortable. Too many common concepts make humans bored while too many new ones deplete human’s cognitive capacity and effective learning can’t be achieved. Although this assumption requires rigorous experiment in cognitive science to verify, this assumption seems empirically verified as beneficial in previous works on English data and CLIP models (where “sweet spot” is used to decide whether a concept is from long-tail and interesting enough to upsample). Compared to setting a hard limit t_lang for each language, this assumption offers a language agnostic way to guide curation of distribution across languages.\n\nDuring the pre-training stage, the target is to enable models to learn general capability and comprehensive knowledge, plus the goal of this work is to present a proof-of-concept recipe for breaking the curse of multilinguality. Thus, we design our algorithm to handle language one-by-one, so that the language distribution in pre-training data preserves and truly reflects the phenomena observed in the real world (e.g., ratio of English vs non-English speakers or example occurrences on the web, the general interests in humans to balance new and common concepts). With this design, the pre-trained model learns general capabilities such as semantic understanding and vision-language alignment.\n\nIt’s very likely such design is not optimal for low-resource languages and may inherit biases on the web (e.g., certain languages or cultures are under-represented due to social or economic reasons). Such bias is more suitable to be handled during fine-tuning (as an example experiment described below) as tuning another upsampler for low-resource languages at pre-training stage may over-complicate the data algorithm and reduce generalizability. Meanwhile, we also observe a broader challenge is that current benchmarks may not fully capture the bias of low-resource languages (see more discussion in the section of “Limitation on Benchmark” in supplementary materials), and we believe it takes community efforts to improve iteratively (i.e., better data algorithm such as this work encourages better benchmarks, which further motivate more improvement in algorithms).\n\nTo show the feasibility of handling the bias with fine-tuning, we design an experiment where we use the pre-trained MetaCLIP worldwide model, and further fine-tune the model on a dataset emphasizing low-resource languages. We downsample each of the top-10 languages in the training set (comprising 94% training data) to the same portion as the 11th language. With just 5k steps of fine-tuning, we see the following boost on low-resource languages from CVQA:\n| Language | Before | After |\n|:------------|:--------:|:-------:|\n| Bulgarian  | 57.1   | 58.5  |\n| Malay      | 62.9   | 67.6  |\n| Igbo       | 34.0   | 42.0  |\n| Oromo      | 29.4   | 36.4  |\n| Hindi      | 66.2   | 70.6  |\n| Sinhala    | 43.1   | 46.7  |\n| Mongolian  | 33.3   | 34.3  |\n\nWe further ablate various ratios (3%, 6%, 10%) for building worldwide data with B/32 (due to time constraint in rebuttal, we can only work on B/32 scale and the full 29B training schedule won’t fit into this period). \n\n| p    | Babel-IN | XM3600        | CVQA (EN/LOCAL)  |\n|:-----:|:---------:|:--------------:|:----------------:|\n| 3%   | 33.7     | 41.2 / 53.7   | 51.0 / 48.1      |\n| 6%   | 33.3     | 41.6 / 53.9   | 50.4 / 47.7      |\n| 10%  | 33.0     | 41.6 / 53.7   | 50.3 / 48.4      | \n\nAs discussed above, we can see there is no common trend in downstream (Babel-IN, XM3600, CVQA) performance when tuning the ratio in different directions, which suggests more sophisticated tuning is required for various downstream use cases. Such tuning is more suitable to be applied at the fine-tuning stage."}, {"review_id": "sKoFv774j9", "reviewer": "Reviewer_wi16", "summary": "This paper presents a worldwide scaling approach for CLIP training, extending MetaCLIP's methodology from 500k english entries to 27M entries covering 300+ languages. Current CLIP models lack proper curation processes for non-English content or rely on models trained on private data sources. The authors address the curse of multilinguality in vision-language models and propose a curation framework for non-English data.", "questions": "- The curse of multilinguality raises fundemental questions about the design choices of CLIP models. If multilingual training hurts performance at low scales, would language specific cLIP models outperform a large unified one? This trade-off between specialization and generalization deserves deeper investigation.\n\n- Is the curse of multilinguality also observed for mSigLIP? Or is it specific to the worldwide data proposed by the authors?\n\n- The authors mention that the code and model will be released, bu will the worldwide metadata and the entry probabilities/counts also be released?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strenghts**\n\n- The paper addresses a relevant and timely problem. Current CLIP models lack proper curation processes for non-English content. The scale of the effort is substantial, expanding from 500k to 27M metadata entries represents significant engineering work that could benefit the community. \n- The proposed curation algorithm extends MetaCLIP's head-tail balancing approach to the multilingual settings via a simple yet efficient language-specific threshold derived from english ratios. \n- The identification of the curse of multilinguality is a noteworthy empirical finding.\n\n**Weaknesses**\n\n- The main limitation is the incremental nature of the contribution. While scaling from English to multilingual is valuable, the core methodology closely follows MetaCLIP without significant algorithmic innovations. \n\n- The paper lacks a proper understanding of the curse of multilinguality (either theoretical or via scaling laws). The authors observe the phenomenon but provide limited analysis of why it occurs or how to predict when it breaks. An analysis of embedding spaces, or language intereference would strengthen the work. \n\n- Although the naming of the paper and the scale of the experiments suggest that the paper is from Meta, it would have been preferable if the authors did not put the logo of their company for their method (Figure 2.) which is a breach of anonymosity. Nevertheless, given the level of contribution, i'm leaning towards acceptance.", "was_revised": true, "final_justification": "The authors addressed all my concerns.", "rebuttal": "We sincerely appreciate your comprehensive review and recognition of our work as addressing 'a relevant and timely problem' and highlighting 'The curse of multilinguality raises fundamental questions about the design choices of CLIP models'. Below, we address the insightful questions raised in your comments:\n> “The main limitation is the incremental nature of the contribution. While scaling from English to multilingual is valuable, ... *(removed for length limit)*”\n\nThank you for sharing your thoughts. Our contribution is on scaling CLIP in a very different dimension: from intra-language (English-only CLIP) to inter-language (all languages on the web). The two dimensions are intrinsically orthogonal in underlying assumptions, core research questions, implementation principles and challenges, and applicable methodologies. Existing English-only recipes such as OpenAI CLIP or MetaCLIP are not applicable to the inter-language case. Through identifying the curse of multilinguality, we raise this fundamental question to the multimodal representation learning community and demonstrate that it can be addressed through a minimal yet effective scaling recipe. To the best of our knowledge, this is the first time this issue has been explicitly investigated and resolved within the research community. \n\nWe also provide a detailed description of our approach and will release multilingual artifacts, which is the first time in the community, to benefit future academic research and reduce the engineering burden. Such emphasis on detail and transparency stands in contrast to a growing trend where large labs with substantial resources adopt increasingly opaque practices (especially with regard to data). We regard our work and openness not just as a practice, but a meaningful scientific contribution  enabling future scaling of CLIP and advocating open research.\n\n> \"The paper lacks a proper understanding of the curse of multilinguality (either theoretical or via scaling laws). ... *(removed for length limit)*\"\n\nThis is a profound question. In this work, we focus on investigating the existence of curse at popular vision-language benchmarks and offering minimal viable recipes to break the curse. We believe formally defining the curse of multilinguality, proving the theory, and designing detection mechanisms can be multiple impactful research projects, and thus in the following we provide intuitive explanations to the existence of curse, and preliminary experiments to encourage future research.\n\nThe major root cause for the curse of multilinguality is the lack of model capacity to learn new capabilities (e.g., understanding new concepts, domains, languages). Thus, a plausible way to detect the existence of curse, as you indicated, is examining the model internal status of “language interference”. We hence design a  “gradient conflict” analysis, inspired by PCGrad [1] which was originally proposed for multitask learning, and later extended to multilingual settings on XLM [2], for this purpose. Specifically, we used the XM3600 dataset, which contains image-text pairs in 36 languages, to compute gradients from our model checkpoints and analyze cross-lingual interference. We measured the cosine similarity between gradients from English examples and those from each non-English language, and then averaged these similarities across all non-English languages.  Here, checkpoints are all pre-trained with Worldwide 29B data, and we take the checkpoints at the 16th (midway) and 32nd (final) epochs.\n\n|Checkpoint|MetaCLIP2-L/14|MetaCLIP2-H/14|\n|-|-|-|\n|Midway (Epoch 16)|0.508|0.688|\n|Final (Epoch 32)|0.546|0.697|\n\nAs we can see, the smaller models (L/14) have lower similarity, i.e., more interference and gradient conflicts when models attempt to learn from different languages , than the larger ones (H/14). With more interference, L/14 performs worse on English tasks when trained on multilingual data than on pure English, since the model needs to spend valuable training steps more in reducing the conflicts throughout training, instead of learning semantics from various languages. In contrast, the larger models (H/14) experience less gradient conflicts even at early training stages, which we believe enables the models to focus on learning from English and non-English properly so that knowledge from both sides are integrated and mutually beneficial, and thus break the curse.\n\nCompared to gradient conflict which has the potential to detect the curse, discovering the scaling law behind is a common approach to predict when the curse will happen. We conducted a “tiny” scaling law experiment where, at smaller models, we gradually increase the sizes of training data and search for the first size that leads to the model learning saturated (i.e., the model trained with English and Worldwide data yields the same performance on ImageNet if not saturated yet, or the curse starts appearing if saturated). We found the sizes are \\~800M Worldwide pairs (or \\~320M English pairs) for ViT-B/16 (\\~86M parameters). Combined with our previous experiments where we found ViT-L/14 (\\~307M) saturates with around 2.5B/1B Worldwide/English pairs, we roughly fit these data points to a linear line for the “scaling law”, and obtain the rough estimate for \\~8 pairs per parameter to break the curse. This well predicts that when we scale the training data to match frontier efforts (i.e., 2B English pairs, and correspondingly 5B Worldwide), ViT-H/14 (\\~632M) is needed. Note this estimation is very rough (only 2 data points to fit a linear line) due to time constraint in rebuttal, specific to our experiment setup, and dependent on factors such as data processing, model architectures, and training schedules. All in all, this empirical study explains why the curse of multilinguality impacts smaller models more severely: it reflects a scaling constraint, where the model lacks sufficient capacity to fully absorb and generalize from the multilingual signals present in large-scale training data. \n\nDue to time constraints, these are preliminary studies aiming at offering insights. With the release of our paper, model and code, we hope to enable the community to conduct more comprehensive investigations into the mechanisms underlying the curse of multilinguality in CLIP-style models.\n\n> \"Although the naming of the paper and the scale of the experiments suggest that the paper is from Meta, ... *(removed for length limit)*\"\n\nWe appreciate your attention to maintain a high standard for paper review. We have removed that accordingly in our current/future preprints.\n\n> “The curse of multilinguality raises fundemental questions about the design choices of CLIP models. If multilingual training hurts performance at low scales, would language specific cLIP models outperform a large unified one? This trade-off between specialization and generalization deserves deeper investigation.”\n\nThis is a very insightful question. Existing evidence shows that B16/L16 SigLIP (English only) is worse than SO400M mSigLIP (76.7/80.5 vs. 80.7 on ImageNet, from SigLIP 1/2 paper), while B/16 SigLIP outperforms B/16 mSigLIP (76.7 vs. 75.1 on ImageNet). This suggests, at small scale (i.e., the size smaller than where models start breaking the curse, in this paper, H/14), the specialization helps when the compute/model size is fixed(B/16), but a specialized model(L/16) underperforms a bigger, unified one(SO400M). It’s exactly one of our contributions that we show with quantitative evidence that there exists a critical point (again H/14 in this paper) where at/beyond this point, English and non-English data can start to benefit each other and an unified model can outperform a specialized one with the same size, since the language-specific models may forgo the advantages of cross-lingual optimization. Note that, we don’t have performance for all model sizes for the above comparison (e.g., L16 mSigLIP is missing in the original paper) and level of model sizes (i.e., B16, L16, SO400M) are discretized. This should affect where the exact critical point is, e.g., some models larger than L16 but smaller than H14 might break the curse, but we believe our argument is still valid at different critical points.\n\nAlso, with the unified model above the critical point (i.e., where the curse is broken) being trained,, it can serve as a teacher for distilling smaller unified CLIP models and expects better performance than from scratch (e.g., we quickly put together an experiment to distill B/32 from H/14 during this rebuttal and our initial result shows the distilled B/32 achieves 65.5+% on ImageNet, compared to 64.7% when training B/32 from scratch).\n\n> “Is the curse of multilinguality also observed for mSigLIP? Or is it specific to the worldwide data proposed by the authors?”\n\nYes. As mentioned in the previous question, for B/16, SigLIP (English) and mSigLIP (multilingual) achieve 76.7 and 75.1 accuracy respectively on ImageNet. For SO400M, SigLIP and mSigLIP achieve 82.2 and 80.7 accuracy respectively on ImageNet. Both results show that mSigLIP (trained for multilinguality) yields worse performance in English tasks (ImageNet) than its counterpart (SigLIP).\n\n> “The authors mention that the code and model will be released, bu will the worldwide metadata and the entry probabilities/counts also be released?”\n\nYes, we will release the metadata and their counts and probabilities. We believe releasing these intermediate artifacts greatly reduces engineering burden and encourages open research.\n\n[1] Yu et al., Gradient Surgery for Multi-Task Learning, NeurIPS 2020  \n[2] Wang et al., On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment, EMNLP 2020"}, {"review_id": "fRWHnyRe20", "reviewer": "Reviewer_HT2c", "summary": "This paper addresses an important challenge in CLIP models, where training on real-world multilingual data typically degrades English performance compared to English-only models, preventing effective utilization of worldwide datasets. The authors propose a successful recipe for training CLIP from scratch on native worldwide image-text pairs across 300+ languages. The main approach includes 3 key components: (1) scaling metadata from 500k English entries to 27M multilingual entries, (2) implementing language-specific curation algorithms with adaptive thresholds, and (3) designing a training framework that scales seen pairs and requires sufficient model capacity. On the experimental side, the authors also provide comprehensive validation on various tasks and achieved new state-of-the-art results on multilingual benchmarks.", "questions": "Please try to address my question in the above Weaknesses section.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths \n- This work addresses a fundamental limitation in current CLIP training that wastes a large portion of worldwide web multilingual data, e.g., English-only models perform better than multilingually trained data.\n- This work proposes a simple yet effective recipe by metadata scaling, curation algorithm, and training framework to achieve success with CLIP models (trained from scratch) that outperform existing models in various benchmarks, such as multiple English and multilingual benchmarks.\n- The overall writing and description is straightforward and easy to follow. The authors have well discussed some of the limitations such as a large money footprint for multilingual tokenizers.\n\nWeaknesses \n- Despite the diverse languages included in the dataset, I wonder if some languages are truly helpful for learning better multilingual capabilities. For example, if language A covers concept X and language B also covers concept X, I understand how the model positively learns from both languages A and B for concept X through this shared knowledge. But would scenarios where one language is missing some knowledge also be beneficial? I am curious to hear the authors' insights.\n- It would be useful for many downstream language-specific tasks to see which languages perform best/worst and how this correlates with training data volume across multilingual benchmarks.\n- I wonder how the authors handle cases with text in images, such as images with text in language A but where the alt-text is in a different language B, and in some cases they are not well aligned.", "was_revised": true, "final_justification": "I have read the authors' rebuttal, and most of my concerns, specifically related to multilingual benchmarks and language-specific performance are properly addressed, and I don't have further questions.", "rebuttal": "We sincerely appreciate your dedicated review and recognition of our work as addressing 'a fundamental limitation in current CLIP training' and for describing our approach as 'a simple yet effective recipe'. We address your profound questions as following:\n> “Despite the diverse languages included in the dataset, I wonder if some languages are truly helpful for learning better multilingual capabilities. For example, if language A covers concept X and language B also covers concept X, I understand how the model positively learns from both languages A and B for concept X through this shared knowledge. But would scenarios where one language is missing some knowledge also be beneficial?. I am curious to hear the authors' insights.”:\n\nYes, we do see that missing concepts in one language can be recovered from adjacent knowledge in other languages. We believe this is one of the major reasons to break the curse of multilinguality (c.f., table 1 in the main paper where Worldwide MetaCLIP H/14 outperforms English only MetaCLIP in English benchmarks, and outperforms non-english only counterpart in multilingual benchmarks).\n\nWe design an experiment to further illustrate how concepts in one language benefit understanding in other languages. We take images from CVQA and prompt in English the H/14 worldwide and English-only models respectively to examine their image understanding capability. We found that the model trained with worldwide data performs better when the images are about culture or region specific concepts, such as local food, transportation methods, or cultural activities. For example:\n* There is one image in CVQA that shows “Tankwa” (boat-like transportation from Ethiopia). The worldwide model correctly recognizes it as a method of water transportation, while the English-only model predicts it as wheeled vehicles like bikes. (Here we prompted the model in random order with: water transportation, wheeled vehicles, aircrafts, or rails)\n* One image shows the full body of “shachihoko” (a mythical Japanese creature with a tiger head and carp body). The worldwide model correctly predicts it as a creature in Japanese folklore; the English-only model predicts a carp. (prompt: creature in Japanese folklore, dragon, carp, tiger)\n* One image shows “Dragon's beard candy” (a Chinese confection made from stretched fine yellow strands of maltose syrup, with strands drawing various patterns). The worldwide model correctly predicts it was made of sugar; the English-only model predicts glass. (prompt: sugar, silk, glass, silicon)\n\nNote that the experiment above is just for extracting some insight but not rigorous evaluation. We believe concepts are hierarchical and continuous, so in almost all cases there is some overlap between concepts (e.g., both water transportation and wheeled vehicles are transportation mechanisms, so both the worldwide and English models get the concept right one layer above). Rigorous evaluation requires careful design of the examples/benchmarks with deep understanding of concept hierarchy.\n\n> “It would be useful for many downstream language-specific tasks to see which languages perform best/worst and how this correlates with training data volume across multilingual benchmarks.”: \n\nThis is a very insightful suggestion. In the following, we summarize downstream task performance in XM3600 for languages from the top-10 languages occurring in training data versus the rest:\n\nTop-10 language in training data\n| Language | Text-to-Image | Image-to-Text |\n|:----------:|:----------------:|:----------------:|\n| en       | 51.6           | 62.2           |\n| es       | 57.2           | 72.5           |\n| fr       | 67.1           | 78.5           |\n| zh       | 61.1           | 72.6           |\n| ru       | 67.8           | 79.9           |\n| ja       | 65.1           | 79.9           |\n| id       | 65.8           | 78.3           |\n| pt       | 60.4           | 72.6           |\n| de       | 69.2           | 83.6           |\n| vi       | 61.1           | 76.2           |\n\nLanguages not in the top-10 training languages:\n\n| Language | Text-to-Image | Image-to-Text |\n|:-----------:|:----------------:|:----------------:|\n| ar       | 47.4           | 60.8           |\n| bn       | 39.4           | 47.1           |\n| cs       | 51.0           | 66.1           |\n| da       | 61.0           | 75.1           |\n| el       | 52.1           | 68.4           |\n| fa       | 56.9           | 70.3           |\n| fi       | 59.3           | 73.7           |\n| fil      | 24.8           | 36.7           |\n| hi       | 26.1           | 41.8           |\n| hr       | 57.3           | 72.9           |\n| hu       | 63.9           | 76.5           |\n| it       | 64.0           | 78.2           |\n| he       | 60.8           | 76.2           |\n| ko       | 54.8           | 70.1           |\n| mi       | 0.5            | 1.2            |\n| nl       | 53.2           | 66.9           |\n| no       | 57.7           | 73.2           |\n| pl       | 61.4           | 75.9           |\n| quz      | 2.5            | 6.5            |\n| ro       | 64.8           | 77.8           |\n| sv       | 57.6           | 73.8           |\n| sw       | 10.0           | 16.6           |\n| te       | 26.1           | 37.1           |\n| th       | 57.7           | 71.4           |\n| tr       | 55.7           | 68.4           |\n| uk       | 60.0           | 74.7           |\n\nWe do see correlation to some extent, e.g., in general the performance from Top-10 languages is higher than 50%, whereas performance of many languages not in the Top-10 (e.g., bn, hi) is lower than 50%. However, the performance is not strictly decided only by volume. For example, we see English, even though with the largest volume, is not the best performing language (which is de) and worse than many others; we also see many non Top-10 languages also achieve 50%+ performance.\n\nBeyond the volume of available data, we believe there are several other factors  affecting performance: (1) the linguistic and cultural proximity to other languages, and (2) the structural characteristics or expressiveness of the language itself. \n\n> “I wonder how the authors handle cases with text in images, such as images with text in language A but where the alt-text is in a different language B, and in some cases they are not well aligned.”:\n\nThis is a great question. We believe that such data is present with a noteworthy amount in our training set due to our Internet scale of data volume. Our data curation process retains such data deliberately, as 1) this type of data enables cross-modal translation capability in our model as by-product, and 2) designing special processing (e.g., OCR and translation to extract the text and check alignment with alt-text) may overcomplicate the data algorithm which hinders generalizability and reusability, and introduces unbounded biases to distribution of curated data. To validate our first argument, we conducted the following preliminary experiments. We take an image which visually showing the Chinese character “狗”, which means dog. Then, we compute cosine similarity with different concepts to classify the image, and show the results (classes and similarities) in the following. As we can see, as expected the class for  “狗” (“dog” character) gets highest cosine similarity. Interestingly, “dog” tops the candidates in English, and “いぬ” (dog in Japanese) tops in Japanese and gets much higher cosine similarity than “dog” in English, probably due to the higher language correlation between Chinese and Japanese. These facts suggest our models do pick up cross-modal translation capabilities from the data, even with potential noises and mis-alignment between OCR texts and alt-texts. We believe at Internet scale, existence of such noises in data is inevitable, but being faithful to the natural distribution mitigates most noises.\n| Word     | Description                                      | Score        |\n|:---------:|:----------------------------------------------:|:----:|\n| 狗       | \"dog\" in Chinese (exactly visualized on image)         | 0.5432458    |\n| 犬       | \"dog\" in Chinese, literary/ancient usage         | 0.04636497   |\n| 猫       | \"cat\" in Chinese                                 | 0.0002509505 |\n| 豺       | \"jackal\" or \"wild dog\" in Chinese                | 0.03426899   |\n| 狼       | \"wolf\" in Chinese                                | 0.014051365  |\n| dog      | English \"dog\"                                    | 0.08239085   |\n| diagram  | Unrelated word                                   | 0.0014264605 |\n| cat      | English \"cat\"                                    | 0.00005071296|\n| puppy    | English \"puppy\"                                  | 0.028257972  |\n| hound    | English \"hound\"                                  | 0.055857323  |\n| いぬ     | \"dog\" in Japanese                                 | 0.19319624   |\n| ねこ     | \"cat\" in Japanese                                 | 0.0006383567 |"}, {"review_id": "vxh41XjoN0", "reviewer": "Reviewer_idkx", "summary": "This paper presents MetaCLIP 2, a comprehensive \"recipe\" for training Contrastive Language-Image Pre-training (CLIP) models on web-scale, worldwide (i.e., multilingual) data from scratch. The authors identify and address two key challenges in scaling CLIP beyond English:\n1. The lack of a transparent, scalable data curation method for non-English data.\n2. The \"curse of multilinguality,\" where adding non-English data often degrades the model's performance on English-centric benchmarks.\n\nThe proposed recipe consists of three main innovations:\n1. Worldwide Metadata: Scaling the metadata used for curation from English-only sources (WordNet, Wikipedia) to their multilingual equivalents, covering over 300 languages.\n2. Worldwide Curation Algorithm: A language-aware curation process that performs per-language substring matching and balancing. A key contribution here is a method to dynamically determine a balancing threshold (t_lang) for each language to ensure a consistent ratio of head-to-tail concepts across all languages.\n3. Worldwide Training Framework: A training methodology that scales the number of seen image-text pairs proportionally to the increase in data size from non-English sources and identifies a \"minimal viable model capacity\" (ViT-H/14) required to overcome the curse of multilinguality.\n\nThrough extensive experiments, the authors demonstrate that their full recipe not only mitigates the curse of multilinguality but reverses it, showing that English and non-English data can be mutually beneficial. The resulting MetaCLIP 2 ViT-H/14 model surpasses its English-only counterpart on ImageNet and sets new state-of-the-art results on several challenging multilingual benchmarks, all without relying on private data, machine translation, or model distillation.", "questions": "1. The finding that ViT-H/14 is the \"inflection point\" to break the curse is fascinating. Could you speculate further on the underlying reasons? Is it purely a matter of parameter count for storing diverse linguistic and visual concepts, or might it be related to other architectural properties of larger ViT models? Have you considered if alternative strategies, such as extending the training duration for the ViT-L/14 model, could potentially yield similar benefits?\n\n2. The proportional scaling of seen pairs (2.3x) appears to be a crucial ingredient in the recipe. This is justified by maintaining the number of seen English pairs. Was this scaling factor ablated? It would be insightful to know if there's a sensitivity to this hyperparameter—for example, if a 1.5x scaling is insufficient or if a 3.0x scaling offers diminishing returns or even harm.\n\n3. Regarding the multilingual tokenizer (Table 3), the 900k-token XLM-V performs best. This is quite large compared to standard tokenizers. Could you comment on the performance drop if a more moderately sized (e.g., ~250k) but still robust multilingual tokenizer was used with the final ViT-H/14 setup? How critical is this massive vocabulary to the overall success of the recipe?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Significance and Timeliness of the Problem: The paper tackles a highly relevant and critical problem. As English-language data for training foundation models approaches exhaustion, developing principled methods for leveraging the vast, multilingual web is essential for future scaling. This work provides a concrete and reproducible path forward.\n2. Principled and Transparent Methodology: The \"recipe\" framing is a major strength. Instead of an ad-hoc collection of tricks, the authors present a systematic, step-by-step approach built upon the well-understood MetaCLIP framework. The commitment to using only public data (Common Crawl) and avoiding black-box filters or distillation is a significant contribution to transparency and reproducibility in the field. The method for deriving language-specific balancing thresholds (t_lang) is particularly clever and well-motivated.\n3. Rigorous and Comprehensive Experimentation: The experimental validation is exceptionally thorough.\n4. Clarity and Presentation: The paper is very well-written and easy to follow. \n\nWeaknesses:\n1. The \"Curse\" is Broken Only at a Very Large Scale: The pivotal finding of the paper is that the curse of multilinguality is an issue of scale. However, this breakthrough is only achieved with a ViT-H/14 model, which is computationally very expensive and inaccessible to most of the academic community. The fact that the curse persists even with a ViT-L/14 model (the largest size in the original CLIP paper) is a significant limitation. While the authors are transparent about this, it does temper the immediate practical applicability of the \"recipe\" for those with limited resources.\n2. Complexity of the Full Recipe: While the individual components are well-explained, the full recipe is non-trivial to implement. It requires constructing massive multilingual metadata, running language identification, performing efficient large-scale substring matching, and managing a complex data balancing scheme across hundreds of languages. The best-performing tokenizer also has a very large vocabulary (900k), which adds to the memory and computational overhead. While the authors promise to release code, the barrier to entry for reproducing or adapting this work remains high.", "was_revised": true, "final_justification": "I have carefully considered the paper and the authors' rebuttal.\nThe authors have thoughtfully addressed my concerns.\nAfter thorough evaluation, I have raised my recommendation to Accept (5).", "rebuttal": "We sincerely appreciate your thorough review and recognition of our work in areas such as 'Significance and Timeliness of the Problem', 'Principled and Transparent Methodology', and 'Rigorous and Comprehensive Experimentation'. Below, we address the insightful questions you raised:\n\n> “The \"Curse\" is Broken Only at a Very Large Scale: The pivotal finding of the paper is that the curse of multilinguality is an issue of scale. ... *(removed for length limit)*”\n\nOur primary goal is to study the existence of the curse of multilinguality, and present a proof-of-concept showing the curse can be broken with common and standard setting in the community (e.g., OpenAI CLIP training setup, 2+ billion English training data). ViT-H/14 marks the first critical point found to break the curse. We choose this standard setup for fair comparison across existing data algorithms and generalizability of our findings. With the critical point we found, this work offers important insights and new opportunities for future research: (1) scaling a multilingual CLIP from scratch with small model capacity is impractical to match existing training scale for English; and (2) the work offering a ViT-H/14 model that no longer suffers from the multilinguality curse to enable alternative and more practical approaches, such as model distillation to effectively train smaller models (e.g., we quickly put together an experiment to distill B/32 from H/14 during this rebuttal and our initial result shows the distilled B/32 achieves 65.5+% on ImageNet, compared to 64.7% when training B/32 from scratch). Releasing our model and data algorithms is the 0 to 1 step to unlock all these future directions.\n\nWe also want to point out that the Standard OpenAI CLIP training setup is not the most compact way of using model capacity and thus it’s very likely more efficient learning algorithms and model architectures can be developed from our work, and the curse may be broken with smaller model sizes.\n\n> “Complexity of the Full Recipe: While the individual components are well-explained, the full recipe is non-trivial to implement. ... *(removed for length limit)*” \n\nThis is a great point. The scaling from intra-language (e.g., English only) to inter-language (worldwide) naturally introduces non-trivial complexity. One of our major contributions is to significantly reduce this complexity with a minimal recipe for this scaling, and making it transparent and accessible to the research community, so that the entry barrier of multilingual CLIP can be greatly reduced. Specifically, we will release our preprocessed metadata so that the community can avoid re-processing large corpora like Wikipedia; we adopt the Aho–Corasick algorithm to accelerate substring matching, which is a preprocessing computation bottleneck, by over 2000 times, and we open source our implementation. Furthermore, by including non-English data in CLIP training, we offer a compact solution to only train one model with a 1.3x increase of its original data volume, but achieving competitive performance in over 300+ languages. On the other hand, this work is part of the recent community endeavors of building foundation models (e.g., Llama, Qwen, SAM, SigLIP). Developing data for foundation is intrinsically complicated. For example, several components in our algorithm, such as language identification (LID), are common across other foundation models even for English-only CLIP or LLMs. \n\nExploring the choice of tokenizers (Table 3 in main paper) and later understanding the limitation (in appendix) is also important to reduce the complexity. We believe distilling smaller tokenizers, which is also discussed in our limitation section in appendix, can greatly improve compactness while maximally preserving performance. Our release of model weights will greatly accelerate the community development on this front. \n\nLastly, in contrast to the recent trends where big labs turn into private mode (e.g., only release model API or opaque about their implementation), we believe our open source efforts, including models, code, intermediate artifacts, significantly reduce the inherent complexity of scaling to worldwide data. By making our data and methods available now, we enable not only current research but we envision to work with the community hand-in-hand on other critical challenges including complexity.\n\n> “The finding that ViT-H/14 is the \"inflection point\" to break the curse is fascinating. ... *(removed for length limit)*”\n\nYes, the major root for curse is the parameter count, or model capacity. The more than doubled data volume, after extending English only data to worldwide, contains much more information (as you mentioned, the diverse linguistic and visual concepts) than a L/14 can hold. Our finding is based on rigorous ablation with the English counterpart trained via standard OpenAI recipe, which is deliberately selected for our conclusion being generalizable. Extending the duration of the worldwide model training alone deviates from the recipe and breaks the ablation: training with more English seen pairs in worldwide data should be compared with an English-only model also with more seen pairs, which is likely also achieving better performance. However, we do see if we relax the constraint on recipe (e.g., data distribution, model architectures, training setup), there should be room for efficiency such as downsample the English examples in data to make room for non-English concepts, model pruning/distillation, more sophisticated design of learning rate, training schedule or losses, or combinations of these. With such a setting, curse might be broken at smaller model size (e.g., the ViT-L/14 that is popular in community), but curse always exists model capacity always kicks in to decide how much a model can learn.\n\nFurthermore, there’s a trend of linear scaling law found in a “tiny” experiment we conducted. In the experiment, with a smaller models, we gradually increase the sizes of training data and search for the first size that leads to the model learning saturated (i.e., the model trained with English and Worldwide data yields the same performance on ImageNet if not saturated yet, or the curse starts appearing if saturated). We found the sizes are \\~800M Worldwide pairs (or \\~320M English pairs) for ViT-B/16 (\\~86M parameters). Combined with our previous experiments where we found ViT-L/14 (\\~307M) saturates with around 2.5B/1B Worldwide/English pairs, we roughly fit these data points to a linear line for the “scaling law”, and obtain the rough estimate for \\~8 pairs per parameter to break the curse. This well predicts that when we scale the training data to match frontier efforts (i.e., 2B English pairs, and correspondingly 5B Worldwide), ViT-H/14 (\\~632M) is needed. Note this estimation is very rough (only 2 data points to fit a linear line) due to time constraint in rebuttal, specific to our experiment setup, and dependent on factors such as data processing, model architectures, and training schedules. All in all, this empirical study explains why the curse of multilinguality impacts smaller models more severely: it reflects a scaling constraint, where the model lacks sufficient capacity to fully absorb and generalize from the multilingual signals present in large-scale training data.\n\n> “The proportional scaling of seen pairs (2.3x) appears to be a crucial ingredient in the recipe. ... *(removed for length limit)*” \n\nWe did conduct such ablation. We experimented with both 1.5× and 3.0× scaling for H/14, and observed that 1.5x just break the curse (c.f., English only H/14 with 80.4 IN) and 3.0x consistently yields more gain:\n\n|Scaling Proportion|ImageNet|Babel-IN|XM3600|CVQA|\n|-|-|-|-|-|\n|1.5x| 80.5|50.0|63.1|56.6|\n|2.3x| 81.3|50.2|64.3|57.4|\n|3.0x| 81.4|50.4|64.5|57.9|\n\nOne thing to note is that this ratio is a constant decided by the English vs non-English ratio in the data pool. We hold this constant to make the worldwide model comparable with the English model. Changing the ratio (as a hyperparameter) downsamples or upsamples English data and thus changes the English baseline. One can still tune it as a hyperparameter as we showed here based on their downstream use cases to emphasize English or non-English performance; this reduces generalizability that this paper aims to hold.\n\n> “Regarding the multilingual tokenizer (Table 3), the 900k-token XLM-V performs best. ... *(removed for length limit)*”\n\nThis is a great question. We did start with moderate sized tokenizers but found suboptimal results. Since we work on worldwide scaled (300+) languages, smaller tokenizers are not able to comprehensively cover all languages to deduct representative multilingual text embeddings, and thus the model shows compromised results in a subset of languages . For example, we trained a ViT-H/14 with an mT5 tokenizer (250k tokens), which is used by mSigLIP. The English performance is the same as our ViT-H/14 with the 900k tokenizer. However, the model drops in Babel-IN accuracy by 1.4% (48.8%) and XM3600 by 2% (62.2%). The drop is not consistent across all languages. In fact, as mentioned above, we do see there is plenty of room to make our recipe more compact due to our emphasis on generalizability and the standard OpenAI CLIP setting. So, we do expect one can distill more compact token embedding and smaller tokenizers with our released model, and we will also release our findings with distilling smaller tokenizers in the final version of this work to show the feasibility of this path."}, {"review_id": "2vVzMRNkSY", "reviewer": "Reviewer_o5PA", "summary": "- The authors have promoted the research on multilingual CLIP, curated global data, and trained standard CLIP towers, ensuring the models' effectiveness and ease of use.\n- They provide detailed description of their data curation pipe.", "questions": "- Is the multilingual data currently collectable by Common Crawl sufficiently representative? I assume that even if we do not discard the non-English parts of the data collected from Common Crawl, the non-English content may still have significant bias. I would like to ask the authors how they view this situation.\n\n- It seems challenging to treat data in different languages equally. The population gaps between different languages are substantial, and the amount of data produced is highly correlated with factors such as population and regional economy. This seems to form a long-tail phenomenon at the multilingual level. I would like to ask the authors how they view the inequality (long-tail) of corpus across different languages.", "limitations": "- It seems that due to considerations of collection difficulty, policies, and licenses, the authors chose data sources like Common Crawl (consistent with the multilingual version of LAION) and so on. However, I believe there exists a big amount of data that is not included in Common Crawl but can better reflect the characteristics of the worldwide context. Many data from platforms actually used by users in their countries remain unadopted.\n\n- I am not sure whether the author has had the opportunity to communicate with users of other languages. I am concerned that the current metrics may not be sufficient to directly reflect how users of other languages feel about the model. When I conduct data research myself, I feel anxious that members of our team can basically only speak no more than three languages. I wonder whether the authors can see the changes of the proposed model in other languages beyond the hard metrics (because I think that especially for long-tail languages, the changes brought by worldwide recipes should be quite significant).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "In my opinion, this article is of great significance. In recent years, our visual language models have consistently focused on a few languages used by a huge number of users. Promoting their scaling to a worldwide level can help artificial intelligence move towards globalization and acquire more extensive world knowledge, etc.\n\n- Findings about the curse of multilinguality (Multilinguality CLIP models get worse English performance than English-only CLIP).\n- A detailed data pipe, the pseudo code is clear.\n- The experiment is well organized.\n\nWeakness:\n\n- It is still not well explained about why the ViT-H break the curse of multilinguality but ViT-L suffers? Is this wholely because of model size? Is there a threshold to judge whether a model suffers from the curse.\n- Due to the curse of multilinguality, the paper mainly focuses on large models. But the ViT-L is still applied by much downstream models (such as many VLMs like [LLaVA](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b/blob/main/config.json#L51)).", "was_revised": true, "final_justification": "I keep the score and confidence. The authors have resolved all questions, including data sources, evaluations, and the details about 'the curse of multilinguality'. Besides, they share their opinions about language bias.", "rebuttal": "We sincerely appreciate your thorough feedback and thoughtful comments, including recognizing that 'this article is of great significance,' that 'promoting scaling to a worldwide level can help artificial intelligence move toward globalization and acquire broader world knowledge,' and that 'the experiment is well organized.' Below, we address the profound questions you raised:\n\n> “It is still not well explained about why the ViT-H break the curse of multilinguality but ViT-L suffers? ... *(removed for length limit)*”\n\nYes, it is because the model size cannot hold the added information from non-English training distribution. To predict when and whether a model suffers from the curse, we conducted a “tiny” scaling law experiment. In the experiment, with a smaller model, we gradually increase the sizes of training data and search for the first size that leads to the model learning saturated (i.e., the model trained with English and Worldwide data yields the same performance on ImageNet if not saturated yet, or the curse starts appearing if saturated). We found the sizes are \\~800M Worldwide pairs (or \\~320M English pairs) for ViT-B/16 (\\~86M parameters). Combined with our previous experiments where we found ViT-L/14 (\\~307M) saturates with around 2.5B/1B Worldwide/English pairs, and 2.5B/1B Worldwide/English pairs for ViT-L/14 (\\~307M) roughly fit these data points to a linear line for the “scaling law”, and obtain the rough estimate for ~8 pairs per parameter to break the curse. This well predicts that when we scale the training data to match frontier efforts (i.e., 2B English pairs, and correspondingly 5B Worldwide), ViT-H/14 (\\~632M) is needed. Note this estimation is very rough (only 2 data points to fit a linear line) due to time constraint in rebuttal, specific to our experiment setup, and dependent on factors such as data processing, model architectures, and training schedules. All in all, this empirical study explains why the curse of multilinguality impacts smaller models more severely: it reflects a scaling constraint, where the model lacks sufficient capacity to fully absorb and generalize from the multilingual signals present in large-scale training data.\n\n> “Due to the curse of multilinguality, the paper mainly focuses on large models. ... *(removed for length limit)*” \n\nOur primary goal is to study the existence of the curse of multilinguality and identify a critical point at which the curse can be broken. Our work serves as a proof of concept using the standard OpenAI CLIP training setup—where English data has already reached a scale of over 2 billion samples. This work offers valuable guidance and new opportunities for future research: (1) it suggests avoiding training smaller models with multilingual data from scratch due to model capacity; and (2) it offers a ViT-H/14 model that no longer suffers from the multilinguality curse to enable alternative approaches, such as model distillation to effectively train smaller models. (e.g., we quickly put together an experiment to distill B/32 from H/14 during this rebuttal and our initial result shows the distilled B/32 achieves 65.5+% on ImageNet, compared to 64.7% when training B/32 from scratch).  Releasing our model and data algorithms is the 0 to 1 step to unlock all these future directions.\n\nAlso note that, we adopt the standard OpenAI CLIP training setup in our recipe to explore the critical point for our findings being generalizable. The training setup is definitely not the most compact way of using model capacity and thus it’s very likely more efficient learning algorithms and model architectures can be developed from our work, and the curse will be broken with smaller model sizes. \n\n> “Is the multilingual data currently collectable by Common Crawl sufficiently representative? ... *(removed for length limit)*”:\n\nOne of our major goals is to promote open research, and selecting a CommonCrawl-based data pool enables the community to reproduce and compare results using open models. Similar to our selection of OpenAI training setup, we acknowledge that our CommonCrawl data pool is not optimal in terms of language and content biases (e.g., CommonCrawl crawler may be built mostly by developers from the Western world, the popularity of content in different languages may be related to social or economic reasons that are not ideal for models to pick up). We believe this is an important question and potentially multiple valuable research projects can grow from. For example, crawler or data sources other than CommonCrawl can be used to augment the data pool and control/mitigate biases introduced by CommonCrawl. However, doing so adds non-trivial complexity in science, engineering, legal compliance etc., and raises the barriers for open research. We consider our open model and data algorithms to serve as a baseline and foundation to spark such research.\n\n> “It seems challenging to treat data in different languages equally. ... *(removed for length limit)*”\n\nThis is a great question. During the pre-training stage, the target is to enable models to learn general capability and comprehensive knowledge, and our goal here is to present a proof-of-concept recipe of breaking the curse of multilinguality. Thus, we design our algorithm so that the language distribution in pre-training data is minimally manipulated and reflect the true language distribution observed in the real world (e.g., ratio of English vs. non-English speakers or occurrences of examples). Although the observation is approximated by the web traffic, which is not ideal, we believe such scaling with the web introduces minimal inductive biases, helps resolving other biases (such as social-economic ones as you pointed out), and enables the pre-trained model to learn general capabilities. See responses to the next question for more related thoughts.\n\nOur data distribution may not be optimal for long-tailed languages, which are affected by inequality, bias, or other reasons as you indicated. We believe it’s more suitable to handle such skew during fine-tuning (as an example experiment described below). Meanwhile, a broader challenge is that current benchmarks may not fully capture the bias and reflect the true importance of languages in long-tails (see more discussion in the section of “Limitation on Benchmark” in supplementary materials).\n\nTo show the feasibility of handling the inequality and long-tailed languages with fine-tuning, we design an experiment where we use the pre-trained MetaCLIP worldwide model, and further fine-tune the model on a dataset emphasizing  long-tailed languages (we downsample top-10 languages in training set to the same portion as the 11th’s language). With just 5k steps of fine-tuning, we see the following boost on low-resource languages from CVQA\n|Language|Before|After|\n|-|-|-|\n|Bulgarian|57.1|58.5|\n|Malay|62.9|67.6|\n|Igbo|34.0|42.0|\n|Oromo|29.4|36.4|\n|Hindi|66.2|70.6|\n|Sinhala|43.1|46.7|\n|Mongolian|33.3|34.3|\n\n> “I am not sure whether the author has had the opportunity to communicate with users of other languages. ... *(removed for length limit)*”\n\nThat's a very good point. As noted in our supplementary material, we are concerned that many current benchmarks rely on translating English data into non-English languages, which fails to capture the cultural diversity inherent to native non-English data and thus limits a full evaluation of our model’s capabilities. We believe establishing a more comprehensive benchmark for long-tail languages is very needed. While creating such benchmarks is beyond the scope of this work, our pretraining recipe is designed with minimal assumptions, that are usually introduced with multiple hand-crafted or model-based filtering mechanisms, to avoid introducing bias. Also, we believe our work encourages the community to build more accurate benchmarks for long-tail languages and facilitate development for downstream use cases in different languages by fine-tuning our pre-trained models.\n\nAlso we appreciate your sharing of the anxiety on researchers’ bias and we had a similar feeling from our previous experience on domain biases (e.g., in designing data curation algorithms, we had explored many assumptions to better cover vast and diverse human knowledge, from one extreme of enumerating data sources to diversify domains in data to another where “no filtering” philosophy was adopted [1]). We learned from the bitter lesson that often it’s better for researchers to have the least involvement in assumption making, which introduces inductive biases of particular interests, and to leave the de-biasing and intelligence boost to scaling (i.e., the wisdom of the crowd). We found that in many cases, proper scaling itself should automatically resolve most biases. Thus, not knowing all languages gave researchers the advantage to avoid designing methods for particular languages and to focus efforts on scaling. On the contrary, building benchmarks should require more domain expertise and be more vulnerable to such biases. That’s why we called out the community should invest more on evaluations.\n\n[1] Pouget, Angéline, et al. \"No filter: Cultural and socioeconomic diversity in contrastive vision-language models.\" Advances in Neural Information Processing Systems 37 (2024)."}]

Author rebuttal:

Meta-review: The paper presents a recipe for training a CLIP model on worldwide, multilingual, web-scale data. By scaling the model capacity and training data the "curse of multilinguality" can be overcome. The resulting model surpasses the English-only variant and achieves several new state-of-the-art results. 

Strong and thorough experimental results are highlighted by the reviewers. Further strengths include addressing the important problem of how to effectively leverage the vast amount of non-English data on the web. Clarity and presentation were praised across reviews.

Transparent methods and code will be of interest and value to the research community.

One weakness raised by several reviewers is that the benefits are mostly demonstrated at large model scale. In addition, the method adds engineering complexity. 

Reviews, rebuttals, and the discussion period were used well.

I strongly recommend to **accept** the paper to NeurIPS, the paper is solid, the results are highly relevant, and there is **agreement to accept among all reviewers**.
Decision: Accept (spotlight)