Title: Meta CLIP 2: A Worldwide Scaling Recipe
Venue: NeurIPS 2025
Accepted: yes
Keywords: CLIP

curation

multimodal

web

multilingual

data curation

pretraining

visual-language

contrastive learning

long-tail distribution

scaling
Abstract: Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary to address the above challenges and present a recipe enabling mutual benefits from English and non-English world data.
In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-of-the-art without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with 64.3% on image-to-text retrieval. Code and model are available at https://github.com/facebookresearch/MetaCLIP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors have promoted the research on multilingual CLIP, curated global data, and trained standard CLIP towers, ensuring the models' effectiveness and ease of use.\n- They provide detailed description of their data curation pipe.\n\nSTRENGTHS AND WEAKNESSES:\nIn my opinion, this article is of great significance. In recent years, our visual language models have consistently focused on a few languages used by a huge number of users. Promoting their scaling to a worldwide level can help artificial intelligence move towards globalization and acquire more extensive world knowledge, etc.\n\n- Findings about the curse of multilinguality (Multilinguality CLIP models get worse English performance than English-only CLIP).\n- A detailed data pipe, the pseudo code is clear.\n- The experiment is well organized.\n\nWeakness:\n\n- It is still not well explained about why the ViT-H break the curse of multilinguality but ViT-L suffers? Is this wholely because of model size? Is there a threshold to judge whether a model suffers from the curse.\n- Due to the curse of multilinguality, the paper mainly focuses on large models. But the ViT-L is still applied by much downstream models (such as many VLMs like [LLaVA](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b/blob/main/config.json#L51)).\n\nQUESTIONS:\n- Is the multilingual data currently collectable by Common Crawl sufficiently representative? I assume that even if we do not discard the non-English parts of the data collected from Common Crawl, the non-English content may still have significant bias. I would like to ask the authors how they view this situation.\n\n- It seems challenging to treat data in different languages equally. The population gaps between different languages are substantial, and the amount of data produced is highly correlated with factors such as population and regional economy. This seems to form a long-tail phenomenon at the multilingual level. I would like to ask the authors how they view the inequality (long-tail) of corpus across different languages.\n\nLIMITATIONS:\n- It seems that due to considerations of collection difficulty, policies, and licenses, the authors chose data sources like Common Crawl (consistent with the multilingual version of LAION) and so on. However, I believe there exists a big amount of data that is not included in Common Crawl but can better reflect the characteristics of the worldwide context. Many data from platforms actually used by users in their countries remain unadopted.\n\n- I am not sure whether the author has had the opportunity to communicate with users of other languages. I am concerned that the current metrics may not be sufficient to directly reflect how users of other languages feel about the model. When I conduct data research myself, I feel anxious that members of our team can basically only speak no more than three languages. I wonder whether the authors can see the changes of the proposed model in other languages beyond the hard metrics (because I think that especially for long-tail languages, the changes brought by worldwide recipes should be quite significant).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MetaCLIP 2, a comprehensive \"recipe\" for training Contrastive Language-Image Pre-training (CLIP) models on web-scale, worldwide (i.e., multilingual) data from scratch. The authors identify and address two key challenges in scaling CLIP beyond English:\n1. The lack of a transparent, scalable data curation method for non-English data.\n2. The \"curse of multilinguality,\" where adding non-English data often degrades the model's performance on English-centric benchmarks.\n\nThe proposed recipe consists of three main innovations:\n1. Worldwide Metadata: Scaling the metadata used for curation from English-only sources (WordNet, Wikipedia) to their multilingual equivalents, covering over 300 languages.\n2. Worldwide Curation Algorithm: A language-aware curation process that performs per-language substring matching and balancing. A key contribution here is a method to dynamically determine a balancing threshold (t_lang) for each language to ensure a consistent ratio of head-to-tail concepts across all languages.\n3. Worldwide Training Framework: A training methodology that scales the number of seen image-text pairs proportionally to the increase in data size from non-English sources and identifies a \"minimal viable model capacity\" (ViT-H/14) required to overcome the curse of multilinguality.\n\nThrough extensive experiments, the authors demonstrate that their full recipe not only mitigates the curse of multilinguality but reverses it, showing that English and non-English data can be mutually beneficial. The resulting MetaCLIP 2 ViT-H/14 model surpasses its English-only counterpart on ImageNet and sets new state-of-the-art results on several challenging multilingual benchmarks, all without relying on private data, machine translation, or model distillation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Significance and Timeliness of the Problem: The paper tackles a highly relevant and critical problem. As English-language data for training foundation models approaches exhaustion, developing principled methods for leveraging the vast, multilingual web is essential for future scaling. This work provides a concrete and reproducible path forward.\n2. Principled and Transparent Methodology: The \"recipe\" framing is a major strength. Instead of an ad-hoc collection of tricks, the authors present a systematic, step-by-step approach built upon the well-understood MetaCLIP framework. The commitment to using only public data (Common Crawl) and avoiding black-box filters or distillation is a significant contribution to transparency and reproducibility in the field. The method for deriving language-specific balancing thresholds (t_lang) is particularly clever and well-motivated.\n3. Rigorous and Comprehensive Experimentation: The experimental validation is exceptionally thorough.\n4. Clarity and Presentation: The paper is very well-written and easy to follow. \n\nWeaknesses:\n1. The \"Curse\" is Broken Only at a Very Large Scale: The pivotal finding of the paper is that the curse of multilinguality is an issue of scale. However, this breakthrough is only achieved with a ViT-H/14 model, which is computationally very expensive and inaccessible to most of the academic community. The fact that the curse persists even with a ViT-L/14 model (the largest size in the original CLIP paper) is a significant limitation. While the authors are transparent about this, it does temper the immediate practical applicability of the \"recipe\" for those with limited resources.\n2. Complexity of the Full Recipe: While the individual components are well-explained, the full recipe is non-trivial to implement. It requires constructing massive multilingual metadata, running language identification, performing efficient large-scale substring matching, and managing a complex data balancing scheme across hundreds of languages. The best-performing tokenizer also has a very large vocabulary (900k), which adds to the memory and computational overhead. While the authors promise to release code, the barrier to entry for reproducing or adapting this work remains high.\n\nQUESTIONS:\n1. The finding that ViT-H/14 is the \"inflection point\" to break the curse is fascinating. Could you speculate further on the underlying reasons? Is it purely a matter of parameter count for storing diverse linguistic and visual concepts, or might it be related to other architectural properties of larger ViT models? Have you considered if alternative strategies, such as extending the training duration for the ViT-L/14 model, could potentially yield similar benefits?\n\n2. The proportional scaling of seen pairs (2.3x) appears to be a crucial ingredient in the recipe. This is justified by maintaining the number of seen English pairs. Was this scaling factor ablated? It would be insightful to know if there's a sensitivity to this hyperparameter—for example, if a 1.5x scaling is insufficient or if a 3.0x scaling offers diminishing returns or even harm.\n\n3. Regarding the multilingual tokenizer (Table 3), the 900k-token XLM-V performs best. This is quite large compared to standard tokenizers. Could you comment on the performance drop if a more moderately sized (e.g., ~250k) but still robust multilingual tokenizer was used with the final ViT-H/14 setup? How critical is this massive vocabulary to the overall success of the recipe?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses an important challenge in CLIP models, where training on real-world multilingual data typically degrades English performance compared to English-only models, preventing effective utilization of worldwide datasets. The authors propose a successful recipe for training CLIP from scratch on native worldwide image-text pairs across 300+ languages. The main approach includes 3 key components: (1) scaling metadata from 500k English entries to 27M multilingual entries, (2) implementing language-specific curation algorithms with adaptive thresholds, and (3) designing a training framework that scales seen pairs and requires sufficient model capacity. On the experimental side, the authors also provide comprehensive validation on various tasks and achieved new state-of-the-art results on multilingual benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- This work addresses a fundamental limitation in current CLIP training that wastes a large portion of worldwide web multilingual data, e.g., English-only models perform better than multilingually trained data.\n- This work proposes a simple yet effective recipe by metadata scaling, curation algorithm, and training framework to achieve success with CLIP models (trained from scratch) that outperform existing models in various benchmarks, such as multiple English and multilingual benchmarks.\n- The overall writing and description is straightforward and easy to follow. The authors have well discussed some of the limitations such as a large money footprint for multilingual tokenizers.\n\nWeaknesses \n- Despite the diverse languages included in the dataset, I wonder if some languages are truly helpful for learning better multilingual capabilities. For example, if language A covers concept X and language B also covers concept X, I understand how the model positively learns from both languages A and B for concept X through this shared knowledge. But would scenarios where one language is missing some knowledge also be beneficial? I am curious to hear the authors' insights.\n- It would be useful for many downstream language-specific tasks to see which languages perform best/worst and how this correlates with training data volume across multilingual benchmarks.\n- I wonder how the authors handle cases with text in images, such as images with text in language A but where the alt-text is in a different language B, and in some cases they are not well aligned.\n\nQUESTIONS:\nPlease try to address my question in the above Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a worldwide scaling approach for CLIP training, extending MetaCLIP's methodology from 500k english entries to 27M entries covering 300+ languages. Current CLIP models lack proper curation processes for non-English content or rely on models trained on private data sources. The authors address the curse of multilinguality in vision-language models and propose a curation framework for non-English data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n\n- The paper addresses a relevant and timely problem. Current CLIP models lack proper curation processes for non-English content. The scale of the effort is substantial, expanding from 500k to 27M metadata entries represents significant engineering work that could benefit the community. \n- The proposed curation algorithm extends MetaCLIP's head-tail balancing approach to the multilingual settings via a simple yet efficient language-specific threshold derived from english ratios. \n- The identification of the curse of multilinguality is a noteworthy empirical finding.\n\n**Weaknesses**\n\n- The main limitation is the incremental nature of the contribution. While scaling from English to multilingual is valuable, the core methodology closely follows MetaCLIP without significant algorithmic innovations. \n\n- The paper lacks a proper understanding of the curse of multilinguality (either theoretical or via scaling laws). The authors observe the phenomenon but provide limited analysis of why it occurs or how to predict when it breaks. An analysis of embedding spaces, or language intereference would strengthen the work. \n\n- Although the naming of the paper and the scale of the experiments suggest that the paper is from Meta, it would have been preferable if the authors did not put the logo of their company for their method (Figure 2.) which is a breach of anonymosity. Nevertheless, given the level of contribution, i'm leaning towards acceptance.\n\nQUESTIONS:\n- The curse of multilinguality raises fundemental questions about the design choices of CLIP models. If multilingual training hurts performance at low scales, would language specific cLIP models outperform a large unified one? This trade-off between specialization and generalization deserves deeper investigation.\n\n- Is the curse of multilinguality also observed for mSigLIP? Or is it specific to the worldwide data proposed by the authors?\n\n- The authors mention that the code and model will be released, bu will the worldwide metadata and the entry probabilities/counts also be released?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes MetaCLIP-2, which is extended from English to multiple languages compared to the previous version of MetaCLIP. This work proposes a dynamic threshold t_lang adjustment method based on English data to balance the concept distribution. It achieves good results on non-English multilingual language benchmarks, while maintaining comparable performace to previous methods on English language tasks. This paper conducts detailed ablation experiments to prove that English data can help improve non-English, and the model performs well in cultural diversity tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**
- MetaCLIP-2 is currently the best multilingual image-text alignment model, which will benefit many domains, such as image generation or cross-regional cultural integration research.

- It provides a complete solution from metadata construction, multilingual training, and dynamic filtering of training data based on the different languages.

- Detailed experimental evaluation and ablation experiments prove its effectiveness.

**Weakness**
- Although the MetaCLIP-2 shows good performance, it is more due to engineering optimization and massive resources, such as data and computing resources, which makes this paper more like a technical report than an academic paper.

- The 6% tail concept ratio is based on the experience of English data and may not be optimal for low-resource languages.

QUESTIONS:
- The 6% tail concept ratio t_en is based on the experience of English data. Has the optimality of this ratio been verified for low-resource languages (such as African languages)? How to alleviate the filtering bias caused by the difference in data volume between languages?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03