Title: Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
Venue: NeurIPS 2025
Accepted: yes
Keywords: scaling laws

open foundation models

open datasets

reproducibility

model comparison

language-vision models

CLIP

MaMMUT
Abstract: In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. Taking language-vision learning as example, we show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. Full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. For the first time, we use derived scaling laws to compare both models and three open datasets, DataComp-1.4B, Re-LAION-1.4B and DFN-1.4B, while ensuring sufficient prediction accuracy on held out points. From comparison, we obtain evidence for (i) MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP (ii) DFN-1.4B outperforming other open datasets. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, classification, retrieval, and segmentation, observing consistently the same scaling trends for models and datasets across tasks. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison on aligned common compute axis across large scale span, avoiding misleading conclusions based on measurements from few isolated single reference scales only. This paves road for guided collective improvement of open foundation models and training datasets, as scaling law based comparisons from various studies executed in common frame can be combined to identify overall better procedures. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/scaling-laws-for-comparison.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates scaling laws for CLIP and MaMMUT across multiple open datasets (Re-LAION-1.4B and DataComp-1.4B). Based on their experiments, the authors find that MaMMUT has both stronger improvement with scale and better sample efficiency than CLIP. They then derive scaling laws for downstream tasks such as classification, retrieval, and segmentation, and show that the observed improvements persist across these applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe paper is clear and well-written. The claims are well supported both by theoretical scaling-law fitting and extensive experiments, with proper methodological rigour. There is great effort spent on ensuring a wide range of parameters are tested across various data regimes. The results will be open-sourced and reproducible, making it useful for the community at large. The release of openMaMMUT-L/14 will provide a good open alternative with performance competitive with proprietary vision-language models.\n\nWeaknesses:\n\nThe scope of this paper is mostly focused on contrastive and captioning objectives – it would be interesting to see results on other alternative architectures and objectives. Evaluation primarily limited to zero-shot settings, with minimal analysis of few-shot learning, fine-tuning transfer, or vision-language understanding tasks. There is also some compute accounting ambiguity, especially when comparing architectures with inherently different parameter counts (Table 12), which raises questions about compute comparison methodology.\n\nQUESTIONS:\n1. The authors limit training to less than 3B samples to avoid repetition effects, but this constrains conclusions about truly large-scale behavior. When trained on 12.8B samples, performance falls below predictions. How can we be certain that this under-performance is actually due to high repetitions?\n\n2. The paper states that SigLIP shows no benefits over CLIP (Section E), contradicting claims in the SigLIP papers. However, this comparison seems a bit limited. It also seems like training hyperparameters such as learning rate, batch size, etc. vary across scales. How were hyperparameters chosen to ensure fair comparison across architectures and scales? Was there a particular methodology for selecting these? How can we be confident that the observed scaling differences reflect architectural properties rather than differences in effort invested in optimization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has widely analyzed scaling laws for vision-language models (VLM), CLIP and MaMMUT, in terms of zero-shot classificaiton, retrieval, and segmentation. The trained VLMs on different massive datasets of DataComp and Re-LAION follow the novel scaling laws in different knowledge from the original paper of scaling laws. Finally, the analyzed VLMs perform well-established zero-shot classification accuracy after the training on a billion-scale dataset.\n\nSTRENGTHS AND WEAKNESSES:\nS1. The paper introduces a great effort to establish novel scaling laws across a couple of scenarios. While maintaining consistency with insights from previously reported scaling laws, the paper additionally covers new findings for the scaling laws in zero-shot image classification, image ritrieval, and image segmentation. The comprehensive analysis derives insights not only for zero-shot image classification but also for zero-shot image retrieval and image segmentation, presenting results that would be valuable to a broader AI community and researchers.\n\nS2. The investigation is conducted with well-deserved strategy. In particular, it appears to have engaged in detailed discussion and try-and-error among themselves regarding the model architectures, datasets, training parameters, and learning strategies for arriving at their results. In terms of experimental design in VLMs, this paper serves as a useful knowledge for a wide readers.\n\nS3. This paper includes excellent trial that the project itself is committed to \"openness.\" Although the computational scale required to explore scaling laws is immense, making reproduction by many researchers challenging, the insights can be publicly and widely shared. Moreover, by open-sourced approaches, the community can directly benefit from the trial. The reviewer believes the paper's execution represents a significant contribution to the AI and realted fields.\n\nS4. The paper shows that the dataset aspect is still tracing a larger scaling. At present, improving and organizing data can still accelerate VLM performance in classification, retrieval, and segmentation. In this point, this paper has the great potential to drive the overall advancement of the AI and related fields.\n\nW1. The paper's constraints are understandable, however, the reviewer believes the paper’s value can be increased if more qualitative evaluations are included in the supplementary material as additional documents. In particular, since this paper uses OpenMaMMUT, which involves not only CLIP-style contrastive learning but also sentence generation, this would be beneficial to show examples of the kinds of captions generated for various images.\n\nW2. Similar contributions such as Molmo & PixMo (CVPR 2025) aim to open-source VLMs. This work compares the accuracy against sophisticated models like GPT and Gemini at the time of paper submission, reporting comparable or slightly superior recognition performance. They also share many practical insights (e.g., introducing overlap among image patches), yielding interesting results. Although this work does not directly affect the contributions of the present paper, discussing their positioning and relationship would enhance paper's overall value.\n\nQUESTIONS:\n1. Although the code release has been mentioned in the paper, what about the pre-trained models? The reviewer assumes these pre-trained models will also be made public. Having models and code available for image classification, image retrieval, and image segmentation would likely accelerate related researches across many subfields.\n\n2. Are there effective pre-training approaches (e.g., parameters and how to pre-train) specialized for zero-shot classification, retrieval, and segmentation, respectively? Or does training on billion-scale datasets describe such specialization less important? If the underlying vision-language tasks share common characteristics, focusing on contrastive learning alone could suffice.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives compute vs performance scaling laws for two open-source language vision procedures CLIP and MaMMUT on two open datasets, DataComp and Re-LAION. They release all the pre-trained models with their intermediate checkpoints and develop their openMaMMUT that achieves 80.3% accuracy on zero-shot ImageNet 1k. When comparing the two models, they use a really dense measurement with $|M| = 15$ model configurations and $|D|=11$ samples configurations. Their contributions are:\n\n- The comparison is fully reproducible and the compute is measured in a dense manner. \n- They find MaMMUT has better scalability than CLIP.\n- Constant learning rate can recover the same trend with less compute. \n- Release all checkpoints, which contribute to the open source community.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has excellent quality, conducted experiments in a dense manner, used significant computing resources, and released all the pre-trained models with their intermediate checkpoints. All the experiments are conducted in a rigorous and reproducible manner. For clarity, there are some typos and inconsistencies mentioned below in the Question section. In terms of significance, I recognize the paper contributes to the open source community in terms of a systematic practice for scaling laws comparison, and release of both checkpoints and open source models. The analysis of the learning rate is also very helpful. However, the question investigated in the paper is specific to the two models and the two datasets. I think more general questions are needed to be investigated, and the details are posted in the Question section. For originality, the paper extends the scaling laws methodology for model selection, and also has its derivative and confidence interval analysis for the power law equation. The conceptual contribution is similar to the original scaling law paper.\n\nQUESTIONS:\n1. In line 216, $\\alpha$ is set to be negative. According to equation (1), it should be positive. Compared with the results in Figure 5, the paper might set $\\alpha$ to be $-\\alpha$ by mistake.\n\n2. Line 62 and Line 186 is conflicting with each other. After looking at the appendix figure 6,7, I found line 62 is correct. This is one of the conclusions of the paper, so please confirm which one is correct.\n\n3. In Figure 8, the paper shows the comparison of SigLIP, CoCa, CLIP, and MaMMUT on the DataComp-1.4B dataset. Everywhere else, the paper also compared the Re-LAION-1.4B dataset. Does the same pattern hold on Re-LAION-1.4B dataset?\n\n4. I wonder whether the paper can be extended to more general questions, such as why MaMMUT has better scalability than CLIP. This can provide more insights into the fundamental question of how to design a more scalable model. Specifically, one can do a study on whether it is the training loss or the model architecture that makes MaMMUT more scalable. This can make the paper more generalizable, and add great significance to the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a rigorous and large-scale empirical study that uses scaling laws as a principled framework for comparing vision-language foundation models and datasets. The authors move beyond traditional single-point comparisons by deriving full scaling laws for two important architectures, CLIP (contrastive) and MaMMUT (contrastive + captioning), across a wide range of model sizes and data scales. The experiments are conducted on two large, open datasets, DataComp-1.4B and Re-LAION-1.4B.\n\nThe core finding is that MaMMUT, which incorporates a generative captioning loss, consistently demonstrates better scalability than the purely contrastive CLIP, meaning its performance improves more rapidly with increased compute. This trend is shown to be robust across different downstream tasks (classification, retrieval, segmentation) and training datasets. The paper also provides a valuable methodology for deriving these scaling laws, including techniques for handling constant learning rate schedules to reduce compute cost. Finally, the authors release all models, checkpoints, and code, culminating in a new SOTA open model, OpenMaMMUT-L/14, which validates the predictions of their scaling analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper's main contribution is to advocate for and demonstrate a more scientific and robust methodology for comparing foundation models. In an era where \"SOTA\" claims are often based on single-point evaluations with murky training details, this work provides a blueprint for principled comparison across scales. This is a significant and much-needed contribution to the community.\n2. The sheer scale of the empirical study is a major strength. The authors have conducted an exhaustive number of pre-training runs, covering multiple model sizes (S/32 to H/14), extensive data scales (1.28M to 3B samples), two model architectures (CLIP, MaMMUT), and two large datasets (DataComp, Re-LAION). This dense grid of measurements allows for highly reliable scaling law fits.\n3. The commitment to reproducibility is exemplary. By using open-source models (OpenCLIP, and their new OpenMaMMUT), open datasets, and promising to release all code, data, and checkpoints, the authors set a high standard. This work is not just a study; it is a valuable and verifiable public resource. The release of OpenMaMMUT-L/14 with 80.3% ImageNet-1k accuracy is a strong practical outcome.\n4. The central result—that MaMMUT out-scales CLIP—is demonstrated consistently and robustly across different evaluation tasks, datasets, and even learning rate schedules. The \"crossover\" effect, where CLIP is better at small scales but MaMMUT overtakes it at larger scales, is a particularly insightful finding that would be missed by single-point comparisons.\n\nWeaknesses:\nThe core comparison is between only two architectural paradigms: CLIP (contrastive-only) and MaMMUT (contrastive + generative). While this is a valuable comparison, the paper's title and claims suggest a more general framework for comparing \"Open Foundation Language-Vision Models.\" The study does not include other important architectural variants, such as (1) SigLIP, which uses a different loss function (sigmoid loss) and has shown very strong performance, (2) BLIP and BLIP2, which are generative large Language-Vision Models, (3) LLAVA, QwenVL, etc which incorporate CLIP encoder with large LLMs.\n\nQUESTIONS:\nIs it possible that the chosen hyperparameter search space or the final selected parameters favor the MaMMUT setup? For instance, the presence of a captioning loss might act as a regularizer, making the model more stable during training with certain learning rates that would be unstable for CLIP alone. Without a more detailed report on the hyperparameter tuning process and sensitivities, a skeptic could argue that the CLIP baseline might not have been trained under its own optimal conditions, thus artificially enhancing MaMM-UT's apparent scaling advantage.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}