Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
Authors: Marianna Nezhurina
Tomer Porian
Giovanni Puccetti
Tommie Kerssies
Romain Beaumont
Mehdi Cherti
Jenia Jitsev
Venue: NeurIPS 2025 poster ()
Keywords: scaling laws
open foundation models
open datasets
reproducibility
model comparison
language-vision models
CLIP
MaMMUT
Primary area: deep_learning
TL;DR: We use scaling law derivation to compare open language-vision foundation models (CLIP, MaMMUT) and datasets (DataComp-1.4B, Re-LAION-1.4B, DFN-1.4B), identifying models and datasets that promise stronger scalability in the pre-training.

Abstract: In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. Taking language-vision learning as example, we show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. Full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. For the first time, we use derived scaling laws to compare both models and three open datasets, DataComp-1.4B, Re-LAION-1.4B and DFN-1.4B, while ensuring sufficient prediction accuracy on held out points. From comparison, we obtain evidence for (i) MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP (ii) DFN-1.4B outperforming other open datasets. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, classification, retrieval, and segmentation, observing consistently the same scaling trends for models and datasets across tasks. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison on aligned common compute axis across large scale span, avoiding misleading conclusions based on measurements from few isolated single reference scales only. This paves road for guided collective improvement of open foundation models and training datasets, as scaling law based comparisons from various studies executed in common frame can be combined to identify overall better procedures. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/scaling-laws-for-comparison.

Scaling Laws for Robust Comparison of Open Visiinn
Foundation Language- Vision Models and Datasets
Marianna Nezhurina 122.8
Tomer Porian 1,2,5+ 1.2.5
Giovanni Puccetti Tommie Kerssies"
Romain Beaumont' Mehdi Cherti 1,2,8 Jenia Jitsey
'LAION JJuelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
Institute of Information Science and Technologies "A. Faedo" CNR Pisa
Eindhoven University of Technology
Open- (Open-Sei) Collective
3 equal first, equal senior, core contribution {m. nezhurina m. cherti, j. jitsse) ofe-eulichhh.cch....oo... de contact@laion.d ai
Abstract
In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. Taking language-vin -viiaageevviionnnnooo learning example, we show here how scaling law deriva- tion can also be used for model and dataset comparison, allowing to decide which procedure is be preferred for pre-training. Full scaling laws based dense measurements across wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. For the first time, we use derived scaling laws compare both models and three open datasets, DataComp-1 1.4,, Re-LAION--1. 1.4B and DFN-1.4B, 1.4,, while ensuring sufficient prediction accuracy on held out points. From comparison, we obtaín evidence for (i) MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP (ii) 1.NB outperforming other open datasets. To strengthen validity of the comparison, we show scaling laws for various down- stream tasks, classification, retrieval, and segmentation, observing consistently the same scaling trends for models and datasets across tasks. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison on aligned common compute axis across large scale span, avoiding misleading conclusions based measurements from few isolated single reference scales only. This paves road for guided collective improvement of open foundation models and training datasets, scaling law based comparisons from various studies executed in common frame can be combined to identify overall better procedures. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUTS MAMMUM--L144, which achieves 80.3% zero-shot ImageNet-11 k accuracy, trained 12.8B samples from DataComp-1144)
Introduction
Foundation models I pre- tre-trained on generic, diverse large datasets enabled massive improvements in transfer learning, showing strong and efficient adaptability across wide range of downstream tasks not shown during pre-training Thanks to learning procedures leading to foundation models,
Code for reproducing experiments in the paper and raw experiments data can be found in the repository
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

transferable learning can be studied across various important domains, including language 2 3 4 vision [5), language-audio [6), and language-vision (7].
Progress in improving learning procedures leading to stronger foundation models crucially depends on the ability to perform systematic and consistent learning procedure comparison. Usually, following the training, various foundation models are compared via performance shown on wide range of standardized reference downstream tasks. Often, this comparison is done on only one or few selected reference model and data scales, without carefully aligning the compute put into pre-training. Further, important parts of learning procedure like training dataset often remain closed. This makes it hard or impossible to discern whether the observed model differences are due to algorithmic, dataset or due to differences in pre- pre-training compute, or a combination of them, leaving also unclear whether the comparison holds for other scales than the few selected ones.
In our work, we make a step towards resolving these issues by using scaling law derivation to enable systematic training procedure, model, and dataset comparison. Foundation models exhibit scaling laws [8] [ IOO [T] that allow to determine dependence of model properties and performance on total pre-training compute from measurements on smaller scales, enabling predictions across wide scale span instead of only one or few selected points. Here, we show how using open datasets for scaling law derivation can enable model and dataset comparison that takes into account model behavior across a wide range of pre-training compute budgets and across different downstream tasks, while offering full control over variations in the whole training pipeline, and being fully reproducible.
We choose language-vision learning as important setting for model and dataset comparison using scaling law derivation. Contrastive language-image pre-training (CLIP) 7 is a well-established learning procedure resulting in models that show impressive robustness and transfer capability, and used routinely as pre-trained components in many setups such as vision- vision-language instruction tuning models (LLaVa [12), In [3, SigLIP [4] and text to image generation models (15). Since the first release of CLIP, many extensions have been proposed such as CoCa [16], MaMMUT [17, and SigLIP I8 14]. These works claim more performant language-vision models than standard CLIP. However, as hinted above, it is still unclear which of the training procedures are better for what reasons, and whether claims of improving on the standard CLIP procedure hold across scales. Here, we conduct large-scale study of the scaling laws of two important procedures, namely CLIP and MaMMUT, pre-trained on open reference datasets, DataComp- 1 ..4B [9] DFN- (2 and Re-LAION-1 4B (1I. which we are able to compare via full scaling law derivation for the first time.
Our study reveals that while CLIP has advantage on smaller compute scales, MaMMUT architecture shows advantage as we increase compute, as illustrated by cross-over of the scaling curves in Fig. 2 (Sec. (Sec. 3.I.. Importantly, we find that the comparison via scaling laws delivers result that is consistent and robust across the pre-training datasets, learning schedules and the downstream tasks.
Our Contributions. 1) We conduct first large-scale study of CLIP 10 and MaMMUT [17] with dense measurements to ensure better prediction to unseen scales and valid model and dataset comparison. Measurements cover model sizes from S/32 to H/14, samples seen from 1.28M to 3B, training on DataComp- 1.4B 19 Re-LAION-- 1,4B 21 and DFN-1.4B (20 datasets and evaluating downstream performance on tasks covering zero-shot classification, retrieval, and segmentation. 2) Based on derived scaling laws, we perform model and dataset comparison. We show validity of this comparison by revealing consistent trends in favor of MaMMUT versus CLIP architecture across different downstream tasks and pretraining datasets, as well as distinct scaling law types (compute- optimal and samples seen based scaling laws). We also show consistent trends favoring DFN- 1.4B over other datasets. Our study is thus the first to provide a fully reproducible systematic comparison of important open foundation models CLenCLIP and openMaMMUT and important reference open datasets Re-LAION--.44B DataComp- 1.4B and DFN-1.4B 1.44. 3) We perform the study using both cosine and constant learning rate schedules, and observe the same consistent trends, revealing that the scaling laws based comparison can be also performed with suboptimal constant learning rate schedule resulting in 98% less compute cost. 4). We provide a detailed procedure to fit scaling laws tailored for model and dataset comparison, sharing our findings on best practices of scaling law derivation which include uncertainty quantification and measuring held-out points on the compute or samples-seen axis. 5) We make our study fully reproducible and release all the intermediate checkpoints from scaling law derivation experiments. For the first time, we provide an open-source implementation of MaMMUT, MpeMMaMMMT, and release MpenMaMMUTTLLLE 14 trained on 12.8B image-text samples from an open dataset DataComp-1.44 1.4B, achieving 80.3% zero-shot ImageNet- Ik accuracy.

---

2 Methods & experimental setup
Our experiments entail model pre-training, evaluation and scaling law derivation, which we describe in the following.
2.1 Pre-training setup
Architecture details. The MaMMUT model is integrated into the openCLIP code base 22 to
take advantage the existing implementation of the CLIP contrastive loss and CoCa captioning loss. Key additions to implement MaMMUT were: (1) using a double forward pass for the text component perform both text encoding (without masking) and decoding (with causal masking); (2) cross-attention between image and text tokens. For an N-layer text decoder, cross-attention layers are inserted every 2 text decoder layers 17.. total of [ cross-attention layers.
Training dataset & objective. We pre-trained CLIP and MaMMUT models on PataComp-1.44 19 Re-LAION-I.4B [1] and DFN-1,4B
20 datasets. Re-LAION-J, ,44B obtained by downloading the Re-LAION-2BBen-- weweeeenreeeeee - subset of Re-LAION [21] which contained R 30% dead links, resulting in total of 1.4B image-caption pairs. DFN-I. .BB was obtained by downloading DFN-2B dataset [20]. which also resulted in discovering = 30% link rot, again providing 1.4B image-text pairs in total. CLIP models are trained with contrastive InfoNCE [23] loss (L Lcontrastive). while MaMMUT models are trained with contrastive and captioning losses (L = Lennrastive + XLeap). we used À 1 in our experiments.
Model & samples seen scales. For both CLIP and MaMMUT architectures, we consider ViT-S, VIT-M, ViT-B, ViT-L, and VIT-H vision encoders. For each vision encoder size, we also vary patch size, and consider patch sizes of 32x32, 16x16, and 14x14, leading to |M = 15 model configurations. We scale text encoders accordingly following previous lit- erature [TO]. For samples seen scales D, we consider a wide range of measurements D {1. (1.28M, 3.07M, 6.4M, 12.8M. 30.7M, 64M, 128M, 307M, 640M, 1.28B, 3.07B). a total of [D] 11 configurations. See App. Tab. [6 for full details about models and number of samples we used in our experiments. To rule effects of training trial-to-trial variance on measurements and scaling law derivation, we estimate variance of downstream performance across training repetitions in the control experiment (App. Tab. 7
Learning rate schedule. In our experiments, we consider both cosine and constant learning rate schedulers. For cosine learning rate schedule experiments, we performed single run for each model-samples seen pair, while for constant learning rate schedule, we only need to train once for each model size.
Optimization. We perform a hyper-parameter sweep for batch size, learning rate and warmup for each training run to avoid suboptimal solutions. We have observed that it is important, especially in small sample seen scales, as large batch sizes usually used in CLIP training will result in small number of optimization steps, making optimization suboptimal. For training, we used Adam (24 as an optimizer with a weight decay of 0.2. To avoid unstable training and loss spikes with larger models (e.g., VIT-L, ViT-H) we followed [TO 25 and used B1 = 0.9, B3 0.95, gradient clip norm of 1, warmup and mixed precision with bfloat16. See more details in App. Tab. 15 and 14
2.2 Downstream evaluation
Zero-shot classification. We evaluate the top-1 zero-shot accuracy on 35 classification tasks from the DataComp evaluation suite [1).. It includes ImageNet- 1k [26). ImageNet robustness to distribution shift datasets (27 28, 29 30 3T] and additional 29 classification tasks covering multiple domains.
We follow the evaluation protocol of [IO), i.e. using the same prompts, class names, and code base The full list of datasets we used in evaluation with description is included in Appendix F
Zero-shot retrieval. We evaluate models on MS-COCOB3) image and text retrieval Recall@5 metrics, following 110.
Segmentation. We fine-tune for semantic segmentation on ADE20K (34, following 35) using 31 epochs and 1500-step warmup (36), with consistent 224 x 224 input and 14 14 patch size.
3

---

2.3 Scaling law derivation and fitting procedure
We vary both model architecture size (number of parameters of both text and vision towers), number of samples seen and patch size. In general, the relationship between compute and performance follows a power law: = aC', where is compute in FLOPs [T0] 37,, with the set of points {C, ((C) located on the Pareto frontier. Since we use error rate on ImageNet-Il Ik zero-shot classification, we follow [37 38 and take into account the saturation (B₂) that occurs at the small compute scales due to the nature of classification tasks the probability of predicting a class even zero-compute is determined by the frequency of this class in the test set. On the other hand, tasks like zero-shot image classification cannot be solved with 100% accuracy due to task- and learning performance ceiling 38 37 Accordingly, we require that our scaling law functional form satisfies:
dL dC
C(C) > 0 (strictly positive),
lim C
[(C)= (irreducible error),
(monotonic decrease).
Given above criteria, we obtain the following functional form for the error that satisfies all three (subscript C' specifies compute dependency):
[(C)) A. (C B.) Fe, 02. >
For each combination of compute scale C' and model architecture, we take point with the minimal eтror rate. In previous works (39], points with minimal loss were found by binning FLOP values into 1500 FLOP logarithmically spaced intervals. We use the same approach to find points with minimal error. Therefore, we obtain mapping from each combination of number of parameters N and samples seen D) to the compute C' (in GFI.Oss). Using measurements {(C,, Y where C; is compute budget (GFLOPs) and Y; is error performance, we fit the curve only on points with compute budget below threshold (C,, Y,), C, < Ciueehold J- To quantify the quality of our fit, we use mean squared error on the remaining (held-out) points: MSE Ei.c,
Cheesral (C(C.) - Y.)².
2.4 Confidence intervals estimation
We estimate 95% confidence intervals for the model parameters by propagating the uncertainty from the estimated parameters. We compute the Jacobian of the model, J, with respect to the parameters at the extrapolated points. We then estimate the variance of our predictions as 0² = J Cov(O) J. Confidence intervals are then given by ÿ + fa/2,n=p" 0, where n-p is the critical value from the Student's t-distribution at 0.05.
2.5 Data efficiency and optimal dataset size estimation
To quantify the data efficiency of CLIP and MaMMUT, we fit scaling law analogous to Equation I
((D) = Ap (D+BD) aa
Bp) + Ep
(2)
Here, LDD denotes the expected error rate function of the number of samples seen D. For each unique data budget D. we extract the corresponding minimal error points (D, L(D) using the Skyline Operator algorithm [40), which selects non-dominated configurations based on error rate.
To estimate the dataset size that is optimal for a given compute budget, we follow the approach of [ and fit pawer-law relationship of the form: Dept Do C" where Dept minimizes L(C, D) for a given compute budget C. The optimal dataset sizes are obtained by identifying (C, pairs that yield minimal error rate under fixed compute constraints.
3 Results
3.1 Scaling laws for model and dataset comparison
We fit function that has a form of Eq. Ton the obtained experimental data using methods described in Section 2.3 To avoid the confound of data repetition [4] we limit the data used for our scaling law fits by selecting only models that were trained on up to 3B. In Tab. [5 estimated parameters for both models (CLIP and MaMMUT) well as for both downstream tasks (INIk classification

---

Model Candidate
Samples Seen Candidate
GELOPS
INIK ace! Predicted (95% CT)
VIT-L-114
12.88
2.14c+12
0.79 (0.788, 0.804)
VIT-L-14
15,58
2.59c+12
0,800 (0.791. 0.808)
mammmut VIT-L-14
10.68
2.14e+12
0.816 (0.811. 0.821)
mammuut V4
12.8B
2.59e+12
(0.811, 0.826)
Table 1: Estimation of INIk performance for CLIP and MaMMUT unseen compute scales using our scaling laws fits. Additionally, for each compute scale, we provide possible models and samples seen (assuming unique samples) sizes.
and MS-COCO retrieval) be found. MaMMUT consistently exhibits better scaling behaviour compared to CLIP. This is reflected in smaller error rates at equivalent compute budget at larger scales after crossing compute threshold that is consistently found to be between 10a0 and 10 GFLOPS across various datasets and tasks (Fig. 2 Note that on smaller scales in the lower performance range, CLIP consistently outperforms MaMMUT, which then consistently takes over CLIP at larger compute scales in the higher performance range. This indicates better efficiency and generalization as we increase compute. This trend holds across:
Pre-training datasets: DataComp- (Fig. (Fig. 3)
I. Re-LAION-1.44 (Fig. 2 and DFN-1.4B
Downstream tasks: ImageNet- Ik shot image classification (see Fig. (a), Fig. (a)
and MS-COCO image retrieval (Fig. T (b), Fig, (b)), ADE20K semantic segmentation (Fig. 6)
Learning rate scheduler: cosine (Fig. and constant (Fig.
We validate our fits by fitting the laws only up to certain compute budgets Citreshold and then extrapo- lating to larger ones. We use these extrapolated points to compute MSE, which allows us to check on quality of the obtained fits, observing that adding more points to the reduces MSE on held-out points and also reduces uncertainty of predictions. The measured performance falls well within the prediction confidence interval (App. Tab. 12). Detailed versions scaling laws plots for CLIP and MaMMUT can be found in Appendix B more details on validating fit quality are in Appendix C As further evidence for the validity of derived scaling laws, we observe same consistent scalability trends across datasets and on further important downstream tasks, for instance DataComp eval suite (Fig.
14 ImageNet robustness (Fig. [5), or on segmentation after fine-tuning (Fig. 6 see Sec.
In Tab. I we provide predictions on DataComp- 1,4B for both MaMMUT and CLIP for unseen compute budgets 2.14e+12 GFLOPs (corresponds to CLIP ViT-L-14 trained on 12.8B image-text pairs) and 2.59e+12 GFLOPs (corresponds to MaMMUT ViT-L-14 trained on 12.8B samples). We see that our predictions favor MaMMUT over CLIP. As a prediction test on larger scales further away, for CLIP ViT-L-14 trained on 12.8B samples of DataComp-1.4B our prediction for ImageNet- lk 0-shot accuracy (79.6%) is close to performance of CLIP ViT-L--14 trained on 12.8B samples reported in the original DataComp work [19 79.2%. Note that the measured performance INIK zero-shot 79.2% in the DataComp original work [19] was done with heavy samples repetitions (12.8B on DataComp-1 1.4 is about 9x), while our prediction is done for unique low repetition scenario, which also might explain tendency to a higher performance in the predictions than observed in experiments 12.8B samples
scale (Tab. Tp.
As evident from Fig. and T0 DFN-1.4B consistently provides stronger scalability compared to DataComp and Re-LAION, for both CLIP and MaMMUT architectures and for both zero-shot ImageNet- Ik classification and MS-COCO retrieval. Despite lower compute used for dataset compar- ison and higher uncertainty for the trends resulting from fewer measurements for DFN, measured trends are clear and consistent, allowing thus to draw conclusions favoring DFN-1.4B over other datasets in the comparison.
3.2 Data efficiency and compute-optimal dataset size
Fig. illustrates that MaMMUT exhibits superior data efficiency relative to CLIP. Fig. (a) we see that MaMMUT achieves better performance on ImageNet-1k zero-shot image classification as the number of training samples increases. Fig. 4 (b) MaMMUT requires fewer training samples to achieve compute optimal performance ImageNet- Ik zero-shot classification. This indicates that
5

---

Imuges Rgg
MANeun B0T
(a) ImageNet-Ik Ik 0-shot classification
(b) MS-COCO image R@S
Figure 1: Scaling on DataComp-1.4B, Comparison of CLIP and MaMMUT via scaling laws on DataComp- 1.... Error rate on downstream tasks is plotted against compute. MaMMUT outperforms CLIP in terms of scalability, with scaling law fit lines crossing close to 10 GFL.OPS.
Inage
13t
2007
199
124
Compute (GELOPs)
Computee CIGFLOP))
(a) ImageNet-IK Ik O-shot classification
(b) MS-COCO image R@S
Figure 2: Scaling Re-LAION-14B. Comparison of CLIP and MaMMUT via scaling laws O0-LAION-I.4B Error rate on downstream tasks plotted against compute. MaMMUT outperforms CLIP in terms of scalability, showing similar trends as in DataComp-1.49
18
MOCOCO mrag wwapehortee FRSS
CUP +227
Compute C [GELOP4)
Compute CLGFLOPS)
(a) ImageNet- 1k 0-shot classification
(b) MS-COCO image R@S
Figure 3: Scaling on DFN-1.4B. Comparison of CLIP and MaMMUT via scaling laws on DFN-1.4B. Error rate downstream tasks is plotted against compute. MaMMUT outperforms CLIP in terms of scalability, indicated by crossing scaling law fit lines, where MaMMUT takes over CLIP in performance from larger compute close to 10 GFLOPS on, again showing similar trend as observed on DataComp and Re-LAION.

---

10
CUP 54106 54 2og 160076422 war
Optimal Numbor Samplics
Compute CTFFL LFFL
(a) IN-1k 0-Shot classification error rate vs. train- (b) Compute optimal dataset size D (Cp ing dataset size
Figure 4: Comparison of data efficiency and optimal dataset size for CLIP and MaMMUT via scaling laws on DataComp-1.44 1.4B. MaMMUT is more data efficient and requires smaller dataset size to be compute optimal.
MaMMUT makes more effective use of training data than CLIP. Therefore, these trends suggest that MaMMUT generalizes better and scales more favorably with additional data. Additionally, we derive scaling laws with respect to number of training samples on two other datasets Re-LAION- LIION-1.4B and 1BB. In Fig. TT we show that MaMMUT shows consistently superior data efficiency across different training datasets compared CLIP. We also provide estimations for optimal number of training samples for unseen compute scales (see Tab. ( for CLIP and MaMMUT trained on Data DataComp-1.4B We use the predicted compute-optimal number of samples
estimate INIK classification error rate using obtained power fit following Eq. (see Fig. 4. We see that MaMMUT a more scalable model which agrees with estimation obtained from fitting Eq. Ton the experimental data (see Fig. [ and Tab.
Prodicted
Model
Predicted INIK
GFLOPss
(C)(C)) (95% CT)
Candidate
#Params
0-shot acc (95% CT)
2.14c+12
2.30e+10 (2.75c+10, 191e+10)
ViT-SO15UM-I 14-05500----mmll--ttee smallleee-tt
279M
0.794 (0.785, 0.803)
2.59c+12
2.64e+ (3. 19, 2.20e+10)
VIT 50150M 14 smaller-test text
295M
0.795 (0.786, 0,804)
2.14e+12
1,23e+10 (1,39+10, 100e+10)
mammnut mammut-Vii---11
522M
0,798 (0.794, 0.803)
2.59e=12
1.42e+10 (1.6le+10, 1.25e+10)
mammnut VIT-L-1
548M
0.799 (0.795, 0.804)
Table 2: Predicted compute optimal samples seen and accuracy for each compute budget and model configuration, with parameter sizes annotated in millions (M).
3.3 Scaling law derivation using constant learning rate scheduler
We follow 42 and show a scaling law derivation based on training with constant learning rate, thus saving 98% compute compared to cosine. We omit points from warmup duration in our derivation, to prevent noise in the low-compute part of the scaling law. In Fig. 5 we visualize our results, showing the measurements density and Tab. 66 we tabulate the coefficients. Our results further support the better scalability of MaMMUT over CLIP, showing consistent trends even when replacing learning rate scheduler.
3.4 Comparison via scaling law for fine-tuning error segmentation dense prediction task
For further comparison evidence, we derive a scaling law (Eq. for ADE20K segmentation error (1 mIoU) after fine-tuning dependent on pre- pre-training compute scale for CLIP and MaMMUT, As shown in Fig. 6 MaMMUT again exhibits stronger scaling than CLIP (a 0.208 vs. 0.354), with an error crossover at approximately 10 GFLOPs. This is below the crossover at approximately 10t1 GFLOPs observed for zero-shot ImageNet classification (Fig. [p), indicating that captioning supervision via fine-tuning improves dense prediction already at lower pre-training scales. See more details in Appendix
H
7

---

MECOCC imae mage
2x
MEDI
Cal -OROLLLI
Compute C [IRLLOSS]
Compute CCGFLOPS)
(a) ImageNet- Ik O-shot classification
(b) MS-COCO image R@5
Figure Scaling law fits using constant learning rate scheduler. Comparison of CLIP and MaMMUT via scaling laws on DataComp-1.45 Error rate on downstream tasks is plotted against compute, Using constant learning rate scheduler for scaling law derivation reveals the same trend as with cosine MaMMUT outperforms CLIP in terms of scalability.
Cal 53 MANMAUTI 221 16--- 0.50
ADDPOK Seervertie Teggmeedetion Rate)
0x38
Compute C[GFLOPS]
Figure 6: Scaling law for semantic segmentation. Downstream error rate (1 mIloU) of openCIIP and openMaMMUT pre-trained on DataComp-1 1.amppp1.444 1.4B and fine-tuned on ADE20K, MaMMUT shows higher performance than CLIP for segmentation at higher scales.
3.5 Scaling up following the comparison: OpenMaMMUT I/14 on 12.8B samples seen scale
We used insights and predictions from our scaling analysis and comparisons to train OpenMaMMUT L/14 12.8B, a large scale open vision-language foundation model. We chose to train 12.8B samples of DataComp-1..4 1.4B to have direct comparison to openCLIP trained in previous work [19]. More info on training hyperparameters can be found in Tab. 13 OpenMaMMUT achieves state of the art performance on zero-shot classification and retrieval tasks among similar-sized models trained only publicly-availl availlable data (MetaCLIP, DataComp. Open Vision, Tab. 3}. It outperforms with 80.3% INIk accuracy as predicted openCLIP pre-trained on same DataComp-1 1.4B budget of 12.8B (79.2%) and even rivals models with much larger pre pre-training compute like SigLIP. OpenMaMMMT represents a highly performant, fully reproducible alternative to other models with openly available data and training code. Note that 12.8B samples seen scale the performance suffers from high amount of repetitions, and therefore is below our prediction of 82% (Tab. [ that is valid for training on unique samples. For further 12.8B sample scale models, see App. Sec. B.]
4 Related work & limitations
Recent work has investigated the performance of vision-language models such CLIP 77. CoCa 16], MaMMUT [17) Cap 45), IiII.IP [14], TULIP[46) or Vision 47 at various scales. These studies analyze different model sizes and highlight either architectural or dataset innovations;

---

ImageNet- 1k
COCO
VIT
Res.
Seq.
Model
Daraset
#Samples
val
v2
T-+1
I-+T
SigLIP
18
WebLI-10B
73.76
75.26
1/16
256
256
Sigl
14
Webl.I-IOB
40B
82.35
76.66
76.84
90.44
OpenCLIP
[0
LATON-2B
34B
75.24
67.73
70.46
84,30
CLIP 7
WIT-400M
12.88
75.54
69.84
59.95
79.56
MctaCLIP
4
MetuCLIP-2.5B
72.64
71.36
84.94
1/14
224
256
EVA CLIP
44
Merged 2B
413
79.75*
72.92*
70.68
85,26
DFN-2B
13R
DataComp
19
DaraComp-1.44
12.88
79.19
72.06
69.86
84,64
OpenMaMMUT (Ours)
DataComp-I 1.4B
12.81
80.34
23.78
71.19
85.88
Table 3: Zero-shot classification (accuracy) and retrieval () results. DFN used ImageNet/MS- COCO-finetuned model for data filtering, WVII was initialized from models pre-trained on ImageNet. We use bold for best overall results, gray for models involving ImageNet/MS-CON data as training data in pipeline, and underlined for best results without ImageNet/MS COCO vement..et.
however, they do not perform a comprehensive scaling law analysis. Furthermore, the datasets used in these works-such WIT for OpenAl CLIP and WebLI for SigLIP-are closed, severely limiting reproducibility and making it impossible to study which of algorithmic or dataset interventions claimed to have beneficial effect learning used in those studies indeed lead to better model performance. In App. Fig. 12, on DataComp-1.4B, finding no significant difference of SigLIP standard CLIP model scalability.
we compare open implementations of CoCa and SigLIP to openCLIP
In 48 authors investigate how various language model families compare in terms of their scaling behavior. While it offers valuable insights into architectural trends, it does not use for comparison full scaling law framework in the sense of jointly modeling loss or downstream task performance as a function of compute and dataset size varied systematically across multiple scales. As accuracy of predictions derived from such trends was not measured, it makes it unclear whether observed trends for various language model architectures are to be trusted and whether comparison would remain valid across various scenarios, which we demonstrate to be the case for scaling law based comparison.
Preparing grounds for this work, [10] derived first reproducible scaling laws for openCLIP using LAION datasets and performed comparison between open LAION-400M/SB and closed WIT. The work used however only few samples seen scales, while also going up to scales that prone to strongly diminishing performance due to heavy sample repetition (6x on 12.8B and 17x on 34B sample seen scales). This affects extrapolations of the scaling law and thus the validity of comparisons based on it. Interestingly, in our work using much denser scaling law derivation without strong repetitions for higher prediction accuracy,
confirm the dataset comparison in [TO] using Re-
LAION-I.4R 4B, observing same trends for WIT being better on zero-shot classification (App. Fig. 7 while worse on retrieval (App. Fig. (). providing further evidence for robustness of scaling law based comparison. Another data-centric work responsible for composing open DataComp- 1.4B dataset
1..4-
19 used measurements on few selected scales compare datasets and decide for benefit of various dataset interventions. No scaling laws were derived to back up the comparisons, leaving unclear whether the observed trends will propagate across larger scales than taken for the comparisons.
Several works explored the effects data constraints on scaling laws. Notably, studies investigating the scaling law behavior under dataset repetitions for language models [88 41 and for CL reported that high repetition factors can lead to heavily diminished performance compared to the Pareto front of scaling with unique data samples. Our experiments carefully limit the repetition factor to less than 3x, minimizing such confounding effects, assuming unique samples or only little sample repetition for comparisons be valid on Pareto front with strong performance scaling.
Building on these foundations, our work provides a unified and empirically grounded scaling law analysis of vision-language models trained on open datasets. We explicitly model performance as a function of data and compute g 9]. and compare multiple architectures under fully controlled, consistent and reproducible settings. Unlike prior work, approach enables rigorous comparison of both data and compute efficiency, ensuring the consistency of the comparison across scales and training conditions.
Limitations. While our work provides a comprehensive analysis for language-vision models such as CLIP and MaMMUT trained on large-scale open foundation datasets, it also has several limitations:

---

1) We mostly use zero-shot setting for model evaluation (for classification and retrieval tasks), using fine-tuning only for segmentation. Linear probing. fine-tuning [T] or vision caaruuttionttunin [[2] of the pre-trained vision encoders can provide further insights into validity and consistency of scaling laws based comparison. 2) While open datasets we use have substantial scale 1.4B unique samples this still limits the ability of derived scaling laws to extrapolate to higher scales, as the effect of sample repetition has to be considered when conducting measurements above 1.4B. To test comparison validity and scaling law predictions are required. 3) In our study we only looked at standard contrastive loss or contrastive and captioning loss objectives and did not incorporate further "loss mixtures" such masking or diffusion- based losses. We also did not derive scaling laws for specific architectural components such as optimal number of parameters in text or vision tower, or other important properties of training procedure like image input resolution and patch size, or context length in general. In its current form, scaling laws based comparison has high computational cost, prohibiting naive incorporation of many factors that influence scalability of the training procedure.
larger reference scales like 12.8B, larger scale open datasets
5 Discussion & conclusion
In this work, we show how systematic learning procedure comparison can be performed via scaling law derivation under fully controlled, reproducible training conditions when using open foundation models and open datasets. We take as example scenario openCLIP 7 22 I0] and openMaMMUT based on [17]. two important open language-vision models relying either on image-text contrastive only or contrastive and captioning loss, trained on three important open reference datasets, DataComp- 1.4B [19] Re-LAION-1 ---N----.4 1.4B [21] and DFN-1.49 1.4B [20). We show that deriving scaling laws gives comparison of model and dataset based on their estimated scalability for wide scale spans and for various downstream tasks, aligned on same total pre-training compute. Such comparison can be validated by checking consistency of scaling trends in different scenarios. For instance, open- MaMMUT scalability is stronger than openCLIP both on zero-shot classification and retrieval, also showing advantage for a wide scale span on segmentation, and across all three open datasets. Also, inconsistencies are insightful- for instance, DataComp-1 1.44 shows stronger scalability for both openMuMMUT and openCLIP for zero-shot classification while being slightly weaker for retrieval. Thus, none of these two datasets is the most scalable candidate across all downstream tasks, and scaling advantage there is task dependent. DFN on the other hand is consistently better than other datasets across downstream tasks and for both MpENMaMMMT and openCLIP.
Comparison via scaling laws offers better protection against misleading conclusions derived from comparison of only few selected points, especially when done on small scales only. On smaller scales, openCLIP outperforms stronger scalable MpenMaMMUT that takes over on larger scales. Remarkably, we observe the compute scale threshold where OpenMaMMUT takes over openCLIP to be consistently settled between 10.0 and 10t1 GFLOPS across datasets, zero-shot downstream tasks and learning schedules. This gives further evidence for the robustness of scaling law based comparison. To properly estimate such crossings, it is crucial to perform dense measurements on smaller scales and use fitting routines that allow for accurate extrapolation to larger scales. Efficient derivation of accurate scaling laws [42 [n] to determine factors affecting scalability of the learning procedure is thus important topic for future work.
In our study, we used open datasets with 1.4B samples. While this is sufficient to demonstrate usefulness of scaling law based comparison, more accurate predictions for training at larger scales on unique samples require larger datasets. Those are also required to train larger scale models with predicted strong capabilities, as too many repetitions on smaller datasets might lead to diminished performance [1 49), which we see in openMaMMUT L-14 trained on 12.8B samples, staying with zero-shot INIk 80.3% below the predicted 82% (Tab. I.. Deriving scaling law correction for diminishing performance due to data repetitions as well as increasing scale of open datasets are important directions for future work.
While we show that robust and reproducible comparison via scaling law derivation is possible, relies crucially on the whole pipeline to be fully open including dataset composition, training itself, and downstream evaluation. We hope that our work will encourage the creation of more open artefacts, especially open datasets as those are still scarce [50] 19 20, 21, to enable collaborative and reproducible progress towards stronger scalable open foundation models guided by independently verifiable and systematic comparison.
10

---

Acknowledgements
MN, TP, MC and JJ acknowledge funding by the Federal Ministry of Education and Research of Germany (BMBF) under grant no, 011$24085C (OPENHAFM), under the grant 16HPC1 177K (MINERVA) and under the grant no. 011S22094B (WestAl AI Service Center West), as well as funding by EU Horizon under grant no. 101214398 (ELLIOT) and co-funding by EU from EuroHPC Joint Undertaking programm under grant no. 101182737 (MINERVA) and from Digital Europe Programme under grant no. 101 (openEuroLLM))
Giovanni Puccetti is fully funded by the project "Italian Strengthening of ESFRI RI RE- SILIENCE" (ITSERR) funded by the European Union under the NextGenerationEU funding scheme (CUP:BS3C2201170000))
We gratefully acknowledge the Gauss Centre for Supereomputing e.V. for funding this work by providing computing time through the John von Neumann Institute for Computing (NIC) on the su- percomputer JUWELS Booster at Jülich Supercomputing Centre (JSC), EuroHPC Joint Undertaking for computing time and storage the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Extreme Access grant EXPC-EXT- 2023E02-068 and in part by Iscra B Project REWATCH (Project Number: HP1OBL,22R) storage resources on JUST granted and operated by JSC and supported by Helmholtz. Data Federation (HDF), computing time granted by the JARA and ISC on the supercomputer JURECA at JSC, and computing time granted on prototype JEDI via JUREAP (JUPITER Early Access Programm) grant at JSC. LAION further acknowledges storage grant by HuggingFace that allows us to provide convenient access to the output of the open-source research to broad community via HF repository.
Further thanks go for support provided by supercomputing facilities and their teams, especially to Damian Alvarez and Mathis Bode from Juelich Supercomputer Center (JSC, Germany) and to Laura Morselli from CINECA (Italy).
We also would like to express gratitude to all the people who are working on making code, models and data publicly available, advancing community based research and making research more reproducible. Specifically, we would like to thank all the members of the LAION Discord s and Open-y/ V/ (Open-Sci) Collective for providing fruitful ground for scientific exchange and open-source development.
References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 1XXv:2208..02258 2021.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18Xiv:1810.04805. 2018.
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-66 2020.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:7 2020.
[5] David Fan, Shengbang Tong. Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. arXiv preprint arXiv:2504.0000 200000000011,,... 2025.
[6] Benjamin Elizalde, Soham Deshmukh, Mahmoud Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
httpe://disacodd.gg gg/ /B/gh/aaooddd.gg////eeeeaaaa https://discord. gg/ www.ggggGeGgeee.eoe

---

[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 9748-8763 PMIR, 2021.
[8] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint XXi::2001.08361, 2020.
[9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[10] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 28 2818-2829, 2023.
[11] Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference Learning Representations, 2025.
[12] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
[13] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic iisual-linguiste tasks. In Proceedings of the IEEE/CVF conference on computer 2024.
vision and pattern recognition, pages
[14] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual lision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.1 14786, 2025.
[15] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 019552 2023.
[16] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
[17] Weicheng Kuo, AJ Piergiovanni. Dahun Kim, xiyang Iuo, Benjamin Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew M. Dai, Zhifeng Chen, Claire Cui, and Anelia Angelova. MaM- MUT: A simple architecture for joint learning for multimodal tasks. Transactions on Machine Learning Research, 2023.
[18) Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986, 2023.
[19] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh. Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092-2271111 2023.
[20] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfih International Conference on Learning Representations, 2024.
12

---

[21] LAION. Releasing ra-laion 5b: transparent iteration on laion-5b with additional safety fixes.
- 2024. Accessed: 30 aug, 2024.
[22] G. Ilharco, M. Wortsman, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip. 2021.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03765 2018.
[24] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[25] Melanie Mitchell. How do we know how smart ai systems are? Science, 381(6654))cud35557 2023.
[26] J. Deng, W. Dong, R. Socher, Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Computer Vision and Patlern Recognition, pages 248-255, June 2009.
[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pages PMLR, 2019.
[28] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis out-of-distribution generalization. International Conference Computer Vision (ICCV), 2021. https://arxiv: orw.ais//rr//////////////
2006. 16241
[29] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
htttp://arxiv.. wttp://..rggbbb/////aeeaee 07174
[30] Haohan Wang. Songwei Ge, Zachary Lipton, and Bric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv. org/ www.aaas9s.. 13549
[31] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[32] Romain Beaumont Mehdi Cherti et al. Clip benchmark. https://github. com LAIeeLLAOON--O AI/ CLIP Jei_chakncmark 2023.
[33] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. European conference on computer vision, pages 750-755. Springer, 2014.
[34] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing through ADE20K Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-5130, 2017.
[35] Tommie Kerssies, Daan De Geus, and Gijs Dubbelman. How to Benchmark Vision Foundation Models for Semantic Segmentation? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1 1162-1171, 2024.
[36] Tommie Kerssics, Niccolò Cavagnero, Alcxander Hermans, Narges Norouzi, Giuseppe Averta, Bastian Leibe, Gijs Dubbelman, and Daan de Geus. Your ViT is Secretly an Image Segmen- tation Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[37] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2214-11213,, 2022.
13

---

[38] Tom Henighan, Jared Kaplan, Mor Katz. Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010. 040220000..47,,,, 2020.
[39] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arX.v:2203..55555 2022.
[40] Stephan Borzsony, Donald Kossmann, and Konrad Stocker. The skyline operator. In Proceedings
1 17th international conference on data engineering, pages 421 1300. IEEE, 2001.
[41] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50333 2023.
[42] Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. Advances Neural Information Processing Systems, 37:100535-100577 2024.
[43] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint (rXXv:2309.1667/ 2023.
[44] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 3333003.339. 15389, 2023.
[45] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances Neural Information Processing Systems, 36:46830046885: 2023.
[46] Zineng Tang, Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M Chan. Tulip: Towards unified language-image pretraining. arXiv preprint arXiv: 1 15485, 2025.
[47] Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, and Cihang Xie. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprins vrXiv:2505.04601, 2025.
[48] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.1055/, 2022.
[49] Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling laws for data filtering-dant curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22702-22711, 2024.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-SB: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims in the abstract are backed up by the experimental results described in the main paper results Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
There are no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Paper works with publicly available open datasets, will provide open-source code and data obtained from experiments to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Paper works with publicly available open datasets, will provide open open-source code and data obtained from experiments reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
public/guides httpe//eeeooeeueeeesssssoooooooa
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co
wetgguddddeeeuuubbueeee ssinoolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Paper contains all details necessary to understand the results, works with publicly available open data and will provide open-source code enabling researchers to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Accuracy of scaling law prediction is estimated via confidence interval as described in in Section 2.4 and reported in Tab. Mand Tab, 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Paper provides information about compute spent for experiments, open-source code and data that will be released will contain information about exact configuration for performed trainings
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics htttss://eeure http:///wrrrp.sccppppc//c/hc.c.t cctpppblccccttiiccssieeeeeeeeeee
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and believe that the research conducted in the paper conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There are no societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The resources used in the paper properly credited and the license and terms of use are explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All assets are properly documented in the paper and will be described in the opne-source repo or at the corresponding HF location.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for conducting this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee http///rrpss.ccooooeeeeeeeee//// ccc/Confereencee/2225/LLL for what should or should be described.
21

---

Appendix: Scaling Laws for Comparison of Open Foundation Language-Vision Models and Datasets
A Estimated parameters for scaling law fits
To complement main results for scaling law based comparison of CLIP and MaMMUT (Sec. 3.1 Fig. 2). we provide exact numbers of scaling law fits for both openCLIP and openMaMMUT measurements on zero shot INIK classification and MS-COCO retrieval downstream tasks (Tab. (5a.. Estimated values of exponents in power laws alone do not tell which models are more scalable, as we use here the functional form with additive terms for both irreducible error and non-zero random model performance (Eq. D Apart from plot visualization attesting Mammut stronger scalability than CLIP (Fig. m scalability can be also compared via computing derivatives of the obtained fit in selected compute points. Derivatives that have larger absolute values stand for larger slope (bigger rate of decrease) and thus indicate stronger scalability. In Tab. 55 we show derivatives computed for scaling law fits, obtaining larger derivatives for openMaMMUTT than for openCIII,, confirming again stronger scalability for MaMMUT over CLIP.
B More details on scaling law derivation experiments
Compute budget and energy consumption for the experiments. Tab. we provide overview over the GPU hours and energy spent for scaling law derivation experiments. We provide separate calculation for different learning rate schedule types (cosine, constant learning rate and constant learning rate + cooldown), for different datasets (Re-LAION-IA 1.4B and DataComp-1.49 different GPU types (A100 and H100). Large fraction resources was spent for reference cosine schedule based scaling law derivation on DataComp- 1.4B. We see that despite higher density of possible measurements, const based schedules use substantially less compute.
and for
Detailed versions of scaling law plots. In the more detailed versions of scaling law plots (Fig. 16 and 17) we see the separate scaling curves for each model size (cooler colors indicate smaller models). The bigger models require larger sample seen scale to unfold their performance advantage, with the performance lagging behind smaller scale models on same smaller compute scale, where larger models suffer from sample seen scale bottleneck. On the other hand, for the higher compute and samples seen seales, smaller models tend to saturate, indicating a bottleneck in model number of parameters.
LR Scheduler
GPU
Dataset
MWh
GPU Hours
NVIDIA A100
cosine
NVIDIA-A100
DataComp- 1.4B
2.59e+05
1.03e+06
const-cooldown
NVIDIA-A100
DataComp- 1.4B
1.43e+05
5.72e+05
const
NVIDIA-A100
DataComp-
9.30e+04
3.72e+05
cosine
NVIDIA-A100
Re-LAION-1.4B
3.91e+04
1.56e+05
const-cooldown
NVIDIA-A100
Re-LAION-1.4B
1.70e+04
6.79e+04
const
NVIDIA-A100
Re-LAION-1.4B
4.61e+03
1.84e+04
A100 subtotal: 5.56e405
2.22e+06
NVIDIA H100
cosine
NVIDIA-HI00 NVIDIA-H100
DataComp- 1.4B
2.09c+04
2.98c+04
cosine
Re-LAION-1.4B
1.06e+04
1.52e+04
H100 subtotal:
3.15c+04
4.50c+04
Total:
5.87c+05
2.27e+06
Table 4: Total GPU compute and energy consumption for scaling law derivation experiments.
22

---

Iriguueaa Sulve.t
STA4 DED
2412
207
Compote ((CPLOPS
Campute (OPLOPS)
(a) CLIP on WIT. Re-LAION, DataComp
(b) MaMMUT on Re-LOI,,, DataComp
Figure 7: Scaling laws for INIK O-shot performance of openCLIP (left) and openMaMMUT (right), comparing training on DataComp- 1.4B and Re-LAION- 1.4.. For CLIP we have additional points for OpenAI CLIP 7 models trained on WIT-400M dataset for reference.
10
Bmags PuttAA
26.7 ORDERAS 222
MANKUP POO
CLEWE
(a) CLIP WIT. Re-LAION, DataComp
(b) MaMMUT Re-LAION, DataComp
Figure 8: Scaling laws for MS-COCO image retrieval performance (I- Recall @5) of openCLIP (left) and MaMMMT (right), comparing training DataComp- 1.4B and AION-1.4B. For CLIP models we have 3 additional points for OpenAl CLIP trained on WTT-400M dataset for reference.
10
28
Anarms
3x3e
Compute C (IEELOP)
Compute CLGTLOPN
(a) CLIP on Re-LAION, DataComp. DFN
(b) MaMMUT on Re-LAION, DataComp. DFN
Figure 9: Scaling laws for INIk 0-shot performance of openCLIP (left) and openMaMMUT (right), comparing training on Re-LAION-1A4 .4B, DataComp- 1.4B and DFN-1.4B. Training on DFN-1.4B results in superior performance across scales consistently for both architectures.
23

---

ImageNet-IK
MS-COCO Retrieval
Model
A.
Be
dke
E.
Gg
Ope.CLIP
57,862
18,391
-0.227
0.111
53.13
18.413
-0.250
0.216
OpenMAMMUT
79,970
19.111
-0223
0.076
119.751
19.122
-0.263
0.212
(a) Fitted scaling law parameters (Ac, Be Cee, Ee)
for enror O-shot ImageNet-Il Ik classification and MS-COCO retrieval tasks, rounded to three decimal places for models trained on DataComp-1.41)
Co GFLOPs
IN-IK Err. Rate C/
COCO RWS Err. Rate dZ C/)
CLIP
5,00e+10
9.85e-13
8.44e-13
1,00++11
4.21c-13
3.00c-13
5,86c-14
4,95c-14
Averoge: IN-IK: 4.882-113, COCO: 4.177e-13
MaMMUT
5.00e+10
1.17e-12
9.65e-13
1,00++11
4,92c-13
4.03c-13
5,00++11
6,54e-14
5,28e-14
Averoge: IN-IK: 5.758e-13, COCO: 4.702e-13
(b) Numerical values of derivatives of fitted functions with respect to compute. in points 5 10"0, 1 1.10", 5- 10" GFLOPs for both ImageNet- Ik error rate and COCO retrieval exhibits higher values of (do(C))/dC) which corresponds higher decrease rate and stronger scalability.
(1-R@5). MaMMUT consistently
Table 5:
Estimated parameters for main scaling law fits for O-shot ImageNet- Ik classification and MS-COCO retrieval, used for openCLIP and openMaMMUT comparison in Fig.
Net-Ik
MS-COCO Retrieval
Model
Ag
Bg
0g
Eg
Ag
Bg
0g
Eg
openC1.IP
14.769
16.725
4.
0.121
6,686
16,209
-0.23
0.089
1850.286
20.521
0,379
0.198
634.190
20.256
0.335
0,249
Table 6: Fitted scaling law parameters (Ae, Be, Cte, Ee) for error rate on O-shot ImageNet- Ik clas- sification and MS-CÓCO retrieval tasks, rounded to three decimal places for models trained on DataComp-1 1.4B with constant learning rate scheduler.
mage Rogs Ert hutal
MANKUT
101
Compute C[CLLPP)
Compute COGTLOPS)
(a) CLIP on Re-LAION, DataComp. DFN
(b) MaMMUT on Re-LAION, DataComp, DFN
Figure 10: Scaling laws for MS-COCO image retrieval performance (1- Recall @5) of openCLIP (left) and openMaMMUT (right), comparing training on Re-LAION-1 1.4B, DataComp- 1..4B and D.N-1.4B. Training DFN-1.4B results again in superior performance across scales consistently for both architectures.
Data efficiency on Re-LAION and DFN. As we see from Fig. II MaMMUT exhibits consistently more superior scaling with respect training data size on both Re-LAION-1 .B and DFN-1.4B. This supports the conclusion that MaMMUT is more data efficient across multiple training datasets.
Training trial-to-trial variance. To perform trial-to-trial variance sanity check for model pre- training, ensuring that trial-to-trial variance of same runs substantially smaller that variance due to scaling or hyperparameter tuning, show downstream task performance on zero-shot INIK as
24

---

OUP 2E2T 2he Mr 325
33
2013339 aptt
MAMNUT 30003 -32
MANU p1358271 64% -0017-300-11 6.22
Imagetent (ERnr PANN!
Image Paate
1x18-1
Samples
Samples
CLIP MaMMUT, Re-LAION-1.44 1.4B
(b) CLIP vs. MaMMUT, DFN-1.4B
Figure 11: Comparison of data efficiency for CLIP and MaMMUT via scaling laws for INIk O-shot classification error on Re-LAION- L.4B (a) and DFN-1.4B (b). MaMMUT is consistently more data efficient on both datasets, which is also in accord with observations from DataComp-1.4B.
observed for 3 training runs of the same configuration for reference scales B-32 and B-16 on 640M samples, using same hyperparameters that correspond to minimum loss obtained in tuning, residing on Pareto front. As the results in Tab. suggest, variance negligibly small compared to difference due to changing the compute scale. This allows us to conclude that measurements we use for scaling laws derivation can be distorted only insignificantly by trial-to-trial training variance, and scaling trends we observe are valid and shaped dominantly by training compute.
Trial/Training
2
3
+
B-32 640M
0,58522
0.57866
0,58014
0.58134 = 0.0034407
B-16 640M
0.66926
0.6668
0.6631
0.666387 0.00310073
I.114 640M
0.72368
0.7203
0.72398
0.722653 0.00204356
Table Trial-to-trial variance control experiment. INIk zero-shot top-1 on DataComp-1 640M samples seen. Mean and standard deviation computed for each reference scale over 3 different training runs. Hyperparameters for each reference scale training run are fixed and correspond to hyperparameters tuned to obtain minimal loss via multiple sweeps for each given reference scale. Trial to trial variance is negligible small compared to performance difference across the scales and is decreasing with increasing performance level.
B.1 Further models on 12.8B sample scale
We trained models of various scales B/32, B/16 on 12.8B sample scale on DataComp and DFN datasets, complementing results for L/14 from Sec. In Tab. g we show that predictions for 12.8B scale are consistently overestimating observed performance (for both B/32 and L/14 taken as examples). As already mentioned in Sec. 3.5, this is expected predictions are done for unique samples seen, while observed performance comes from training with repetitions. We show further IN1-k 0-shot performance numbers for DataComp and DFN 12.8B scale for openMaMMUT B/32 and B/16 in Tab. 19, in addition to MpMM.MMMMTT L/14 12.8B reported in Sec. 3.5 As predicted by scaling laws based comparison, DFN trained models consistently outperform DataComp on 12.8B scale.
B.2 Effect of the number of points used for the scaling law fits
For scaling law derivation on DataComp- 1.4B 4B (Fig. D. we used different number of points for MaMMUT (1010 points) and CLIP (672 points). Since MaMMUT architecture was never trained before on open data like DataComp, we had to perform more rigorous hyper-parameterr search than for CLIP, hence we ended up with a larger set measurements. To understand whether the number of measurements that have obtained affects our conclusions conduct additional experiment to
25

---

Type
model
samples
gllops
accl, w. rpt.
acc predicted, unique samples (CI 95%)
CLIP
B/32
12.8B
1889+111
0,6864
0.699 (0.693, 0.706)
MaMMUT
B/32
12.8B
3,50e+11
0.69468
0.710 (0. (0.702, 0.718)
CLIP
L/14
12.8B
2.14e+12
0.792
0.796 (0.788, 0.804)
MaMMUT
1/14
12.8B
2.59e+12
0.8034
0.820 (0.815, 0.826)
Table 8: Predictions for compute budget corresponding 12.8B samples seen scales, ViT B/32 and
L/14 model scales derived from scaling law measured on DataComp-1 1.4.. Shown predictions for IN O-shot classification accuracy, assuming compute-optimal training and unique or low repetitions samples, compared to measured performance on 12.8B samples from DataComp-1.44 1.44, which results in about 9x repetition factor. As it known that repetitions may result in diminished performance, the measured performance is expectedly lower than predictions for unique samples. Discrepancy between predicted and measured performance be also due training with given scale combinations residing on compute-ope optimal Pareto frontier.
Dataset
Model
Samples
INTK O-shot Acc
B/32
12.8B
0.6947
DataComp
B/16
12.8B
0.7546
L/14
12.8B
0.8034
DFN
B/32
12,8B
0.7213
B/16
12.8B
0.7773
Table 9: ImageNet- IK zero-shot classification accuracy for openMaMMMUT models trained on 12.8B samples from DataComp-1.49 and DFN-1.4B datasets. Following predictions from scaling laws, DFN trained models consistently outperform DataComp larger 12.8B scale.
double check whether there is any difference the obtained scaling law working with same number
of points for MaMMUT for CLIP on DataComp- 1.4B. We perform bootstrapping, sampling randomly 672 points from 1010 available points for MaMMUT, doing 10 trials, fitting scaling law
for each trial and averaging the obtained scaling law coefficients. We observe that the obtained fit
coefficients have no significant difference from scaling law obtained with 1000 points I0
Thus, the
Model CLIP
A
B.
O2
E
57.862083
18.391321
-0.226604
0.111169
MaMMUT (full points)
125.356572
19.289384
-0.255670
0.101112
MaMMUT (same points num. as CLIP)
125.267163
19.301461
-0.255934
0.108208
Table 10: Comparison of different models with their corresponding parameters.
comparison reduced points shows similar trends. Measurements are balanced for Re-LAION (750 vs 703 points), well as for DFN (737 732 points).
C Evaluating scaling law fit quality
To validate our scaling law fits, we use a threshold Chureshold to up which we take the data for the fit. We compute RMSE for the held-out points to get a measure of how good each fit We compare two Chrrehold values (see Tab. and Fig. 20 for DataComp- 1.4B dataset). We see that both RMSE and uncertainty (the width of the confidence intervals) decreases as we take more the more pointsfor the fit.
We also compare different functional forms that can be used to fit the data: model with double saturation (L(C") A. (C + B.) + E.) and without a term for irreducible error Eg:
((C) A. (C B.)
We choose first Chrrehoold = 2.5- GFLOPs and the second Cihreshold = 5-10" GFLOPs. As we see from Tab. TT and Tab. for both values of Cihreshold double saturation form (Eq. I has consistently lower RMSE than the function without irreducible error. RMSE on held out points provides thus way to select among various scaling law fits the candidate that provides better prediction accuracy for unseen scales, which in our case is the fit obtained via double saturation functional form (Eq.
26

---

We see that the same trend of reducing confidence intervals and thus reducing uncertainty of the predictions when taking more points for the scaling law fit holds also for other tasks like MS- COCO image retrieval and other pre-training dataset Re-LAION-1 (see Fig. 20 and Fig. 13
for comparison between ImageNet-Ik lk classification and MS-COCO retrieval and Figs. 18] Re-LAION-
19 for
When comparing predictions with actually measured downstream task performance, we see that accuracy for the held-out points is high (Tab. TT.. For instance, we measure for 3B samples seen scale on held-out points for openMaMMUT ViT L- zero-shot I 0.784, with prediction 0.777 and 95% confidence interval (0.771, 0.783), and for openMaMMUT ViT H-14 14 0.795, with prediction 0.801 and 95% CI of (0.793, 0.809). Similar accuracy is observed for openCLIP, with actual measurements falling within predicted confidence intervals. The derived scaling laws provide thus solid ground for comparison on unseen scales that have low amount of repetitions (less than 3x in case of 3B samples seen scale when training on DataComp- 1.4B or Re-LAION-1.4)
As already discussed in Sec. 3.5 the prediction for performance of MaMMUT L-14 on the larger 12.8B samples seen scale (Tab. I zero-shot INIK 0.820, 95% CI (0.815, 0.826)) is therefore only made for low repetition scenario, and to validate it, dataset size larger than currently used 1.4B samples (which gives around 9x repetitions for 12.8B samples seen scale) is required. The measured 0.803 for openMaMMMT L-14 on 12.8B (Tab. 3) is thus expectedly below the prediction. as performance is diminished due to high amount of repetitions, in line with observations by previous works 49
INIk
Predicted INIk
Predicted (more points) INIK
Model
Samples
GFLOPs
0-shot acc
D-shot (95%
0-shot (95%
CLIP
16
3.07e+9
4.07e+11
0.761
0.747 (0.738, 0.755)
VIT-I--14
3.07c49
5.18ce11
0.766
0.753 (0.744, 0.762)
0.759 (0.751, 0.766) 0.779 (,79)
VIT-H-14
3.07c+9
1,14e+12
0.784
0.773 (0.761,0..7888
RMSE: 1.26e-02
RMSE (more points):
3.90e-03
MaMMUT
mammut- ViT.L- 14
1.28et9
2.59ct11
0.749
0.743 (0.737, 0.748)
mammur- mammur-ViT-L-HM 14
3.07e49
6.22e+11
0.784
0.773 (0.765, 0.781)
0.777 (0.771, 0.783)
maminut- WIT-H-14
3,07c49
1,43c+12
0,798
0.797 (0,787, 0.807)
0.801 (0.793, 0.809)
RMSE: 7.576-03
RMSE (more points): 7.37e-03
Table 11: Predicting held-out points on compute-optimal Pareto front based on scaling law derivation for the functional form with double saturation (Eq. [ To check prediction accuracy when extrapo- lating beyond points taken for the fit, we predict starting from different compute threshold values MUT 2.59 10 Citreshold points themselves are predicted by of CCLLP = 4.07 10 Chrespold
taking smaller Ceutoit = 2.5 2. The last column contains updated predictions made after taking additional data points up to Ciurehold, showing predictions that extrapolate 2.4 4 and 5.5 compute factor beyond the fit for MaMMUT, and 1.3 and 2.8 for CLIP. Both confidence interval and RMSE decrease as we take more points. RMSE is consistently lower than RMSE measured for functional form without irreducible error (Tab. 12
INIK
Predicted IN1K
Predicted (more points) INIK O-shot (95% CI)
Model
Samples Seen
GFLOPs
O-shot
D-shol (95% CI)
CLIP
VIT-I.-16
3.07e19
4.07e=11
0.761
0.769 (0.764, 0.773)
14
3.07e+9
5.18e+11
0.766
0.778 (0.774, 0.783)
0.777 (0.773, 0,782) 0.805 (0.801, 0.809)
V4
3.07e+9
1.14e=12
0.784
0.806 (0.802, 0.811)
RMSE: 1.55e-02
RMSE (more pointsj:
1.72e-02
MaMMUT
mammur- 14T-L-14
mammur-ViT-L-14
I. 128849
2.59ce11
0.749
0.757 (0.754, 0.760)
mammut. ViT.L-14
3.07et9
6.22e+11
0.784
0.795 (0.792, 0.798)
0.794 (9.791, 0.796)
mammut- mammut-ViT-H..L 14
3.07e49
124
0.794
0.825 (0.822, 0.828)
0.824 (0.822, 0.827)
RMSE: 1.98e-02
RMSE (more points):
2.26e-02
Table 12: Predicting held-out points on compute-optimal Pareto front based on scaling law derivation for the functional form without irreducible error (Eq. 33) Comparing prediction quality to the functional form with double saturation (Tab. IT). using same values for Ccutoit and Cibreshola- The last column contains updated predictions made after taking additional data points up to Cirreshola- Both confidence interval and RMSE decrease as we take more points. RMSE consistently higher than
RMSE measured for functional form with double saturation that includes irreducible error (Tab.
27

---

CUl Suieeii ued 2ig-mart
cook 0121--14
Compure C (GRLOOS)
Compute COGFLOPS)
(a) CLIP vs SIGLIP on DataComp-1.c 1.48
(b) MaMMUT vs CoCa vs CLIP on DataComp- 1.4B
Figure 12: Scaling laws for ImageNet- 1 O-shot classification, comparing SigLIP (left) and CoCa (right) with standard CLIP and MaMMUT using open DataComp-1.44 C. dataset. SigLIP shows benefit over standard CLIP, contrary to claims previous work. CoCa is predicted to less scalable than MaMMUT, while crossing CLIP is possible, although is clear due to high uncertainty for CoCa estimates on larger scales, as measurements on smaller scales for CoCa are not dense enough.
Image Ruttoel (10
Compute C [GELOPs)
Combute CCGTLOP))
(a) Scaling law fit for MS-COCO image retrieval error rate (1-Recall @ C-CCC image retrievat error rate (1-Recall @5) for DataComp- (Crveshold 2.5 10th GFLOPs)
(b) Scaling law fit for
for DataComp-1.4p using more points
(Crusshola 10 GFLOPs)
Figure 13: Comparison of the fit quality for MS-COCO image retrieval error rate for OpenMaMMUT and openCLIP. Adding more points in (b) reduces the uncertainty of the fit, indicated by the width of bands around each curve.
D Datasets comparison
Scaling laws can be also be used as tool for dataset comparison. Here, we compare performance of models trained on two reference datasets (DataComp- 1.100mpp1.4BBB and Re-LAION-1.4B) for both model architectures CLIP and MaMMUT, for two downstream tasks ImageNet-1k k 0-shot classification and MS-COCO retrieval. For CLIP, we additionally plot OpenAI CLIP models' performance that were trained the WIT-400M dataset. As see from Fig. 7 for both CLIP and MaMMUT, training DataComp-1.4B provides superior scalability for zero-shot ImageNet-lk classification, compared to training on Re-LAION-I.44 At the same time, training either on Se-LAION-1.45 or DataComp- 1.4B leads to similar scalability and performance on MS-COCO retrieval, with Re-LAION-1.4B being for retrieval slightly more beneficial (Fig. {).
Using much denser measurements for scaling law derivation, we can also confirm findings from previous work [10), which showed that the closed dataset WIT-400M 7 has better scaling trend on zero-shot classification, but worse scaling trend on zero-shot retrieval when compared LAION-2B. We observe the same for Re- Re-LAION-I.4B, which is safety update of LAION-2B used in [10).
28

---

otherwise being same dataset with less samples due to link rot [2T. This provides further evidence for robustness of scaling law based comparison, showing consistent trends despite major difference in scaling law derivation. Previous work [IO] used few samples seen scales of 3B, 12.8B and 34B, which results in high repetition given that only 2B unique samples are contained in LAION-2B [50), while our work used denser lower samples seen scales up to 3B, doing derivation with unique samples or low repetition only, Despite these differences, derived scaling laws agree the dataset comparison for each downstream task, predicting same scaling trends in favor of WIT-400M on classification and in favor of Re-LAION-IL4D on retrieval. DataComp-1 1.4B can be seen this comparison as an improved version of Re-LAION-L.AR 1.4B, with stronger scalability classification (Fig. 7 that matches WIT-400M, while obtaining performance for retrieval (Fig. that matches Re-LAION-1.4B, outperforming WIT-400M.
E Scaling behavior of other architectures
We investigate additional model architectures: SigLIP [8] (CLIP with the sigmoid loss instead of softmax), CoCa [16] and Cap [45 (pure captioner). We train these models on DataComp-1.44 in
order to compare with openCLIP and openMaMMUT. Fig. 12 shows the fitted lines for these models.
We see that CLIP and SigLIP have very similar scaling behavior on ImageNet- Ik classification (Fig. 12 (a)) while MpeMMaMMUT consistently overtakes CoCa on the same compute scale Fig. 12 (b). Notably, analysis shows that SigLIP has similar even worse scalability than CLIP which contradicts recent claims of SigLIP being a better choice for a vision encoder 18 14) due to its architectural advantages. Thus, when properly controlling for same training data in experiments, no benefits for SigLIP can be derived from the obtained scaling law trends. We also observe that text decoder-only MaMMUT overtakes encoder-decode CoCa on the same compute scale, indicating that simpler and more parameter efficient architecture of MaMMUT might be preferable.
Moreover, we see (Fig. 21 that MaMMUT has superior scaling compared to Cap, showing that combination contrastive and captioning losses is advantageous. We see Cap also underperform- ing standard CLIP, hinting that Cap captioner only based architecture is not good candidate for strong scalability O-shot regimes, making another case for contrastive losses being important part of scalable architectures for 0-shot classification. It is further important to note that Cap can use only log-likelihood based evaluation for zero-shot classification task, as opposed to CLIP and MaMMUT embedding similarity based evaluation thanks to their contrastive loss. As evident from Fig. 21 embedding similarity based evaluation used in openCLIP and openMaMMUT has strong advantage over log-likel likeliodldood based one. It in addition also much cheaper in execution. Cap has thus architectural disadvantage in not being able to use similarity based evaluation due to missing contrastive loss, which leads to inferior performance in 0-shot regime.
that in addition
For both comparisons, we see uncertainty getting high when extrapolating to larger scales, which makes it for instance hard to predict whether CoCa might still cross CLIP or not. To reduce uncertainty, it is thus important to both conduct dense measurements at smaller scales and not to cut off measurements scales too small to be used for proper extrapolation.
Hyperparameter
Value
Model Architecture
mammut- ViT-L-14
Samples Seen
12.8B
Warmup Steps
6000
Global Batch Size
180,224
Learning Rate
2.5 x 10-
GPU Hours
3.53 x 1
Number NVIDIA A.00 GPUs
1024
Table 13: Training hyperparameters for openMaMMUT-L-11,
29

---

Model
Samples Seen
Warmup
Global Batch Size
Learning Rate
mammut-ViT-SS
1.28c+06
1000
512
1.00e-03
HammuttViT---32
1.28c+06
1500
512
5.00c-04
mammut-Vitss ViT----1
1.28e+06
1000
512
5.00e-04
mammut- ViT-S-32
3.07e+06
4000
512
5.00e-04
mammut- ViT-S-16
3.07e+06
4000
512
1.00e-03
mammut- ViT-S-32
6.40e+06
4000
1024
1.00e-03
manmut-ViT--S32
1.28c+07
4000
2048
2.00e-03
ViT---1
1.28e+07
3000
2048
2.00e-03
mammut- ViT-S-32
nammuu-Vii--S32
3.07e+07
4000
4096
2.00e-03
mammut- 16
3.07c+07
3000
4096
2.00c-03
Nammuu--Vi---33
6.40c+07
4000
4096
2.00c-03
VIT----6
6.40c+07
4000
4096
1.50e-03
mammut- ViTT-S33
1.28e+08
4000
8192
2.00c-03
mammut- ViT-S-14
1.28e+08
4000
8192
2.00e-03
mammut- ViI--uu-ViT---16 ViT-M-16
1.28e+08
4000
8192
2.00e-03
mammut- 14
3.07e+08
4000
16384
2.00e-03
HammuttviT----I
3.07++08
4000
16384
2.00e-03
mammut- ViT-S-144
6.40e+08
4000
16384
1.50e-03
mammut- B-6
3.07++08
4000
16384
2.00e-03
mammut- ViT-B-32
1.28c+09
4000
16384
2.00e-03
mammut- - ViT-B-16
6.40e+08
4000
32768
2.00c-03
mammut- V1T-B-14
1.28e+09
4000
90624
2.00e-03
mammut- ViT-L-166
6.40e+08
6000
45056
2.00e-03
mammut- ViT-L-14
6.40e+08
6000
45056
2.00e-03
mammut- ViT-L-14
1.28e+09
4000
90624
2.00c-03
mammut- ViT-L-16
3.07c+09
4000
91136
2.00e-03
mammut- VIT-L-1
3.07e+09
4000
91136
2.00e-03
Table 14: Hyperparameters for MaMMUT models trained DataComp- 1.4B Pareto frontier
1.44B that are located on the
F Results additional benchmarks
We also fit scaling laws the data for other downstream tasks. In the Fig. 14 we show the scaling behavior on DataComp eval suite, which is constituted by averaging over 35 classification tasks from DataComp (see Tab. 15 from 19) Additionally, we provide scaling law fits for ImageNet-Vi V2 and full ImageNet robustness set O-shot classification performance for both openMaMMUT and openCLIP (Fig. 15). For all of these tasks we see the same trend openMaMMUT stronger scalable than CLe and has higher performance given the same compute at larger compute scales. This is also valid for the important robustness metrics that reflects diitt-iiiitiittiionn generalization (Fig. 15) MaMMM shows stronger scalable robustness and outperforms openCLIP in robustness at larger compute scales.
G Additional training details
In Tab. 13 we provide hyperparameters that were used for training openMaMMUT L/14 at larger 12.8B scale. Additionally, in the Tab. [5] and [4 we provide training hyperparameters for all models and sample seen scales that were used for scaling law fits (i.e. models that are located on the Pareto frontier) for openMaMMUT and openCLIP respectively. In Tab. T6 we provide overview of hyperparameters used for the openCLIP and openMammut models that reside close to compute-optimal Pareto frontier. In Tab. 18] we show for selection of various model, sample seen scales and pre-training datasets hyperparameter combinations used for training for openCLIP [7 shows hyperparameter ranges used for scaling law measurements for
and open MaMMMT Tab. [7 various architecture types. Finally, Tab. contaiins samples of IN1K 0-shot accuracy numbers for the selection of model, sample scales and pre-traine pre-training datasets for openCLIP and opeMMaMMUT.
30

---

Model
Samples Seen
Warmup
Global Batch Size
Learning Rate
ViT-S-32
1.28e+06
1500
512
5.00e-04
ViT-S-16
1.28c+06
1500
512
5.00e-04
ViT-S-16
1.28e+06
1500
512
2.00c-03
ViT-S-32
3.07e+06
1500
1024
5.00e-04
ViT-S-32
6.40e+06
4000
1024
1.00e-03
ViT-S-32
1.28e+07
4000
2048
1.00e-03
ViT-M-32
1.28c+07
3000
2048
1.00e-03
ViT-S-32
3.07e+07
4000
4096
2.00e-03
ViT-S-32
6.40e+07
4000
4096
2.00e-03
ViT-M-32
6.40e+07
10000
4096
1.00e-03
ViT-S-32
1.28c+08
6000
8192
2.00e-03
ViT-S-16
1.28c+08
6000
8192
2.00e-03
ViT-S-32
3.07c+08
8000
16384
2.00c-03
ViT-S-32
6.40e+08
4000
16384
2.00e-03
ViT-S-14
3.07e+08
4000
16384
2.00e-03
ViT-M-32
6.40e+08
6000
32800
2.00e-03
ViT-B-32
1.28c+09
15000
16384
1.00e-03
ViT-L-32
6.40e+08
4000
45056
2.00e-03
ViT-B-16-text-plus
6.40e+08
6000
32768
2.00e-03
ViT-L-32
1.28e+09
4000
90624
4.00e-03
ViT-L-16
6.40c+08
4000
45056
2.00c-03
ViT-L-32
3.07c+09
4000
91136
4.00c-03
VIT-L-14
1.28c+09
4000
90624
4.00c-03
ViT-L-16
3.07e+09
4000
91136
4.00e-03
ViT-L-14
3.07e+09
4000
91136
4.00e-03
Table 15: Hyperparameters for CLIP models trained on DataComp-1.44 1.44 that are located the Pareto frontier.
21a
DENCCMT Tave Suate Dubbet BaNe
CEP 7277-4177
MAMMUT CO HApCEIANED 2117
10
101r
Compute [SFLORs]
Figure 14: Scaling law on DataComp evaluation suite (average over 35 tasks, 0-shot classification), openCLIP vs. openMaMMUT comparison on DataComp-I.4B
31

---

6x12
23.01
Cur 2.A
MANPLU
CEGELOPS]
CEGFLOPS)
(a) CLIP MaMMUT, ImageNet V2
(b) CLIP vs. MaMMUT, ImageNet Robustness
Figure 15: Scaling laws for ImageNet-v2 (left) and ImageNet robustness set (right, averaged perfor- mance across datasets ImageNet-v2222 (27, ImageNettR([8) ImageNet-Sketch (30, ObjectNet(3]. and ImageNet-A 22 O-shot classification for openCLIP and openMaMMUT comparison on DataComp-1.4B
10
VT-S-14
VIT-5-16
VIT-5-32
ImageNet1k O-shot [Error Rate]
VIT-M-16
VIT-B-14
10-1
VTT-B-16-text
VTT8-32
VIT-L-14
VIT-L-16
VIT-L32
4 10
VIT-H-14
VIT-H-16
VIT-H-22
x 10
CUP: 57.86 (x exx(18.39) 9227 -
2 x 10
107
10a
1010 Compute [GFLOPS]
109
10
10
Figure 16: Detailed version of the scaling law fit for ImageNet 0-shot classification error rate for DataComp-I.4B for openCLIP. Cooler colors indicate smaller models. Bigger models are bottlenecked by samples seen scale (require larger samples models saturate with increased data and compute scale (over-training regime). Pareto front is composed by taking for each compute budget the points corresponding models reaching minimal error rate for the given compute. Fit is performed through points on Pareto front.
than the smaller ones) and smaller
32

---

100
mammut VIT-S444
mammut VIT-S-16
Image 0-shol [Error Rate]
mammut VITMM:14
mammut VIT-M-16
mammut
6
mammut VIT-B-14
mammut VIT-8-16
mammut
mammut VIT-L-14
xx
mammut VIT-L-16
mammut VIT-L32
ViT+-14
mammut VIT+--16
x 10
mammut VIT--32
(x exp(19.299) 6251
10-1
10a
10
10
1011
1012
Compute [GFLOPS]
Figure 17: Detailed version of the scaling law fit for ImageNet O-shot classification error rate for DataComp-1 1ammp-1.44 I.4B for OpenMaMMUT,. Cooler colors indicate smaller models. Bigger models scale (require larger samples
bottlenecked by samples smaller models saturate with increased data and compute scale (overtraining regime). Pareto front is composed by taking for each compute budget the points corresponding models reaching minimal error rate for the given compute. Fit is performed through points on Pareto front.
than the smaller ones) and
10
-0-
6000
Compute (CHLOPN)
Computt CI (OLLOPS)
(a) Scaling law fit for ImageNet O-shot classification error for Re-LAION-L 1.0N-1144B (Catvobold
(b) Scaling law fit for ImageNet O-shot classification error rate for Re-LAION- 1.4B using more points
10St GFLOPs)
(Ciheesold = 10th GFLOPs)
Figure 18: Comparison of the fit quality for ImageNet- 1kk 0-shot classification error rate for openMaM- MUT and openCLIP trained on Re-LAION-1 1.4B. Adding more points in (b) reduces the uncertainty of the fit compared to (a), indicated by the width of bands around each curve.
33

---

Name
Width
Emb
Depth
Params (M)
GFLOPs
ViT-S-32
384/384
384
12/12
63.09
5.51
mammut- VVT---22
384/384
384
12/12
85.62
13.91
ViT-S-16
384/384
384
12/12
62.26
11.75
mammut- ViT-S-16
mammut-vin-S-16
384/384
384
12/12
84.79
20.72
ViT-S-14
384/384
384
12/12
62.21
14.3
mammut- ViT-S-14
384/384
384
12/12
84.74
23.5
ViT-M-32
512/512
512
12/12
103.12
9.74
nammut-ViT-M-32
512/512
512
12/12
134.73
22.1
ViT-M-16
512/512
512
12/12
102.02
20.84
-iimuut-iiT---6
512/512
512
12/12
133.63
34.2
ViT-M-14
512/512
512
12/12
101.95
25.37
mammut-vit---4
512/512
512
12/12
133.57
39.14
ViT-B-32
768/512
512
12/12
151.28
14.54
mammut- mammut-Vit-B-3.
ViT-B-32
768/512
512
12/12
183.02
26.91
ViT-B-16
768/512
512
12/12
149.62
39.51
ViT-B-16-text-plus
768/768
768
12/12
210.04
46.78
nammut-Vit-B-16
768/512
512
12/12
290.52
79.7
ViT-B-14
768/512
512
12/12
149.53
49.7
mammut- mammut-Vit-B-14
768/512
512
12/12
181.27
63.54
ViT-L-32
1024/768
768
24/12
429.95
43.59
mammut-ViT-L-32
1024/768
768
24/12
510.63
74.28
ViT-L-16
1024/768
768
24/12
427.74
132.37
mammuu-ViT-L-16
1024/768
768
24/12
508.42
165.37
ViT-L-14
1024/768
768
24/12
427.62
168.61
mammut- mammut-ViT-L-IA ViT-L-14
1024/768
768
24/12
508.29
202.56
ViT-H-32
1280/1024
1024
32/24
989.02
109.81
mammut- -VT----
1280/1024
1024
32/24
1191.06
192.97
ViT-H-16
1280/1024
1024
32/24
986.26
294.78
mammut-Vit-H-16
1280/1024
1024
32/24
1188.3
385.72
ViT-H-14
1280/1024
1024
32/24
986.11
370.28
mammut-ViT-H-14
1280/1024
1024
32/24
1188.1 14
464.39
Table 16: Hyper-parameters of architectures consider. Width refers encoder width, Emb refers to embedding size, Depth refers to number of layers. Params refer to the number of parameters in millions, and GFLOPs refer total GFLOPs per forward pass. Entries in the form of A B denote image and text parameters respectively. There are more parameters in MaMMUT models because of the additional cross-attention layers.
H More details on fine-tuning for segmentation and scaling laws
Following prior work on how to benchmark vision foundation models for semantic segmentation 35 we evaluate CLIP and MaMMUT on semantic segmentation by fine-tuning them end-to-end using linear decoder on ADE20K 34]. Regardless of the patch size used during pre-training, we interpolate the patch size all models to 14 14. ensure fair comparison. We use an image input size of 224 x 224 and thus interpolate the positional embedding to 16x 16. Hyperparameters used for training are consistent with (35), except the use of linear learning rate warmup of 1,500 steps, an epoch- based schedule of 31 epochs, and batch size of 16 without gradient accumulation, following [36]. We fine-tune pre-trained models up to and including VIT-L and 3B samples seen, with different pre-training hyperparameters. We evaluate using sliding window approach, again following 335
Fig. 22 and Fig. 23 show the fitted scaling laws for CLIP and MaMMUT, respectively. Tab. 20 shows
the corresponding estimated scaling law fit parameters.
34

---

imagge
3
CUP 28 L27 ME 4026
3x30
CUT 47.371N WPERR
38 BOTS AgCAATT
(CGPLOPs
(a) Scaling law fit for MS-COCO image retrieval rate (I-Recall@ M)ecall@@MMS-OGR COCO image retrieval rate (I-Recall ( for Re-LAION--LAH
(b) Scaling law fit for
for Re-LAION-1.41
(Chtrehold 2.5 10t GFLOPs)
(Chhesholld = 5 10 GFLOPs)
Figure 19: Comparison of the fit quality for MS-COCO image retrieval error for openMaMMUT and openCLIP trained on Re-LAION- 1.B.. Adding more points in (b) reduces the uncertainty of the fit compared (a), indicated by the width of bands around each curve.
10
2412
Compute (GPLOPs)
Comsute (CPLOPS)
(a) Scaling law for ImageNet O-shot classification error rate for DataComp-1.44 (Carebold 2.5 10th GFLOPs)
(b) Scaling law fit for ImageNet O-shot classification error rate for DataComp- Comp-1.4B 1.4B using points (Cilreshold 10th GFLOPs)
Figure 20: Comparison of the fit quality for ImageNNe- 1k O-shot classification error rate for openMaM- MUT and openCLIP trained on DataComp- 1.4B. Adding more points in (b) reduces the uncertainty of the fit compared to (a), indicated by the width of bands around each curve.
Table 17: Hyperparameters for different model architectures used for scaling law measurement experiments.
Model architecture
Learning Rate
Global Batch Size
Warmup Steps min
min
max
min
max
max
Cap
lc-04
3e-03
512
90,624
1,000
20,000
CLIP
le-04
le-02
512
181,248
1,000
15,000
CoCa
lc-04
2e-03
512
45,056
1,500
10,000
MaMMUT
le-05
le-02
256
181,632
50
20,000
SigLIP
5e-05
2e-03
64
45,056
500
10,000
35

---

Table 18: Hyperparameters used for training to obtain scaling law measurements for openCLIP and openMaMMUT, shown for selection of different combinations of model, samples seen scales and pretraining datasets. NaN indicates that training for the particular combination was not performed.
Dataset
Model Size
Samples Seen
Learning Rate
Warmup
Batch size
clip
mammut
clip
mammut
clip
mammut
DFN-1.4B
ViT-B-16
307M
2.00e-03
1.50e-03
4.00e+03
4.00e+03
1.64e+04
1. 1.64++04
DFN-1 ,4B
ViT-B-16
640M
2.00e-03
2.00e-03
4.00e+03
6.00e+03
4.51e+04
3. 3.28e+04
DFN-1.45
ViT-B-16
1.00e-03
1.50e-03
6.00e+03
6.00c+03
1.6c+044
3.28c+04
DFN-1.4B
ViT-B-16
3B
NaN
2.00e-03
NaN
6.00e+03
NaN
9.01e+04
DFN-1.44 1.4B
ViT-B-32
307M
1. 1.50e-03
2.00e-03
4.00e+03
4.00e+03
1.64++04
1.64e+04
DFN-1.45
ViT-B-32
640M
1.00e-03
2.00e-03
6.00e+03
6.00e+03
1.64e+04
3.28e+04
DFN-1 1.4B
ViT-B-32
1B
2.00e-03
2.00e-03
6.00e+03
6.00e+03
9.01e+04
6.55e+04
DFN-1.4 1.4B
ViT-B-32
3B
2.00c-03
2.00e-03
1.00c+04
1.00e+04
6.55c+04
6.55e+04
DFN-1 .BB
VIT-H-14
307M
1.50e-03
1.50e-03
4.00e+03
4.00e+03
3.28e+04
1.64++04
DFN-1 1.4B
VIT-H-14
640M
2.00e-03
2.00e-03
4.00e+03
4.00e+03
4.51e+04
4.51e+04
DFN-1 1.4B
VIT-L-14
307M
2.00e-03
2.00e-03
4.00e+03
4.00e+03
3.28e+04
1.64e+04
DFN-1.4B
ViT-L-14
640M
2.00e-03
2.00e-03
6.00e+03
6.00e+03
3.28e+04
4.51e+04
DFN-1.4B
ViT-L-14
IB
4.00e-03
2.00c-03
4.00c+03
4.00e+03
9.01e+04
9.01e+04
DataComp-Ia 1.4B
ViT-B-16
307M
2.00e-03
2.00c-03
4.00c+03
4.00e+03
1.64e+04
1.64e+04
DataComp-1 1.44
ViT-B-16
640M
2.00e-03
2.00c-03
6.00e+03
4.00e+03
3.28e+04
3.28e+04
DataComp-1.44 1.4B
ViT-B-16
IB
1.00e-03
1.00e-03
4.00e+03
4.00c+03
1.64e+04
1.64e+04
DataComp-1.44 1.4B
ViT-B-16
3B
2.00e-03
2.00e-03
4.00c+03
4.00e+03
4.53e+04
9.06e+04
DataComp-1.4e
ViT-B-32
307M
2.00c-03
2.00c-03
4.00e+03
4.00c+03
1.64e404
1.64e+04
DataComp-1.4B
VIT-B-32
640M
NaN
1.50e-03
NaN
4.00e+03
NaN
1.64e+04
DataComp-1,4B
ViT-B-32
1. 1.00e-03
2.00e-03
1.50e+04
4.00e+03
1.64e+04
1.64e+04
DataComp-1.4B
ViT-B-32
3B
3.00e-03
2.00e-03
4.00e+03
4.00e+03
9.06e+04
4.53e+04
DataComp-1,4B
V1T-H-14
307M
2.00e-03
1.50e-03
6.00e+03
6.00e+03
1.63e+04
1.63e+04
DataComp-1.4B
ViT-H-14
640M
2.00e-03
2.00e-03
4,00c+03
6.00e+03
4.51e+04
3.20e+04
DataComp-1.4B
V1T-H-14
2.00e-03
2.00e-03
6.00e+03
6.00e+03
9.01e+04
9.02e+04
DataComp-1.4B
VIT-H-14
3B
1.50e-03
1.50e-03
4.00e+03
6.00e+03
9.02e+04
9.02e+04
DataComp-1.48
VIT-L-14
307M
1.50e-03
1.50e-03
3.00e+03
4.00c+03
3.28e+04
1.64e+04
DataComp-1.4B
VIT-L-14
640M
2.00c-03
2.00c-03
4.00c+03
6.00e+03
4.51e+04
4.51e+04
DataComp-1.49
VIT-L-14
4.00c-03
2.00c-03
4.00c+03
4.00c+03
9.06c+04
9.06c+04
DataComp-1.4B
VIT-L-14
3B
4.00e-03
2.00e-03
4.00e+03
4.00e+03
9.11e+04
9.11e+04
Re-LAION-1.4B
VIT-B-16
307M
1.50e-03
1.50e-03
4.00e+03
4.00e+03
1.64e+04
1.64e+04
Re-LAION-1.48
VIT-B-16
640M
2.00e-03
2.00e-03
6.00e+03
4.00e+03
4.51e+04
4.51e+04
Re-LAION-1.4B
ViT-B-16
IB
2.00c-03
2.00c-03
6.00e+03
6.00e+03
6.55e+04
6.55e+04
Re-LAION-1.44
ViT-B-16
3B
2.00e-03
2.00c-03
1.00e+04
6.00e+03
6.55e+04
9.01e+04
Re-LAION-1.4B
ViT-B-32
307M
2.00e-03
2.00c-03
4.00e+03
4.00e+03
1.64e+04
1.64e+04
Re-LAION-1.4B
ViT-B-32
640M
2.00e-03
2.00e-03
4.00e403
6.00++03
3.28e+04
3.28e+04
Re-LAION-
ViT-B-32
IB
2.00c-03
1.50e-03
6.00c+03
6.00e+03
6.55e+04
3.28c+04
Re-LAION-
VIT-B-32
3B
2.00c-03
2.00c-03
6.00c+03
6.00c+03
9.01e+04
9.01c+04
Re-LAION-I.A
V1T-H-14
307M
2,00c-03
1.50c-03
4.00c+03
4.00e+03
3,28c+04
3.28c+04
Re-LAION-1
V4
640M
2.00e-03
1.50e-03
4.00e+03
4.00e+03
4.51e+04
4.51e+04
Re-LAION-
VIT-H-14
1B
2.00e-03
1.50e-03
6.00e+03
6.00e+03
9.01e+04
9.01e+04
Re-LAION-1.44
VIT-H-14
3B
NaN
4.00e-03
NaN
6.00e+03
NaN
1.80e+05
Re-LAION-1..0
VIT-L-14
307M
2.00e-03
2.00e-03
3.00e+03
3.00e+03
3.28e+04
3.28e+04
Re-LAION-I.A4
ViT-L-14
640M
2.00e-03
2.00e-03
4.00e+03
4.00e+03
4.51e+04
4.51e+04
Re-
Re-LAION-1.41
VIT-L-14
IB
4.00e-03
2.00e-03
6.00e+03
4.00e+03
9.01e+04
9.01e+04
Re-LATON-1.4B
VIT-L-14
2.00e-03
2.00e-03
4.00e+03
4.00e+03
9.01e+04
9.01e+04
36

---

Table 19: IN-IK O-shot accuracy for selection of different scale combinations and pretraining datasets for openCIIP and openMaMMUT. NaN indicates that training for the partieular combination was not performed.
Dataset
Model Size
Total Samples Seen
clip
mammut
DFN-1 14B
ViT-B-16
307M
0.6274
0.6516
DFN-1.44
ViT-B-16
640M
0.6779
0.6991
DFN-1.45
ViT-B-16
IB
0.7059
0.7268
DFN-1.4B
ViT-B-16
3B
NaN
0.7541
DFN-1 1,4B
ViT-B-32
307M
0,5524
0,5682
DFN-1.4D
ViT-B-32
640M
0.6104
0.6218
DFN-1 1.4B
ViT-B-32
IB
0.6402
0.6599
DFN-1.44
ViT-B-32
3B
0.6943
0.6947
DFN-1 1.4B
V4
307M
0.6955
0.7190
DFN-1 1 1.4B
V4T-H-14
640M
0.7477
0.7666
DFN-1 1 1,4B
VIT-I-14
307M
0.6777
0.7021
DFN-1.44 1.4B
VIT-L-114
640M
0.7258
0.7470
DFN-1.4B
VIT-L-14
IB
0.7609
0,7777
DataComp-Ia 1.4B
V6T-B-16
307M
0,5777
0.6148
DataComp-1. 1.4B
ViT-B-16
640M
0,6405
0.6693
DataComp- 1.44B
VIT-B-16
IB
0.6679
0.6908
DataComp-1.44
ViT-B-16
3B
0.7133
0.7267
DataComp-1.48
VIT-B-32
307M
0.5028
0.5257
DataComp-I.4B
VIT-B-32
640M
NaN
0.5852
DataComp-1.48
VIT-B-32
1B
0.6076
0.6209
DataComp-1,4B
VIT-B-32
3B
0.6529
0.6613
DataComp-1.48
ViT-H-14
307M
0.6483
0.6912
D.t
ViT-H-14
640M
0.7178
0.7410
DataComp-1.4B
V1T-H-14
IB
0.7517
0.7682
DataComp-1.4B
ViT-H-14
3B
0.7840
0.7986
DataComp-1.4B
ViT-L-14
307M
0.6323
0.6685
DataComp-1.4B
VIT-L-14
640M
0.6987
0.7240
DataComp-1.4B
ViT-L-14
IB
0.7317
0.7488
DataComp-1.4B
ViT-L-14
3B
0.7656
0.7845
Re-LAION--144B
VIT-B-16
307M
0.5104
0.5434
Re-LAION-1.4B
ViT-B-16
640M
0.5810
0.6094
Re-LAION-1144
ViT-B-16
IB
0.6257
0.6439
Re-LAION-1.4B
ViT-B-16
3B
0.6677
0.6830
Re-LAION-1.44B
ViT-B-32
307M
0,4401
0,4574
Re-LAION-I.4B
ViT-B-32
640M
0.5122
0.5181
Re-LAION-1.4B
VIT-B-32
IB
0.5601
0.5634
Re-LAION-
VIT-B-32
3B
0.6065
0.6075
Re-LAION-1.45
ViT-H-14
307M
0.5898
0.6214
Re-LAION-I.44
ViT-H-14
640M
0.6575
0.6843
Re-LAION-1.44
VIT-H-14
IB
0.7029
0.7258
Re-LAION-IE
VIT-H-14
3B
NaN
0.7615
Re-LAION-1.4B
VIT-L-14
307M
0.5662
0.6020
Re-LAION-1.A4
ViT-L-14
640M
0,6365
0.6652
Re-
Re-LAION-1.A4
VIT-L-14
IB
0.6818
0.7061
Re-LAION-1.A4
ViT-L-14
3B
0.7206
0.7369
37

---

30
Imagetatt G-abbo Iimos BAAN)
CAP: 3822
cLP
Soameull Lo rap.aaaaan. 2.222
337
105
201
Compute (GGLLOPS)
Figure 21: Scaling law fit for ImageNet-Ik 0-shot classification, comparing MaMMUT, CLIP and Cap (captioning only). Cap can be only evaluated via log-likelihood, which is more expensive similarity based evaluation used by CLIP and MaMMUT, Cap misses contrastive loss in its architecture, which makes it disadvantageous for 0-shot setting.
ADE20K Semantic Segmentation [Error Rate]
VIT-S-14
10
VIT-S-16
VIT.S332
VIT-M-16
VIT-M-32
8 x 10
VIT-B-14
VIT---16-tex--llus
VIT-L-14
7 10
VIT-L-16
CLIP: 17.93 exp(17.49)) 920
10-1
107
10a
10
10
10
10
Compute [GFLOPs]
Figure 22: Downstream semantic segmentation performance of CLIP pre-trained on DataComp-1.4B and fine-tuned ADE20K. Error rate (1 mloU).
Ac
Be
Qc
Eg
CLIP
18.407549
17.577295
-0.209187
0.468456
MaMMUT
352.152176
18.759619
-0.356718
0.497617
Table 20: Fitted scaling law parameters (Ac, Bes Ocey Fe.) for segmentation rate.
38

---

ADE20K Semantic Segmentation [Error Rate]
mammut VIT:S:44
mammut_Vi::-144
9 10
mammut VIT-5-16
mammut VIT.S:32
mammut VIT-M-16
8 x 10
mammut VIT-M-32
mammut
mammut VIT-8-16
mammut VIT-B-32
10
mammut VIT.L-14
mammut VIT-L-16
mammut VIT-L-32
MAMMUT: 331.17 (x (x+=)x(1..73)) +
10-1
x 10-1
10a
109
10
10
10a2
Compute [GFLOPS]
Downstream semantic segmentation performance of MaMMUT pre-trained on DataComp- 1.4B and fine-tuned on ADE20K. Error rate (I (1-mloU).
Figure 23:
39

---

I Author contributions
Marianna Nezhurina: established major part of scaling law fitting procedures. Performed analysis of scaling law fit quality, derived predictions and confidence intervals. Conducted major part of data analysis. Performed initial training experiments with openCLIP and openMaMMUT. Established environments for experiments across various supercomputers. Supported compute resource acquisition. Established infrastructure for distributed dataset acquisition via Ray. Obtained Re-LAION and part of DFN dataset. Co-organized automated experiments data collection and analysis. Wrote the manuseript.
Tomer Porian: co-designed and performed const Ir schedule scaling law derivation ex- periments. Extended automated experiments execution for const Ir schedule experiments. Collected and analyzed data, provided further input for scaling law fitting procedures. Co-wrote the manuscript.
fine-tuning experiments for dense prediction segmentation, evaluating segmentation via different modes, scaling law derivation for segmentation, data collection and analysis. Co-wrote the manuscript.
Tommie Kerssies:
Giovanni Puccetti: openMaMMUT implementation openCLIP, initial experiments with openCLIP and openMammut training. Initial co-design and implementation of automated experiments execution. Proof reading the manuscript.
Romain Beaumont Re-LAION safety maintenance, hash filtering and re-packaging. Toolsets for dataset download and composition. Proof reading the manuscript.
Mehdi Cherti: led the project, supported compute resource acquisition. Co-established environments for experiments across various supercomputers. Obtained DataComp. DFN and part of Re-LAION dataset. Designed and implemented automated experiments execution and evaluation. Wrote procedure for const Ir schedule experiments. Conducted scaling law derivation experiments (DataComp. Re-LAION, DFN; openMammut, openCLIP, Cap). Designed and implemented evaluation. Organized automated experiments data collection and analysis. Collected and analysed the experimental data. Wrote the manuscript.
Jenia Jitsev: led and coordinated the project, acquired compute resources. Organized data transfer (DataComp. Re-LAION) across the supercomputers. Co-established environments for experiments across various supercomputers. Co-designed automated experiments exe- cution. Defined, designed and conducted scaling law derivation experiments (DataComp. Re-LAION, DFN; openMammut, OpenCLIP, CoCa, SigLIP). Collected and analysed the experimental data. Trained 1penMammut-L-14 12.8B of DataComp-1 following the scaling law predictions. Led manuscript writing, wrote the manuscript.
40


Reviews:
[{"review_id": "8t1P8YCvUh", "reviewer": "Reviewer_Nr5N", "summary": "This paper presents a rigorous and large-scale empirical study that uses scaling laws as a principled framework for comparing vision-language foundation models and datasets. The authors move beyond traditional single-point comparisons by deriving full scaling laws for two important architectures, CLIP (contrastive) and MaMMUT (contrastive + captioning), across a wide range of model sizes and data scales. The experiments are conducted on two large, open datasets, DataComp-1.4B and Re-LAION-1.4B.\n\nThe core finding is that MaMMUT, which incorporates a generative captioning loss, consistently demonstrates better scalability than the purely contrastive CLIP, meaning its performance improves more rapidly with increased compute. This trend is shown to be robust across different downstream tasks (classification, retrieval, segmentation) and training datasets. The paper also provides a valuable methodology for deriving these scaling laws, including techniques for handling constant learning rate schedules to reduce compute cost. Finally, the authors release all models, checkpoints, and code, culminating in a new SOTA open model, OpenMaMMUT-L/14, which validates the predictions of their scaling analysis.", "questions": "Is it possible that the chosen hyperparameter search space or the final selected parameters favor the MaMMUT setup? For instance, the presence of a captioning loss might act as a regularizer, making the model more stable during training with certain learning rates that would be unstable for CLIP alone. Without a more detailed report on the hyperparameter tuning process and sensitivities, a skeptic could argue that the CLIP baseline might not have been trained under its own optimal conditions, thus artificially enhancing MaMM-UT's apparent scaling advantage.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper's main contribution is to advocate for and demonstrate a more scientific and robust methodology for comparing foundation models. In an era where \"SOTA\" claims are often based on single-point evaluations with murky training details, this work provides a blueprint for principled comparison across scales. This is a significant and much-needed contribution to the community.\n2. The sheer scale of the empirical study is a major strength. The authors have conducted an exhaustive number of pre-training runs, covering multiple model sizes (S/32 to H/14), extensive data scales (1.28M to 3B samples), two model architectures (CLIP, MaMMUT), and two large datasets (DataComp, Re-LAION). This dense grid of measurements allows for highly reliable scaling law fits.\n3. The commitment to reproducibility is exemplary. By using open-source models (OpenCLIP, and their new OpenMaMMUT), open datasets, and promising to release all code, data, and checkpoints, the authors set a high standard. This work is not just a study; it is a valuable and verifiable public resource. The release of OpenMaMMUT-L/14 with 80.3% ImageNet-1k accuracy is a strong practical outcome.\n4. The central result—that MaMMUT out-scales CLIP—is demonstrated consistently and robustly across different evaluation tasks, datasets, and even learning rate schedules. The \"crossover\" effect, where CLIP is better at small scales but MaMMUT overtakes it at larger scales, is a particularly insightful finding that would be missed by single-point comparisons.\n\nWeaknesses:\nThe core comparison is between only two architectural paradigms: CLIP (contrastive-only) and MaMMUT (contrastive + generative). While this is a valuable comparison, the paper's title and claims suggest a more general framework for comparing \"Open Foundation Language-Vision Models.\" The study does not include other important architectural variants, such as (1) SigLIP, which uses a different loss function (sigmoid loss) and has shown very strong performance, (2) BLIP and BLIP2, which are generative large Language-Vision Models, (3) LLAVA, QwenVL, etc which incorporate CLIP encoder with large LLMs.", "was_revised": true, "final_justification": "The author's response addressed almost all of my concerns, and I am willing to maintain my “accept” rating.", "rebuttal": "We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s high appraisal of the study, some of the summaries are so precise and well written that we consider adapting it for our own communication. In following, we would like to address the relevant points raised by the reviewer:\n\n1. > … The core comparison is between only two architectural paradigms: CLIP and MaMMUT … the paper's title and claims suggest a more general framework for comparing \"Open Foundation Language-Vision Models.\" \n\nThe general question our work addresses is how to perform robust, reproducible comparison of training procedures for open foundation models across scales, both with regard to architectures and datasets involved. We choose CLIP and MaMMUT and open datasets DataComp-1.4B and Re-LAION-1.4B as an example study showcasing how our method works for comparing important vision-language learning procedures. The method is though not specific to the chosen study subjects (and is also not restricted to language-vision scenario only) and can be applied to compare any learning procedures that feature open foundation models and open datasets by conducting scaling law derivation as described in our work. \n\n2. > The study does not include other important architectural variants, such as (1) SigLIP, which uses a different loss function (sigmoid loss) and has shown very strong performance … \n\nWe would like to clarify that we do look at SigLIP in our study, deriving scaling law on DataComp-1.4B for this architecture, where softmax in CLIP is replaced by sigmoid transfer function, and comparing with original CLIP architecture (App. Sec. E, “Scaling behavior of other architectures”). By executing controlled comparison on same open dataset DataComp 1.4B we obtain evidence that replacing softmax with sigmoid transfer function does not provide any benefit over CLIP, as we do not see any difference in scaling trends obtained from scaling laws derived on same open dataset (App. Fig. 8 (a), App. Sec. E). Original SigLIP study featured a closed dataset (WebLI), such that independent controlled comparison with CLIP there is in contrast not possible. Our result shows that strong performance reported for SigLIP previously is highly likely due to the closed dataset WebLI, and not due to replacing softmax with sigmoid transfer function as claimed in the original work. \n\n\n> 3. … (2) BLIP and BLIP2, which are generative large Language-Vision Models, (3) LLAVA, QwenVL, etc which incorporate CLIP encoder with large LLM …\n\nWhile methods from our study can be applied to compare various foundation models and datasets, we have deliberately chosen a scenario where 1) pretraining is happening from scratch from randomly initialized models, excluding confounds bound to taking already pre-trained components 2) pretraining procedure is simple single stage . This allows for fully controlled setting to perform scaling law derivation, providing same reference frame for the comparison of learning procedures. Learning procedures that lead to BLIP3, Qwen-VL and other similar works are using various pre-trained components, often trained under unknown conditions and performing training in various multiple stages, which necessitates further hyperparam and dataset decisions per stage. This is in contrast to our showcase, where all components are fully controlled, open and hyperparams can be tuned in the single stage, allowing to derive scaling law for the full extent of a given learning procedure to serve as base for validated, reproducible comparison to other procedures. Another consideration is available compute budget – each comparison requires scaling law derivation and we have chosen to use the available compute budget to demonstrate our method in clear way for comparison of important reference vision-language learning procedures CLIP and MaMMUT. We hope that our method can be used to compare other important learning procedures in search of more scalable open foundation models in follow up works, taking additional necessary compute required to execute such comparisons.\n\n> 4. Is it possible that the chosen hyperparameter search space or the final selected parameters favor the MaMMUT setup? … a skeptic could argue that the CLIP baseline might not have been trained under its own optimal conditions, thus artificially enhancing MaMMUT's apparent scaling advantage.\n\nFor our main comparison showcase, CLIP vs MaMMUT, we used wide intervals to scan for both models hyperparams extensively. We provide here overview for the hyperparam intervals and resulting scans for both models on datasets used in the study (for rebuttal, we also performed experiments on another important open dataset, DFN-1.4B):\n\n| Model  | **Learning Rate** |       | **Global Batch Size** |       | **Warmup Steps** |       |\n|--------------------|------------------|-------|----------------------|-------|-----------------|-------|\n|                    | min              | max   | min                  | max   | min             | max   |\n| CLIP               | 1e-04           | 1e-02 | 512                  | 181 248 | 1000         | 15 000 |\n| MaMMUT             | 1e-05           | 1e-02 | 256                  | 181 632 | 50            | 20 000 |\n\n\n| Dataset   | Model | Points |\n|-------------------|-------------|--------|\n| **DataComp-1.4B** | CLIP        | 672    |\n|                   | MaMMUT      | 1010   |\n| **Re-LAION-1.4B** | CLIP        | 703    |\n|                   | MaMMUT      | 750    |\n| **DFN-1.4B**        | CLIP        | 732    |\n|                   | MaMMUT      | 737    |\n\nAs evident from the table, the measurements are balanced for Re-LAION (750 vs 703 points), as well as for DFN (737 vs 732 points). For DataComp, we have 672 scans for CLIP and 1010 scans for MaMMUT on DataComp-1.4B. More measurements were performed on DataComp for MaMMUT to avoid unfair advantage for CLIP, as CLIP architecture was already extensively tuned in numerous previous works, which is not the case for MaMMUT that was not open sourced before and was not available for tuning by the broad community as was the case for eg openCLIP.  Confirming there is no tuning advantage, predictions from scaling laws on held out points from DataComp have same high accuracy and low uncertainty for both CLIP and MaMMUT, also showing similar RMSE  - App. Tab. 7, here a short version:\n\n| Model             | Samples Seen | GFLOPs    | IN1k 0-shot acc |  Predicted IN1k 0-shot acc (95% CI) |\n|------------------|--------------|-----------|:------------------:|:--------------------------------------------------:|\n| **CLIP**                                                                                                                                            |\n| ViT-H-14         | 3.07e+9      | 1.14e+12  | 0.784            | 0.779 (0.770, 0.789)                             |\n| *RMSE: 5.90e-03*                                                                                                    |\n| **MaMMUT**                                                                                                                                         |\n| mammut-ViT-H-14  | 3.07e+9      | 1.43e+12  | 0.798    | 0.801 (0.793, 0.809)                             |\n| *RMSE: 7.57e-03*                                                                                                    |\n\nThe predictions are accurate using held-out points on compute-optimal Pareto front for both CLIP and MaMMUT, as we see actual measurements on compute-optimal Pareto front falling well into the 95% confidence interval for the predictions on held-out points. (For the fit, points up to $C^{MaMMUT}_{threshold} = 2.59\\cdot 10^{11}$ GFLOPS for MaMMUT and up to\n\n$C^{CLIP}_{threshold} = 4.07\\cdot 10^{11}$ GFLOPS for CLIP were taken, which extrapolates for MaMMUT 5.5 and for CLIP 2.8 times; see App. Tab. 7). Similar high prediction accuracy for both models again hints on absence of tuning advantage for either CLIP or MaMMUT.\n\nFurther, we conduct an additional experiment to double check whether there is any difference in obtained scaling law if working with the same number of points for MaMMUT as for CLIP on DataComp-1.4B. We perform bootstrapping, sampling randomly 672 points from 1010 available points for MaMMUT, doing 10 trials, fitting scaling law for each trial and averaging the obtained scaling law coefficients. We observe that the obtained fit coefficients have no significant difference from scaling law obtained with 1000 points: \n\n| Model                                           | Ac        | Bc        | αc        | Ec        |\n|------------------------------------------------|-----------|-----------|-----------|-----------|\n| CLIP                                            | 57.862083 | 18.391321 | -0.226604 | 0.111169  |\n| MaMMUT (full points)                            | 125.356572| 19.289384 | -0.255670 | 0.101112  |\n| MaMMUT (same points num. as CLIP, 10 random samples) | 125.267163| 19.301461 | -0.255934 | 0.108208  |\n\nThus, comparison on reduced points shows again the same trends. Further, comparison via scaling laws on Re-LAION (Fig. 2) and DFN (will be included in the revised manuscript) which have balanced number of scanned points gives same trends as on DataComp, showing MaMMUT stronger scalable than CLIP, again confirming no influence of tuning on comparison. We can thus conclude from the gathered evidence that observed trends and performed comparison are valid and are due to differences in learning procedure (MaMMUT having contrastive+captioning loss mixture vs contrastive only for CLIP), and cannot be due to differences in tuning. We will update the revised manuscript with corresponding material."}, {"review_id": "I4xlbtILcD", "reviewer": "Reviewer_TkVk", "summary": "The paper derives compute vs performance scaling laws for two open-source language vision procedures CLIP and MaMMUT on two open datasets, DataComp and Re-LAION. They release all the pre-trained models with their intermediate checkpoints and develop their openMaMMUT that achieves 80.3% accuracy on zero-shot ImageNet 1k. When comparing the two models, they use a really dense measurement with $|M| = 15$ model configurations and $|D|=11$ samples configurations. Their contributions are:\n\n- The comparison is fully reproducible and the compute is measured in a dense manner. \n- They find MaMMUT has better scalability than CLIP.\n- Constant learning rate can recover the same trend with less compute. \n- Release all checkpoints, which contribute to the open source community.", "questions": "1. In line 216, $\\alpha$ is set to be negative. According to equation (1), it should be positive. Compared with the results in Figure 5, the paper might set $\\alpha$ to be $-\\alpha$ by mistake.\n\n2. Line 62 and Line 186 is conflicting with each other. After looking at the appendix figure 6,7, I found line 62 is correct. This is one of the conclusions of the paper, so please confirm which one is correct.\n\n3. In Figure 8, the paper shows the comparison of SigLIP, CoCa, CLIP, and MaMMUT on the DataComp-1.4B dataset. Everywhere else, the paper also compared the Re-LAION-1.4B dataset. Does the same pattern hold on Re-LAION-1.4B dataset?\n\n4. I wonder whether the paper can be extended to more general questions, such as why MaMMUT has better scalability than CLIP. This can provide more insights into the fundamental question of how to design a more scalable model. Specifically, one can do a study on whether it is the training loss or the model architecture that makes MaMMUT more scalable. This can make the paper more generalizable, and add great significance to the paper.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper has excellent quality, conducted experiments in a dense manner, used significant computing resources, and released all the pre-trained models with their intermediate checkpoints. All the experiments are conducted in a rigorous and reproducible manner. For clarity, there are some typos and inconsistencies mentioned below in the Question section. In terms of significance, I recognize the paper contributes to the open source community in terms of a systematic practice for scaling laws comparison, and release of both checkpoints and open source models. The analysis of the learning rate is also very helpful. However, the question investigated in the paper is specific to the two models and the two datasets. I think more general questions are needed to be investigated, and the details are posted in the Question section. For originality, the paper extends the scaling laws methodology for model selection, and also has its derivative and confidence interval analysis for the power law equation. The conceptual contribution is similar to the original scaling law paper.", "was_revised": true, "final_justification": "I now understand that the focus of the paper is more on how to perform robust, reproducible comparison of training procedures for open foundation models, rather than solely related to the comparison of several models and datasets in the paper. The rigorous experiments and procedures can be referenced for later experiments and offer valuable guidance. I appreciate the new experiment from the authors for rebuttal. I thank the authors for their clarification that the MaMMUT vs Cap experiment helps to identify that the contrastive and captioning loss mixture creates a more scalable training procedure. I have also raised my score as the author clarified the general question the paper is answering and provided answers to my questions.", "rebuttal": "We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s remarks on the excellent quality of the paper, emphasis on rigorous and reproducible experiment execution and on contributions to the open-source community. In following, we would like to address the relevant points raised by the reviewer:\n\n1. Thanks for careful reading and catching the typos. At line 216, we will remove the incorrect minus sign for $\\alpha$, reading \"($\\alpha=0.208$ vs. $\\alpha=0.354$)\". At line 186, Re-LAION-1.4B will be replaced by DataComp-1.4B, reading \"... stronger scaling for DataComp-1.4B on classification ...\"\n\n2. >  the question investigated in the paper is specific to the two models and the two datasets …\n\nThe general question our work addresses is how to perform robust, reproducible comparison of training procedures for open foundation models, both with regard to architectures and datasets involved. To showcase how such comparison based on scaling law derivation can be conducted, we choose CLIP and MaMMUT and open datasets DataComp-1.4B and Re-LAION-1.4B as an example study subjects – the posed question and method are though not specific to the models and datasets we took in our example. Already in this study, we look at further models, deriving scaling laws for comparison of CLIP and MaMMUT with SigLIP (App. Fig. 8 (a)), CoCa (App. Fig. 8 (b)), CaP (App. Fig. 17). For the rebuttal, using the same method, we conduct additional experiments on another important open reference dataset, DFN-1.4B [1], obtaining further model and dataset comparison on DFN-1.4B. We again obtain same trends from scaling laws derived on DFN-1.4B as observed for DataComp (Fig. 1) and Re-LAION (Fig. 2) – MaMMUT being stronger scalable than CLIP, providing further support for consistency of the comparison across datasets and downstream tasks. Dataset comparison reveals DFN-1.4B [1] leading to better training for both CLIP and MaMMUT than DataComp and Re-LAION. This shows that our method is generic and can be applied for robust model and dataset comparison in various scenarios. We will update the revised manuscript with corresponding material. \n\n[1] Data Filtering Networks. Fang et al, ICLR 2024, arXiv:2309.17425\n\n3. > In Figure 8, the paper shows the comparison of SigLIP, CoCa, CLIP, and MaMMUT on the DataComp-1.4B dataset. Everywhere else, the paper also compared the Re-LAION-1.4B dataset. Does the same pattern hold on Re-LAION-1.4B dataset?\n\nLarge fraction of available compute budget for the study was used to obtain measurements for the main comparison for CLIP and Mammut, deriving scaling laws and showing consistent trends across datasets and downstream tasks  (Fig. 1, 2). Across datasets, we executed 4604 measurements (full training runs) to derive scaling laws for both models, see summary table below: \n\n| Dataset   | Model Type | Points |\n|-------------------|-------------|--------|\n| **DataComp-1.4B** | CLIP        | 672    |\n|                   | MaMMUT      | 1010   |\n| **Re-LAION-1.4B** | CLIP        | 703    |\n|                   | MaMMUT      | 750    |\n| **DFN-1.4B**        | CLIP        | 732    |\n|                   | MaMMUT      | 737    |\n\n\nAs compute cost for executing dense measurements for scaling law derivation is substantial (see App. Tab. 4), the budget was not sufficient to repeat the same extensive measurement procedure across datasets for further architectures, comparison for which was thus restricted to DataComp-1.4B. To show consistent patterns for other datasets, we conduct for the rebuttal additionally for the main comparison showcase in our study - CLIP vs MaMMUT - comparison of data efficiency on Re-LAION-1.4B, which confirms observations on DataComp. and complement this with another data efficiency comparison for DFN-1.4B. Both confirm the patterns observed on DataComp - MaMMUT being a more data efficient model than CLIP. Together with the additional experiments on DFN-1.4B described above, we can confirm the same pattern of consistent comparison outcomes for CLIP vs MaMMUT, pointing to MaMMUT as stronger scalable and more data efficient learning procedure than CLIP across all studied open datasets. We will update the revised manuscript with corresponding material. \n\n4. > why MaMMUT has better scalability than CLIP … fundamental question of how to design a more scalable model … study on whether it is the training loss or the model architecture that makes MaMMUT more scalable.\n\nEvidence obtained from comparisons in our work allows us to draw conclusions about what makes MaMMUT scale stronger than other measured architectures. On the one hand, we have a comparison to CLIP, which has in contrast to MaMMUT only contrastive loss, thus ablating captioning loss and cross-attention from vision to text tower. On the other hand, we also compare to Cap (see App. Fig. 17 and App. Sec. E “Scaling behavior of other architectures”; for rebuttal, we additionally perform comparison MaMMUT vs Cap using log likelihood evaluation based on logprobs of the text tower), which has captioning loss and cross-attention, but no contrastive loss, thus ablating for contrastive loss. As both CLIP and Cap underperform MaMMUT especially at larger scales, we can conclude that neither contrastive loss nor captioning loss + cross-attention alone are responsible for providing stronger scalability, and it is the contrastive and captioning loss mixture which creates more scalable training procedure (see also App. Sec. E.).\n\nIn our study, the comparisons between the various models are performed in a common reference frame (eg same open datasets) and across scales, while fully controlling for adding and removing components of the learning procedure for each scaling law derivation. This allows us to identify the effects of various ablations on learning procedure scalability. This is in contrast to other works, where on the one hand, unknown confounds like closed datasets make it hard or impossible to infer which of the differences between the learning procedures (algorithmic or dataset based) led to better performance, while on the other hand performance is measured only on few selected scales without estimating scaling trends, making unclear whether comparison holds for wider span at larger scales. Designing better learning procedures requires an approach to conduct guided search for more scalable procedures given the existing ones as reference baselines. This approach requires a robust method for training procedure comparison valid across scales that can operate in the same reference frame to control for any interventions - including controlling the datasets which requires open data, - and this is where our study makes progress.  We will update the revised manuscript with corresponding material and make our conclusions about why MaMMUT is a more scalable learning procedure clearer in main text’s discussion section."}, {"review_id": "5XZHOOqHRa", "reviewer": "Reviewer_jP4c", "summary": "This paper has widely analyzed scaling laws for vision-language models (VLM), CLIP and MaMMUT, in terms of zero-shot classificaiton, retrieval, and segmentation. The trained VLMs on different massive datasets of DataComp and Re-LAION follow the novel scaling laws in different knowledge from the original paper of scaling laws. Finally, the analyzed VLMs perform well-established zero-shot classification accuracy after the training on a billion-scale dataset.", "questions": "1. Although the code release has been mentioned in the paper, what about the pre-trained models? The reviewer assumes these pre-trained models will also be made public. Having models and code available for image classification, image retrieval, and image segmentation would likely accelerate related researches across many subfields.\n\n2. Are there effective pre-training approaches (e.g., parameters and how to pre-train) specialized for zero-shot classification, retrieval, and segmentation, respectively? Or does training on billion-scale datasets describe such specialization less important? If the underlying vision-language tasks share common characteristics, focusing on contrastive learning alone could suffice.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "S1. The paper introduces a great effort to establish novel scaling laws across a couple of scenarios. While maintaining consistency with insights from previously reported scaling laws, the paper additionally covers new findings for the scaling laws in zero-shot image classification, image ritrieval, and image segmentation. The comprehensive analysis derives insights not only for zero-shot image classification but also for zero-shot image retrieval and image segmentation, presenting results that would be valuable to a broader AI community and researchers.\n\nS2. The investigation is conducted with well-deserved strategy. In particular, it appears to have engaged in detailed discussion and try-and-error among themselves regarding the model architectures, datasets, training parameters, and learning strategies for arriving at their results. In terms of experimental design in VLMs, this paper serves as a useful knowledge for a wide readers.\n\nS3. This paper includes excellent trial that the project itself is committed to \"openness.\" Although the computational scale required to explore scaling laws is immense, making reproduction by many researchers challenging, the insights can be publicly and widely shared. Moreover, by open-sourced approaches, the community can directly benefit from the trial. The reviewer believes the paper's execution represents a significant contribution to the AI and realted fields.\n\nS4. The paper shows that the dataset aspect is still tracing a larger scaling. At present, improving and organizing data can still accelerate VLM performance in classification, retrieval, and segmentation. In this point, this paper has the great potential to drive the overall advancement of the AI and related fields.\n\nW1. The paper's constraints are understandable, however, the reviewer believes the paper’s value can be increased if more qualitative evaluations are included in the supplementary material as additional documents. In particular, since this paper uses OpenMaMMUT, which involves not only CLIP-style contrastive learning but also sentence generation, this would be beneficial to show examples of the kinds of captions generated for various images.\n\nW2. Similar contributions such as Molmo & PixMo (CVPR 2025) aim to open-source VLMs. This work compares the accuracy against sophisticated models like GPT and Gemini at the time of paper submission, reporting comparable or slightly superior recognition performance. They also share many practical insights (e.g., introducing overlap among image patches), yielding interesting results. Although this work does not directly affect the contributions of the present paper, discussing their positioning and relationship would enhance paper's overall value.", "was_revised": true, "final_justification": "The reviewer is grateful to the authors for great efforts and the information. In particular, additional experimental results are very reasonable to further understand the paper and effectiveness. After the rebuttal phase, the reviewer has a better understanding on the research project. This has enough contributions to the topic on vision-language models. Hopefully the proposed method can be widely shared in the research community and employed diverse applications. Of course the reviewer keeps the accept-side rating in the final submission.", "rebuttal": "We thank the reviewer for time dealing thoroughly with our work. In following, we would like to address the relevant points raised by the reviewer:\n\n1. > Although the code release has been mentioned in the paper, what about the pre-trained models? The reviewer assumes these pre-trained models will also be made public.\n\nThis is correct. We will release the pretrained models (for the rebuttal, we additionally trained DataComp 12.8B openMammut B/32 which will complement  openMammut L/14 already described in the study) from all points measured for scaling law derivation (in total over 4500 models trained on DataComp, Re-LAION and DFN [1], for which we conducted further scaling law derivation experiments for the rebuttal), including intermediate training checkpoints for each model, their evaluation and source code to reproduce training, evaluation  and fitting of scaling laws. In the table below is the overview of all points measured during the study (including experiments on DFN [1] performed for the rebuttal) for which models will be released.\n| Dataset   | Model | Points |\n|-------------------|-------------|--------|\n| **DataComp-1.4B** | CLIP        | 672    |\n|                   | MaMMUT      | 1010   |\n| **Re-LAION-1.4B** | CLIP        | 703    |\n|                   | MaMMUT      | 750    |\n| **DFN-1.4B**        | CLIP        | 732    |\n|                   | MaMMUT      | 737    |\n\n[1] Data Filtering Networks. Fang et al, ICLR 2024, arXiv:2309.17425\n\n2. > … since this paper uses OpenMaMMUT which involves not only CLIP-style contrastive learning but also sentence generation, it would be beneficial to show examples of the kinds of captions generated for various images …\n\nWe agree that it will be a good reminder that openMaMMUT is also a functional captioner by providing examples of caption generation. Main focus of the study was to demonstrate the capability of our method to compare pure contrastive CLIP with contrastive+captioning loss mixture of MaMMUT, and check where scaling law based comparison can identify which procedure is the more scalable in this fully controlled setting. Thus, adding captioning loss served the purpose of demonstrating comparison between two controlled learning procedures, and while we gather evidence that contrastive+captioning loss mixture is what makes openMaMMUT perform better at larger scales on various downstream tasks compared to CLIP, we do not assume strong function of captioning itself. However, it is indeed helpful to present captioning examples nevertheless. In our additional experiments, we see examples of good captioning, eg correctly identifying multiple objects on the image or performing simple OCR from written text, as well as examples of failure, for instance not being able to parse calligraphic writing. We will update the revised manuscript with corresponding material and also provide a openMammut based demo for captioning on HuggingFace Spaces to try captioning on any dropped image.\n\n3. > Similar contributions such as Molmo & PixMo (CVPR 2025) aim to open-source VLMs … discussing their positioning and relationship would enhance paper's overall value … \n\nWe have a different focus than works like Molmo & PixMo, in 1) describing a novel generic method for robust comparison of various learning procedures (models & datasets) across scale span, while the mentioned work presents a single model and dataset showing performance on few selected scale points 2)  dealing with fully controlled scenario of single stage pre-training from scratch without relying on already pre-trained components, which makes training procedure and comparison fully reproducible, in contrast to Molmo which eg uses pre-trained language model and pre-trained vision encoder (with data for vision encoder being closed, making whole procedure not fully controllable and reproducible). In follow up works,  an interesting direction would be to apply scaling law based comparison to VLM post-training setting with fully open pre-trained components like studied in our work. This would enable VLM training procedure comparison that takes properly into account all stages, without having closed components (models & datasets) introducing non-controllable confounds. We will extend discussion in revised manuscript with the outlook to future work on scaling laws based comparison for multi stage VLM learning procedures with pre-trained components.\n\n4. > … pre-training approaches … specialized for zero-shot classification, retrieval, and segmentation? If the underlying vision-language tasks share common characteristics, focusing on contrastive learning alone could suffice …\n\nIn the study, we look at CLIP as a model using contrastive loss only and at MaMMUT as a model using contrastive+captioning loss to provide a compelling showcase for comparing two learning procedures using our method based on scaling law derivation across different settings. The comparison allows us to predict which of the procedures provides better performance at larger scales. It turns out to be MaMMUT that shows stronger scalability and outperforms CLIP at larger scales. Using our comparison method, we can thus decide whether contrastive learning alone could suffice to perform strongly on  those downstream tasks, which turns out to be not the case – as according to scaling law derivation based comparison, contrastive+captioning loss mixture performs consistently better across all the downstream tasks from certain compute threshold on (while on smaller scales on the other hand, contrastive learning via CLIP prevails), and thus poses the better, stronger scalable alternative to contrastive learning alone."}, {"review_id": "lE66JIC7MM", "reviewer": "Reviewer_Mi8q", "summary": "This paper investigates scaling laws for CLIP and MaMMUT across multiple open datasets (Re-LAION-1.4B and DataComp-1.4B). Based on their experiments, the authors find that MaMMUT has both stronger improvement with scale and better sample efficiency than CLIP. They then derive scaling laws for downstream tasks such as classification, retrieval, and segmentation, and show that the observed improvements persist across these applications.", "questions": "1. The authors limit training to less than 3B samples to avoid repetition effects, but this constrains conclusions about truly large-scale behavior. When trained on 12.8B samples, performance falls below predictions. How can we be certain that this under-performance is actually due to high repetitions?\n\n2. The paper states that SigLIP shows no benefits over CLIP (Section E), contradicting claims in the SigLIP papers. However, this comparison seems a bit limited. It also seems like training hyperparameters such as learning rate, batch size, etc. vary across scales. How were hyperparameters chosen to ensure fair comparison across architectures and scales? Was there a particular methodology for selecting these? How can we be confident that the observed scaling differences reflect architectural properties rather than differences in effort invested in optimization?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nThe paper is clear and well-written. The claims are well supported both by theoretical scaling-law fitting and extensive experiments, with proper methodological rigour. There is great effort spent on ensuring a wide range of parameters are tested across various data regimes. The results will be open-sourced and reproducible, making it useful for the community at large. The release of openMaMMUT-L/14 will provide a good open alternative with performance competitive with proprietary vision-language models.\n\nWeaknesses:\n\nThe scope of this paper is mostly focused on contrastive and captioning objectives – it would be interesting to see results on other alternative architectures and objectives. Evaluation primarily limited to zero-shot settings, with minimal analysis of few-shot learning, fine-tuning transfer, or vision-language understanding tasks. There is also some compute accounting ambiguity, especially when comparing architectures with inherently different parameter counts (Table 12), which raises questions about compute comparison methodology.", "was_revised": true, "final_justification": "The authors have effectively addressed my concerns. The clarification on compute accounting resolves my questions about fair architectural comparison, while the extensive hyperparameter search and additional 12.8B sample experiments validate scaling law predictions and confirm persistent MaMMUT advantages.\n\nWhile small concerns on having only zero-shot evaluation methods remain, I think these are acceptable given compute constraints and given justifications based on prior works. \n\nOverall, this paper makes strong empirical contributions, and I think it will provide valuable open-source resources to the community. Thus, I am updating my rating to a 5.", "rebuttal": "We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s remarks on the clarity and writing quality of the paper, highlighting the effort spent for methodologically rigorous and reproducible experiments, also mentioning usefulness for the open-source community. In following, we would like to address the relevant points raised by the reviewer:\n\n1. > The scope of this paper is mostly focused on contrastive and captioning objectives - it would be interesting to see results on other alternative architectures and objectives.\n\nAvailable compute budget for the study allowed us to demonstrate execution of validated, reproducible comparison using scaling laws derivation for important architectures such as MaMMUT, CLIP, SigLIP, CoCa, CaP and for open datasets DataComp, Re-LAION and DFN (for DFN, we executed additional experiments for the rebuttal that will enter the revised manuscript). Comparison of further learning procedures requires additional compute budgets, and we hope that the introduced method will be used in follow up works to perform further useful, reproducible comparisons, guiding validated search for stronger scalable learning procedures.\n\n2. > Evaluation primarily limited to zero-shot settings, with minimal analysis of few-shot learning, fine-tuning transfer, or vision-language understanding tasks\n\nChoosing zero shot evaluation for scaling laws (with exception of dense segmentation task where we performed fine-tuning) was due to two considerations. First is grounded in works like [1] and [2] stating that zero shot performance is a strong predictor for fine-tuning performance, and thus can be taken as good proxy for model quality. That is, stronger scalability for zero-shot would highly likely imply stronger scalability for full fine-tuning.  Second comes from available compute budget considerations: zero shot evals are cheap in compute, while full fine-tuning is expensive, especially given the large number of obtained points for scaling law derivation it has to be executed for. We thus think that zero-shot evals provide a strong proxy for model quality on the one hand, while allowing to keep compute costs down for evals on the other hand. In follow up studies, given further compute budgets, it will be interesting to perform fine-tuning evals for control as well.   \t\n\n[1] Robust Fine-Tuning of Zero-Shot Models, Wortsman et al, CVPR 2022, arXiv:2109.01903 \\\n[2] Reproducible Scaling Laws for Contrastive Language-Image Learning, Cherti et al, CVPR 2023, arXiv:2212.07143\n\n3. > There is also some compute accounting ambiguity, especially when comparing architectures with inherently different parameter counts (Table 12), which raises questions about compute comparison methodology.\n\nWe would like to clarify that there is no ambiguity wrt compute accounting. openCLIP profiler routine (relying on fvcore library) computes exact FLOPS for each model scale employed in our work. The calculated compute is then used for scaling law derivation and for reporting GFLOPS as listed in Table 12, taking into account image resolution, patch size, text context length and number of parameters in vision and text towers (compute does not depend on param count only!). This avoids any ambiguity, assigning for each model and samples scale its respective compute. The calculations can be reproduced and checked by installing openCLIP and calling profiler, eg ```python src/open_clip_train/profiler.py --model mammut_ViT-B-32 --results-file compute_profiling.csv``` for any given model config.\n\n4. > … limit training to less than 3B samples to avoid repetition effects … constrains conclusions about truly large-scale behavior. When trained on 12.8B samples, performance falls below predictions. How can we be certain that this under-performance is actually due to high repetitions?\n\nDeriving scaling laws based on measurements up to 3.07B samples seen allows us to make predictions for behavior of compute-optimal training on unique or low repetition (2.2x for 1.4B) samples. We provide evidence that these predictions are accurate using held-out points on compute-optimal Pareto front for both CLIP and MaMMUT in App. Tab. 7, as we see actual measurements on compute-optimal Pareto front falling well into the 95% confidence interval for the predictions on held-out points. Predictions for MaMMUT extrapolate 2.4 and 5.5 compute factor beyond the fit (taking points up to $C^{MaMMUT}_{\\text{threshold}} = 2.59\\cdot 10^{11}$ GFLOPS for the fit). Factor 5.5 would allow us to extrapolate from 3B samples seen to 16.5B samples, which is well above 12.8B. As we confirm scaling law derivation giving accurate predictions when extrapolating far ahead, the most likely explanation for the deviation from predictions on 12.8B scale is thus sample repetition, as it is also known to cause diminishing returns from previous works [3], [4]. Further reason for observed performance drop can be also due to training procedure not residing on compute-optimal Pareto frontier. For rebuttal, we train in addition openMammut B/32 on 12.8B samples of DataComp-1.4B, resulting in following overview on 12.8B scale: \n\n| Type   | Model | Samples | GFLOPs    | acc1, w. rpt. | acc1 predicted, unique samples (CI 95%) |\n|--------|--------|---------|-----------|----------------|------------------------------------------|\n| CLIP   | B/32  | 12.8B   | 1.89e+11  | 0.6864         | 0.699 (0.693, 0.706)                     |\n| MaMMUT | B/32  | 12.8B   | 3.50e+11  | 0.69468        | 0.710 (0.702, 0.718)                     |\n| CLIP   | L/14  | 12.8B   | 2.14e+12  | 0.792          | 0.796 (0.788, 0.804)                     |\n| MaMMUT | L/14  | 12.8B   | 2.59e+12  | 0.8034         | 0.820 (0.815, 0.826)                     |\n\nFor both B/32 and L/14, the measured IN1k 0-shot accuracy is below the predictions for unique samples, due to 9x sample repetitions (1.4B unique samples, 12.8B total), as elaborated above. The measured performance can be still projected from the predictions for unique samples by taking offset of 2\\% for MaMMUT and 1\\% for CLIP downwards from the confidence interval of the predicted performance on unique samples. Testing on unique samples is currently not possible, as it requires open datasets with scale of at least 6B unique samples to keep repetitions low. For accurate prediction on repeated samples, follow up work is required to derive scaling laws with corrections for sample repetitions which requires additional training experiments similar to [3], [4]. Important for our work is that comparison is valid also at larger scales, MaMMUT consistently outperforming CLIP as predicted from scaling laws, also for high sample repetition. \n\n[3] Scaling Data-Constrained Language Models. Muennighoff et al, NeurIPS 2023, arXiv:2305.16264 \\\n[4] Scaling Laws for Data Filtering. Goyal et al, CVPR, 2024 arXiv:2404.07177\n\n5. > … SigLIP shows no benefits over CLIP … How were hyperparameters chosen to ensure fair comparison across architectures and scales? … How can we be confident that the observed scaling differences reflect architectural properties rather than differences in effort invested in optimization?\n\nFor our main comparison showcase, CLIP vs MaMMUT, wide hyperparams interval was scanned for both models extensively, resulting in 672 scans for CLIP and 1010 scans for MaMMUT on DataComp-1.4B. We invested more effort into MaMMUT tuning as CLIP architecture was already extensively tuned in previous works and we could afford to tune CLIP based on already known hyperparam settings, which is not the case of MaMMUT. MaMMUT as the previously less optimized architecture was thus tuned more to avoid unfair advantage for CLIP. Confirming there is no tuning advantage, predictions from scaling laws on held out points have same high accuracy and low uncertainty for both CLIP and MaMMUT, also showing similar RMSE (App. Tab. 7, as also elaborated above).\n\nFurther, we conduct additional experiment to double check whether there is any difference in obtained scaling law if working with same number of points for MaMMUT as for CLIP on DataComp-1.4B. We perform bootstrapping, sampling randomly 672 points from 1010 available points for MaMMUT, doing 10 trials, fitting scaling law for each trial and averaging the obtained scaling law coefficients. We observe that the obtained fit coefficients have no significant difference from scaling law obtained with 1000 points: \n\n| Model                                           | Ac        | Bc        | αc        | Ec        |\n|------------------------------------------------|-----------|-----------|-----------|-----------|\n| CLIP                                            | 57.862083 | 18.391321 | -0.226604 | 0.111169  |\n| MaMMUT (full points)                            | 125.356572| 19.289384 | -0.255670 | 0.101112  |\n| MaMMUT (same points num. as CLIP, 10 random samples) | 125.267163| 19.301461 | -0.255934 | 0.108208  |\n\nThus, comparison on reduced points shows again same trends. Measurements are balanced for Re-LAION (750 vs 703 points), as well as for DFN (737 vs 732 points). Comparison via scaling laws on Re-LAION (Fig. 2) and DFN gives same trends as on DataComp, showing MaMMUT stronger scalable than CLIP, again confirming no influence of tuning on comparison.\n\nFor SigLIP and CLIP, we also have balanced tuning situation on DataComp-1.4B with 590 tuned points for SigLIP, and thus can rely on the scaling laws derived for the comparison (Fig. 8a). Executing controlled comparison on same open dataset DataComp 1.4B is sufficient to see that replacing softmax with sigmoid transfer function does not provide any benefit over CLIP. Original SigLIP study featured a closed dataset (WebLI), such that controlled comparison there is not possible. Our result shows that original SigLIP strong performance is highly likely due to the closed dataset WebLI, and not due to replacing softmax with sigmoid transfer function as claimed in the original work."}]

Author rebuttal:

Meta-review: The paper presents a scaling law study on CLIP (contrastive) and MaMMUT (contrastive + captioning) architectures with  DataComp-1.4B and Re-LAION-1.4B datasets. The scaling law shows that MaMMUT consistently demonstrates better scalability than the purely contrastive CLIP, where MaMMUT performance improves more rapidly with increased compute, on a diverse downstream tasks (classification, retrieval, segmentation) and training datasets. The authors also release models, checkpoints, and code. 

Strength of the paper: 
- Working on a popular and important area (vision language representation learning)
- Reasonable scale of experiments and number of different configurations to support generalizability of findings
- Generic experiment design and good selection of downstream evaluation tasks provides useful knowledge for a wide range of readers
- Due to the scale of the investigation (dataset and model size etc.) , it's hard to reproduce experiments for academic community anyway. However, releasing the best model obtained in the investigation can facilitate many practitioners in their works, and open sourcing the implementation code etc. still encourage the community to be more open.

Reviewers align on most of the strengths of the paper from the high level and see its novelty, contribution, and relevance to NeurIPS venue. Thus we recommend this paper to be accepted to the conference.

After most questions being addressed and clarified during the rebuttal and discussion stage, reviewers feel a few areas that can improve the paper.  

To improve:
- Foundation Language-Vision Models is a wide area while this paper explored two examples (CLIP and MaMMUT) of the models. It's better to have a title better describing the work
- Many discussion between reviewers and authors are valuable to include in the final manuscript
Decision: Accept (poster)