MobileCLIP2: Improving Multi-Modal Reinforced Training
Authors: Fartash Faghri
Pavan Kumar Anasosalu Vasu
Cem Koc
Vaishaal Shankar
Alexander T Toshev
Oncel Tuzel
Hadi Pouransari
Venue: Accepted by TMLR ()

Abstract: Foundation image-text models such as CLIP with zero-shot capabilities enable a wide array of applications. MobileCLIP is a recent family of image-text models at 3-15ms latency and 50-150M parameters with state-of-the-art zero-shot accuracy. The main ingredients in MobileCLIP were its low-latency and light architectures and a novel multi-modal reinforced training that made knowledge distillation from multiple caption-generators and CLIP teachers efficient, scalable, and reproducible. In this paper, we improve the multi-modal reinforced training of MobileCLIP through: 1) better CLIP teacher ensembles trained on the DFN dataset, 2) improved captioner teachers trained on the DFN dataset and fine-tuned on a diverse selection of high-quality image-caption datasets. We discover new insights through ablations such as the importance of temperature tuning in contrastive knowledge distillation, the effectiveness of caption-generator fine-tuning for caption diversity, and the additive improvement from combining synthetic captions generated by multiple models. We train a new family of models called MobileCLIP2 and achieve state-of-the-art ImageNet-1k zero-shot accuracies at low latencies. In particular, we observe 2.2% improvement in ImageNet-1k accuracy for MobileCLIP2-B compared with MobileCLIP-B architecture. Notably, MobileCLIP2-S4 matches the zero-shot accuracy of SigLIP-SO400M/14 on ImageNet-1k while being 2× smaller and improves on DFN ViT-L/14 at 2.5× lower latency. We release our pretrained models and the data generation code. The data generation code makes it easy to create new reinforced datasets with arbitrary teachers using distributed scalable processing.

Published in Transactions on Machine Learning Research (08/2025)
MobileCLIP2: Improving Multi-Modal Reinforced Training
Fartash Faghri' Apple
Jurtash@apple.com
Pavan Kumar Anasosalu Vasu" Apple
panssosakerasuu@pppee.coom
Cem Koc
Oem Gemn__oee@apple.comm
Apple
Vaishaal Shankar Work done while Apple
Alexander Toshev Apple
tosheo@apple.com com
Oncel Tuzel Apple
otuzel@apple.co com
Hadi Pouransari Apple
meporransariapppll.com
Reviewed on OpenReview:
https: openreview. net/ forum? id= NeF9zolnge
Abstract
Foundation image-text models such as CLIP with zero-shot capabilitios enable a wide array of applications. MobileCLIP a recent family of image-text models at 3 15ms latency and 50-150M parameters with state-of-the-ate zero-shot accuracy. The main ingredients in MobileCLIP were its low-latency and light architectures and novel multi-modal reinforced training that made knowledge distillation from multiple caption-generators and CLIP teachers elficient, scalable, and reproducible. In this paper, we improve the multi-modal reinforced training of MobileCLIP through: 1) better CLIP teacher ensembles trained on the DFN dataset, 2) improved captioner teachers trained on the DFN dataset and fine-tuned on diverse selection of high-quality image-caption datasets. We discover new insights through ablations such as the importance of temperature tuning contrastive knowledge distillation, the effectiveness of caption-generate fine-tuning for caption diversity, and the additive im- provement from combining synthetic captions generated by multiple models. We train new family of models called MobileCLIP2 and achieve seate-of-the-att ImageNet- 1k zero-shot accu- racies at low latencies. In particular, we observe 2.2% improvement in ImageNet-1k accuracy for MobileCLIP2-B compared with MobileCLIP-B architecture. Notably, MobileCLIP2-S4 matches the zero-shot accuracy of SigLIP-SONE -0400M/14 14 on ImageNet- while being 2x smaller and improves on DFN VIT-I./14 2.5x lower latency. We release our pretrained models [ and the data generation code The data generation code makes it casy create reinforced datasets with arbitrary teachers using distributed scalable processing.
'Equal contribution.

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
CLIP [Radford et al. 2021) is an image-text model that maps images and text inputs to a shared embodding space, where text describing an image, also called caption, is mapped close to image matching its description but far from dissimilar images. Building
vast literature (Frome et al. 2013 Socher al.) 2014 Karpathy & Fei-Fee 2015 Kiros et al. 2014 Faghri et al. 2018 CLIP substantially increased the scale of training data and models. Consequentially, along with improved image-text retrieval performance, new zero-shot classification capabilities emerged with non-trivial accuracy on classification tasks without. any explicit supervised training with classification labels through linear probing. The image-encoder can be further specialized to new task by either linear probing (fixed encoder), or full tinnttnnig to achieve state-of-the-at performance on diverse set of tasks (Wortsman al. 2022). CLIP is one of the first to be called foundation model given the diversity of its capabilities and applications (Bommasani et al. 2021
The success of CLIP resulted in an in- crease in the sizes of models and datasets,
85
84
leading to a gradual increase in perfor-
B8
mance (Fang et al..
2024b
Zhai et al.
ImageNet-1k Zero-shot Accuracy
80
B/16
2023 Gadre et al.,
2023 Fang et al.
PF2
2024a). Recently, this trend has been re-
75
Boz
versed to models with small size and low
50-
latency for applications
mobile devices.
70
Notably, TiPy (Wu et al.) 2023) and
50
B/32
B/16
MobileCLIP Vasu et al. 2024c) proposed 65
B/92
OpenAl
SigLIP2
models with as few as 50M total param- eters (sum of image and text encoder pa- rameters). For example, MobileCLIP-SO. with total latency of 3ms (sum of image and text encoder latencies), achieves sim- ilar average performance to the original OpenAT ViT-B/16 CLIP while being 3x smaller and 5x faster. It also demon- strates improved performance comparod prior state-of-tho-ar larger models, such SigLIP (Zhai et al. 2023).
61M82
5/32
DataComp
MobileCLIP
RN101
SigLIP
MobileCLIP2 (ours)
60
40M832
LAION
20M
TinyCLIP
RNISM
ACED
200M
55
15
20
25
Latency Sum Image/Text Encoders (ms)
Figure 1: MobileCLIP2 models trained on DFNDR- 2B achieve state-of-the-art accuracy at low latencies. MobileCLIP2-S4 matches the accuracy of SigLIP-S0400M. with 2x fewer paramcters and surpasses DFN ViT-L/14 at 2.5x lower latency measured on iPhone12 Pro Max. MobileCLIP- S3/S4 are our new architectures trained on MobileCLIP's training datasct, DataCompDR-11 (dashed lines).
In this paper, we present ablations of modti-modal reinforced training and present an improved training recipe. We train a new family of models, MobileCLIP2, that establishes new state-of-thheart ImageNet- 1k accuracy at a range of latencies matching the performance of larger SigLIP (Zhai et al.) (023) and DFN (Fang al.] 2024a) models while up 4x smaller (our MobileCLIP2S22 compared with SigLIP2-B/32. and up to 2.5x faster (our MobilcCLIP2SS compared with DFN ViT-L/14). Morcover, release efficient distributed code for generating reinforced datasets using arbitrary leacher models.
2 Improved Training
MobileCLIP introduced a family of low-latency image-text models consisting of SO, S1, S2, B, and B-LT variants with aggregate image-text latencies spanning 3.8-13. 1ms. These low latencies were achieved with specialized architectures based on FastViT (Vasu et al.! 2023b) and an improved training method called multi-modal reinforced training. We seek to further explore and improve each step of multi-modal reinforced training. We additionally consider a more diverse family of architectures that cover wider range of latencies.
Reinforced training a method for achieving better performance from a base dataset through improvements from additional sources such as pretrained models (Faghri al.) 2023). Multi-modal reinforced training introduced in Vasu et al. (002c)) adds information to image-text dataset from pretrained image-text. models
2

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Summary of MobileCLIP2 training improvements. CoCa models are pretrained on a large dataset for 13B seen samples then fine-tuned for 12M seen samples (denoted by +). The architecture for all CLIP teachers in this table is ViT-L/14. We report mean and standard deviations of 5 runs when available.
Name
Datasct
CLIP Teacher Datasets
CaCa Datoset
IN-val
Flidkr30k
Avg. 38
MobileCLIP
(Vasu al. 2024ec
DaisComp- 1B12M
OpenAI + DataComp-XI XL.
LAION-2B MSCOC0-1228
61.6
72.8
53.5
Table
DFN-5B1MM
Open.A DataComp-XX XL
LAION-2B MSCOC0-1224
63.1aa
73.30u
51..u.:
Table
DFN-5B12M
DFN-2B + DFN-2B-439B
LAION-2B - MSCOC0-1133
66.464
75.8.s
56. 56.20.
Mobine
(Tab. 6
DFN-5B12M
DFN-2B DEN-2B-s39B
DEN-2B MSCOCO-38k
65.9ua
75.40z
56.5u.
Table
DFN-5B12M
DFN-2B DFN-2B-439B
DFN-2B Syn.x10
66.0o.
b6.bo.3
well as pretrained synthetic caption generator. Specifically, they add the following additional information to DataComp-1B dataset: 1) image embeddings from two CLIP teachers on 10 random augmentations of each image 2) text embeddings from two CLIP teachers on the original text as well as 5 synthetic captions generated from a CoCa caption generator. Given a reinforced datasel, they modify the training loss to include a knowledge distillation loss given the embeddings from teachers on each sample (Hinton et al. 2015). To ensure consistency between the tcacher and student, the same image augmentation reproduced via stored augmentation parameters (Beyer et al. 2022: Faghri et al., 2023). They perform ablations to find the set of CLIP teachers, caption generator, and image augmentations that provide the largest performance gain ImageNet as well as the average accuracy on 38 evaluations from DataComp (Gadre et al. 2023
We follow a similar multi-modal reinforced training to MobileCLIP while improving all aspects and call the resulting model family MobileCT.IP2. Table [ summarizes the gains from each major improvement. Tn short, we use better training data, better CLIP teacher models, and better and diverse synthetic caption generators compared to MobileCLIP. In all ablations, we train MobileCLIP-B for 30k iterations (~20 epochs) datasets with 12.8M images. We provide summary of datasets in this paper in Tab. 15
Figure 2 demonstrates the efficiency gains compared with DFN (Fang et al. 2024a). Data Gaarr et al. 2023 and DataCompDR (Vasu et al.) 2024ch datasets during training. Training on DFNDR-2 for 30M seen samples is 5x more efficient than training on DataComp- (apuam--11220 1B12M, i.e., we reach the ImageNet- Ik zero-shot accuracy of training on DataComp-IB12h for 30M after seeing only 6M samples of DFNDR-2B12M. Similarly, training on DataCompDRR122) is 3.3x more efficient compared to DFN-2B12M and 1.3x more efficient compared with DataCompDRR120 12M. We also observe 1.6x speedup when training on DFNDR-23 compared with training on DataCompDR-IB for 13B seen samples. Similar to DataCompDR, training on DENDR datasets do not have any wall-clock time overhead, i... each training step of training on DataComp, DFN, DataCompDR, and DFNDR takes the same amount of time. That means any efficiency gains in terms of the number of samples and training iterations directly translate to wall-clock time efficiency gains.
2.1 Multi-Modal Reinforced Training
Datasct Reinforcement (DR) (Faghri et al., 2023) is a method for improving a datasct to achieve higher accuracy with minimal changes to the training code and minimal computational overhead. DR was first introduced for training image classifiers where Faghri et al.) (2023 improved the ImageNet dataset by storing classification probabilities efficiently from strong ensemble of classifiers. Given stored probabilities, the training was essentially Knowledge Distillation (Hinton et al., 2015) with no overhead for computing the teacher predictions. The cost efficiency makes it feasible to train longer for larger gains as observed in [Beyer et al. 2022). Vasu et al. (2024c) adopted DR for training image-text CLIP models by storing knowledge from a strong ensemble of CLIP models and additionally synthetic captions from an image caption generator. They demonstrated up to 1000x improved learning efficiency compared with non-reinforced CLIP training.
Given batch of b image-text pairs, we denote the embeddings of target student model by Pung: Pixe E R.xd where d is the dimensionality of the shared embedding space. We utilize two types of teachers, an image- teacher ensemble that maps images and texts to a shared space similar to CLIP (Radford et al. 2021 | and CoCa-based captioners that generate a caption given an image using an encoder-decoder architecture YYu E Rbxds denote the image-text embeddings from the k-th CL.P-based teacher et al. 2022 Let. (b)) (((k)
i
3

---

Published in Transactions on Machine Learning Research (08/2025)
ImageNet-1k Zero-shot Accuracy (%)
80.0
ImageNes 1-1kk Tero-shot Accuracy
1.7x
+3.2%
60
76.0
5.0x
72.5
70.0
30
67.5
20
DFNDR-12M
DataComp- Comp-12M
DFN-12M
DataCompDR-12MM
DFNDR-2B
DataCompDR-1B
60.0
0000
10000
16000
20000
25000
30000
20000
40000
60000
80000
100000
120090
140000
160000
180000 200000
Training step
Training step
Training on DFNDR-12M is up to 5x more efficient compared with SaaaComp-1112MM 3.3x compared with DFN-12M, and 1.3x compared to DataCompDR-12M All models are trained for 30k iterations and global batch size 8192 (246M seen samples). DFN-12M consists of 12M uniformly sampled image-text pairs from DFN-2B and DFNDR- 12M consists of additional reinforcements per sample in DFN- 12M. Right: Training on DFNDR-2B is up 1.7x more efficient compared with DataCompDR-11 Models are trained for 200k iterations and gloabl batch size 65536 (13B seen samples).
Figure 2: Left:
where de the dimensionality of the shared space. The distillation loss is defined as
D
1 26K
Σ KL (v ( L (s Pix)) + KL(S, (w txt ( (|5 o @mmg))
(1)
Image to Text
Text to Image
where KL. denotes Kullback-Leibler divergence, and S+UU, V) is the row-wise Softmax operation applied to UVT/T with temperature т. The total loss is LTota = (1 X)LLLPP + AC that is the sum of a standard CLIP loss and the distillation loss with coefficients and X, respectively.
2.2 Better Base Dataset: DFN
Multi-modal reinforced training starts from a base dataset containing real image-text pairs commonly collected from the web. DataComp (Gadre et al., 2023 demonstrated that the quality of large-scale image-text datasets can be significantly improved through filtering based on scores such compatibility of image and text. Their proposed BestPool filtering applied pool of 12B samples resulted in the DataComp-1B dataset that benchmark
used the base dataset in MobileCLIP. DataComp also released the original 12B samples for dataset curation and filtering methods. DFN (Fang et al. (2024a) proposed to filter data using filtering network trained on high-quality data. Applying their model on DataComp-12B pool resulted the DFN-2B dataset. They additionally collected a larger set of images from the web disjoint from DataComp-12B and after filtering resulted in another 3B samples and collectively created the DFN-5B dataset.
We study the impact of replacing the base dataset in MobileCLIP with DFN-5B. We ablate using 12M uniformly sampled subset of DataComp-1B referred to
SasaComp-11121 that was introduced in (Vasu et al. 2024ch for rapid experimentation. We similarly sample 12M subset from DFN-5B referred to DFN-5B12M. Table 2 compares the performance of training with and without distillation iitillatiion/ymtheeiiic captions. We observe that DFN-5B12M results in up to 1.4% gain together with distillation and synthetic captions. Although this gain smaller compared to the up to 6% gain without distillation/syntlen captions, is still more than standard deviation.
2.3 DFN CLIP Teachers
One source of reinforcement in multi-modal reinforced training is the embeddings from CLIP teachers that are used as targets in CLIP distillation. (Vasu et al., (2024c) performed comprehensive study of existing strong CLIP teachers the time of publication and found the ensemble of ViT-L-14-openai
4

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Training on DFN is better than DataComp with and without distillation/syythetic captions. CLIP teachers and synthetic caption generators are the same as MobileCLIP (OpenAI-DataCome XL CLIP-ViT-L/14, and CoCa- CCC--Vi-L/11)) For distillation, the coefficient À is set 1.0 (no CLIP loss) and use strong image angmentations.
Dataset
Distill.
Syn. Caps.
IN-val
Flickr30k
Avg. 35
DataComp-1BI2M
X
44.6
42.4
40.1
DFN-5B12M
x
49.9
48.5
43.5
DataComp- 1B12M
x
51.9
71.8
47.8
DPN-5B12M
x
54.9
70.7
DataComp-IB 1B12M
/
56.3
57.8
48.7
DFN-5B12M
59.5
60.4
50.0
DataComp- 1B12M
/
61.6
72.8
53.7
DFN-5BI2M
63.0
74.1
51.6
b90k to result the best student performance. Here we investigate the effectiveness of - models as teachers. DFN-pretraing CLIP models with ViT-L-14 and ViT-H-14 achieve the stato-of-the-art performance Avg. 38 evaluations of DataComp (Fang al. 2024a better than other popular models such as SigLIP (Zhai et al. 2023)
and NUT-L-14-dattcoopp___
As the choice of the caption generator and the CLIP teachers may depend on each other, we reduce the complexity of our analysis by analyzing the effect of the CLIP teachers on synthetic captions from a CoCa model without fine-tuning (See Sec. 2.4 We explore the diversity of synthetic captions through fine-tuning in Sec. 2.5
Table 3: Optimal logit scale for distillation varies across teachers. The dataset DFN-5BI2M with
synthetic captions generated from CoCa-DFN-2B Sec. The loss coefficient set to 1.0 and trained using strong image augmentations.
Teacher
Logit Scale
IN-val
Flickra0k
Avg. 38
datacomp x1 x1_s13b_ 190x-CLIP-V1T---14
50
62.6
65.6
53.3
APW2B-CLIP-ViT-L-11
70
65.5
68.0
56.5
-L-B--CLIP-VVT---14
90
64.0
54.7
DENGB-CLID-WAT-H-14--33
55
64.6
67.6
54.4
60
65.2
67.5
54.8
Table 4: Ensemble of DFN CLIP teachers improve ImageNet-1k validation accuracy by 2.8%. The dataset is DFN-5B12M with synthetic captions generated from CoCa-DFN-2B in Sec. 2.4 The loss coefficient À is set to 1.0 and trained using strong image augmentations. The optimal logit scales for each model is set independently based on Tab.
Teacher
Teacher 2
IN-val
Flickr30k
Avg.
VIT-L-14-opena1
VIT-L-14-datacomp [_____13bb900 b90k
63.1
64.7
55.2
ViT-L-14-dataccop x1, ______3b_______ b90k
DEXSB-QLIP-VII--------2
64.5
67.8
54.5
ViT-L-id-dataccopp. _l__slb_______________________ b90k
11
64.4
67.3
55.3
117-L-14-dataccmp__t_
CIVBB-LLI--V1T---144
65.3
68.1
56.2
3PNSS-CLLI--ViT-----------
64.7
66.9
DPWSa-cL.1--Vii----1-----8
DP12B-C111-V3T----11
65.8
56.2
DFNGB-C..P-VVT-TR-14
DFV2B-C.1T--:T----144
68.0
DRi28-CUL1P-E11--114--338
aatacomp_xl_stbb.b
67.6
DF128-0UL1E--11---1444336
DPXGB-0U11P-VTT-4414---88
65.7
67.3
DF12B-0UL1--V1----14-333
DEN5B-CLIP-V-T-H-14
65.7
68.2
DF128-OLIT-VAT---14--333
DEN2B-CLIP-VAT-L-14
65.9
68.7
55.9
Logit scaling. CLIP models are trained with a logit scale that tuned during the training in the range 0-100. MobileCLIP used the same logit scalar the temperature scaling in the KD loss. We observe that.
5

---

Published in Transactions on Machine Learning Research (08/2025)
the logit scalar in DFN and DataComp models is not optimal for KD and tune that further. Table 3 shows the optimal logit scale used for each teacher to train MobileCLIIP-B model. We observe that the logit scale is not sensitive hyperparameter where values within a range of 5 points achieve similar performance.
Ensemble teachers. We construct ensembles of size two using DataComp and DFN teachers. Table 4 shows the performance of training MobileCLIP-B model using embeddings from various ensembles. We observe significant improvements compared with teachers used in MobileCLIP. Specifically, IN-val and Flickr30k improve by up to 3%. We choose the ensemble for MobileCLIP2 based on its performance and cost efficiency compared other larger or higher resolution ensembles. We utilize the optimal logit scales for each member of the ensemble that is found independently. It is possible that the optimal logit scales for ensemble would vary when used together but do not further optimize logit scales jointly.
1PN-B--LI--VII---1---333 and DFN2B SEIZB-CCII-VIT-L-1.
2.4 DFN Caption Generators
Another source of reinforcements for training MobileCLIP2 is synthetic captions generated from an image caption generator. MobileCLIP used a single CoCa captioner which has a two-tower image-text architecture coupled with a text decoder (Yu et al., 2022). Compared with most recent VLMs, the text-decoder is fairly weigh, that results in an overall relatively faster caption generator compared with more recent. VLMs (Liu et al., [2024b Vasu al., 2024a). As MobileCLIP generated multiple synthetic captions on billions of images, the cost of rumning CoCa
important decision factor. They did not provide analysis the choice of caplioner but observed significant. gains from training synthetic captions compared with not using synthetic captions (7.4% for 30k training iterations). MobileCLIP generated 5 synthetic captions per image although they observed the majority of the gain comes from the first 1-2 synthetic captions.
We explore training a new CoCa model using the DEN dataset to improve the quality of synthetic captions. We adopt the same architecture as the CoCa. model utilized MobilcCLIP based on the VIT-L//4 image encoder. They utilized the model trained on LAION-2B dataset and fine-tuned on MSCOCO-128k dataset. We pretrain the same architecture DFN-2B for 13B seen samples using OpenCLIP (llharco et al. 2021
Table Pretraining CoCa on DFN-2B without fine-tuning results in similar IN-1k performance but worse robustness and retrieval. The dataset is DFN-5B12M, CLIP teachers are the same as MobileCLIP (OpenAL-DataComp.
CLIP- ViT-L/11) and the architecture of the CoCa model the same CoCa-ViT-I/11. For distillation, the coefficient À is set to 1.0 (no CLIP loss) and use strong image augmentations. Values within one standard deviation of the best of each group are highlighted.
IN-al Flickrr0k Avg.
Distill
High Aug.
LAION-20 2B MSCOC0-1288
DFN.2B
19.9
48.5
43.5
54.9
70.0
19.6
x
X
51.1
65.7
45.3
54.6
55.1
46.2
56.8
67.2
48.4
/
59.5
60.3
50.0
/
X
63.0
74.1
54.6
63.1
64.7
65.2
63.4
72.0
56.1
Table 5 demonstrates the impact of DFN-CoCa synthetic captions on the performance with and without distillation. We observe that utilizing DFN-CoCa synthetic captions results in improved IN-val and Avg. 38 performance but negatively impacts retrieval. As we observe in Sec. [.5] the retrieval performance recovers with fine-tuning on high-quality datasets such as MSCOCO. We further observe the synthetic captions from the original CoCa model can be used together with DFN-CoCa captions to provide additional gains but these gains small with distillation.

---

Published in Transactions on Machine Learning Research (08/2025)
2.5 Fine-tuning Caption Generators
In Sec. 2.4 we showed that pretraining CoCa. model DFN-2B results in improved IN-val and Avg. 38 performance when utilized for multi-modal reinforced training. However, the retrieval performance falls behind which is due to the lack of fino-tuning on a high-quality datasct. MobileCLIP used a CoCa model fine-tuned on MSCOCO (Chen et al., 2015). MSCOCO-200 contains 123k images with captions that have higher quality compared to average image-text pairs in DataComp and DFN datasets.
In this section, we study the impact of fine-tuning on various high-quality datasets. In addition to 123k samples from MSCOCO which we refer to as MSCOCO- -23k,, also consider subset of 38k samples with permissive licenses (CC Attribution 2.0, CC
coarbbuttoo-SharAAlliii 2.0, and CC Attribution-NoDenn 2.0) which we refer to as MSCOCO-38k. We also consider GBC-IM/10M (Hsieh et al., 2024), DOCCI-9K- short/ "extended//ooe completee (Onoe et al.) 2025), DCI-8k (Urbanek et al. 2024), and SeCCap-CCCO-31 Li et al. 2024). We fine-tune DFN-CoCa on each dataset for 12M seen samples using the same loss as CoCa pretraining.
Table 6:
The dataset is DFN-5B12M, CLIP teachers are selected DFN models DDFN2B-CLI--VII-IE s39b and DFN2B-CLIP CLIP-VI-----I1 and the architecture of the CoCa model is the same as CoCa-VIT-L/14. For distillation, the coefficient is set to 1.0 (no CLIP loss) and use strong image augmentations.
Base Dataset
FT Dataset
Context len
IN-val
Flickr30k
Avg. 38
LATON-2B
MSCOCO-123K
77
65.46.4
75.8a.s
56.2gs
DFN-2R
77
65.9
68.7
55.9
DFN-2B
MSCOCO-123K
77
65.9
76.0
56.2
DFN-2B
MSCOCO-38k
17
65.9u.
75.4g
56.3u3
DFN-2B
GBCIM-short
77
DFN-2B
DOCCI
77
66.3
72.6
57.3
DFN-2B
DCI-short
77
65.9
74.0
56.3
DFN-2B
DCT-extended
77
73.5
56.1
DF
DCI-complete
77
65.8
73.8
56.2
DFN-2B
Recap- Cocp-CO00-30K
77
65.1
73.5
55.5
DFN-2B
GBC-IM-long
255
61.7
72.4
55.1
DFN.2B
TRC-10M-ahott-rlaattn
255
73.8
DFN-2B
GBC-10M-lang
255
64.6
71.9
54.6
DF
DOCCI
255
66.1
74.0
57.2
DFN-2B
DCI-exteuded
253
65.7
75.1
55.9
DFN-2B
DCI-complete
255
74.0
56.8
DFN-2B
3x2
77
65.96
74.7o.a
56.36
DF
10x1
77
66.0p.
75.1
56.55
Fine-tuning on MSCOCO38K and MSCOCO128K. We observe that restricting fine-tuning to MSCOCO samples with permissive licenses does not have a negative impact on performance.
Ablation on number of synthetic captions and beam search. (Vasu et al., 2024c) observed that even though one can generate multiple synthetic captions from a CoCa model, their effectiveness saturates at 2 per sample for classification tasks. We observe similar results using single CoCa model with various sampling strategies. We explore varying the generation method and hyperparameters. Specifically, we used top-p, top-k, and beam-search and observed that beam-search results in qualitatively more diverse captions, however, we did not observe any improvement in downstream performance when utilized for reinforced training.
Fine-tuning on GBCIM, GBCI2M, DOCCI, DCI, ReCap-COCO30k We observe that most finc-tuning datasots underperform MSCOCO fine-tuning or perform on-par within one standard deviation. An exception is fine-tuning on DOCCI results in 0.8% improvement in average of 38 evaluations which is more than one standard deviation from the MSCOCO-355 results.
Effect of context length. The context length for training CLIP and CoCa models is typically set to 77. We explore training CoCa models lo generale longer captions by setting the context length for training and generation to 255. Most results stay within one standard deviation. Recent works have improved the support for long captions in CLIP models with improved loss functions and training strategics (Zhang et al. 2024 Zheng et al. 2024 Najdenkoska et al. 2024). We leave extending these modifications CoCa models for future work.
7

---

Published in Transactions on Machine Learning Research (08/2025)
1,
Nare Rasamitt Eritos S44
Parc Driber 200e
10a
Stage 2
Stage 3
Sha
Stage
Stage
Projection Layer
Latency (ms)
10
MCH-3,4
TMan brite siner?
Plach Erites Sises
Stem
Stage
Siage
Suige
Stage 5
To Projection Layer
MC12 Scaled
10a
MCI3
256
512
768
1024
Commolutional Stemm
HepMiterr Stage
Selt Attention Stage
Image Resolution (px)
(b) 5-Stage achieves lower latency at higher resolutions. Both models MC12- Scaled and MC13 are of the same size, with MC13 using 5-stage design.
(a) MCi architectures for new larger variants. For smaller
variants, the image encoder has four distinct stages compute and for larger variants, use five stages. The projection layer for MCi models include Global Avg. Pooling layer followed by a linear layer.
Figure 3: MobileCLIP2 architecture and latency.
Effect of synthetic caption diversity. We further explore training with diverse collection of captions generated from an ensemble of CoCa models fine-tuned on different datasets. The motivation is the diversity in fine-tuning datasets would increase the divresity in synthetic captions and hence increase in the effectiveness of additional synthelic captions. We observe that utilizing up to 10 different CoCa models results in performance that is still within one standard deviation of the best performance.
Reinforced DFN datasets. Our final datasets small DFNDR-5B12M and DFNDR-2B12M consist of synthetic captions with MSCOCO-35k fine-tuning, and embeddings from the ensemble two DFN2R-ViT- L/14 teachers discussed Soc. 2.3 for image augmentations as well as ground-truth and synthetic captions. We explored training on only the 2B subset of DFN versus the full 5B set that was expanded with 3B samples outside of the 12B pool of DataComp. Tab. ? shows that the average performance on 38 evaluations is within standard deviation for both datasets while ImageNet-1k validation accuracy is better with the 12M samples from 5B. However, we did not observe the improvement to hold when training at larger scales and restricted our recipe to the 2B dataset.
Table DFNDR-5B12M and DFNDR-2B12M perform similarly on average 38 evaluations.
Dataset
IN-val
Fliekr30k
Avg. 38
DENDR-GB12M
65.90.
56.50.3 65.5
75.46.2
DFNDR-2B12M
74.8
56.4
3 Architecture
Our MobileCLIP2 consists of similar architectures MobileCLIP well as two new variants. Specifically, train MobileCLIP2-S0, MobileCLIP2-S2. and MobileCLIP2-B where utilize the standard "Base" text encoder for MobileCLIP2-SO and drop the S1 variant. In addition to architectures introduced in MobileCLIP, we introduce two new variants in MobileCLIP2 family, i.c., MobilcCLIP2-S3 and MobileCLIP2-S4 The text encoders for Uhese variants are pure Iransformer-based architectures and the image encoders are based on Vist (Vasu et al.) 2023b), which uses train-tinn overparameterization blocks introduced in (Vasu et al.) 2023a). The smaller variants, MC10, MC11, and MC12 are hybrid vision transformers with four distinct stages of compute. We introduce an additional transformer stage for MCB3 and MC14 preceded by 4x down-sampling of input tensor shown in Fig. 3a The 5-stage design has two advantages when scaled up: First,, the
8

---

Published in Transactions on Machine Learning Research (08/2025)
parameters can be distributed across five stages with the largest layers operating four times fewer tokens. Second, the design scales more effectively to higher resolutions.
We empirically validate our design choices across various image resolutions. In Fig. 3b we scale MCI2 to match the size of MC13 (125 million parameters) and benchmark its performance across four input resolutions. Our results show that MCI3, with its five-stage design, offers a significantly better trade-off compared to a scaled MC12. At low image resolution, i.e, 256x256, MC13 1.9x faster than similar sized MC12 and for larger imput resolutions, i.c., as 0024x1024, MCB3 is 7.1x faster than similar sized MCi2. Responsiveness at higher resolutions is particularly important when the image encoder is fine-tuned for dense prediction tasks such as image segmentation, where the input image resolution is 512x512.
4 Experiments
In this section, train new family of efficient CLIP models, MobileCLIP2, and evaluate diverse set of tasks. Following our findings in Sec. 2 we create the reinforced dataset, DFNDR-2B, which contains five synthetic captions generated from our CoCa-ViT-I./14 model pretrained on DFN-2B and fine-tuned on MSCOCO-38K. DFNDR-2B also contains image-text emboddings from an ensemble of CLIP modols, DPV2B-LLI--VTT--144-333
and DEN2B-QLIP-ViT---14 for all images, ground-truth captions, and synthetic captions. We train a more diverse family of architectures compared with MobileCLIP and evaluate their performance 38 zero-shot classification tasks (Gadre et al., 2023) Particularly, we introduce MobileCLIP2- S3 and MobileCLIP2-SA architectures trained on DENDR-2B as well as variants trained on DataCompDR-1B which we refer to as MobileCT.IP-S3 and MobileCTIP-S44 Table S shows our results compared with other models with similar latencies. Details of training and hyperparameters are described in Appx.
We compare MobileCLIP2 to prior small CLIP architectures TinyCLIP Wu et al. 2023 trained on LATON (Schuhmann et al. 2022 2021) and ACED (Udandarao al. 2024). We also compare with larger models from OpenAl's CLIP RRdford et al.) 2021), DataComp (Gadre al. 2023). VeCLIP (Lai al.)
2023
EVA (Sun et al., 2023). DFN (Fang et al 2024ab. SigLIP (Zhai et al. 2023). and SigLIP2 (Tschannen) 2025 We evaluate all models using OpenCLIP (Tlharco et al. 2021 and DataComp (Gadre et al. gaps with reported results in their paper.
et al.
2023
In some cases such as SigLIP2, we observe
MobileCLIP2 achieves state-of-the-al ImageNet-1k validation zero-shot accuracies at various latencies. Notably, MobileCLIP2SS4 matches the zero-shot accuracy of SigLIP-S0400M//4 on ImageNet validation set while being 2x smaller and improves on DFN ViT-L/14 at 2.5x lower latency. We also improve on ImageNet-1k performance of ACED models considering their latencies. As ACED optimized their models for low inference flops, the latency of both ACED-F1 and ACED-F2 comparable to our MobileCLIP2-S2 architecture while still have higher latency and more parameters. SigLIP-B/ 16 and SigLIP2-B/ 16 models are more comparable in size and latency to our new larger architectures. Particularly, SigLIP2 models have inbstantially larger text-encoders compared to SigLIP models.
We note that our models pretrained on DFNDR-2B do not always achieve state-of-the-art retrieval performance. We attribute this to the bias of DENDR-2B dataset towards zero-shot classification tasks and particularly TmageNet- We observe that models trained DataComp, WebLI, and their derivatives may achieve higher retrieval performance compared to DEN datasets and derivatives while lower on Avg. 38 performance. As such, we also train our new architectures on DataCompDR-1B referred to as MobileCLIP-SS. and MobileCLIP-SA The combination of these two families of architectures will provide flexibility for broader applications.
4.1 VLM evaluations
We report vision-language evaluations using MobileCLIP2 pretrained models in the LLa VA-1.5 setup Liu ct al. 2024a). We keep the vision backbone frozen for all the runs and use Qwen2-7B instcad of Vicuna-7B. All other training details are the same as the original LLaV A.LVA--. VA-1.5 setup, more details are provided in appendix. We evaluate ViT-B/16 16 models pretrained on DataComp, DFN, DataCompDR, and DENDR for 13B seen samples. In Tab. 9 we observe that on average training on DFNDR achieves 3.5% higher accuracy compared with DFN pretrained model, 1.6% better than DataComp pretrained model, and 0.6% better than DataCompDR pretrained model.
9

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: MobilcCLIP2 family of models has the best average performance at various latencies. Retrieval performances are reported @1. Last column shows average performance on 38 datasets as in OpenCIIIP Mharco et al., 2.21). Models are grouped by their total latency in increasing order and by performance within each group. "Base" refers to standard CLIP Transformer-basee based (Vaswani et al, 2017) text encoder with 12 layers, and " "Custom" stands for customized text encoder used in the respective method. Models with substantially higher latencies and/or larger model sizes are grayed out.
Namu
Datnoet
Soen Sumplees
Roselutionn
Eucoder
Imsage
Toxt Encoder
Parums
(iug (x4)
Latency (mux)
(lug (xX)
Zero-shot
IN-val
DN-dhiht
T-1
Flickröck Ret.
1--T
COCO Ret.
Avg. un
TLP RNIMM
LAION- 00M
15.20
BesNet- 1994
Chatonn
15:
56.3
55.0
15.4
47.6
45.3
TLL SNN0M
LAION- 00M
BosNect-100 2004
Chat.on
06.1
63.0
90.2
TTxCCLPP C01//
LAION- 000M
VIII-00M//22
Cuation
14.5
30
50.5
05.0
58.1
51.8
MsbilKCTUP-00
DetaChomp1i-1111
1215
MC20
MUS
12.4
07.8
03.3
07.7
58.1
ACRRI-F
BaraChmp-1 13
1315
356
VIT-8/32
Small
12.7 58.8
21
08.3
71.4
87.0
01.2
[-)
LIPS 20
DENDR-20
3CCT
OpenAA-ENNo
OpenAE-4ao 1005
18B
204
Bas 50
Bose:
15.8 4 68.4
83 3.8
20.3
45.1
67.4
80.0
26.5
45.8
45.1
TLyCLIP 6LM/22
LAZON- unid
15
204
VII-61M//22
Chaloon
61.4 1 54.0
4.8 2.0
02.4
45.7
02.6
78.7
30.3
12.8
G3.0
LAZON-UT uni
VPCC-15M
15.50
204
VIT-G3M/227
Chatom
())
(4)
61.3
(9)
65.0
61.9
38.5
56.9
(+)
McARKCTPP-S]
DencCompDD-111
1315
206
Bose
21.0 + 14.4
25
72.6
60.7
71.0
59.2
44.0
02:2
61.3
OpesA3-RNDD RN10
OpeaAAL-100S
14B
284
Besbees-0 101
Bose
56.8 1 68.4
4.1 8.8
02.3
45.5
55.0
73:0
23.7
40.5
00.3
OpesAA-B/10
OpenAA--1058
13B
284
G1.3
45.5
58.8
76.9
531.4
50.1
92.5
LAION-11/27
LARUN-20 201
2215
284
Bose
86. + 13.
5.9 33
65.77
51.9
18.4
391
56.8
54.5
Seng-1-----1/000 11/27
DensCompp1118
1306
003.1
55.3
61:1
79.0
371
53.5
59.0
Danal Ht/-0////-----
DansComp-118
3:B
V-r--9/29
Powe
86.2+ 63-1
71.8
56.7
61..
81.
39-9
S7.8
00:9
Webll-i0B
V-T--8/55
Chsteen
91.0 262.5
73.8
57.8
75.2
R5-0
17.9
61.9
61.9
M003
Boser
35.7 + 65-
36 5.5
14.4
03.1
75.4
991-3
05.4
65.4
63.2
ACED-FI
DetsComp-13
V-T--1000
Small
26.2 +
6-2 1.8
14.3
1-)
77.0
42.3
74.9
[-)
ACED-F2
DwnaCeap-13
V:T-B//2 /34
Small
26.2 +
1.8
78.0
79.5
91.1
49.7
06.9
[-]
MobileCLIPS-S2
DFND8-2B
30022
Bosk
8.8
77.2
64.7
74.5
93:4
68.8
06.7
64.1
+
[-)
+
56-1
1.40NN-20
EVAIS-B/10
Mersyd-25
DEN-B/M
DFN-2B
V-T-8/16
DataCaur-B/11
DasaComp-13
DenaCompDD-11
98.8
05.3
350bACCLPP-
De 1B
77.2
MobilcCLIPS-B
DENDR-28
SaLIP-B/16
Welll
d1B
204
V:T-B/166
Chatton
52.9
110.5
0.0
76.8
01.0
74.7
80.1
47.8
66.7
02.3
SaLLP-B/115-25
Weld
d1B
204
V:I-B/16
Chalion
52.9
110.8
13.4 + 6.5
78.5
02.0
75.0
90.4
68.4
66.1
62.3
SgLIP2-B/16
WoBLL-10B
41B
204
V:I--//16
Chalion
92.9
225.8
9.01 6.3
78.8
63.5
79.8
93.3
14.2
00.4
64.6
SgLIP11///0 13/66/2666 200
WobLL-30B
41B
206
V.3--8/16
Chalion
92.9
225.3
13.4 6,3
70.1
05.3
50.2
01.2
G4.1
70.5
G4.6
MobileCLIPPS3
1215
256
Large
125.1
1131.
8.0 6.6
76.1
65..2
77.9
80.1
51.8
05.5
65.1
MobileCMPP2--3
DENDE--0
1315
256
Lange
135.1 + 15516
80--
80.7
68.9
17.5
41.6
50.9
05.4
66.8
WebLi
d1B
254
WT-L111
Chotsen
315.0 + 559.2
56.2 19.1
80.1
05.5
79.0
91.5
12.3
70.6
65.6
DEN-L. Li-nnkarrs
DFN-2B
1SB
204
VTT-L114
Lange
304.5 + 123.8
67.0 + 6.6
81.1
05.5
75.5
R9.0
69.7
68.6
05.9
MobiliCLiPP-L/I
DanCCmppD 133
1BB
254
VIT-1.114
Lung
204.5 à 123.8
67.9 + 6.6
79.5
01:5
75.5
01.3
67.8
66.5
06.5
MobileCLIPP2-3 S4
DENDR-20 2B
1RB
206
32514
Luge
221.6
123.8
15.6 + 6.6
$1.0
70.3
75.0
92.4
61.6
69.8
07.8
MobileCLIPS-1A 1,14
DEND8-21
18B
284
VIT-L.114
Lung
304.5
123.6
07.9 1 6.6
$1.0
70.2
77.2
92.0
51.8
69.0
67.8
MobileCLIP.S4
DeteChompD--111 313
1:B
206
M0134
Lorg
223.6
123.6
15.6 1: 6.6
70.4
09.7
79.5
94.9
52.1
70.8
68.1
S0
WebL1
Sc-0000
Custom
427.7
420.7
82.0
03.5
15.2
08.1
ScLIP2
WHALI-10
VIT-L/16
Chattm
315.0
505.8
38.2
19.5
82.1
70.5
81.8
94.5
68.3
ScUP2 50400/14
WHLL--00
So-0004
Chetom
427.7
707.8
50.3
73.0
88.
03.9
55.5
73.9
09.1
Table 9: VLM evaluations in LLaVA-1.5 setup. ViT-B/16 pretrained models reach 3.5% higher accuracy compared with DFN pretrained model, 1.6% better than DataComp pretrained model, and 0.6% better than DataCompDR pretrained model.
Dataset
CQA
SQA
TexeVQA
POPE
MMMU
MMB
VizWiz
VQAv2
Avg.
D8taCommp-1
59.6
71.5
50.5
81.8
51.8
70.7
61.0
DFN-2B
56.9
71.3
16.0
81.4
41.9
52.2
56.1
66.9
59.1
DataCompDD-.1
60.3
73.1
50.4
81.7
54.9
72.1
62.0
DENDR-2B
60.4
72.9
49.9
83.3
45.2
61.9
54.5
72.4
62.6
4.2 Dense Prediction tasks
We evaluate the quality of the visual representations learned by finetuning the image encoder on dense prediction tasks like object detection, semantic segmentation and depth estimation. In Table [0 we report performance of VIT-B/16 16 model with MaskRCNN He et al. (217)) head for instance segmentation on MS-COCO Chen et al. (015)) dataset. All models were trained using MMDetection library Chen et all (2019) using 1x schedule with single scale testing as described in Wei et al. (023). We follow finctuning setup described in Wei el al. (2023), more details in appendix. In Table Table 11 we report performance of VIT-B/11 16 model with UperNet Xiao et al. (2018) head, trained using the same setup described in Liu et al. 2224cc on ADE20k (Zhou et al. (0017) dataset. In Table 12 we report Root Mcan Square Error (RMSE) on
NYUv2 dataset Nathan Silberman & Fergus (2012). We use the same settings as described in Vasu et al.
2024bb, more details are provided in appendix.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 12: Results on NYUv2 for depth estimation following the same settings as Wei et al. are for ViT-B/16 models.
2023 All results
Method
Dataset
RMSE(L)
CatLIP Mehla el al 2024
DataComp
0.394
MAE
He et al 2022
IN-IK
0.383
MAWS Singh et pl. 2025
IG-3B
0.371
FD-CLIP Wes el a. 22125
OpenAI-WIT + IN-1K
0.352
MAE
Singh el al.
IG-3B
0.348
CLIP Radford et al.
2021
OpenAI-WIT DFNDR-28
0.116
MobieCLIP2
0.356
Table 13: Comparison pretraining methods for semantic segmentation on ADF-20k. For reference, we have included recent state-of-the-art semantic segmentation models (in gray).
Encodor
Decoder
Pro- Tras-rriinng
Resolution
Parana((M)
Interalmuge-
Wine et al.
2028
UperNe
Xito al.
2018
Sup. IN-1K
612x512
128.0
00.3
VIT- VIT-Adapter-
SemsoticFPP
Kurlloy
7019
Sup. IN-22I
612x512
101.8
50.7
VTT-Adepte--
UperNe
Sup. IN-22K
612x512
Swin
Lea
UperNet
A3to 0 3]
2015
Sup. IN 22-22K
640x610
234.1
52.1
MC3
SemantieTT
Kirilly et al 2019
Sup. IN IK
512x512
14.5
48.8
MCH2
Semaati:FP
IN-1K
612x3312
Semanti:
MobileCLIP2
(42:2)
MOR2
SemsnticFP PX
K Kiiloy vi 90
20
MobileCLI1P
012x512
38.5
51.6 (4227)
Additionally, assess the performance of smaller MobileCLIP2 variants dense prediction tasks. Popular pretraining methods like MAE (He et al.) 2022), are not directly applicable to hicrarchical convolutional and hybrid architectures such as our MCI models, hence we compare MobileCLIP2 pretraining with supervised pretraining for the same architectures. In Tabs. 13 and 14 we see that MobileCLIP2 pretraining is significantly better than supervised pretraining and can serve as a good pretraining choice for hicrarchical architectures.
Table 10: Object detection and instance segmenta- tion results MS-COCO with Mask-RCM head trained for 1x schedule. All models are ViT-B/16.
Table 11: Semantic segmentation results on ADF20k using UperNet decoder. All models are VIT-B/ /66
Method
Datacet
mioo
Method
Dataset
mAp
2022
CatLir
DataComp
aBO
IN-1K
49.5
90.77
MAP Fe al 3022
IN-IK [G 30
46.5
60.9
MAWS Sinch of
IG-30
50.4
61.5
MAE Siagl 2122
16.4
12.1
CaLLiP
DelaClump
MAWS
2212
1G-30
48.0
43.4
FD-CLIF
OpenAL-WII'
51.7
FD-CLIP Werdt
OpenAL-WII WIT
62.5
CLIP Macked el al.
21121
OpesAL-WFI'
49.5
CLIP Radford et
2021
OpeuAL-WII DENDR 2B
45.0
39.8
MobiecLiPg
DENDR 2B
52.8
64.0
Mobire DIP2
47.0
41.8
5 Related Work
Improving the training of multi-modal models focus on three aspects: data, objective function and architecture. Our MobileCLIP2 builds on MobileCLIP and provides improvements in all three aspects.
Data approaches either filter dataset or augment it with additional information. Basic filtering methods begin by selecting or crawling large dataset of candidate image-text pairs and filter using ad-hoc rules based on the URLs statics of the images and captions (Radford et al. 2021: Schuhmann et al. 20211 2022 Xu et al. 2024). More advanced filtering methods involve filtering models trained high-quality data utilized to remove low-quality image-text pairs. These methods may utilize pretrained CLIP model (Gadre et al. 2023) or more specialized filtering models (Fang et al. 2024a). The challenge with data methods is that the biases introduced by ad-hoe rules or pretrained models. For example, most publicly available datasets such as DataComp are filtered for English-only data which limits the capabilities of models on non- English tasks (Carlsson et al. 2022) Nguyen et al.) 20024] Pouget et al. (024)). Alternatively, protrained models may be used for active data selection based on the sample difficulty (Evans et al.] 2024a b). It has also been observed that repeating high-quality data achieves higher utilization (Goyal et al. 2024
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: Comparison pretraining methods for object detection task on MS-COCO using MaskRCNN He et al. (2017) detection head. All models are trained for 1x schedule. For reference we have included recent state-of-the-art object detection models (in gray).
Model
Pre- Fre-Training
# (Mrams(M)
mAp
mAP
ViT-Adapter-II B
Chen et al:
2023
Sup. IN-IK
284
47.0
41.8
InternImage--
Wang et AL.
2123
Sup. IN-1K
115
48.8
41.0
VIT-Adapter-I
2023
Sup. IN-22
317.9
48.7
43.3
MC30
Sup. IN-IK
31.0
41.8
38.0
MC32
Sup. IN-IK
55.0
46.6
41.7
MC30
MobileCLIP2
31.0
44.4
(+2.6)
39.6
(+1.6)
MC32
MohileCLIP2
55.0
49.1
(+2.5)
43.2 (+115)
More broadly, the output of pretrained models be stored part of new augmented dataset. For example, various works utilize image-captioning models to generate synthetic captions for images in dataset (Yang
ct al. 2023a Nguyen et al. 2023 Lai et al. 2023 Liu et al. 2024d Li ct al. 2024). Large langnage models
can also be used to rewrite ground-truth captions Fan et al. 2023 as well as together with text-to-image
models to generate fully synthetic datasets (Hammoud et al. 2024j. MobileCLIP introduced the multi-
dataset reinforcement where they utilized an image-caption model to generate synthetic captions as well as an ensemble of large CLIP models store CLIP embeddings for multiple image augmentations and synthetic captions and store them efficiently (Vasu et al. 2024c) We follow similar approach while improving both
the caption generator and CLIP embedding generators through better DFN models (Fang et al. 2024a
Another approach is to improve the objective function of multi-modal training. The original CLIP paper utilized a contrastive loss that encourages the representations of images and texts paired in the dataset to be kept close to each other while staying farther away from other images and texts in a mini-batch (Radford et al. 2021 SigL.IP introduced a variant based on Sigmoid instead of Sof that achieves higher training efficiency at larger batch sizes (Zhai et al.
2023 Tschannen et al. 2025).. Other methods utilize objectives
based on image masking (Yang et al. 2023b
Fang et al. 2023 Sun et al. 2023 Li ct al., 2023b and unimodal as well as multi-resol ion training Li el al. 2023a for achieves more significant. improvements, particularly for 2022b Kuang et al. 2023 Wang et al. 2022a Wu et al. 2023
self-supervision (Mu et al., 2022; Li et al.
2021
cost-effective training. Multi-modal distilaaionn
smaller architecture variants (Wang et al.
Notably, MobileCLIP (Vasu et al.) 2024c)
achieved high training efficiency by utilizing an offline knowledge
distillation method (Shen & Xing 2022 Yun et al. 2021 Faghri al. 2023). We utilize a similar objective
function as MobileCLIP that includes embedding distillation on image-text pairs and synthetic captions.
Lastly, architectural improvements seek improved inference efficiency and higher performance given parameter, flops, or latency budget. CLIP architectures are often borrowed from uni-modal image and text models. Particularly, the original CLIP and various followup works utilized standard VIT architectures
together with a modified BERT text encoder (Dosovitskiy el al. 2020 Devlin el al. 2019 Radford el al.) 2021
Efficient architectures for CLIP include TinyCLIP that prunes ViT (Wu et al. 2023). Cao et al. (2023 that
roduce tokens, and Evans et al. (2024b) that reduce the paramcters for lower flops. MobileCLIP introduced efficient architectures specifically design for CLIP where they introduced low latency comvolutionn-traasffonne hybrid architectures for both their image and text encoders. We further improve on their architectures by introducing two new variants that fill the large latency gap between common B and L architectures.
6
Conclusion
We introduce MobileCLIP2, new family of low latency image-text models, achieving state-of-the-art ImageNet-1k zero-shot validation accuracy. Our methodology improves multi-modal reinforced training by utilizing stronger CLIP teachers as well as our newly trained image-captioning models. We particularly perform a comprehensive study of tuning and ensembling CLIP teachers as well as training and fine-tuning efficient image-captioning models. Notably, MobileCLIPP2SS matches the zero-shot accuracy of SigL.IP- S0400M/14 14 on ImageNet- 1k while being 2x smaller and improves on DFN VYT-L/14 at 2.5x lower latency. We release our model checkpoints and data generation code that facilitates dataset generation at scale.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Broader Impact Statement
Our work introduces a family of foundation models particularly optimized for deployment on mobile and edge devices. As such, it aacilitates broader use of foundation models and development of applications for wider user bases. MobileCLIP2 may be used for various applications such as image classification where its output is impacted by the existing biascs of the training datasets and teacher models.
Acknowledgments
We would like to thank Albin Madappally Jose, Barry Theobald, Chen Huang, Rick Chang, and Apple Machine Learning Research team for their help and discussions throughout this project.
References
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: good teacher is patient and consistent. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10925-10934) 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprinl Str:2108.07255 2021.
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. PuMer: Pruning and merging tokens for efficient vision language models. Tn Proceedings of the 61st. Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), 2023.
Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magmus Sahlgren. Cross-lingual and multilingual clip. Proceedings of the thirteenth language resources and evaluation conference, pp. 6848 6854, 2022.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1eeii.90660755 2019.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupla, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arviv:150 00322, 2015.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
MMSSegmentation Contributors. MMSegmentattinn: Openmmlab semantic segmentation toolbox and bench- 2020.
mark. http://///gthtbbbcooooooooee com/ www.:pen aaaaanammmm/mmmssmnntttattt htt.g://.ggghhha...coooo//o/aaaa
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North Americon chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171 4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesníkov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiu:2010.1190 11929, 2020.
Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier J. Hénaff. Data curation via joint example selec- tion further accelerates multimodal learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advonces in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Neur IPS 2024, Vancouver,
13

---

Published in Transactions on Machine Learning Research (08/2025)
BC, Canada, December 10 15, 2024, 2024a. URL http::/ppaprr..niiii../ https//ppppppillllppppppeeee
and Benchmarks Crack.htm.
Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tamno, and Olivier J Henaff. Bad students make great teachers: Active learning accolerates large-scale visual understanding. In Europeon Conference on Compuler Vision, pp. 264 280. Springer, 2024b.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. +: Improving visual-semantic embeddings with hard negatives. 2018. URL httt::://tttuub.cc//fatttttttttt
Fartash Faghri, Hadi Pouransari, Sachin Mehta, Mehrdad Farajtabar, Ali Farhadi, Mohammad Rastegari, and Oncel Tuzel. Reinforce data, multiply impact: Improved model accuracy and robustness with dataset reinforcement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advonces Neural Information Processing Systems, 36:35544-35575, 2023.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T. Toshev, and Vaishaal Shankar. Data filtering networks. 2024a. URL httpps://openreviie
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, pp. 105171, 2024b.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In NewrlPS, pp. 2121 2129, 2013.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14 108, 2023.
Sachin Goyal, Pratyush Maini, Zachary Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-des curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 22702-22711, 2024.
Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthelip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402 01832, 2024.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming Hc, Xinlci Chen, Saining Xic, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. In CVPR, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint arXiv:150s 020311 2015.
Yu-Guan Hsieh, Cheng- Yu Hsieh, Shih- Ying Yeh, Louis Béthune, Hadi Pour Ansari, Pavan Kumar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph-based captioning: Enhancing visual descriptions by interconnecting region captions. arXie preprint w=Xin:2407.0672, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongsook Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL https::///t orgo/i...rr/1/1. 52811.20o0 5143773 If you use this software, please cite it as below.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Andrej Karpathy and Li Fei-Fei. Deep visnal-eemantic alignments for generating image descriptions. In CVPR, pp. 3128-3137, 2015.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic emboddings with multimodal neural language models. 2014.
Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xucfeng Xiao, Rui Wang, Min Zheng, and Rongrong Ji. Dlip: Distilling language-iman image pre-traininn training.. arXiv preprint arXiv:2308. 12956, 2023.
Zhengleng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al. From scarcity to efficiency: Improving clip training via visual-cnriched captions. arXiv preprint ar11::2310.07699, 2023.
Xianhang Li, Zeyu Wang, and Cihang Xic. An inverse scaling law for clip training. arXiv preprint arXiv:2305.0700 2023a.
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? ar preprinl arXiv:2406.084 2024.
Yangguang Li, Feng Liang Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. pre-training paradigm. arXiv
Supervision exists everywhere: A data efficient contrastive preprinl ar Xrv::221 10. 10.05208, 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13390-23400, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pollern Recognition, pp- 26296 26306, 2024a.
Haotian Liu, Chunyuan Li, Qingvang Wn, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. In ICLR, 2024c.
Yanqing Liu, Xianhang Li, Zeyu Wang, Bingchen Zhao, and Cihang Xie. Clips: An enhanced clip framework for learning with synthetic captions. ar preprint arXiv:2411 16828, 2024d.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint arXiv:108.140300 2021.
Sachin Mchta, Maxwell Horton, Fartash Faghri, Mohammad Hosscin Sckhavat, Mahyar Najibi, Mchrdad Farajtabar, Oncel Tuzel, and Mohammad Rastegari. Cailip: Clip-level visual recognition accuracy with 2.7x faster pre-training on web-scale image-text data, 2024.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pro-training. In European Conference on Computer Vision, pp. 529-544. Springer, 2022.
Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne van Noord, Marcel Worring, and Coes GM Snock. Tulip: Token-length upgraded clip. arXiv preprint arXi::2410.11000 10034, 2024.
Pushmoet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. arviv preprint arXim:2307. 10350, 2023.
Thao Nguyen, Matthew Wallingford, Sebastin Santy, Wei-Chiu Ma, Sewoong Oh, Ludwig Schmidt, Pang Wei W Koh, and Ranjay Krishna. Multilingual diversity improves vision-language representations. Advances in Neural Information Processing Systems, 01014440-14599 2024.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuuet, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp. 291-309. Springer, 2025.
Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Steiner, Xiaohua Zhai, and Tbrahim M Alabdulmohsin. No filter: Cultural and socioeconomic diversity in contrastive vision-language models. Advances in Neural Information Processing Systems, 37:106474-106496, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katla, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02 02114. 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Thoo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open largo-scale datasel for training next generation image-text models. Advances in Newral Informalion Processing Systems, 35:25278- 25294, 2022.
Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In Europeon Conference on Compuler Vision, pp. 673 690. Springer, 2022.
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of mae pre-pretraining for billion-scale pretraining, In ICCV, 2023.
Richard Socher, Andrej Karpathy, Quoe V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional semantics for finding and describing images with sentences. 2:207-213 2014.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXio:2303. 15389, 2023.
Michael Tschannen, Alexcy Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint Xiv:25002 14786, 2025.
Vishaal Udandarao, Nikhil Parthasarathy, Muhamuna Ferjad Naeem, Talfan Evans, Samuel Albanie, Federico Tombari, Yonggin Xian, Alessio Tonioni, and Olivier J Hénaff. Active data curation effectively distills large-scale multimodal models. ar:iio preprint arXiv:2/11.1867, 2024.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano.
A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26700-26709, 2024.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Amurag Ranjan. Mobilcone: An improved one millisccond mobile backbone. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 7907-7917, 2023a.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterivation. Tn Proceedings of the TERE/CVF International Conference on Computer Vision (ICCV), October 2023b.
Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koe, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastvlm: Efficient vision encoding for vision language models. orXiv preprinl Xrxiv:24 12. 121.111131. 13303, 2024a.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, and Oncel Tuzel. Clip with quality captions: A strong pretraining for vision tasks. arXiv preprint urXiv:2405.88911, 2024b.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15963-15974, 2024c.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. aiviv preprint arXiv:2409.1211 2024.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowci Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023.
Zhecan Wang, Noel Codella, Ven-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang, Haoxuan You, Kai-Wei Chang, Shih-fu Chang, et al. Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks. arXiv preprint ar 2rv: 10496, 2022a.
Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan. Clip-td: Clip targeted distillation for rision-language tasks. arXiv preprint ar Xix:2201.05722 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine-tuning performance. In ICCV, 2023.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7959 1759-7971,-0 2022.
Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengehen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21970- 21980, 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CL.I data. 2024. URL
https:///peerreview hwtp:enneeeoooournddsssoooaa1
Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pro-training with synthetic caption. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2922 2931, 2023a.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Shusheng Yang, Yixiao Ge, Kun Yi, Dian Li, Ying Shan, Xiaohu Qie, and Xinggang Wang. Rils: Masked visual reconstruction in language semantic space. In Proceedings of the IEEF/CVF Conference on Compuler Vision and Pottern Recognition, pp. 23304 23314, 2023b.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. Trans. Mach. Learn. Res., 2022, 2022. URL
http::://pperrerreee.eeeeet/ nett/forumm1dd5527777717
Sangdoo Yun, Scong Joon Oh, Bycongho Heo, Dongyoon Han, Junsuk Choc, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IERE/CVF Conference Computer Vision and Pattern Recognátion, pp. 2340-2351 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pro-training. arXiv preprint arXiv:2303.18 15343, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pp. 310 325. Springer, 2024.
Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-inage pre-training with long captions. In European Conference on Compuler Vision, pp. 73 90. Springer, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
18

---

Published in Transactions on Machine Learning Research (08/2025)
A
Experimental Setup
Table
15 provides summary of datasets used in our ablations and experiments.
Table 15: Summary of pretraining datasets.
Num. Samples
Syn. Captioner
Image Angs.
Syn. Capa.
(TDss
DalaCoomp B82MM
12.8M
0.9
12.5M
DEN 5B12M
12.8M
X
1.5
1ataCComp
OpenA1 DataComp-Xi.
LAION-2H MSS0000-1234
DENDR-2B12M
12.8
DFN-2B DFX-2D-39B
DPN-2B MSCOCO-18I:
30
1.3
DENDR-MB12M
12.8M
DFN-2B DFN-2B
DFN-2B MSCOCO-38k
30
2.6
DataCoup-- 1B
1.3B
90
DFN-2B
1,08
65
DacCComp13-118
1,33
OpenA1
DataComp-XL
LAION-2H MICD00-1233
140
DFNDR-2B
1.9B
DFN-2B DFN-2B-300
DFN-2B MSCOCO-38l
162
Table 16 summarizes the hyperparameters we used to train MobileCLIP2. For training on 13B seen samples, we use either a setup with 32x8x A 100-40GB GPUs or a setup with 16x8xH100-80GG GPUs. For our ablations we train for 30k seen samples using 4x8xH100-80GB GPUs and global batch size 8192.
Table 16: Training hyperparameters for our CLIP experiments on DFNDR-2B.
Hyperparameter
S0
$2
S3
S4
Input resolution
256
256
2242
256
256
Context length
77
angmentation
RandAugment
Random resize crop scale
[0.08, 1.0
Random resized crop ratio
0.75, 1.33]
RangeAugment target value
(40, 20)
Train iterations
200k
Warmup iterations
10k
10k
2k
2k
2k
Global batch size
65536
65536
65536
114688
11 414688
Optimizer
Adamm
AdamW betal
0.9
AdamW beta2
0.95
Max learning
1e-3
Min learning rate
1o-6
1e-6
L.R. decay schedule
cosine
Weight decay rate
0.2
Gradient clipping
Mixed precision
16
EMA decay rate
0.9995 0.0
No EMA
No EMA
No EMA
No EMA
CLIP loss weight
0.0
0.0
0.0
0.0
loss weight
1.0
GT caption weight
1.0
Synth. caption weight
1.0
Synth. teacher
CoCa- VIT-L/14 DFN-2B
MSCOCO-35L
Teacher
MEN2B-CLLI-VVT-LLL-E
Teacher 2
DENIB-CLIP-VI-LLII.
Teacher logit scale
70.0
Teacher logit scale
60.0
Teacher resolution
224x224
A.1 Training details for CoCa caption generators
We use OpenCLIP train CoCa- VIT-L-1 architecture (coca_ Vit_L-1)- We pretrain models on DFN-2B and fine-tune on various datasets. Table 17 summarizes the hyperparameters for our CoCa pretraining and fine-tuning.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Table 17: Training hyperparameters for our CoCa models trained on DFN-2B.
Hyperparameter
DEN-2B Pretrain
Fine-tune
Input resolution
224
224a
Context length
77
77, 255
Seen samples
12.8B
12M
Train iterations
195k
3k, 6k
Early stop iterations
143k
N/A
Warmup iterations
10k
Global batch size
65536
4092, 2048
Optimizer
AdamW
AdamW betal
0.9
AdamW beta2
0.95
Max learning rate
le-3
1e-5
Min learning rate
0
LR. docay schodule
cosine
Weight decay rate
0.2
0.1
Gradient clipping
1.0
Mixed precision
amp
CoCa caption loss weight
2.0
CoCa contrastive loss weight
1.0
GPU Setup
32x8x 100-40GHs
1x8xH100-800GH
A.2 Training details for VLM
To assess the quality of the vision encoders, we adopt the LLaVA-1.5 (Liu et al. 2024a
training framework.
This framework consists of two stages: (1) projector training, and (2) joint Lune-tuning of the projector and the language model on 665K instruction-tuning samples. The hyperparameters used in our experiments are summarized in Table 18 For the language model, we use Qwen2-7B- Instruct Wang et al. (0224) as opposed to Vicuna-70 7B. In both the stages the vision encoder remains frozen.
Stage-1
Stage-2
Data
VA-15 558K
LLaVA-1.5 665k
Learning Rate
1e-3
2e-5
Global Batch Size
256
128
Epochs
1
LR. schedule
cosine decay
cosine decsy
LR. warmup ratio
0.03
0.03
Optimizer
AdamW
Adamw
Trainable
Projector +
Projector
modules
Language Model
Table 18: LLaVA-1.5 training setup used in ablations for Table 9
A.3
Training details for dense prediction tasks
A.3.1
Object detection
We train object detection models with MaskRCNN detection heads. Along with detection, these models also perform instance segmentation. We follow the settings prescribed in recent works like Liu et al. (2024c); Wei ct al. (2023); Singh et al. (2023): Vasu et al. (0044b). All evaluations reported in the main paper are from single-scale evaluations on MS COCO validation set following prior works. We sweep through stochastic depth rate in steps of 0.05 and peak learning rate for all the results reported in the paper and the ranges are listed in Table 19 For ViT-B/16 models, we use ViTDet style feature pyramid network. For MCi architoctures, we follow the setup described in Vasu et al. (2023b). All models were trained using MMDetection library Chen et al. (2019) single node with 8 A100 NVIDIA GPUs.
20

---

Published in Transactions on Machine Learning Research (08/2025)
A.3.2 Semantic Segmentation
We train segmentation models with UperNet and SemanticFPN heads. These models are trained on ADF20k Zhou et al. (2017) dataset following the settings prescribed in Liu et al. (2024c); Wei et al. (2023); Singh et al. (2023); ; Vasu et al.) (2024b). All evaluations reported in the main paper are from single-scale evaluations validation set following prior works. For ViT-B/16 modcls, we use ViTDet style fcature pyramid work with UperNet head. For MCi architectures, we follow the setup described in [Vasu el al. (2023b) and train models with only SemantieFPN head. We sweep through stochastic depth rate in steps of 0.05 and peak learning rate for all the results reported in the paper and the ranges are listed in Table 20 All models were trained using MMSegmentation library Contributors (2020) on a single node with 8 A100 NVIDIA GPUs.
;
A.3.3 Depth Estimation
We follow the experimental setup and architecture as described in Wei et al. (0223) Vasu et al. (2024b). The models are trained and evaluated on NYUv2 dataset, Nathan Silberman & Fergus 2012). We sweep through stochastic depth rate in steps of 0.05 and peak learning rate for all the results reported in the paper and the ranges are listed in Table 21
Table 19: Training hyperparameters for object detec- tion and instance segmentation experiments on MS COCO. "RRC" is RandomResizedCrrp. We sweep through stochastic depth rate in steps of 0.05.
Table 20: Training hyperparameters for semantic segmentation experiments on ADE20k. "RRC" is RandonResizedCropp We sweep through stochastic depth rate in steps of 0.05.
Hlyerrpaaantters
MaskRCNN
Hyperparameters
UperNet SemanticFPN
Stochastic depth rate
00.0, ... 0.3]
Stochastic depth
[0.0, ... 0.2]
Data augmentation
Multi scale RRC 12
Data augmentation
RRC
Train epochs
Crop Size
512x
Batch size
16
Train Heraaiios
160k 16
40k
Batch size
Optimizer
AdamW
Optimizer
AdamW
Peak learning rate
[5e-4, 2e-4, le-4]
Posk leurning rale
5e-4, 2e-4, le-4]
LRR. drcny schedule type
Step-wise
LR, deray schedule type
Polynomial
LR. decay schedule
8. 11]
Warmup iterations
1500
Weight decay
0.1
Weight decay rate
0.01
5e-4
Table 21: Training hyperparameters for depth estimation experiments on NYUv2 dataset. "RRC" is RandomResizedCrop. We sweep through stochastic depth rate in steps of 05.
Hyperparameters
Value
Stochastic depth rate
[0.0, .... 0.2]
Data augmentation
RRC
Crop Size
x80x480
Train epochs
25
Batch size
24
Optimizer
AdamW
Peak learning rate
[7e-4, 5e-4, 2e-4, le-4] 0.8
Layer decay rate
Weight decay rate
0.05
21


Reviews:
[{"review_id": "UivscPitVC", "reviewer": "Reviewer_hu5S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The experiments are comprehensive and the descriptions are clear. The paper is easy to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Proposes MobileCLIP2, a family of CLIP models built on top of MobileCLIP, achieving state-of-the-art zero-shot accuracy on ImageNet-1k. Key improvements include leveraging stronger CLIP teachers, training better captioning models, and conducting a comprehensive study on tuning and ensembling methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are some formatted issues in a few table headers (like table 8)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rUJczL3qjs", "reviewer": "Reviewer_cmuV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper propose a new family of efficient and low-latency image-text models through improved multi-modal reinforced training.\n2. The authors find that without fine-tuning, caption generators lead to worse retrieval performance, while additional fine-tuning on datasets like MSCOCO significantly improves results.\n3. Extensive experiments demonstrate the model's efficency.\n4. The authors commit to releasing their data generation code and pretrained models to support scalable and reproducible research.\n\n\nWeakness:\n1. The paper lacks quantitative discussion of training efficiency compared to baseline methods.\n2. Given that modern vision encoders are frequently integrated into large-scale vision-language models (e.g., CLIP, SigLIP), it would be beneficial to include experiments demonstrating the advantages of replacing the vision encoder in models such as LLaVA with MobileCLIP-2, highlighting any improvements over baseline performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents MobileCLIP2, a new family of efficient and low-latency image-text models. It improves upon MobileCLIP through enhanced multi-modal reinforced training. Specifically, the authors investigate the effectiveness of DFN-pretrained models as teachers compared to those pretrained on DataComp. Then, they ablate serval critical aspects of the caption generator, including the pretraining and fine-tuning datasets, the number of synthetic captions, and the context length of CoCa models. Based on these analyses, the authors select the best solution and build MobileCLIP2, which achieves state-of-the-art zero-shot performance on ImageNet-1k at low latency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness. It is crucial to demonstrate that the newly proposed visual encoder not only achieves superior performance on standard vision benchmarks but also provides broader benefits, such as enhancing multimodal large language models. Therefore, I recommend including such experiments to further strengthen the paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HWKuD9RU2M", "reviewer": "Reviewer_4B8F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper demonstrates the effectiveness of improved training components, such as pretraining dataset, distillation framework and new architectures, over MobileCLIP through a systematic ablation study.\n- MobileCLIP2 achieves superior performance on zero-shot image classification tasks compared to previous approaches while maintaining low latency and a small model size.\n- The detailed description of hyperparameters in the Appendix ensures the reproducibility of this work.\n\nWeaknesses\n\nThere are no major weaknesses worth pointing out. The lower performance on retrieval tasks also seems attributable to biases in the training dataset (DFN-5B), as the authors noted in the main draft. One suggestion for further experimentation is to evaluate a key motivation behind joint training of image and text representations, that is, such vision encoders can be paired with pretrained LLMs to build performant and versatile vision-language models. Given the growing interest in multimodal LLMs that can run on mobile and edge devices, it would have been more compelling if the authors had demonstrated how the MobileCLIP2 vision encoder performs when integrated with 1B or 3B scale LLMs across various VQA tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new family of low latency image-text foundation models, MobileCLIP2. Building upon MobileCLIP, MobileCLIP2 improves multi-modal reinforced training by training on a better pretraining dataset (DFN-5B) and utilizing stronger CLIP teachers as well as an improved synthetic caption generator (CoCa). It also introduces two new architectures, L and XL, by adding an additional transformer stage. The resulting MobileCLIP2 family achieves state-of-the-art performance at a range of latencies, especially on zero-shot image classification tasks including ImageNet-1K and 38 datasets from DataComp.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please fix typos (e.g., \"multi-resolution training (Li et al., 2023a) proposed training at for cost-effective training\" in Section 5) and answer the following questions.\n\n- Section 5 \"Improving the training of multi-modal models focus on three aspects: data, objective function and architecture. Our MobileCLIP2 builds on MobileCLIP and provides improvements in all three aspects.\": What are improvements in the objective function compared to MobileCLIP?\n- Table 4 caption \"CLIP teachers are the same as MobileCLIP (OpenAI+DataComp-XL CLIPViT-L/14): This should be removed (it seems to have been taken directly from Table 5 caption).", "broader_impact_concerns": "The paper has addressed broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)