CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization
Authors: Luong Tran
Lan-Cuong Nguyen
Huynh Dang Nguyen
Dat Nguyen Cong
Dung D. Le
Van Nguyen
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Mixture of Experts
Fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The manuscript proposes a novel fine-tuning CLIP method using Mixture-of-Experts.

Abstract: Mixture-of-Experts (MoE) architectures have emerged as a promising approach for scaling deep learning models while maintaining computational efficiency. However, existing MoE adaptations for Contrastive Language-Image Pre-training (CLIP) models suffer from significant computational overhead during sequential training and degradation of zero-shot capabilities. To address these limitations, we propose CLIP-FMoE, a novel approach that integrates MoE architecture into CLIP fine-tuning. Our method uses Isolated Constrained Contrastive Learning, a pipeline that trains specialized experts on cluster-based data partitions to accelerate expert specialization. Additionally, we introduce a Fusion Gate mechanism to mitigate catastrophic forgetting of pre-trained knowledge. Extensive experiments across multiple benchmarks demonstrate that our approach achieves consistent improvements on downstream tasks while preserving zero-shot capabilities. Furthermore, our method demonstrates robust performance across varying context lengths, making it particularly suitable for diverse real-world applications.

Published as a conference paper at ICLR 2026
CLIP-FMOE: SCALABLE CLIP VIA FUSED MIXTURE- OF -F-ERPESSS WITH ENFORCED SPECIALIZATION
Luong Tran'
Lan-Cuong Nguyen'
Huynh Dang Nguyen'
Dat Nguyen Cong'
Dung D. Le
Van Nguyen' 1+
'FPT Software Al Center Center for Al Research, VinUniversity
ABSTRACT
Mixture-of-Experts (MoE) architectures have emerged as a promising approach for scaling deep learning models while maintaining computational efficiency. However, existing MoE adaptations for Contrastive Language-Image Pre-training (CLIP) models suffer from significant computational overhead during sequential training and degradation of sero-shot capabilities. To address these limitations, we propose CLIP-FMoE, novel approach that integrates MoE architecture into CLIP fine-tuning. Our method uses Isolated Constrained Contrastive Learning, a pipeline that trains specialized experts cluster-bases data partitions acceler- ate expert specialization. Additionally, we introduce a Fusion Gate mechanism to mitigate catastrophic forgetting of pre-trained knowledge. Extensive experiments across multiple benchmarks demonstrate that our approach achieves consistent improvements on downstream tasks while preserving zero-shot capabilities. Fur- thermore, our method demonstrates robust performance across varying context lengths, making particularly suitable for diverse real-world applications.
INTRODUCTION
Contrastive vision- language models, most notably CLIP (Radford et al. (021), demonstrate strong zero-shot generalization across diverse tasks. A natural approach to enhancing CLIP is to scale up the model size, as demonstrated by follow-up studies such as EVA-CLIP (Fng et al. 2023), which show that larger variants consistently outperform their smaller counterparts. However, pure parameter scaling comes at a steep cost: keeping large models stable requires much longer training schedules, significantly larger batch sizes, and sprawling GPU clusters, turning it into highly resource-intensive effort that soon outstrips the means of most practitioners.
These scaling challenges motivate the need for alternative approaches that enhance CLIP's pacity without the prohibitive cost of uniformly increasing model size. One promising direction is the Mixture-of-Exxeets (MoE) architecture (Jacobs al.) 1991). which has recently gained attention as an efficient way to scale large models sparsely. Instead of activating the entire network for each input, MoE selectively activates only a subset of "experts", enabling faster processing, improved scalability for large models, and greater computational efficiency.
In the context of vision-language models, MoE offers compelling solution: It allows to expand CLIP's modeling power while keeping inference efficient. Recent works such as CLIP-MoE (Zhang al. (024c) and CLIP-UP (Wang al., (2255)) have demonstrated the potential of MoE im- prove performance without fully retraining massive models. However, prior works do not ensure the specialization of experts in MoE architectures (i.e., avoiding trivial expert behavior) or address scalability of the methods (i.c., the number of concurrently trained experts). This is our first primary research focus.
On the other hand, CLIP struggles with fine-grained visual features, likely because its web-crawled pre-training captions are short and noisy, often failing to capture detailed image content. To address this, researchers have tumned to fine-gra fine-grained long-caption datasets. Works like DreamLIP (Zheng et al. 2024) and FLAIR (Xiao et al. 2025) recaptures roughly 30M web-scraped images using
" *Corresponding author.

---

Published as a conference paper at ICLR 2026
multimodal large language models (LLMs), achieving strong results in fine- finegrained tasks, but still lagging behind CLIP in coarse- coarse-grained zero-shot classification. This gap can be attributed to differ- ences in data volume. Alternatively, some methods fine-tune CLIP synthetic fine Aie-grained data. FG-CLIP (Xie et al. 2025), for example, augments CLIP-400M with 1.6B L.MM-generaed samples, but heavy reliance synthetic captions risks a "garbage-in, garbage-out" eycle (as the multi-modal LLMs use vision encoders trained on web-scale datasets) and may introduce hallucination noise, Other approaches such as LongCLIP (Zhang al. 2024a). TULIP (Najdenkoska al. 2025). and FineCLIP ęJing et al. 2024h) train on smaller tine-graine sets and extend context lengths, but often sacrifice generalization on zero-shot benchmarks.
To mitigate catastrophic forgetting, the continual learning literature has explored a diverse set of strategies. Significant attention has been directed toward Mix-ure-of-Experts (MoE) and prompt- based frameworks, such as MoB--Aaaptess Yuu et al. 2024) and 1.2P (Wang et al. (022), which dy- namically adapt model architecture or input prompts. Parallel efforts have focused on regularization- based techniques, notably ZSCL (Zheng et al. 2023), to constrain weight updates, while others have employed orthogonality-
constraints like PGP (Qiao et al. 2024 to minimize interference be- tween tasks. Additionally, feature compression methods such as LADA (Luo et al. (2025) have been proposed to enhance scalability. Distinguishing itself from these existing paradigms, our work in- troduces a fusion gate mechanism, designed explicitly preserve pretrained knowledge during the learning process while allowing experts to acquire new complementary representations.
Those existing approaches either neglect the valuable pre-trained knowledge embedded in vision- language models or fail to preserve the robust Internet keyword labeling knowledge that remains unaffected by hallucination issues. Our second primary objective addresses this gap by developing a solution to mitigate catastrophic forgetting while simultaneously improving performance through training on additional fine-gres aen-igrined datasets.
Our main contributions are as follows.
We introduce CLIP-FMoF, a novel approach to scale vision-language models that com- bines knowledge-fusion with Isolated Constrained Contrastive Learning (ICCL) mech- anism. This design preserves and leverages pre-trained knowledge while efficiently adapt- ing to new data under limited computational resources.
We demonstrate that CLIP-FMoE helps ensure the specialization of experts by learning abundant, distinct knowledge, through data clustering, and reducing the effect of token imbalance routing MoE, thanks to the
pipeline.
CLIP-FMoE shows consistent improvement on various downstream tasks across different settings of context length and dataset size.
RELATED WORKS
2.1
VISION-LANGUAGE MODELS
Vision Vision-Language foundation models like CLIP (Rafford al. 2021), trained on large-scale datasets, show impressive generalization across multiple benchmarks. However, despite CLIP's strong zero- shot performance, its reliance short and noisy pre- pre-training captions limits effectiveness in specific scenarios requiring fine-grained understanding and constrains its scalability. Recent approaches to improve CLIP can be categorized into several directions. Improvements in contrastive learning focus on revisiting the original contrastive objective, particularly by improving the quality of negative sam- ples within training batches (Ma al. 2024). The data approach leverages datasets with improved captions generated by multi-mon modal large language models (MLLMs). For pre- trainingg works like DreamLIP (Zheng al. 2024) and FLAIIR (Xiao al. 2025) train on 30M high-quality samples to improve detailed understanding and boost generalization. For fine-tuning, FGCLIP (Xie et al.) 2025) further fine-tunes CLIP on 1.6B synthetic LLM-generated samples. Context extension meth- ods such as LongCLIP (Zhang et al. (2024)), TULIP (Najdenkoska et al. 2025), and FineCLIP (Jing et al. 2024) focus on extending context windows to utilize high-quality datasets containing long, de- scriptive captions effectively. An orthogonal approach is to scale model capacity. The EVA-CLIP series grows model size from 5B parameters DEVA-02-CLIP (Fang et al. 2023)) to 18B parameters (EVA-CLIP-II) 18B (Sun et al. 2024)) achieving 80% zero-shot top-1 accuracy across 27 benchmarks.
2

---

Published as a conference paper at ICLR 2026
2.2 MIXTURE-OF-EXPEEE
The Mixture-of-Experts (MoE) paradigm (Jacobs et al. 1991) enables dramatic increases in model capacity without proportional rise in computation by activating only a sparse subset of experts per token (Shazeer et al. 2017 Mu & Lin 2023). Empirically, this sparsity mechanism has unlocked large performance gains across vision, language, and multimodal tasks (Rajbhandari et al. 2022] Dai et al. 2024 Wu et al., 2024; Mustafa et al. (022), and enables expert specialization for im- proved task coverage (Shen et al. 2023] Zhu et al., 2025 Lu et al., 2024). A central bottleneck in MoE development lies in the construction and initialization of effective experts. Training MoE To address this, Sparse Upcycling (Komat- suzaki et al. 2023) introduces a cost-efficient initialization scheme by cloning the MLP layers of dense pretraindd model to serve as multiple experts. However, the homogeneity of replicated MLPs reduces further specialization, limiting the upper bound of the model capabilities and leading to
from scratch can be prohibitively essource-inennsin
suboptimal performance (He et al., [024]] Jiang et al.] 2025; Nakamura et al., (025). To address this in vision-lange vision-language models, CLIP-MoB (Zhang al. (222c) sequentially trains experts via penter-and-contrra scheme, while MODE (Ma et al. (2024) uses ensemble learning, training sepa-
rate experts in disjoint subsets without an MoE. layer.
2.3 MODEL MERGING
The problem of combining pre-trained models with task -psc-ipe knowledge has been explored through various paradigms. Model merging techniques such as task arithmetic (Ilharco et al.] 2023) enable linear combinations of model weights to compose capabilities, while LiNeS (Wang et al., 2025a) introduces post-training layer scaling to prevent catastrophic forgetting during merging. Temporal merging approaches (Dziadzio et al. 2025) address how to combine multimodal models trained at different time points, and recent work task arithmetic in tangent space (Ortiz-Jmenne) et al. 2023) demonstrates improved editing of pre-trained models through geometric considerations. Beyond static merging, dynamic composition methods have gained attention: mixture-of-expene architectures (Fedus et al.) (022b) employ gating mechanisms to route inputs across specialized modules, while Model Soups (Zimmer al. (2024) demonstrate that averaging weights of multiple fine-tuned models improves accuracy. Unlike prior merging methods that operate praitrained components, our approach learns the fusion dynamically and hierarchically throughout the training process.
PRELIMINARIES
3.1 CONTRASTIVE LEARNING
Contrastive learning (van den Oord al. 2018) has become a foundational technique for aligning representations across different modalities. CLIP (Radford et al. (021)) measures the similarity of )B images and text pairs {I, T. f- in batch B using function s(,;)), which pulls matching pairs closer in the embedding space while pushing mismatched pairs apart.
To achieve this, symmetric InfoNCE objective is applied across the batch. For each image text pair, the loss encourages the model to assign higher similarity to the true pairing than to all other combinations in the batch:
1
B
log exp (s(L,,
InfoNCE(I,T)
(1)
B
2==1 exp (spps T;))
3.2 MIXTURF-OF-EXPERTS
In Transformer architecture, Mixture-of- (MoE) layer replaces the conventional MLP with collection of N experts { controlled by a router It, where each expert E(i) is an inde- pendent MLP. The routing process begins by computing expert assignment logits of a given input token representation through R parameterized by weight matrix W. The K experts out of N with the highest logits are then selected by TopK operation, and their corresponding logits are then normalized using the softmax function. This yields the routing probability distribution R(x)
3

---

Published as a conference paper at ICLR 2026
Semantic Clustering
Training Layer For model
Training MoE
Stage
Faoel Scees Licke
Stage
Pount Tiper
Fass MiC Levkr
Figure 1: Illustration of our main training pipeline. Stage performs semantic clustering to group original data into distinet clusters (Co, Ct. C2 ... Cx) (left). Data from each cluster trains corresponding MLP layer, with all MLPs training concurrently (middle). In Stage 2, the trained MLP weights from Stage I serve initialization for the MoE experts. During this stage, only the router network is trained while the expert weights remain frozen (right).
the selected experts among all N available experts:
Emoc R(a); (x), R(x) Softmax(TooK(x: W)),
where R(x); represents the i-th routing weight produced by the router network W.
This framework allows models to scale in size and capacity without proportionally increasing com- putation costs compared to dense counterparts. To balance token utilization across experts. |Fedus ctt al. (2022a) proposed regularization term. Given N experts indexed by = .,..., . and a set of tokens processed by routers across all layers in one mini-batch B denoted as Q, the load balancing loss for a transformer block containing an MoE layer is defined as:
Lialance = N. Se. Ji = के Σ ifie TopK(p(:))}, P. = प Ep((t)
(3)
BEQ
=EQ
where f. is the fraction of tokens assigned to expert i, p(x) = Softmax(:: W) denotes the router probability of token across N experts (without applying TopK), and P: is the fraction of the router probability allocated to expert i.
4 METHODOLOGY
This section describes the proposed CLIP-FMoE method. We outline the complete training pipeline and then provide stage-wise explanation of our approach, highlighting the of Isolated Con- strained Contrastive Learning and Fusion Gate model design.
4.1 TRAINING PIPELINE
Recently, Sparse Upeyeling (Komatsuzaki al.) 2023) was proposed to initialize MoE models by replicating the MLP layers of pretrained dense model into multiple experts, thereby enabling efficient model scaling without training from scratch. However, the replication ML.P hinders the development of expert specialization, which in turn restricts the model's ultimate capacity and produces suboptimal performance (He et al. 2024 Jiang et al. 2025 Nakamura et al. 2025).
To address this limitation, CLIP-MoB (Zhang al. 2024c) adopts the Multi-stage Contrastive Learning (MCL) (Zhang et al. 2024b) strategy, which trains MLP series sequentially before in- tegrating them into unified MoE model. Specifically, this approach encourages each expert to learn distinct feature representations at each stage through an inheritance mechanism. where em- bedding clusters from the previous step determine the negative samples for the subsequent stage.
4

---

Published as a conference paper at ICLR 2026
Pused Dencs Loyer
Dense Saper
MyP Beer
Liver
Frozen
Trai
Figure 2: Mixture-of-experts architecture pipeline. Vanilla Transformer Block (left) is initialized from the pretrained model. Each Expert and Fusion Gate in N models (middle) is fine-tuned. N experts are aggregated in a unified block (right) while the Router and Fusion Gate are fine-tuned. Layers with solid borders are trainable, while layers with dashed borders remain frozen.
Samples belonging the same cluster in the previous step are assumed to represent the same fea- ture type and are assigned as negative samples in the next stage to facilitate learning of new feature representations. However, this design suffers from accumulated clustering errors and computational inefficiency in sequential learning. To enable efficient MoE scaling while ensuring expert special- ization, we introduce CLIP-FMoE, a two-stage training paradigm, illustrated in Figure
Stage 1: Isolated Constrained Contrastive Learning. We apply an ICCL mechanism by perform- ing two-level semantic clustering to create specialized training data subsets for each MLP series. An MLP series comprises set of MLPs from multiple layers of the base model that are jointly fine-tuned on the same data subset. These ML. series are then trained independently and in parallel to ensure expert diversification.
Stage Unification of Domain-specific Experts. We integrate all trained MLP series into a unified MoE model as domain-specifi experts through lightweight training of the router and fusion gate components. This approach enables efficient consolidation of specialized knowledge while maintaining computational efficiency.
4.2 ISOLATED CONSTRAINED CONTRASTIVE LEARNING
Data semantic clustering, At the beginning of Stage 1, we use the original base model to extract feature embeddings from the entire dataset for semantic clustering. This choice of embedding model is motivated by the intuition that the base model's representations can help identify challenging domains where its features tend collapse or become indistinguishable. In the context of CLIP, we can utilize both text and image modalities for embeddings. However, based on experimental observations, we use the CLIP image encoder for all settings due to its superior robustness.
In the first level of clustering, we apply K-means to the embedding features of the original dataset D. partitioning it into N semantic clusters {C. that potentially capture coarse-grained attributes useful for expert specialization. At the second level, we further apply K K-means within each cluster C, dividing it into M sub-clusters ( J==1 to capture more fine-grained semantic distinctions.
Model initialization. For each cluster C, we fine-tune |A] dense layers of the image encoder and (B] dense layers of the text encoder, where A and B are sets of chosen layer indices. We denote
the resulting MLP networks as { {E(,) }KE,AAB. Stage 1, to encourage each MLP network E(!) to learn complementary knowledge that builds upon pre-trained representations rather than redundant features, we leverage an additional component called the Fusion Gate, which is introduced in the model architecture section. As illustrated in Figure [ for each k-th layer in both the image and text encoders, the fused dense layer is initialized from the corresponding dense layer in the base model by adding replicated base MLP (B(() and newly initialized Fusion Gate. During training,
we keep only the MLP series (E(.) TKEAA8 and their corresponding gates trainable, while all other parameters remain frozen.
Contrastive learning. Following semantic clustering and model initialization, we train N MLP networks in parallel, independently of each other, each in one of the N clusters. By performing
5

---

Published as a conference paper at ICLR 2026
Model
Cars
Coun
FGVC
Food
Sun
Cal
Euro
Flow
Pets
IN-IK
IN-v2
Avg.
OpenAI CLIP
77.93
31.88
31.77
93.07
67.56
83.27
62.59
79.20
93.21
75.54
69.84
69.62
Fine-tuning
69.12
25.43
25.83
91.34
71.17
79.67
65.20
69.56
90.24
72.89
66.39
66.08
Up-cyeling
68.98
25.36
27.66
90.78
70.82
81.22
69.98
68.52
91.11
72.92
66.27
66.69
CLIP-MoF
71.20
25.31
27.81
91.27
71.01
79.19
65.13
69.90
90.35
73.19
66.86
66.47
CLIP-FMoB
74.62
28.02
29.58
92.32
71.86
81.79
68.31
73.54
91.58
74.87
68.66
68.65
Table 1: Zero-shot classification results across common datasets. All methods use CLIP VIT-L/14, trained on CC3M (LLaVA-ReCap) dataset. The reported metric is top-1 accuracy. Bold denotes the best performance among CLIP-finetuning methods.
isolated training in each cluster C, each MLP series {E(} is guaranteed to learn diverse, non- overlapping knowledge representations, thereby achieving specialization within its domain.
To leverage the fine-grained semantic clustering within sub-clusters { LM Jj=1 of C, we constrain the batch sampling process during contrastive learning to ensure all samples within a batch belong to the same sub-cluster, denoted as Constrained Sampler (see Appendix Cfor more details) shown in Figure [ This approach contrasts with vanilla training processes that employ random sampling, which can compromise the quality of in-batch negatives. [Zheng et al. (2024); Xiao et al. (2025) demonstrated the benefits of matching images with multiple corresponding captions. We adopt this strategy by computing contrastive loss between images and their associated caption sets, which contain various captions (e.g., split from longer descriptions).
Given varying caption quality, we use weighted combinations across loss terms. For each image paired with L positive captions, let {xe)-n be the corresponding weights, where % 2 and Els To 1. We then aggregate the contrastive losses across all positive captions bidirection- ally, computing both image-to-captie and caption-to-image similarities to obtain final objective for Stage-1 training as follows:
L
Lmulti-pe pas
Ya InfoNCE T9) +
Σ %o InfoNCE(T", n),
where embedding batch {1, T. is sampled from the same sub-cluster.
4.3 UNIFICATION OF DOMAIN-SPECIFIC SPECIFIC EXPERTS
Knowledge- Beeedde-reeeriingggg Fusion Mechanism. We introduce the Fusion Gate mechanism that inte- grates pre-trained knowledge with the newly learned, domain-specific information from each seman- tic cluster C,. The mechanism is applied in both training stages, as illustrated in Figure 2 In Stage the fusion occurs within the representation space of the MLP series and their base counterparts Ebase. which function as the experts denoted Enew in Stage 2. In Stage 2, the fusion is between the base MIP, which represents pre-trained knowledge, and a mixture of domain-specific experts from the MLP series (also denoted as Enews known as Emoo). Through this two-stage process, the Fusion Gate enables experts to not only inherit and effectively align with pre-trained knowledge in the first stage, but also be further enhanced by the MoE architecture in the second stage. The fusion controlled by a learnable gating network producing G(z) E [0. 1]
Engo (x) = Erase (x) + (1 - G(x)) (x), G(x) = (Linear((x))
where o is sigmoid and Linear maps tokens to D-dimensional gating coefficients. The layer output Engo (x) combines pretrained and domain specifcc knowledge, then added the attention block's
residual connection.
Fused MoE. After Stage we obtain set of MLP series (E, | E AxB, i [.]]. The process of Fused MoE initialization, MoE along with Fusion Gate, is described as follows: from base CLIP model. specifically, for a chosen Transformer layer l in either image or text encoder, we initialize
N experts Ex( ..... E N by copying the weights from Stage 1, and initialize the MoE router from
in Stage to connect scratch. While Fusion Gate is initialized from average weights of {((1) i=1

---

Published as a conference paper at ICLR 2026
MS COCO
Flickr 8K
Flickr 30K
IR
TR
IR
TR
IR
TR
Model
Avg.
R@1
R@5
R@1
R@5
R@1
R@5
R@I 1
R@5
R@1
R@5
R@1
R@5
OpenAl CLIP
36.51
61.01
56.38
79.36
61.60
86.36
77.40
94.00
65.22
87.26
85.10
97.30
73.96
Fine-tuning
49.04
73.36
63.76
83.94
73.24
93.14
83.20
96.80
78.16
94.85
90.90
99.00
81.62
Up-cycling
48.83
73.19
63.12
84.36
73.10
93.48
83.50
97.00
77.98
94.66
91.20
98.90
81.61
CLIP-MoE
49.46
73.80
65,30
85.44
73.28
93.48
85.70
96.90
78.22
94.94
92.00
99.00
82.29
CLIP-FMoE
49.60
73.69
66.66
86.84 73.68
93.40
86.10
97.40
78.42
94.58
92.30
99.50 82.68
Table 2: Standard image-text retrieval results (R@1, R@5) on MS COCO, Flickr&K, and Flickr30K benchmarks. All methods use CLIP ViT-L/14 architecture with 77-token context length, trained on CC3M with LLaVA-generater captions. Bold denotes the best performance.
base MLP with a mixture of domain-specifir experts, as illustrated in Figure 2 Fusion Gate and the MoE router are set to learnable, which not only keeps the specialization of experts but also is an efficient training strategy. In Stage 2, the Fused MoE model is trained on the entire dataset without constraints on sample size or batch sampling strategy. As in Stage 1, we train the model using the Multi-Positivy InfoNCE loss, together with the token balance regularization defined in equation 3 The final objective is given by
In this stage, only
Lmce = Lmulti-pos + aL
(5)
where we set a 0.01 for all experiments.
5
EXPERIMENTS
5.1
MAIN RESULTS
5.1.1
EXPERIMENTAL SETTINGS
Dataset. For our main experiments, we fine-tune pretrained OpenAl CLIP ViT-L/14 model on LLaVA-ReCap variant of the CC3M dataset (Sharma et al.] 2018 Li al. 2024). Due to the original CLIP model's context length limitation of 77 tokens, we split long captions that exceed this limit into multiple semantically coherent sub-captions (typically four per image, constructed at the sentence level). In addition, we incorporate the raw caption version of CC3M. These combined sources (a total of L = 5 captions) constitute the set of positive captions used in our training dataset.
Implementation. Following Wang et al. (2025b), we fine-tune the MLP layers at odd-numbered po- sitions and in the second half of both encoders of CLIP ViT-L/14. The L.IP-FMoE model employs 4 experts, with 2 experts activated during both training and inference. In Stage 1, the training data are clustered into N 4 primary clusters, each further subdivided into M = 64 sub-clusters, yield- ing total of 256 sub-clusters. We batch size of 1024 and all experiments on 8 NVIDIA A 100 GPUs Additional details are provided in the appendix.
Baselines. We choose Vanilla Fine-tuning, Up-eycling MoE (Komatsuzaki et al., 2023). and CLIP- MoE (Zhang et al.] 2024c) as our baselines. All are fine-tuned from the pretrained CLIP ViT- L/14 using the same training objectives described in Section 4.2 For fair comparison,
fine- tune or apply up-cycling only on the same MLP layers used in our proposed model. To match the number our activated parameters during training and inference, Up-cycling MoE and CLIP-MoE are configured with 5 experts and a top-3 routing.
5.1.2 ZERO-SHOT CLASSIFICATION
Table presents the zero-shot accuracy of our CLIP-FMoE compared with OpenAl CLIP, vanilla fine-tuning, Sparse Up-cycling, and CLIP-MoE. The evaluation is conducted on 11 benchmarks, encompassing fine-grained classification (Stanford Cars (Krause et al.) 2013), FGVC-Aircraft (Maji cet al. 2013)), scene and remote sensing (SUN397 (Xiao et al. 2010). EuroSAT (Helber et al. 2019), Country-211 (Radford et al. 2021)), domain-specific tasks (Flowers 102 (Nilsback & Zisserman
7

---

Published as a conference paper at ICLR 2026
MS COCO
Flickr 8K
Flickr 30K
DOCCI-Long
IIW-Long
Urban
DCI
Model
Avg.
IR
TR
IR
TR
IR
TR
IR
TR
IR
TR
IR
TR
IR
TR
OpenAl CLIP
36.5
56.4
61.6
77.4
65.2
85.1
63.3
66.0
90.5
90.2
55.8
68.3
33,9
36.0
63.3
Fine-tuning
46.6
64.3
72.1
84.6
74.8
90.8
77.7
77.6
97.5
96.8
88.9
89.8
54.4
57.6
76.7
LongCLIP
47.0
63.3
72.5
83.4
76.1
89.3
78.8
66.7
97.2
94.5
86.1
82.4
52.7
45.3
73.9
TULIP
45.7
61.2
70.6
81.5
74.6
88.0
79,6
78.0
97.2
96.2
91.2
90.2
54.7
54.3
75.9
CLIP-FMoB
47.7 65.6
72.4
84,9
75.9
91.6
79.2
78.4
97.3
98.0
88.2
90.7
54.7
55.9
77.2
Table 3: Retrieval performance on long-context scenarios. All methods use CLIP VIT-L/14 with 248-token context (except OpenAl CLIP), trained on ShareGPT4V. Bold indicates best performance, underline indicates second best.
2008
Oxford-IIIT Pets (Parkhi et al. 2012 and general-purpont datasets (Caltech101 (Li et al., ImageNet-1K (Russakovsky et al. 2015), and ImageNet-v2 (Recht et al. 2019)).
2022
Vanilla fine-tuning suffers from significant catastrophic forgetting, with its average accuracy drop- ping by nearly 4% compared to the original CLIP. This is especially evident on fine-grained datasets, such 25,83% accuracy FGVC-Airerafe In contrast, both Sparse Up-cycling and CLIP-Mot mitigate this issue, achieving average accuracies around 66.5%. However, both methods still under- perform the original CLIP model by average of approximately 3%.
On the other hand, CLIP-FMoE nearly closes the performance gap, reducing the average accuracy drop less than 1% compared to the original CLIP. Our method consistently achieves the highest accuracy among all adaptation methods across every benchmark. These results demonstrate that the fusion-gate mechanism in CLIP-FMoE is highly effective preserving pretrained knowledge.
5.1.3 ZERO-SHOP RETRIEVAL
We further evaluate our method on retrieval tasks using several established benchmarks: the standard image-text datasets MS-COCO (Lin et al., 2014) and Flickr30K (Plummer et al.) 2015); the fine- grained retrieval sets DOCCI (Onoe al. 2024) and IIW (Bell et al. [2014); and the long-caption datasets Urban- K (Zhang et al. 2024a) and DCI (Urbanek et al. 2024). following the protocol of (iiao et al. 2025). Table and Table g in Appendix togethher
Appendi demonstrate that CLIP-FMoE. con- sistently achieves the best performance across diverse settings and attains the highest average scores on both standard and fine-grained retrieval. Therefore, we hypothesize that the fusion gate goes beyond simple combination: for each token, it dynamically balances pretrained representations and domain-speciff expert adaptations, utilizing whichever more effective. Furthermore, this process is enhanced by our ICCL mechanism, which sharpens expert specialization on semantically coherent clusters. The combination of these components allows CLIP-FMoE to strike an effective balance, maintaining the pre-trained knowledge generalization while boosting performance on retrieval tasks.
5.2 LONG CONTEXT UNDERSTANDING
In this experiment, we evaluate the long context understanding capabilities of our proposed method. To handle longer captions, we follow LongCLIP (Zhang al. (2024)) and extend the original context length of the base ViT-L/14 text encoder from 77 to 248 tokens by stretching positional embeddings through linear interpolation. For fair comparison with the existing LongCLIP (Zhang et al. (024a) and TULIP (Najdenkoska et al. 2025). ) we use ShareGPT4v (1.2M samples) (Chen et al. (024) as our training dataset. Our experimental setup employs
4 clusters (equivalent to 4 experts) with 2 activated during both training and inference, M 16 sub-clusters per cluster, and batch size of 2048. We also fine-tune CL.IP with extended context length as an additional baseline. Following LongCLIP's approach, we compute the objective loss using two versions of each caption: the full long caption and a shortened version consisting of only its first sentence. This results in a total of L == 2 captions per sample. We set 7 0.1 and 7/2 0.9. We evaluate on 7 datasets, comprising 3 standard image-text retrieval benchmarks (MS COCO, Flickr 8K, Flickr 30K) and 4 specialized long-caption retrieval datasets (DOCCI-Long, IIW-Long, Urban-1k. DCI).
Table compares the retrieval performance on CLIP-FMoE with recent long-context methods. CLIP-FMoE outperforms others, with the highest average performance 77.2%, outperforming the
8

---

Published as a conference paper at ICLR 2026
Dataset
Ex.
Ex.
Ex. 3
Ex.
MoE
Cars
62.95
61.56
58.72
58.08
61.15
Food-101
85.62
87.24
86.97
84.68
87.24
Sun.997
66.73
66.11
65.63
66.12
67.87
Classification
Barosat
43.33
50.36
53.34
$5.25
55.47
Caltech-101
62.38
80,92
82.81
82.79
82.95
Standard
Retricval
Imagenet-a
40.64
38.83
37.99
36.32
40.11
Imagenet-o
41.65
43.35
42.4
41.55
42.55
Imagenet-r
76.54
75.59
74.23
75.92
75.96
uung-contad Netrioval
No Fuston
Pusimn in both stages Fusion stage
Imagenet-s
46.58
46.84
46.04
46.43
46.61
Imagenet-Ik 1k
64.45
64.75
65.36
64.85
65.99
00.0
62.5
05.0
67.5
70.0
72.5
75.0
77.5
80.0
COCO IR
39.48
39.55
Retrieval
COCO TR
56.68
57.34
58.4
59.42
58.8
Figure 3: Ablation on Fusion Gate configurations and data variants. The rr-axis represents the aver-
Urban IR
80.6
84.4
84.4
86.2
87.2
87.80
DCI IR
47.69
49.25
48.81
51 84
51.98
aged Accuracy/R@1
DCI TR
47.73
49.16
54.45
53.4
54.6
Table 4: Result Experts and Unified Model
across classification and retrieval datasets.
strong baseline TULIP by 1.3%. This consistent improvement across diverse datasets demonstrates the robustness of CLIP-FMoE, showing its effectiveness even in long-context settings.
5.3 EXPERT SPECIALIZATION
We present analysis of expert-level performance within CLIP-FMoEE alongside the unified model results. The evaluated model was trained on ShareGPT4V with an extended 248-token context length, built upon CLIP ViT-B/16. Individual expert evaluation was conducted by disabling the MoE routing mechanism and forcing all input tokens through a single expert. The evaluation protocol includes 10 zero-shot classification and zero-shot retrieval datasets.
Table demonstrates that CLIP-FMoE achieves meaningful expert specialization across domains. Expert excels in automotive and adversarial scenarios (Cars, ImageNet-A/R),, Expert special- izes in food recognition and out-of-distribution data (Food-101, ImageNet OOD variants), Expert 3 1 K), and Expert dominates retrieval tasks (COCO IR/TR. Urban IR). The unified MoE model consistently matches exceeds individual ex- pert performance through dynamic expert combination. Crucially, experts learn complementary rather than redundant representations, with performance compensation between experts. This pat- tern confirms that ICCL effectively enhances model capability through expert diversification.
handles general classification (Caltech- 101, ImageNet-
5.4 FUSION GATE ABLATION STUDY
To examine the role of Fusion Gate in CLIP-FMoE, we evaluate three variants: (1) no Fusion Gate, (2) Fusion Gate in both stages (standard design), and (3) Fusion Gate only in Stage 2. We port average performance across zero-shot classification (top- accuracy, datasets from Table [
standard retrieval (top-1 1 recall, 3 datasets from Table 2 and long-context retrieval (top-1 recall, datasets from Table 8 Appendix (A). The results are shown in Figure
Design (1) performs well retrieval tasks but shows the weakest zero-shot classification, validat- ing the necessity of Fusion Gate preserving pretrained knowledge. Design (3) achieves superior classification but fails on diverse retrieval tasks, suggesting that Stage 2-only Fusion Gates prevent complementary expert learning and allow pre-trained representations to dominate. Our standard design (2) provides optimal balance across all tasks, effectively integrating with ICCL while main- taining expert specialization benefits.
5.5 EFFICIENCY AND SCALABILITY ANALYSIS
To evaluate the computational efficiency of our proposed CLIP-FMoE against the alternatives in Table I we report model size, per-sample FLOPs, trainable parameters per stage, clustering time (for CLIP-MoE and CLIP-FMoE), and total training time. All experiments were conducted under identical hardware, batch size, and precision settings. The results are summarized in Table

---

Published as a conference paper at ICLR 2026
Model
Parameters (M)
Cluster (min)
Total Training (min)
Inference (GFLOPs)
CLIP (Base)
427.62 64.53
24.59
84.38
Up-Cycling
685.78 322.69
40.41
112.44
CLIP-MoE
64.53 685.78 64.53 / 0.04
60.32
193.51
11..4
CLIP-FMoE 693.83 / 72.59 8.09
14.54
53.24
128.22
Table 5: Efficiency comparison of CLIP (Fine-tuning). Up-Cyeling, CLIP-MoF, and our CLIP- FMoE. Parameters are reported as either (base / trainable) or (base / trainable in Stage 1 trainable Stage 2), in millions (M). FLOPs are measured per sample (forward pass only) GFLOPs. Training times are wall-clock measurements under identical hardware and training settings (total training time including clustering time).
CLIP-FMoE attains favourable pereeromnee trade-off: reduces total training time by 72.5% compared with CLIP-MoE while increasing inference FLOPs by only 14%. Relative to Up-Cycling, CLIP-FMoE requires roughly 80% fewer peak trainable parameters. The resulting reduction in memory pressure and optimizer-state size improves optimization stability, making the modest inference overhead an acceptable compromise in scenarios where training time and GPU memory are the primary constraints.
CONCLUSION
This paper presents CLIP-FMoE, novel CLIP scaling method that integrates Mixture-of- -fepeeetssss architecture into base CLIP models. Our approach distinguishes itself from existing methods by preserving zero-shot capabilities through a Fusion Gate mechanism while ensuring expert special- ization via an Isolated Constrained Contrastive Learning training pipeline. Experimental results demonstrate the effectiveness of CLIP- across diverse settings, including maintaining zero- shot classification performance and improving fine-grained and long-context understanding capa- bilities. However, we have limited understanding of how the semantic data clustering impacts final model performance. Future work could extend CLIP--MMEE to other multimodal alignment scenar- ios such as video-text, audio-text, or multilingual vision- langua-la tasks.
REPRODUCIBILITY STATEMENT
All the models and datasets used in experimental results are publicly available. We have described detailed experimental settings includes the dataset, model settings, and baselines at the beginning of each section. Moreover, provide detailed hyperparameter settings in Table g and 10 in the Appendix. The full code will be released upon acceptance.
REFERENCES
Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In (I7), pp- 370-387, 2024. URL htttttt///oooorrr 10. 1007/978 031 72643-2222 222222992222222
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expertt language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1280-1299 1297. Association for Computational Linguistics, 2024. doi: 10.18653/V A/200/ 70. URL https://doi.or//1.. 8653 htt.:/1....66331111111 accl ong.
10

---

Published as a conference paper at ICLR 2026
Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Al- banie, and Matthias Bethge. How to merge multimodal models over time? In ICLR 2025 Work- shop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025. URL htttps://openeewee
net forrm/orrmmmmm LBBKKTBBR
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xin- long Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 19358-192555 19369, June 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022a. ISSN 1532-4435.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022b. ISSN 1532-4435.
Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan. Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. CoRR, abs/2410.07524, 2024. doi: 1044550/ARXXV2241000772 URL
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. The Eleventh International Confer- ence Learning Representations, 2023. URL https:: httt::://openrevew..eett hwtpleewwwweeeeee..eeeeeee net/forumfifii
6t CKCf8 jrrj
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-88 87, 1991. doi: 11111///nco..991..11.1
Wangyi Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. Improved sparse upcycling for instruction tuning. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI- Al-Khalifa, Bar- bara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguisties, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 9485- 9498. Association for Computational Linguistics, 2025. URL https: /taps///cclattto holgooy. ogy. org/2025. co iing-maiii. 636/
Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang, Wei Wei, Huiwen Zhao, and Zhiwu Lu. Fineclip: Self-distilled region-based clip for better fine-grained understanding. In A. Glober- son, L. Mackey. D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in
Curran Associates, Inc., Neural Information Processing Systems, volume 37, pp. paper fi wttp:fffpeeeeeeae//eeeaae 2024. URL https: / httppps://pocccced neurips.eeppp......
file 21222322a222efffscclll2f55eeeee2 Paw.eseeeeeepppeee-ooeoeeeeeeeee .ddf
Aran Komatsuzaki. Joan Puigcerver, James Lee-Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints, 2023. /B//:tttpe///arxivvorr//aasa
05055
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554- 561. 2013.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuan- han Zhang, Ziwei Liu, and Chumyuan Li. Llava-next: What else influences vi- sual instruction tuning beyond data? https:/ qithub httiauuu...bbbbb/...... ic/blog/ 2024-05-25-1120 stava-nnett ablations. httiiinnss/untnnneeeeme wawtummsuuee-cchaageeee cessed 02-08-2025).
2024. [Ac-
11

---

Published as a conference paper at ICLR 2026
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng, Twin-merging:
Dynamic integration of modular expertise in model merging. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.),
Advances Neural Information Processing Systems 38: Annual Conference on Neural Sormation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024. URL. http: wtwpppaprsssseeeae n nips. wwt.pprss.ipscc/ppppppp......pee cc/paper fi pttple//ppppe////a/eeeeeee 8fcal wet.l17e99bbaeee www.baaeeed222222 776226666666666b6e 799 Abstract
- www.:aactt-ooofennn....o html
Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. Lada: Scalable label-specific clip adapter for continual learning. arXiv preprint arXiv:2505.23327/ 2025.
Jiawei Ma, Po- Yao Huang, Saining Xie, Shang-Wer Wen Li, Luke Zeutlemoyer, Shih-Fu Chang, Wen- Tau Yih, and Hu Xu. Mode: Clip data experts via clustering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26354-26363, June 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-gus Fine-grained visual classification of aircraft, 2013.
Siyuan Mu and Sen Lin. comprehensive survey of mixture-of-experrss: Algorithms, theory, and applications. CoRR, abs/2503.07137,, 2025. doi: 1048550/ARXIV.5500071177 URL https:
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL 3ac728/8aaF0000022e0000000000000
nips
/ttp:e/essssssaaaee///////////// 22ah2/hh/hhh///// html
Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M Asano, Nanne Van Noord, Marcel Worring, and Cees G. M. Snoek. TULIP: Token-length upgraded CLIP. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https: //ppprevii.......//
Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. Drop- upcycling: Training sparse mixture of experts with partial re-initialization. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singupore, April 24-28, 2025. OpenReview.net, 2025. URL httttt::://peeeeeeeeeeee
www. wwwpeeeeeeetoooooggggo...... www.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, 2008
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp. 291-309. Springer, 2024.
Guillermo Ortiz-limenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tan- gent space: Improved editing of pre-trained models. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL http://///ppeneeeee..ee.e. 0A9E212DGW
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar, Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. -505,, 2012.
12

---

Published as a conference paper at ICLR 2026
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region to- to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2641-2649, 2015.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representatiins, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL 33zum?1d=d1522RBxC2
net/
Jingyang Qiao, zhizhong zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xie. Prompt gradient projection for continual learning. In The Twelfth International Confer- ence on Leurning Representations, 2024. URL pestps::/openreweee
net/ / forum? di
EH203h7sBT
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18-24 Jul 2021. html
URL httpp:: httpsss//pocceeddeee ngs mlr wttpppsss/v////affofoo
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe Advancing mixture-of-cxperrs inference and training to power next-generation Al scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Con- ference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol- ume 162 of Proceedings of Machine Learning Research, pp. 18332-18346. PMLR, 2022. URL https: http://ppocee.....ee/e www...ppes./llle. ra httpr/aaaaaaaaaaaaaaaaaaa h
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-54890 5400. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 111(3):211-222, December 2015. ISSN 0920-5691.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2556-2565. Association for Computational Linguistics, 2018 doi: 10.18653/V1/P18. 12288 URL https:tpp//// /aa antaolooogyl www.y..rrg.P.. wtwoooog.............. 123111111111111111
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017 Toulon, France, April 24.26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL net/ tttt://oommmmdmllomaa..o.
Sheng Shen, Le Hou, Yan-Quan Zhou, Nan Du, S. Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Web- son, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-expent meets instruction tuning: A winning combination for large language mod- els. In International Conference on Learning Representations, 2023. URL https: //app. semant cschol ar aww.ct/..rr.oog//oorurustt///sSs
13

---

Published as a conference paper at ICLR 2026
Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip- Scaling clip to 18 billion parameters, 2024. URL aattps://arxiv. org/ abs/
14422.02252
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, CVPR 2024, Seattle, WA, USA, June 16- 22, 2024, pp. 9568-9577 IEEE, 2024, doi: 10. 109C 7/doi. [/doi.org//00 1wt.:11109//222120 2024. 00914
URL https:
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense cap- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26700-26709, 2024.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1807 .07748, 2018. URL http::t/ppaxi...ooot abaaaaass0010 .0374474
Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Lines: Post-training layer scaling prevents forgetting and enhances model merging. In The Thirteenth International Conference on Learming Representations, 2025a. URL https://openrevve wttp://wwwweetetettteerraaoe fote://fooommmmmo/ss/0o
Xinze Wang, Chen Chen, Yinfei Yang, Hong-Your You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, and Xianzhi Du. Clip-up: A simple and efficient mixture-of-expere clip training recipe with sparse upcycling. arXiv preprint arXiv:2502.0099 2025b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingehao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang. Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseck-v12: Mixture-of-experts vision-language models for advanced multimodal understanding. CoRR, abs/2412. 2024. doi: 11.4850/ARXXIV22122100000 URL https: //diis 4222/110..555//rxxx 2412.
10302]
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. pp. 3485-3492, 2010.
Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, and Stephan Alaniz. Flair: sengguageiinfrrmede image representations. In Proceedings of the Computer
Vlm with fine-grained Vision and Pattern Recognition Conference (CVPR), pp. 24884 -4894,, June 2025.
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. FG-CLIP: Fine-grained visual and textual alignment. In Forty-second Interna- tional Conference on Machine Learning, 2025.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixtureoof-expere adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-232.33 2024.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision pp. 310-325. Springer, 2024a.
Jihai Zhang, Xiang Lan, Xiaoye Qu Yu Cheng, Mengling Feng, and Bryan Hooi. Learning the unlearned: Mitigating feature suppression in contrastive learning. In European Conference on Computer Vision, pp. 35-52. Springer, 2024b.
14

---

Published as a conference paper at ICLR 2026
Jihai Zhang, Xiaoye Qu, Tong Zhu, and Yu Cheng. Clip-moe: Towards building mixture of experts for clip with diversified multiplet upeycling. 2024c.
Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen.
Dreamlip: Language-image pre-training with long captions. In ECCV, 2024.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 09125-19136, 2023.
Tong Zhu, Daize Dong, Xiaoye Qu Jiacheng Ruan, Wenliang Chen, and Yu Cheng. Dynamic data mixing maximizes instruction tuning for mixture-of-experts. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1663-1677, Albuquerque, New Mexico, April 2025. Association for Computational doi: 10.18653/v 1/2025. ma225.aaal--oog.800 URL https:
Linguisties. ISBN 979-9-999-918188 18--
//aclaathollooy..crr//255... wwwp:////........aaaaaaa....aa.. -ongilloonn...m 80/
Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Sparse model soups: A recipe for improved pruning via model averaging. The Twelfih International Conference on Learning Representa- lions, 2024. URL https: httpt:::://peenreieeeee wwwpeerwwwwwwwwweeeeeeeeeeeeeeee ner/forum? 1ddxxxxIxTpp33
15

---

Published as a conference paper at ICLR 2026
A
MAIN EXPERIMENTAL RESULTS
Text-to- -Image Generation
There is green shore and lake in the front and then huge background. the big tree, and most what you see the bark of tree, The building reflecting in the water. is mostly cream and brick colored. Around the monsion are green lowns. There are trees the background on the left and night. In the background wide gray sky.
the left
CLIP
TULIP
CLIP FMEE
are two building front. one and has few windows. There are trees in front of it. The ground is poved, and there
There bigger building the
right. It has several doors and windows, It also has rails in front of
OpenAI
Long-G,10
it. is least three stories. In the background that mostly white and red. There tree the very right. The sky is bright blue, and it is daytime.
Fernis
TULIP
CLIP-FMoE
Figure 4: Generated image samples from Stable-Diffusion-Xe conditioned the output of the text encoder. Highlighted texts indicate the aspects that our method successfully captures compared to the baselines.
Model
Cas
Cous
PGVC
Foxal
Sun
Cll
C10
C100
Fum
Flow
Fas
IN-4
IN-6
IN-r
IN-IK
IN-Ske
IN-y2
Arg.
CLIP OpenAl
77.93
31.88
31.73
93.07
67.56
$5.27
95.39
75.85
62.59
79.20
93.21
70.75
32.25
$7.82
75.54
59.58
69.54
69.36
Fine-tuning
69
25.83
67.00
Upeeycling
CLIFMMB
21.20
25.31
79.19
77.00
69
75.19
Table 6: Top-1 accuracy across 17 benchmark datasets (extended version of Table D). Models trained on LLL Re-Rec CC3M with ViT-L/14 backbone (77-token context). Bold denotes the best method out of reported fine-tuning methods
A.1 DATASET
For the main results, we use the LLaVA-Recaa CC3M dataset (Li et al. 2024), which contains approximately 2.8M samples with long captions. For each image, the set of positive captions consists of one raw caption from the original CC3M dataset and four detailed captions.
The detailed captions are generated as follows: we first split each long caption from the LLaVA- Recap CC3M dataset into sentences and compute the token count for each sentence. We then domly combine sentences to construct detailed captions of approximately 77 tokens each. Although initially generate five detailed captions, retain only four to avoid potential sentence duplica- tion the final set. The complete process will be documented in the code repository to released later.
The weights of the loss term y are set to 0.1 for the raw caption and 0.225 for each the four detailed captions, encouraging greater diversity.
16

---

Published as a conference paper at ICLR 2026
Avp
OpenAt
50.25
14.36
28.08
85.00
LengCau
28
30.78
71.54
96,10
79.81
69.09
72.35 39
91
37,00
86.44
06.38
68.1
TULIP
63.25
21.75
19.95
83.46
67.34
83.
96
71.91
55.43
61.64
$7.03
53.25
33.90
81.42
63.48
52.51
57.56
61.87
CLIP-FMeE
75.38
30.17
29.07
92.32
71.54
84.63
95.26
76.02
67.11
76,86
93.08
65.34
35.00
87.27
73.90
58,84
67.72
69.51
Table Top-1 accuracy across 17 common benchmark datasets. Results are obtained using models trained on ShareGPT4V dataset with ViT-L/14 architecture and standard tok-noken context length.
DOCCI
IIW
Urban
DCI
Model
Avg.
IR
TR
IR
TR
IR
TR
IR
TR
OpenAl CLIP
17.1
48.5
35.0
84.3
55.8
68.3
33.9
36.0
47.4
Vanilla Fine-tuning
23.9
53.6
46.8
88.5
87.3
85.9
51.1
49.2
60.8
CLIP Up-cycling
23.9
52.9
46.6
89.0
87.6
85.4
53.1
51.3
61.2
CLIP-MoE
24.1
56.1
47.2
89.8
86.1
86.0
53.5
52.0
61.8
CLIP-FMoE
23.7
58.2
45.4
92.5
85.5
85.7
53.9
53.0
62.2
Table 8: R retrieval results fine-grained (DOCCI, IIW) and long-caption (Urban-1k and DCT) datasets.
A.2 EXPERIMENTAL SETUP
The main experiment uses the OpenAI CLIP ViT-L/14 (Radford et al. 2021) model; we use standard context length 77 tokens. We apply CLIP-FMoE (Zhang al., (202cc) only odd layers in the last half of both encoders to reduce computational requirements and demonstrate the lightweight nature of our method. Detailed hyperparameters provided in Table
A.3 RESULTS
We present extended zero-shot classification results across 17 datasets (compared to the con- densed results in the main paper). These results demonstrate CLIP-FMoE's consistent knowledge- preserving ability across a wide range of benchmark datasets, outperforming existing methods by significant margins.
B LONG CONTEXT UNDERSTANDING
Hyperparameter
Value
Hyperparameter
Value
Global batch size
1024
Global batch size
2048
Epochs (per-expert)
Epochs (per-expert)
Epochs (full MoE)
Epochs (full MoE)
Positive captions (L)
5
Positive captions (L)
2
Warm-up steps
3% total steps
Warm-up steps
3% total steps
Learning rate
5 10-5
Learning rate
x 10-5
Optimizer
AdamW W
Optimizer
AdamW W
31
0.9
B1
0.9
82
0.999
B2
0.999
1 10
E
x 10
Weight decay
0.5
Weight decay
0.5
Precision
AMP mixed
Precision
AMP mixed
Data loader workers
16
Data loader workers
16
Experts per cluster (N)
Experts per cluster (N)
Sub-clusters per cluster (M)
16
Sub-clusters per cluster (M)
16
Table 9: Hyperparameters for CC3M.
Table 10: Hyperparameters for Long Context.
17

---

Published as a conference paper at ICLR 2026
B.1 DATASET
We use the ShareGPT4V dataset (Chen et al., 2024) for this experiment. Following LongCLIP's experimental setting, the positive caption for each image consists of the original long caption and its first sentence. We set the loss weights to 0.1 and 0.9 respectively for these two caption types.
B.2 EXPERIMENTAL SETUP
The long context understanding experiment uses the OpenAl CLIP ViT-L/14 model. Following LongCLIP, extend the context length from 77 248 tokens. We apply CLIP-FMoE: only odd layers in the last half of both encoders to reduce computational requirements and demonstrate the lightweight nature of method. Detailed hyperparameters are provided in Table I0
B.3 RESULTS
7 the zero-shot classification results for each method in the long context understand- ing experiment. These results indicate that other long-context understanding methods also experi- ence decreased generalization performance when trained on new data, such long-caption datasets, despite achieving improvements on benchmarks involving longer input captions. In contrast, CLIP- FMoE performs well in all benchmarks compared TULIP (Najdenkoska al., 2025) and Long- CLIP (Zhang et al., 2024a), which rely more complex training techniques such as coarse-gras coanse-rrained feature and short caption alignment learning or text encoder distillation, while our approach employs straightforward training objective. This suggests that our method has potential for integration with other techniques to enhance CLIP's capabilities.
B.4 STABLE DIFFUSION QUANTITATIVE RESULTS
Following the approach of LongCIIP (Zhang al. 2024a). we integrated our trained CL.IP Text Encoder into Stable Diffusion XI. (Podell et al. 2024) as its text encoding component. The results are presented Figure
The figure demonstrates that CLIP-FMoE exhibits superior performance compared to previous long- context approaches in capturing fine-grained textual details during image generation.
C DETAIL MECHANISM OF CONSTRAINED SAMPLER
In Stage 1, for each cluster C2 containing M sub-clusters (5))) 1-=1- ) we fine-tune only the MLP (FFN) modules selected set of layer indices: A in the image encoder and B in the text encoder.
At each training iteration the Constrained Sampler is responsible for producing a mini-batch size B that contains only samples from single sub-cluster Sy C C₁ described in Zhang al. (2024c). The sampling procedure is implemented as follows:
1. Shuffle each sub-cluster. We permute the indices of every sub-cluster Sj using a fixed seed for reproducibility.
2. Trim to full batches. For each sub-cluster S, with n, = |Sy| samples, keep only the first
n2/ B B. indices from the permuted order. This ensures each retained sub-cluster yields integer number of full-size mini-batches and avoids partial batches.
3. Batch Sampling. Cycle through the trimmed sub-clusters in round-robin order. For each selected sub-cluster, pop samples and append them to the current mini-bar until the batch reaches size B. When full batch is assembled, yield it and proceed to the next sub-cluster. Continue until no sub-cluster can produce another full batch.
D ABLATION STUDY OF CLUSTERING TYPES
To better understand how clustering types affect final performance, we examine several clustering variants beyond our standard image-based clustering: (1) random clustering with only level (2)
18

---

Published as a conference paper at ICLR 2026
MS COCO
Flickr 8K
Flickr 30K
Cluster Type
Image Retrioval
Text Retricval
Image Retrieval
Text Retrieval
Image Retrieval
Test Retrieval
Avg.
R
R@5
Random
40.8
66.0
58.8
$1.8
64.9
88.4
80.1
95.0
69.3
89.3
86.6
97.9
76.5
82.4
88 6
70.8
98.2
77.3
Image + Text
41.1
66.5
59.6
82.3
63.4
88.3
80.6
95.3
70.7
89.9
89.2
98.6
77.3
Image (level 1)
41.3
66.6
59.8
82.8
65.9
88.4
80.7
95.3
70.6
90.3
87.5
98.5
77.3
98.5
77.7
Table 11: Standard text-image retrieval results for clustering method ablation study of CLIP-FMoE. All experiments were conducted with a 248-token context length and ViT-B/16 architecture on ShareGPT4V dataset.
Fine srans-gained Text Image Retrieval
Long Text Imagg Retrieval
DOOCI-FG
IWWFF
DCT
Type
TR
TR
Aug:
R:81
R405
RB1
RA95
R@1
R.05
R:81
R-05
R&1
R005
R:01
R405
R@1
RO95
R@1
RAB5
Bundom
15.7
29.5
47.1
72.9
32.2
52.9
96.0
98.0
$4.5
96.7
$7.0
97.0
50.4
69.2
52.2
71.0
65.3
Ted
16.0
30.4
47.8
74,8
32.6
52.9
$7.5
98.5
56.1
97.7
$9.9
98.6
51.5
78,5
53.3
71.8
66.2
Image + Yeat
15.0
90.0
48.1
71.
12.5
53.0
59.8
08.1
65.5
96.9
88.1
98.1
509
69.
534
71.9
66.9
Image dovel 1)
16.1
30.2
47.4
73.8
32.6
50.2
$8.5
98.0
558
96.1
$7.9
98.5
51.6
69.7
54.0
72.7
66.0
Stndard
16.1
30.3
479
74.1
32.
52.4
38.0
98.0
87.2
90.3
88.2
98.6
52J1
70.3
54A
752
66.3
Table 12: Fine-grained and long-context retrieval results for clustering method ablation study of CLIP-FMoE. All experiments were conducted with a 248-token context length and ViT-B/16 archi- tecture ShareGPT4V dataset.
text-based clustering, (3) multimodal clustering using both image and text features, and (4) image clustering with only level 1. Note that standard, text-based, and multimodal clustering employ the two-level clustering approach described in the main paper. All embedding steps are performed using the same OpenAl CLIP ViT-B/16 model used for training. For multimodal clustering, we use Cartesian product approach. For example, to obtain 4 clusters, features into 2 clusters and separately cluster based on image features into 2 clusters, then use their Cartesian product form 4 multimodal clusters. This is analogous to performing level 2 clustering on each cluster to obtain sub-clusters. For this ablation study, we use OpenAl CLIP ViT-B/16 as the base model, with all other configurations identical to those listed in Table TO
first cluster data based on text
Results from Tables [ and 12 indicate that our standard clustering approach achieves the best per- formance. Random clustering, while being a naive divide-and-conquer approach, cannot ensure high-quality in-batch negatives. Similarly, image clustering with only level may achieve better performance by directing experts learn distinct domains as categorized by the base model, but fails to ensure in in-batch negative quality due to free sampling in the contrastive learning objective. For multimodal and text-based clustering, performance remains worse than image-only clustering. This may be attributed to the limited representation quality of CLIP's text encoder, which was orig- inally trained on web-crawled text captions rather than high-quality corpora used in modern LLMs. In our current work, motivated by enabling experts to learn challenging domains identified by the base model, we utilize the base model itself for clustering. However, alternative embedding models could also be explored for this purpose.
E ANALYSIS ON FINE-TUNING DATASET
In this section, we provide analysis on the choice of fine-tuning dataset. We compare the clas- sification and retrieval performance between fine-tuning on ShareGPT4V (Chen et al.) (024) and LLaVA-Reeap (Li et al. 2024). The results are presented in Tables [3 and 14 The results show that fine-tuning on LLaVA LLaaap-eeap dataset yield better performance in both tasks, showcasing the benefits of scaling the number of data samples, and detailed description.
F ANALYSIS ON THE NUMBER OF CLUSTERS (EXPERTS) AND SUB-CLUSTERS
We conduct experiments to investigate the impact of the number of clusters (experts) N and the number of sub-clusters M. The results presented in Tables 15 and 16 Our analysis reveals that
19

---

Published as a conference paper at ICLR 2026
type
MS
SK
Flickr
DOCCI-oong
IMW-Foog
Ave.
TR
TR
TR
TR
400M OpenAl
56.36
63.31
1.3M
ShareGPTea
07.57
72.08
$4.90
75.96
91.10
78.85
77.61
97.50
98.25
88.00
90.50
54,50
55.43
76.96
2.8M LLaVA Recup
89.50
67.00
74,02
85.39
78.54
93.10
84.92
$1.90
97.75
96.75
93,40
94,00
59.88
58.07
79,57
Table 13: Ablation study of CLIP-FMoE on standard text-image retrieval tasks with varying fine- tuning dataset sizes. Results obtained using the VIT-L/14 architecture with context length of 248.
Data type
Curs
Coun
DGVC
Food
Sin
Cal
C10
C100
Euro
[low
Pets
DN
IN-0
IN-r
IN-IK
IN Ske
IN-2
Avg.
400M OpenAl
77.9
31.9
31.8
93.1
57.6
83.3
95.6
75.0
62.6
79.2
93.2
70.8
323
87.8
75.5
99.6
69.8
69.
+ SiorCPPT4
75.8
30.1
19.8
92.4
55.1
64.4
95.4
78.3
67.2
76.0
93.0
65.1
35.6
87.2
73.9
56.9
678
69.
+
288 LLAVA Recap
74.6
28,0
29.7
92.8
71.9
82.5
95.7
79.0
68.0
74.2
922
66.1
358
87.4
74.7
59.0
68.7
69.4
Table 14: Ablation study of CLIP-FMoE Top-1 zero-shot classification accuracy across 17 bench- mark tasks with varying fine-tuning dataset sizes. Results are reported using the ViT-L/14 architec- ture with context length of 248.
MS COCO
Flickr 8K
Flickr 30K
Urban
DCI
M
N
Avg.
IR
TR
IR
TR
IR
TR
IR
TR
TR
4
41.6
60.2
65.4
80.5
70.9
88.4
87
88
52.2
54.3
68.9
8
8
40.6
59.4
64.8
79.7
70.1
86.9
83.5
86.3
50.8
53.2
67.5
4
42.2
60.3
66
80.9
70.7
88.6
87.2
88.2
54.6
69.1
16
8
40.6
58.8
64.8
79.7
69.8
87.1
82.7
86.5
50.5
52.8
67.3
4
41.9
60.4
66
80.8
70.8
87.8
87.2
89.6
52.1
54.4
69.1
32
8
40.5
59.2
64.7
79.6
69.6
87
83.3
87.5
50.7
52.8
67.5
4
42
60.9
66
81
70.8
88.1
87.7
90.2
52.9
55.1
69.5
64
8
40.4
58.6
64.9
79.4
69.9
87.7
83.4
88.1
50.9
52.5
67.6
Table 15: Hyperparameter analysis on the number of clusters/experts (N) and the number of sub- clusters (M). The setting is CLIP-FMoE on standard text-image retrieval tasks. Results are obtained using the ViT-B/16 architecture with a context length of 248.
IN-x2
Avg.
89.9
76.2
59.1
61.5
22.2
22.5
67.6
82.7
90.8
68.4
57.9
68.8
89.3
41.8
44.3
76.4
66.5
47.6
59.2
62.1
16
61.2
62.2
22
22
22.2
87.2
67.9
67.6
82.8
90.4
90.6
66.6
68.3
55.7
58.4
68.3
69.5
89.3
89.6
4012
41.6
42.6
43.9
76.2
76
66.4
66
47.5
47.5
59.5
58.9
61.6
62.1
61.9
67.4
88.9
44.2
76.4
59.3
62.0
64
61.4
61.5
21.5
21.5
22.3
21.8
87.8
87.1
67.6
67.6
82.8
82.9
90.4
91
67.4
67.9
57.3
56
68.9
67.7
88.6
89.2
40.6
40.8
42.6
43.6
76.2
76.1
63.7
66.2
47.3
47.2
59.6
58.5
61.5
61.7
Table 16: Hyperparameter analysis on the number clusters/experts (N) and the number sub- clusters (M). The setting is CLIP-FMoE zero-shot image classification task. Results are obtained using the ViT-B/16 architecture with a context length of 248.
increasing the number of experts N leads to improved classification performance but diminished performance on retrieval tasks. We conclude that scaling the number of experts introduces a trade- off between classification and retrieval performance. The performance is robust with the choice of number of sub-clusters, with their results differs from each other less than 1.
G EXPERT SPECIALIZATION
To further investigate the expert specialization, conduct additional analysis the MMVP benchmark (Tong et al., 2024) using the same experts (ViT-B/16) as in Table 4 and show that they capture diverse and complementary attributes. MMVP evaluates CLIP model by asking it to select
20

---

Published as a conference paper at ICLR 2026
Attribute
Expert
Expert
Expert 2
Expert
Orientation and Direction
13.33
20
0
6.67
Presence of Specific Features
6,67
20
13.33
6,67
State and Condition
40
40
40
46.67
Quantity and Count
20
13.33
13.33
13.33
Positional and Relational Context
20
6.67
Color and Appearance
53.33
66.67
73.33
66.67
Structural Characteristies
20
26.67
33.33
26.67
Texts
6.67
6.67
13.33
13.33
Viewpoint and Perspective
13.33
6.67
20
6.67
Table 17: Attribute distribution across experts for different characteristics.
the correct image, given a textual statement, from a pair of visually similar images. The evaluation set is carefully categorized by human annotators into nine fine-
fine-grained grained visual attributes.
The results Table 17 indicate clear attribute-level specialization across experts. For example, Expert 0 performs best on Quantity and Count. Expert excels in Orientation and Direction and Presence of Specific Features. Expert focuses on Color and Appearance and Structural Character- istics, while Expert 3 is strongest State and Condition and shares the lead Texts. This pattern supports our claim that semantic clustering induces meaningful routing and complementary feature learning rather than redundant experts.
Dataset
Cars
Food
Sun
Eurosat
Cal
IN-a
IN-o
IN-r
IN-s
IN-1K
IN-v2
FLow
Avg.
Worst among experts
65.69
88.49
64.71
48.83
75.22
54.88
32.2
84.72
55.3
67.34
60.77
65.95
63.68
Best among experts
71.33
91.71
69.44
64.57
82.5
61.55
33.2
86.6
56.91
70.38
64.47
72.03
68.72
Top-2Top2
73.22
91.66
71.19
66.61
81.45
64.11
34.7
87.22
58.61
72.98
66.52
72.63
70.08
MoE-Top-4
73.48
91.85
71.36
67.33
81.74
64.52
35.05
87.33
58.78
73.31
66.95
73.07
70.40
Table 18: Zero-shot classification performance on the 13.6M dataset.
among expens MoE-Top expers
73.6
45.28
73.8
Table 19: Zero-shot retrieval performance the 13.6M dataset.
Table in the main text presents results from the model trained at the smallest scale using the ShareGPT4V dataset with a ViT-B/16 backbone, which may not fully capture expert collaboration. Additionally, due computational constraints, employed only TopK=2 with 4 experts in this experiment. Since each expert specializes different feature domains, the limited TopK and small number of experts (2 and 4) restrict the number of possible collaborations and may lead one expert dominating, which aligns with the reviewer's observation. To better examine expert collabo- ration, decided to scale up the number of experts and train the CLIP ViT-L/14 model on the "Merged Dataset" consisting of ShareGPT4V, ALaVA-ReCap ReCaR-CCaaa... and LLaVA-ReCap-CC 2C (Total: 13.6M images). To optimize computational resources, we TopK=2 during training, and used TopK=2 and TopK=4 during inference. We compare the worst and best performance of in- dividual experts across each dataset with the corresponding performance of the MoE. model. The results presented in the two tables below (top-1 accuracy and retrieval).
These results in Tables 18 and 19 demonstrate that with a larger number total and activated experts, the unified model benefits from more diverse combination of expert knowledge, leading to a clearer gap in performance between the unified model and individual experts in both classification and retrieval tasks. We also observe that using larger TopK during inference performance, even when a smaller TopK is used during training. We hypothesize that training with larger TopK, such TopK=4, could further narrow the gap and lead to even better results.
improve overall
21

---

Published as a conference paper at ICLR 2026
H DETAILS OF LARGE LANGUAGE MODELS USAGE
We employ Large Language Model (LLMs) solely to assist with polishing the writing of this manuscript. The LLMs are used improve grammar, clarity, and style in certain sections, but they do not contribute to the conceptual development, experimental design, data analysis, or inter- pretation of results. All scientific content, ideas, and conclusions are derived entirely by the human authors.
22


Reviews:
[{"review_id": "kgM1o9LUa5", "reviewer": "Reviewer_sfPq", "summary": "This paper introduces CLIP-FMoE, a method for scaling Contrastive Language-Image Pre-training (CLIP) models using a Fused Mixture-of-Experts (MoE) architecture, with the aim to reduce computation cost when scaling. It involves two stages: Isolated Constrained Contrastive Learning (ICCL) where data is semantically clustered for separate experts training & specialization, and the integration of these experts into an MoE layer, by introducing the Fusion Gate mechanism to dynamically blend pre-trained knowledge with new, domain-specific expert knowledge. The method is evaluated on CC3M and achieves better performance consistently across tasks. Though, the method is not evaluated on billion-scale dataset training while the generalization of this method is expected to be further examined if more computational resource is available.", "questions": "I am curious on how your approach generalize when trained on OpenAI CLIP ViT-Base. After all, the visual embedding quality of ViTs with different scale varies, which may hurt the performance.\n\nMeanwhile, I am happy to increase my score further if the authors can address all my questions & weakness part properly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea to use CLIP model to split the data via clustering is interesting. \n- The performance is good and the result is convincing & promising. \n- The fusing mechanism is indeed needed and this paper provides a good methodology to prevent forgetting and enable efficient scaling. \n- Though not clearly stated in the method, but this paper consider shared and specialized parameters, which is good.", "weaknesses": "- **Ablation on Clustering Strategy**: The impact of the specific clustering algorithm and hyperparameters (e.g., choice of N and M clusters) on final performance is not thoroughly explored. It is unclear how sensitive the method is to the quality and granularity of the semantic clusters.\n- **Single-modal clustering**: The paper uses image modality only for the clustering, but more discussion is needed. To fully discover the finegrained visual concept, the same image may need to be compared with different set of negative samples that focus on different aspects, thus it is unclear how the clustering over visual embeddings can help highlight the fine-grained visual elements. Also, as discussed in MoDE, the limited description in text is the primary cause of false negative. As such, more discussion is needed. \n- **Generalization**: The method is only evaluated on small-scale CLIP training, and the generalization over much larger-scale dataset is hard to determine. I completely understand that the computation resources is a primary constraint, but I would still keen to know how your method can generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kCtztNbV8W", "reviewer": "Reviewer_Wka5", "summary": "This paper addresses the limitations of existing MoE adaptations for CLIP (high sequential training overhead and degraded zero-shot capabilities) by proposing CLIP-FMoE, an approach integrating MoE into CLIP fine-tuning via two core designs: Isolated Constrained Contrastive Learning (ICCL), a two-stage pipeline that performs two-level semantic clustering to train specialized experts in parallel and reduce token imbalance in MoE, and a Fusion Gate mechanism to mitigate catastrophic forgetting of pre-trained knowledge ; extensive experiments on benchmarks (11 zero-shot classification datasets, standard/fine-grained/long-context retrieval tasks) show CLIP-FMoE preserves zero-shot capabilities (average accuracy drop <1% vs. original CLIP), outperforms baselines like Vanilla Fine-tuning and CLIP-MoE, and maintains efficiency (72.5% less training time than CLIP-MoE, 14% higher inference FLOPs) ; its main contributions include introducing CLIP-FMoE with ICCL and Fusion Gate, ensuring expert specialization, and demonstrating robust performance across context lengths and dataset sizes .", "questions": "# Questions for the Authors  \nGiven that this paper presents an engineering-sound work on CLIP scaling via CLIP-FMoE, i have two questions:  \n1. What new insights do you believe this work provides to the field of CLIP scaling and MoE adaptation for vision-language models, which were not sufficiently addressed or recognized in prior studies?  \n2. The current framework is tailored for CLIP fine-tuning. Do you think the core design (e.g., ICCL’s two-level clustering, Fusion Gate) can be extended to train non-CLIP models? If so, what key adjustments might be needed; if not, what inherent limitations of the framework restrict such extension?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Clear Motivation**: The paper designs a two-stage training pipeline (Isolated Constrained Contrastive Learning, ICCL) and a Fusion Gate mechanism to address key issues in CLIP scaling: it uses two-level semantic clustering (coarse-grained primary clusters + fine-grained sub-clusters) for parallel expert training in Stage 1, ensuring experts learn non-overlapping domain knowledge and avoiding the accumulated errors of sequential training in prior CLIP-MoE; the Fusion Gate dynamically balances pre-trained CLIP knowledge and expert-specific expertise in both training stages, effectively mitigating catastrophic forgetting.\n\n\n\n2. **Solid Experimental Design**:  The experiments cover diverse task scenarios: 11 zero-shot classification benchmarks (including fine-grained, scene, and general datasets), 6 retrieval tasks (standard MS COCO, fine-grained DOCCI, long-context Urban-1K), and long-context understanding (extending context length from 77 to 248 tokens). All methods use the same CLIP ViT-L/14 backbone and fine-tune only odd layers in the second half of the encoder, ensuring fair comparison detailed hyperparameters (e.g., batch size, learning rate) for different datasets (CC3M, ShareGPT4V) and extended experimental results (e.g., 17 classification benchmarks in the appendix) are provided, enhancing result reproducibility.\n\n 3. **High computational efficiency**: CLIP-FMoE achieves an excellent trade-off between performance and computational cost: compared with CLIP-MoE, it reduces total training time  through parallel expert training; its inference FLOPs only increase by 14%, while the peak trainable parameters are 80% fewer than Up-cycling, reducing GPU memory pressure. Additionally, it maintains robustness across scenarios—performing well in both resource-constrained environments (due to low memory usage) and diverse practical tasks (e.g., fine-grained product recognition, long-text image retrieval), expanding the application scope of CLIP-style models.", "weaknesses": "1. **Incremental Contribution**: CLIP-FMoE’s core components are all combinations of mature existing technologies in the field: the MoE architecture , contrastive learning with InfoNCE loss , K-means semantic clustering, and the Fusion Gate —no breakthrough technical ideas  and insights are proposed. In terms of performance, its improvement is only incremental: in zero-shot classification tasks (Table 1), its average accuracy (68.65) is only 2.18 percentage points higher than the second-best baseline CLIP-MoE (66.47), and even 1.19 percentage points lower than the original CLIP (69.84); in retrieval tasks (Table 2), the average R@1 is only 0.39 percentage points higher than CLIP-MoE, with a tiny improvement margin that fails to open up new technical directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sg0uKRAYxO", "reviewer": "Reviewer_2spG", "summary": "This paper presents CLIP-FMoE, a scalable MoE framework for CLIP to handle diverse visual-text data efficiently. The key idea is a two-stage clustering mechanism that partitions data by visual similarity, enabling each expert to specialize in a subset of the modality space. A lightweight Fusion Gate is introduced to adaptively aggregate expert outputs, balancing specialization and generalization. The experiments demonstrate improvements in zero-shot and retrieval benchmarks over several baselines.", "questions": "1. Could the authors provide details on the sample distribution across the generated data clusters? Specifically, how balanced were the clusters?\n2. In Stage 2, only the router and fusion gate are trained. Have the authors experimented with also fine-tuning the expert weights with a small learning rate? It seems plausible that allowing for minor adjustments during this unification stage could further improve performance.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and the proposed method is presented clearly, making it easy to follow.\n2. Experimental results are consistent across multiple benchmarks, showing solid improvements over vanilla CLIP and other MoE-based baselines.", "weaknesses": "1. The novelty of the fusion gate is somewhat incremental, as gating mechanisms for interpolating between network modules are a common way in deep learning. The paper could strengthen its contribution by better positioning this component with respect to prior art in model merging or dynamic network composition.\n2. The analysis of expert collaboration is limited. The expert specialization results in Table 4 suggest the unified model often matches the performance of the single best expert, rather than demonstrating strong synergistic gains from combining multiple experts.\n3. The sensitivity of balancing loss coefficient alpha in Eq. 5 has not been evaluated. The authors use a fixed value of 0.01, but an ablation study would be necessary to understand its impact and the robustness of the model to this choice.\n4. In the context of preventing catastrophic forgetting, continual learning literature offers several relevant strategies. Besides MoE-based approaches such as MoE-Adapters [1] and L2P [2], there also exist regularization-based approaches e.g. ZSCL [3], orthogonality-based approaches e.g. PGP [4] and feature compression approaches e.g. LADA [5]. These methods should be discussed in the related work section.\n\n\n\n[1] Yu, Jiazuo, et al. \"Boosting continual learning of vision-language models via mixture-of-experts adapters.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2] Wang, Zifeng, et al. \"Learning to prompt for continual learning.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2022.\n\n[3] Zheng, Zangwei, et al. \"Preventing zero-shot transfer degradation in continual learning of vision-language models.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2023.\n\n[4] Qiao, Jingyang, et al. \"Prompt gradient projection for continual learning.\" *The Twelfth International Conference on Learning Representations*. 2024.\n\n[5] Luo, Mao-Lin, et al. \"LADA: Scalable Label-Specific CLIP Adapter for Continual Learning.\" *Forty-second International Conference on Machine Learning*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsBGCTgbFs", "reviewer": "Reviewer_jDHU", "summary": "This paper proposes CLIP-FMoE, a mixture-of-experts scaling method for CLIP models. CLIP-FMoE combines an Isolated Constrained Contrastive Learning stage and a fused MoE training stage to preserve pretrained knowledge. Experiments on zero-shot classification, image-text retrieval, and long-caption benchmarks show modest improvements over existing baselines.\n\n**The assigned paper is outside my area of expertise. My reviews are based on my limited understanding.**", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. This paper addresses an important challenge and is valuable for multimodal foundation model development.\n2. The method is evaluated on diverse benchmarks, including standard and fine-grained classification, retrieval tasks, and long-context understanding.\n3. The paper is clearly written and well-organized.", "weaknesses": "1. It seems like novelty is incremental: Both ICCL and Fusion Gate largely reuse familiar concepts (e.g., semantic clustering of data, constrained batch sampling, and gating-based interpolation).\n2. The authors should provide stronger evidence for expert specialization. The current evaluation of expert behavior relies only on limited task-wise performance comparisons (Table 4) and does not convincingly show that semantic clustering leads to meaningful routing or complementary feature learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)