CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

## Abstract

Abstract In recent years, Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in multimodal intelligence.
However, recent studies have identified that the information loss in the encoding process of CLIP is substantial.
Such deficiency significantly limits the ability of a single CLIP model to handle images rich in visual detail.
In this work, we propose a simple yet effective model-agnostic strategy, Diversified Multiplet Upcycling (DMU) for CLIP.
It integrates multiple CLIP models that capture diversified, complementary information into a Mixture of Experts (MoE) architecture.
Inspired by the recently proposed Multistage Contrastive Learning (MCL), which constructs multiple CLIP models that share the same structure while capturing different complementary information, Diversified Multiplet Upcycling efficiently fine-tunes a series of CLIP models from a dense pre-trained CLIP checkpoint to capture different feature distributions, sharing parameters except for the Feed-Forward Network (FFN). These models are then transformed into a CLIP-MoE with a larger model capacity but minimal computational overhead.
Extensive experiments demonstrate the significant performance of CLIP-MoE across various zero-shot retrieval, zero-shot image classification tasks, and downstream Multimodal Large Language Model (MLLM) benchmarks by serving as a vision encoder. Furthermore, Diversified Multiplet Upcycling enables the conversion of any dense CLIP model into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner without requiring further adaptation in downstream frameworks.
Through Diversified Multiplet Upcycling, we aim to provide valuable insights for future research on developing more efficient and effective multimodal learning systems.
Codes and models are released at: https://github.com/OpenSparseLLMs/CLIP-MoE .

## 1 Introduction

Contrastive Language-Image Pre-training (CLIP) (Radford et al., )
is a strong vision-language foundation model that utilizes large-scale datasets to learn comprehensive visual representations by bridging vision and language via contrastive image-text pre-training.
Beyond traditional tasks like classification, CLIP has been broadly applied in areas such as image (Wang et al., ; Zhang et al., ), audio (Guzhov et al., ), and video (Rasheed et al., ) understanding, cross-modal retrieval (Ma et al., ; Zhao et al., ), multimodal generation (Ramesh et al., ; Xie et al., ), and data filtering (Schuhmann et al., ). Additionally, CLIP serves as the vision encoder for various Multimodal Large Language Models (MLLMs) (Alayrac et al., ; Liu et al., ; ; Chen et al., ; Li et al., ).

However, existing CLIP models still face inherent limitations. Recent studies have highlighted that CLIP often encodes inputs in a very coarse-grained manner, overlooking much useful information (Tang et al., ; Tong et al., ; Bleeker et al., ).
As a result, CLIP frequently produces blind pairs (Tong et al., ), where two semantically different images with similar visual components are encoded into the same representation. This leaves downstream models with insufficient information, especially when CLIP serves as a vision encoder. Such substantial information loss negatively impacts downstream tasks and can confuse downstream models, such as the base LLMs in Multimodal Large Language Models (MLLMs).
To address this issue and enhance CLIP’s ability to encode richer information, efforts have been made to improve the quality of training data and to scale up model size. However, these works typically involve retraining the CLIP model from scratch  (Li et al., ; Ma et al., ; Xu et al., ), which is both resource-intensive and costly.
Additionally, there are attempts to ensemble different types of vision encoders (Tong et al., ; Shi et al., ), which makes the entire model heterogeneous and total parameters grow explosively.

To address the above limitations, we propose a simple yet effective model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP, which leverages the sparsely activated Mixture of Experts (MoE) framework to extend model capacity while fully utilizing off-the-shelf pre-trained dense checkpoints, avoiding the need for training from scratch. MoE has proven effective in scaling large pre-trained models by using fixed activated parameters, enhancing both performance and robustness (Jiang et al., ; Dai et al., ; Chen et al., ).
In Diversified Multiplet Upcycling, we first fine-tune the base dense CLIP model to produce a series of multiplet CLIP models using the recently proposed Multistage Contrastive Learning (MCL) (Zhang et al., ). MCL generates models that encode diversified information through a multistage clustering and fine-tuning process. By multiplet, we refer to CLIP models that share all parameters except for the feed-forward network (FFN) layers during MCL fine-tuning. By diversified, we mean that these models yield a series of FFN experts, each capturing different aspects of the input information, which are then used to initialize a CLIP-MoE model.
Finally, through fine-tuning the router in CLIP-MoE, we ensure the full utilization of all experts, enabling CLIP-MoE to capture richer and more useful information than the base model, while leveraging sparse activation to avoid the explosion of activated parameters.

We demonstrate that using a small high-quality image-caption dataset, our MCL-initialized CLIP-MoE significantly improves CLIP’s performance. Notably, on retrieval tasks, CLIP-MoE outperforms the base OpenAI CLIP model by about 20%, while incurring minimal additional training overhead—less than 2% of the total computational cost of training the base CLIP model from scratch. When serving as a vision encoder for MLLMs, CLIP-MoE also shows substantial improvements in most benchmarks simply by replacing the original vision encoder. Our experiments show that CLIP-MoE not only outperforms other fine-tuning baselines but also surpasses popular MoE-construction methods like Sparse Upcycling (Komatsuzaki et al., ). To the best of our knowledge, this work is the first to introduce sparsely activated MoE into CLIP foundation models, whereas previous methods have focused either on vision representation (Li et al., ) or model-wise ensembling (Ma et al., ).

In summary, the contributions of this work are as follows:
First, we propose a novel method, Diversified Multiplet Upcycling for CLIP, which initializes a CLIP-MoE using FFN experts obtained through multistage fine-tuning, offering a new pathway to effectively scale the CLIP foundation model.
Second, we demonstrate that our model-agnostic Diversified Multiplet Upcycling significantly improves model performance by fully leveraging new high-quality data and pre-trained CLIP checkpoints, while avoiding the high computational costs associated with training from scratch.
Third, we conduct extensive experiments, showing that our upcycled CLIP-MoE achieves significant performance improvements over the original CLIP and other baselines with lower computational costs across various downstream tasks, including classification, retrieval, and serving as a vision encoder for MLLMs.

## 2 Related Works

### 2.1 Contrastive Learning

In contrastive learning, the core objective is to minimize the distance between positives and the anchor while maximizing the distance between negatives and the anchor within the representation space. This objective compels the model to effectively encode sufficient information of the inputs to distinguish anchors from their negatives.

Contrastive learning has become a central technique in self-supervised learning, aiming to learn representations by bringing semantically similar samples closer in the embedding space while pushing dissimilar samples apart (Chen et al., ; He et al., ). This approach has been particularly successful in multimodal settings, where models like Contrastive Language-Image Pre-training (CLIP) (Radford et al., ) have emerged as foundational tools. CLIP aligns visual and textual representations by training on vast datasets of paired images and text, enabling the model to bridge different modalities effectively.

Despite its success, CLIP is not without its limitations. One significant shortcoming is its tendency to encode only coarse-grained visual concepts, which can lead to the loss of fine-grained information that is crucial for certain downstream tasks (Tang et al., ; Tong et al., ). To address these limitations, recent works mainly focus on improving the quality of training data (Li et al., ; Ma et al., ; Xu et al., ; Zhang et al., ). However, most of these approaches require retraining the model from scratch, which is computationally expensive, time-consuming, and not easily extendable when better data becomes available.

### 2.2 Mixture-of-Experts

The Mixture-of-Experts (MoE) architecture could scale the model capacity without additional computational cost (Fedus et al., ).
For each input token, only top-$k$ best experts are selected to obtain an aggregated representation (Shazeer et al., ).
This sparsity allows MoE models to scale to trillions of parameters while maintaining the computational efficiency (Lepikhin et al., ; Fedus et al., ).
Due to the large model capacity, the performance could be improved by large margins (Rajbhandari et al., ; Dai et al., ).
Besides, specialized experts in MoE models are good at handling a wide range of tasks (Shen et al., ; Zhu et al., ; Lu et al., ) with high robustness (Chen et al., ).

However, one challenge in MoE training is expert initialization.
Sparse Upcycling (Komatsuzaki et al., ) has been proposed as a technique to initialize MoE models by copying Feed-Forward Networks (FFN) from dense models as multiple experts.
It selectively activates and fine-tunes only a sparse subset of parameters.
This method significantly reduces the training cost.

In this work, we explore the integration of Multistage Contrastive Learning (MCL) with the MoE architecture.
By using MCL to initialize the experts, we aim to capture complementary information across different CLIP experts, which can then be leveraged by the MoE structure to enhance overall performance with minimal additional computational cost.

## 3 Preliminaries

### 3.1 Multistage Contrastive Learning (MCL)

Multistage Contrastive Learning (MCL) (Zhang et al., ) is designed to obtain a series of contrastive models, each capturing different and complementary information from the input data through multiple cluster-and-contrastive processes.
Specifically, at each stage, the learned representations are clustered. In the following stage, for each anchor, negative samples are drawn only from the same accumulated cluster from the previous stages. In this way, the model learns new information beyond what was captured in earlier stages.
For example, consider a dataset containing objects with varying shapes, colors, and textures. In the first stage, the contrastive model might focus on learning color information. After clustering, samples within the same cluster will share the same color. In the second stage, since the anchor and its negative samples share the same color, the model is compelled to learn other features, such as texture, to differentiate between them. After clustering in the second stage, samples in the same accumulated cluster will now share both color and texture. Consequently, in the third stage, the model must focus on other attributes, such as shape, to distinguish between samples. After three stages, we obtain three contrastive models, each encoding distinct information: color, texture, and shape.

Formally, let $\bm{X}=\{\mathbf{x}_{i}\}_{i=1}^{M}$ represent a dataset. After training the encoder in the first stage, we obtain encoded representations $\bm{Z}_{0}=\{f_{0}(\mathbf{x}_{i})\}_{i=1}^{M}$. By clustering $\bm{Z}_{0}$, we obtain cluster assignments $\bm{Y}_{0}=\{\mathbf{y}_{(i,0)}\}_{i=1}^{M}$. In the $j^{th}$ stage, after the cluster-and-contrastive process, each sample $\mathbf{x}_{i}$ is assigned to an accumulated cluster $\hat{\mathbf{y}}_{(i,j)}=[\mathbf{y}_{(i,0)},\cdots,\mathbf{y}_{(i,j-1)}]$. The objective at the $j^{th}$ stage is:

$$ $\displaystyle\mathcal{L}=\mathbb{E}_{\mathbf{x},\mathbf{x}^{+},\{\mathbf{x}_{i }^{-}|\hat{\mathbf{y}}_{j}=\hat{\mathbf{y}}_{(i,j)}^{-}\}_{i=1}^{m}}\left[- \log\frac{e^{s(\mathbf{z},\mathbf{z}^{+})/\tau}}{e^{s(\mathbf{z},\mathbf{z}^{+ })/\tau}+\sum_{i=1}^{m}e^{s(\mathbf{z},\mathbf{z}^{-}_{i})/\tau}}\right],$ (1) $$

where $\hat{\mathbf{y}}_{j}$ represents the accumulated cluster assignment of the anchor $\mathbf{x}$ at the $j^{th}$ stage; $\hat{\mathbf{y}}_{(i,j)}^{-}$
denotes the accumulated cluster assignment of the negative sample $\mathbf{x}_{i}^{-}$ at the $j^{th}$ stage; and $s(\cdot,\cdot)$ denotes cosine similarity. In our proposed Diversified Multiplet Upcycling, we leverage the MCL framework to fine-tune a base model and extract a series of experts for the MoE, whereas the original MCL results in a series of standalone CLIP models.

### 3.2 Mixture of Experts (MoE)

Mixture of Experts (MoE) is an efficient architecture designed to scale large models by dynamically routing inputs through a subset of specialized sub-models, or “experts”. This structure allows the model to maintain high overall capacity while only utilizing a fraction of its parameters for any given input, thereby optimizing both computational efficiency and performance.

In the context of Transformer, an MoE layer (Jiang et al., ) typically replaces the standard feed-forward network (FFN) with a set $\{E_{i}\}_{i=1}^{N}$ of $N$ experts, each of which is an independent FFN. Given an input token representation $\mathbf{x}$, it
first passes through a gating network $\mathbf{W}_{r}$ to obtain the logits corresponding to each expert, then the largest Top-K experts will be chosen, and finally, the probabilities of these selected experts are normalized using Softmax. In this way, we can obtain the probability $R(\mathbf{x})$ of selected experts among all $N$ experts. Notably, the probability of non-

$$ $\displaystyle\mathbf{x}_{\text{out}}=\sum_{i=1}^{N}R(\mathbf{x})_{i}\cdot E_{i }(\mathbf{x}),\quad R(\mathbf{x})=\text{Softmax}(\text{TopK}(\mathbf{x}\cdot \mathbf{W}_{r})),$ (2) $$

where $R(\mathbf{x})_{i}$ denotes the $i$-th routing weight vector produced by the router network $\mathbf{W}_{r}$.

To ensure that all experts are utilized effectively and prevent the model from overfitting to a small subset of experts, a load balancing loss (Fedus et al., ) is often added to the primary loss function. This loss penalizes imbalanced expert usage by encouraging a more uniform distribution of the input tokens across all experts.

## 4 Diversified Multiplet Upcycling for CLIP

Figure: Figure 1: Overview of Diversified Multiplet Upcycling: Our approach involves three key steps. (a) Fine-tuning the base CLIP model using the MCL framework while freezing all parameters except for the FFN layers. This process yields a new set of FFN layers at each stage of MCL. (b) Using the obtained FFN layers as experts to initialize a CLIP-MoE. (c) Continuously fine-tuning the CLIP-MoE using both contrastive learning loss and a router balancing loss to optimize the routers. The terms ‘color’, ‘shape’, and ‘texture’ are metaphorical representations of abstract features.
Refer to caption: x1.png

### 4.1 Expert Extraction

We begin by extracting a series of Feed-Forward Network (FFN) layers utilizing Multistage Contrastive Learning (MCL) to fine-tune a pre-trained base CLIP model for multiple stages. During fine-tuning, we freeze all parameters of the base CLIP model except for the FFN layers within each transformer block in both the image and text encoders. Because the distributions of contrastive negative samples in different MCL stages are distinct, the FFN layers at each stage will learn diversified and complementary information distinct from previous stages.
For clarity, we use superscripts to index the transformer blocks and subscripts to index the MCL stages or MoE experts. Suppose we are fine-tuning a transformer-based CLIP model, where the image encoder contains $A$ transformer blocks and the text encoder contains $B$ transformer blocks.
The FFN layers in the original base model are denoted as $\{E_{0}^{(i)}\}_{i=1}^{A+B}$. As illustrated in Figure 1, the base model might initially focus on color-related information. During MCL Stage 1, only the FFN layers are fine-tuned. After the cluster-and-contrast process in MCL, the FFN layers $\{E_{1}^{(i)}\}_{i=1}^{A+B}$ in the fine-tuned model learn new information beyond color, such as texture. In MCL Stage 2, the model further fine-tunes the FFN layers, resulting in $\{E_{2}^{(i)}\}_{i=1}^{A+B}$, which now encodes additional features such as shape. Through two stages of MCL, we obtain FFN layers where $\{E_{0}^{(i)}\}_{i=1}^{A+B}$ focus on color, $\{E_{1}^{(i)}\}_{i=1}^{A+B}$ on texture, and $\{E_{2}^{(i)}\}_{i=1}^{A+B}$ on shape.

### 4.2 Initialization of Mixture of Experts

Once a series of FFN layers $\{E_{j}^{(i)}\}_{j=0}^{N}$ have been obtained through $N$ stages of MCL, we utilize these FFNs as the experts in a Mixture of Experts (MoE) model, as depicted in Figure 1. According to Equation 2, in the $i^{th}$ transformer block of the base CLIP model, the original FFN layer is replaced with a randomly initialized router and a set of experts:

$$ $\displaystyle\mathbf{x}_{\text{out}}^{(i)}=\sum_{j=0}^{N}R^{(i)}(\mathbf{x}^{( i)})_{j}\cdot E^{(i)}_{j}(\mathbf{x}^{(i)}),\quad R^{(i)}(\mathbf{x}^{(i)})= \text{Softmax}(\text{TopK}(\mathbf{x}^{(i)}\cdot\mathbf{W}^{(i)}_{r})),$ (3) $$

where $R^{(i)}(\mathbf{x})_{j}$ denotes the $j$-th component of the routing weight vector produced by the router network $\mathbf{W}^{(i)}_{r}$ in the $i^{th}$ transformer block. This setup results in a CLIP-MoE model where different experts within different transformer blocks specialize in distinct aspects of the input.

### 4.3 Continuous Fine-Tuning of CLIP-MoE

To enable the model to learn optimal routing strategies while preserving the information learned by the FFN layers during MCL, we further fine-tune the routers while freezing all other parameters. We apply the standard contrastive learning loss while incorporating an auxiliary load balancing loss, following the approach from Fedus et al. (), to encourage a balanced load across experts.
Given $N+1$ experts indexed by $j=0$ to $N$, and a batch $\mathcal{B}$ with $T$ tokens, the load balancing loss for the $i^{th}$ transformer block is defined as:

$$ $\displaystyle\mathcal{L}_{balance}=N\cdot\sum^{N}_{j=0}f_{j}\cdot P_{j},\ \ f_ {j}=\frac{1}{T}\sum_{x\in\mathcal{B}}\mathbbm{1}\{\operatorname{argmax}p(x)=j \},\ \ P_{j}=\frac{1}{T}\sum_{x\in\mathcal{B}}p_{j}(x),$ (4) $$

where $f_{j}$ is the fraction of tokens assigned to expert $j$, and $p(x)$ is the logits output from the router network; $P_{j}$ represents the fraction of router probability allocated to expert $j$, which is the mean of $p_{j}(x)$, the probability of routing token $x$ to expert $j$. For simplicity, we omit the transformer block index $i$ in the equation. Since $f_{j}$ and $P_{j}$ are positive and both their sums are equal to $1$, $\mathcal{L}_{balancing}$ is minimized if and only if $f_{j}=\frac{1}{T},\ P_{i}=\frac{1}{T}$. This balancing loss encourages not only a uniform distribution of actual tokens routed to each expert (i.e., ensuring that all experts have equal importance), but also a uniform distribution of router confidence across tokens (i.e., preventing the router from being overly confident for some tokens and underconfident for others). With this auxiliary load balancing loss, the total loss is given by:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{CLIP}+\alpha\cdot\frac{1}{A+B}\sum_{i=1} ^{A+B}\mathcal{L}^{(i)}_{balance}.$ (5) $$

Following Fedus et al. (), we set $\alpha=0.01$ by default. By applying MoE-Packing to CLIP, we obtain a CLIP-MoE model that is capable of capturing more useful information than the base model, with minimal computational overhead, resulting in a robust and efficient enhancement of CLIP.

## 5 Experiments

### 5.1 Datasets

To fully showcase the potential of our MCL-initialized CLIP-MoE, we implement our experiments on the following two image-caption datasets respectively.

Recap-DataComp.
Recap-DataComp-1B (Li et al., ) is a large-scale dataset comprising 1.3 billion high-quality image-caption pairs. This dataset is derived from the original DataComp-1B dataset, with all images re-captioned using a fine-tuned LLaVA-1.5 model powered by LLaMA-3 (Dubey et al., ). Li et al. () utilized this dataset to train CLIP models from scratch, resulting in significant improvements in retrieval performance. Due to computational constraints, our experiments use a randomly sampled subset of 1 million pairs from Recap-DataComp-1B, referred to as Recap-DataComp-1M, to demonstrate the data efficiency of our proposed pipeline.

ShareGPT4V.
ShareGPT4V (Chen et al., ) is a high-quality image-text dataset containing 1.2 million highly descriptive captions. The captions are generated by a Multimodal Large Language Model (MLLM) fine-tuned on 100k image-text pairs produced by GPT4V, resulting in well-aligned image-text pairs.

### 5.2 Baselines

Direct Fine-tuning.
As our experiments incorporate additional data, we use direct fine-tuning as a basic baseline to evaluate the performance contributions from the additional data.

Sparse Upcycling.
Sparse Upcycling (Komatsuzaki et al., ) is a widely adopted method for initializing a Mixture of Experts (MoE) model using a pre-trained dense checkpoint. It is a simple yet effective approach for scaling up a pre-trained model and is much more efficient than training an MoE from scratch.

Long-CLIP.
Long-CLIP (Zhang et al., ) introduces an efficient pipeline to enhance CLIP performance through fine-tuning on high-quality image-caption datasets with long captions. It aligns the long caption of an image with the encoded image features and the short caption with the primary components of the image features.
While effective on the ShareGPT4V dataset, Long-CLIP is limited to datasets with a similar structure, where each image has both one short and one long caption. Moreover, it requires significantly more computational resources compared to our approach.

LLaVA-1.5.
LLaVA-1.5 (Liu et al., ) is an improved version of LLaVA (Liu et al., ), commonly used as a baseline for MLLMs. It bridges a pre-trained CLIP vision encoder with a pre-trained LLM using a simple MLP, enabling the LLM to gain visual understanding with minimal fine-tuning on image-text pairs. We evaluate the representation quality of our CLIP-MoE by replacing the vision encoder in the original LLaVA-1.5 with our CLIP-MoE and fine-tuning it following the same pipeline as LLaVA-1.5.

### 5.3 Training Setup

By default, we use OpenAI CLIP-ViT-L/14 (Radford et al., ) as the base model for our Diversified Multiplet Upcycling approach. During the clustering process at each stage of MCL, we cluster the image features into 3 clusters and the text features into 3 clusters, resulting in 9 clusters per stage (the Cartesian product of the image and text feature clusters). To accommodate longer text inputs, we interpolate the positional embeddings following the approach in (Zhang et al., ). The global batch size is maintained at 800 unless otherwise specified. To balance performance and computational cost, we set the number of experts to 4 and use top-2 activation.

### 5.4 Training Cost

We use 8 A100 GPUs for training. To train the CLIP-MoE model with four experts, we introduce three additional MCL fine-tuning stages, each trained for 1 epoch. When using the ShareGPT4V dataset, each MCL stage takes approximately 0.5 hours, and the router fine-tuning stage also takes about 0.5 hours. In total, the training time is less than 2.5 hours. In comparison, Long-CLIP training under the same conditions takes around 6 hours, making our approach significantly more efficient. Our maximum GPU memory usage is 8×65955MB, which is comparable to Long-CLIP’s 8×63581MB. When training on the Recap-DataComp-1M dataset, the training cost is even lower. During inference, with top-2 activation, the activated parameter size of our CLIP-MoE is approximately 1.7 times that of the base model (OpenAI CLIP-ViT-L/14).

### 5.5 Evaluation

We begin by evaluating the performance of CLIP-MoE on Zero-Shot Image-Text Retrieval, a key task for assessing whether the CLIP model can capture rich fine-grained information, following Zhang et al. (). All baselines are trained and compared using the Recap-DataComp-1M (Recap-DC) and ShareGPT4V (ShareGPT) datasets, with the exception of Long-CLIP. Long-CLIP is incompatible with the Recap-DataComp dataset, as it requires both a short and long caption for each image, whereas Recap-DataComp provides only one caption per image.
Next, we assess the effectiveness of CLIP-MoE as a vision encoder within LLaVA-1.5, a representative Multimodal Large Language Model (MLLM). LLaVA-1.5 serves as an effective visual representation evaluator, helping to mitigate potential biases present in traditional evaluation tasks (Tong et al., ). Finally, we test CLIP-MoE on traditional Zero-Shot Image Classification tasks, which rely more on coarse-grained features.

Zero-Shot Image-Text Retrieval.

**Table 1: Performance comparison on image-to-text (I2T) and text-to-image (T2I) retrieval tasks using the COCO and Flickr30k datasets. The models were trained and evaluated on the Recap-DataComp-1M (Recap-DC) and ShareGPT4V (ShareGPT) datasets, respectively. The best performance for each dataset is highlighted in bold. Our proposed CLIP-MoE consistently outperforms all baselines across all tasks.**
|  |  | COCO I2T | COCO T2I | Flickr I2T | Flickr T2I |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Model | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 |
|  | OpenAI | 56.1 | 79.5 | 86.8 | 35.4 | 60.1 | 70.2 | 48.5 | 72.6 | 80.8 | 28.0 | 49.3 | 58.7 |
| Recap-DC | Direct FT | 58.9 | 81.5 | 88.5 | 44.3 | 69.5 | 78.8 | 41.6 | 66.5 | 76.1 | 37.2 | 60.4 | 69.5 |
| Upcycling | 59.2 | 81.7 | 88.7 | 45.8 | 70.9 | 79.9 | 42.1 | 67.3 | 77.0 | 39.4 | 62.9 | 71.7 |  |
| CLIP-MoE | 64.0 | 85.1 | 90.8 | 45.2 | 70.2 | 79.4 | 56.8 | 80.1 | 87.0 | 40.8 | 63.8 | 72.5 |  |
| ShareGPT | Direct FT | 63.3 | 84.9 | 91.0 | 44.5 | 70.0 | 78.9 | 50.5 | 74.4 | 82.3 | 38.5 | 61.3 | 69.9 |
| Upcycling | 62.9 | 84.6 | 90.8 | 45.2 | 70.6 | 79.6 | 49.6 | 73.8 | 82.1 | 39.5 | 62.4 | 71.1 |  |
| Long-CLIP | 62.8 | 85.1 | 91.2 | 46.3 | 70.8 | 79.8 | 53.4 | 77.5 | 85.3 | 41.2 | 64.1 | 72.6 |  |
|  | CLIP-MoE | 65.0 | 86.0 | 92.0 | 46.8 | 71.7 | 80.4 | 60.5 | 82.3 | 88.8 | 42.1 | 64.7 | 73.2 |

Following the methodology outlined in Zhang et al. (), we evaluate text-to-image (T2I) and image-to-text (I2T) retrieval on the 5k COCO validation set (Lin et al., ) and the 30k Flickr30k (Young et al., ) dataset. The results are presented in Table 1. Given that both Recap-DataComp-1M and ShareGPT4V datasets offer higher caption quality and longer average caption lengths compared to web datasets, Direct Fine-Tuning, Sparse Upcycling, and CLIP-MoE demonstrate superior performance over the original OpenAI model across most tasks, including COCO I2T, COCO T2I, and Flickr T2I. However, for Flickr I2T, Sparse Upcycling, and Direct Fine-Tuning show significant performance degradation on the Recap-DC dataset. In this fine-tuning context, Sparse Upcycling only provides a limited advantage over Direct Fine-Tuning. Although Long-CLIP clearly outperforms both Direct Fine-Tuning and Sparse Upcycling, it is incompatible with the Recap-DataComp dataset, because it requires each image to have both a short and a long caption. In contrast, our proposed CLIP-MoE surpasses all baselines on both Recap-DataComp and ShareGPT4V, maintaining consistent performance by leveraging the diverse information extracted by MoE experts initialized through different stages of MCL.

Performance in LLaVA-1.5

We further evaluate CLIP-MoE as the vision encoder within the LLaVA-1.5 model. The original vision encoder for LLaVA-1.5 is OpenAI’s CLIP-ViT-L/14@336px (Radford et al., ), which is trained on images with a resolution of 336x336 pixels. To ensure a fair comparison, we use OpenAI’s CLIP-ViT-L/14@336px as the base model for MCL and train our CLIP-MoE on the ShareGPT4V dataset at the same 336x336 resolution. After obtaining CLIP-MoE, we freeze it as the vision encoder and follow the same two-stage training procedure as LLaVA-1.5, using Vicuna-7B (Chiang et al., ) as the base LLM for CLIP-MoE-LLaVA1.5-7B and Vicuna-13B (Chiang et al., ) as the base LLM for CLIP-MoE-LLaVA1.5-13B. The evaluation results, shown in Table 2, demonstrate that by simply replacing the vision encoder with our CLIP-MoE, the final MLLM achieves significant performance improvements across most downstream tasks. This supports the conclusion that our CLIP-MoE is capable of extracting more useful information from image inputs and encoding higher-quality image representations.

**Table 2: Performance comparison between OpenAI CLIP and CLIP-MoE as vision encoders in LLaVA1.5. The best performance for each dataset is highlighted in bold.**
| Method | MME | VQAv2 | TextVQA | POPE | MMBench |
| --- | --- | --- | --- | --- | --- |
| LLaVA1.5-7B | 1510.7 | 78.5 | 58.2 | 85.9 | 64.3 |
| CLIP-MoE-LLaVA1.5-7B | 1486.2 | 79.2 | 58.8 | 86.4 | 66.1 |
| LLaVA1.5-13B | 1531.3 | 80.0 | 61.3 | 85.9 | 67.7 |
| CLIP-MoE-LLaVA1.5-13B | 1593.7 | 80.0 | 60.9 | 86.3 | 69.1 |

Zero-Shot Image Classification.

**Table 3: Performance comparison on zero-shot image classification. The models were trained and evaluated on the Recap-DataComp-1M (Recap-DC) and ShareGPT4V (ShareGPT) datasets, respectively. The best performance for each dataset is highlighted in bold. CLIP-MoE achieved the highest average performance across both Recap-DC and ShareGPT.**
| Dataset | Model | ImageNet | ImageNet-O | ImageNet-V2 | Cifar10 | Cifar100 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | OpenAI | 75.5 | 31.9 | 69.9 | 95.4 | 76.8 | 69.9 |
| Recap-DC | Direct FT | 57.0 | 32.8 | 51.3 | 91.6 | 68.7 | 60.3 |
| Upcycling | 61.1 | 32.3 | 55.3 | 93.6 | 71.0 | 62.7 |  |
| CLIP-MoE | 74.3 | 32.2 | 68.7 | 95.5 | 79.3 | 70.0 |  |
| ShareGPT | Direct FT | 59.8 | 34.5 | 53.3 | 87.8 | 63.1 | 59.7 |
| Upcycling | 62.5 | 34.4 | 56.5 | 91.3 | 67.5 | 62.5 |  |
| Long-CLIP | 73.5 | 33.7 | 67.9 | 95.3 | 78.5 | 69.8 |  |
|  | CLIP-MoE | 74.6 | 33.5 | 68.5 | 95.7 | 79.6 | 70.4 |

We evaluated the zero-shot image classification accuracy on ImageNet (Deng et al., ), ImageNet-O (Hendrycks et al., ), ImageNet-V2 (Recht et al., ), CIFAR-10 (Krizhevsky et al., ), and CIFAR-100 (Krizhevsky et al., ). The results are shown in Table 3. Both Direct Fine-Tuning and Sparse Upcycling exhibited significant performance degradation across most classification tasks, which is consistent with the observations in Zhang et al. (). This decline in performance may be attributed to model overfitting, as both the Recap-DataComp and ShareGPT4V datasets contain approximately 1 million samples, a substantially smaller dataset compared to the 400M samples used for training OpenAI’s CLIP. While Direct Fine-Tuning and Sparse Upcycling successfully learned more fine-grained information from the improved and lengthier image captions, leading to enhanced retrieval performance, they also lost the original model’s ability to encode more coarse-grained information, resulting in decreased classification accuracy. In contrast, our proposed CLIP-MoE demonstrated a superior ability to preserve classification performance compared to Long-CLIP and even surpassed the original OpenAI CLIP on ImageNet-O, CIFAR-10, and CIFAR-100. Additionally, CLIP-MoE achieved the best average performance when trained on both Recap-DC and ShareGPT datasets.

### 5.6 Discussion

Ablation Study on MCL

To further validate the effectiveness of expert extraction utilizing MCL in Diversified Multiplet Upcycling, we conducted an ablation study on ShareGPT4V by training a CLIP-MoE model with only two experts: one from the original OpenAI CLIP and one from fine-tuning FFN layers on ShareGPT4V. As seen in Table 4, the performance of CLIP-MoE on the retrieval tasks is consistently higher than the model without MCL stages 1 and 2, demonstrating that more MCL stages do obtain experts that capture more useful information. The slight degradation in ImageNet zero-shot classification performance is expected, as not all of the additional learned information is beneficial for classification, which tends to rely on more coarse-grained features (Zhang et al., ).

**Table 4: Ablation study on the impact of MCL stages 1 and 2 in CLIP-MoE performance.**
|  | ImageNet | COCO I2T | COCO T2I | Flickr I2T | Flickr T2I |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Top-1 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 |
| w/o S1 S2 | 75.4 | 62.6 | 84.2 | 90.3 | 43.4 | 68.3 | 77.8 | 56.4 | 79.3 | 86.3 | 37.6 | 60.3 | 69.3 |
| CLIP-MoE | 74.6 | 65.0 | 86.0 | 92.0 | 46.8 | 71.7 | 80.4 | 60.5 | 82.3 | 88.8 | 42.1 | 64.7 | 73.2 |

Computation and Data Efficiency
We compare the performance gains of our CLIP-MoE, trained on a 1M randomly sampled subset of Recap-DataComp-1B, to the CLIP-ViT-L-16-HTxt-Recap (Li et al., ), which was trained from scratch on the entire Recap-DataComp-1B dataset. The activated parameter size of our CLIP-MoE, with 4 experts and top-2 routing, is 0.69B, which is comparable to the 0.64B parameter size of CLIP-ViT-L-16-HTxt-Recap. Thanks to MoE-Packing and leveraging the OpenAI CLIP dense checkpoint, our total training computation cost is less than $2\%$ of that for CLIP-ViT-L-16-HTxt-Recap. As shown in Table 5, CLIP-MoE demonstrates comparable performance gains on retrieval tasks relative to CLIP-Recap, with even superior text-to-image retrieval performance on the Flickr30k dataset, highlighting the efficiency of our proposed MoE-Packing for CLIP. It is worth noting that CLIP-Recap uses an even larger text encoder.

**Table 5: The performance gain of CLIP-MoE and CLIP-ViT-L-16-HTxt-Recap compared to the OpenAI CLIP-ViT-L-14 on the retrieval tasks**
|  | COCO I2T | COCO T2I | Flickr I2T | Flickr T2I |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 |
| CLIP-MoE | +7.9 | +5.6 | +4.0 | +9.8 | +10.1 | +9.2 | +8.3 | +7.5 | +6.2 | +12.8 | +14.5 | +13.8 |
| CLIP-Recap | +10.8 | +7.7 | +5.5 | +12.3 | +12.3 | +10.7 | +10.9 | +8.3 | +6.8 | +11.9 | +12.9 | +11.9 |

Figure: Figure 2: Proportion of tokens assigned to each expert on the COCO and ImageNet validation dataset. Here, we consider experts that are either selected as a first or second choice by the router.
Refer to caption: x2.png

Routing analysis
To evaluate whether all the experts learned through MCL are utilized by CLIP-MoE, we perform an analysis of the routing strategy. We use the CLIP-MoE model with 4 experts and top-2 routing trained on ShareGPT4V, and compute the proportion of tokens assigned to each expert. For retrieval tasks, we use the COCO validation dataset, and for zero-shot image classification, we use the ImageNet validation dataset. The analysis results are presented in Table 2. From the results, we observe that for experts from each MCL stage (represented by each column in the heatmap), there are consistently yellow areas (indicating heavily utilized experts). No column is entirely dark blue, which indicates that all MCL stages contribute useful experts to CLIP-MoE. This further validates the effectiveness of our MCL initialization in MoE-Packing.

Case Study

Figure: Figure 3: Example cases comparing the performance of CLIP-MoE and OpenAI CLIP on the MMVP-VLM Benchmark, illustrating differences in their ability to capture fine-grained semantic information.
Refer to caption: x3.png

We demonstrate the comparison between CLIP-MoE and OpenAI CLIP on samples from the MMVP-VLM Benchmark (Tong et al., ). MMVP-VLM contains manually filtered image pairs with different semantics that are difficult to distinguish using the vanilla OpenAI CLIP. We task the models with matching the corresponding statement to the image. As shown in Figure 3, OpenAI CLIP struggles to distinguish fine-grained details in these image pairs. In cases like the alarm clock, OpenAI CLIP matches both images to the statement “hour hand points at 10.” In other cases, such as the rabbit pair, OpenAI CLIP completely misinterprets the information and matches the opposite statement to the images. However, CLIP-MoE captures more fine-grained details and makes the correct match in most cases. It can accurately capture camera perspectives, as seen in the coffee example, orientation information in the rabbit example, and it demonstrates a superior ability to distinguish relations between objects, such as differentiating between “animal inside the basket” and “animal outside the basket.”

## 6 Conclusion & Future Work

In this paper, we proposed a novel Diversified Multiplet Upcycling for CLIP to enhance the model with minimal computational overhead. Our method enables the extraction of diversified and complementary experts across multiple fine-tuning stages, which are then utilized within the MoE framework to capture richer information from the inputs. This approach is straightforward to apply, model-agnostic, and provides a new path to scale and improve CLIP foundation models. By leveraging off-the-shelf CLIP checkpoints and newly constructed high-quality image-text datasets, our method avoids the costly process of training CLIP models from scratch. For future work, our current experiments are limited to image and text modalities. We plan to extend our method to additional modalities, such as audio and video. Beyond the fine-tuning settings explored in this paper, we aim to experiment with larger datasets and test large-scale continuous training settings to further explore the boundaries of Diversified Multiplet Upcycling. Additionally, while we tested CLIP-MoE as a vision encoder for MLLMs, we will also investigate its potential as a text encoder in generative tasks, such as in stable diffusion.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: please see weaknesses
soundness: 3
strengths: The experiments conducted in this study cover a wide range of tasks including zero-shot retrieval, classification, and MLLM understanding. By comparing against the original OpenAI CLIP large model, LongCLIP, and Llava-1.5, the results demonstrate superior performance of the proposed method. Additionally, this work shows that the Diversified Multiplet Upcycling (DMU) is more efficient compared to simply upcycling the FFN to a MoE.
summary: This work introduces a novel method, Diversified Multiplet Upcycling (DMU), for constructing Mixture of Experts (MoE) on CLIP models. The study showcases the effectiveness of DMU by presenting results on zero-shot image retrieval benchmarks, zero-shot image classification benchmarks, and MLLM understanding benchmarks by integrating it as the vision encoder in Llava-1.5. The findings demonstrate that DMU is indeed effective in enhancing the performance of CLIP models.
weaknesses: 1. The zero-shot image classification results should be more diverse. Only including ImageNet, ImageNet-O, ImageNet-V2, CIFAR-10, and CIFAR-100 is not sufficient. Please refer to the CLIP benchmark [1]. I believe that including ImageNet, ImageNetV2, ImageNet-A, ImageNet-R, ImageNet-Sketch, and ObjectNet datasets is essential for a more comprehensive evaluation.

2. I'm interested in the performance on the MM-Vet benchmark.


Reference:
[1] https://github.com/LAION-AI/CLIP_benchmark

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. In the MCL process, is each FFN initialized from the previous stage’s FFN or from the pre-trained CLIP FFN?

2.  Why does upcycling appear less effective than direct fine-tuning in Table 1, yet w/o S1 S2 in Table 4, it outperforms both methods significantly, such as on Flickr I2T?
soundness: 3
strengths: 1. This work claims to be the first attempt to apply MoE to a CLIP-style model. I also believe it is an early attempt at incorporating MoE into contrastive learning.
2. The authors propose a novel method for initializing multiple experts in MoE, using MCL to first train and obtain diverse yet meaningful experts.
3. The proposed method is efficient, as it does not require retraining from scratch.
4. The paper is well-written, allowing me to quickly understand the authors’ key ideas.
summary: This work applies the recent popular concept of Mixture of Experts (MoE) in LLMs to CLIP. Specifically, it first uses Multistage Contrastive Learning (MCL) to lightly train various diversified FFNs. Then, these FFNs are used as experts in a CLIP-MoE, and the router is fine-tuned to obtain the final model. The work achieves impressive results in zero-shot image-text retrieval and zero-shot image classification tasks.
weaknesses: 1. The motivation behind this work—that CLIP often encodes inputs in a very coarse-grained manner—is neither novel nor particularly compelling. Additionally, the paper lacks an in-depth analysis of why this issue arises and does not clearly explain how the proposed method addresses it. The introduction could benefit from further refinement, as it currently seems to have chosen a weak motivation simply to justify applying MoE to CLIP.

2. The rationale for using Multistage Contrastive Learning (MCL) is not fully convincing. The paper suggests that clustering can simulate grouping raw data by attributes, such as color or shape. However, as clustering is conducted entirely within CLIP’s feature space, this outcome is not necessarily assured. The authors could strengthen their argument by visualizing each expert’s focus on distinct details, using methods such as t-SNE. Additionally, the routing analysis and case study presented in the experiments are too general to effectively illustrate MCL’s advantages. Beside, a more targeted comparison that highlights each expert's independent downstream task performance would provide clearer insights into the effectiveness of MCL.

3. The zero-shot tasks for CLIP lack sufficient comparisons to other state-of-the-art methods. This work mainly compares against Long-CLIP, which was primarily designed for sequence extension rather than enhancing CLIP’s zero-shot performance in downstream tasks.

4. The design of the ablation study is somewhat unclear. The sole comparison (w/o S1, S2) introduces too many variables, such as the number of experts, whether the experts were fine-tuned, and whether fine-tuning occurred on the original dataset or the clustered data (assuming clustering aims to mine hard negatives). The authors should design more distinct ablation studies to clearly demonstrate the contribution of each component.

5. Minor typo: in line 488, "Table 2" should refer to "Figure 2."

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weakness section.
soundness: 2
strengths: 1. It introduces a new perspective on fine-tuning CLIP with Multistage Contrastive Learning (MCL) and validates this method through proper experiments. 

2. The results show improvements across various downstream tasks, indicating its potential effectiveness in practical applications, supported by coherent writing and a relative reasonable experimental setup.
summary: The paper presents a novel model-agnostic fine-tuning strategy called Diversified Multiplet Upcycling (DMU) for enhancing the CLIP framework. This approach utilizes the sparsely activated Mixture of Experts (MoE) architecture to extend model capacity without the need for training from scratch, instead leveraging pre-trained dense checkpoints. DMU fine-tunes a base dense CLIP model to create multiple multiplet experts through Multistage Contrastive Learning (MCL), which encodes diverse information via clustering. These multiplet models share all parameters except for the feed-forward network (FFN) layers, resulting in specialized FFN experts that capture different input aspects.
weaknesses: 1. A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

2. Firstly, there is a mislabeling in the paper. In Section 5.5, Table 1 presents the results of the text-to-image retrieval experiment on the COCO dataset. The CLIP-MoE model trained on the Recap-DC dataset achieved an @10 value of 79.4, which is lower than the Upcycling's 79.9. However, the value of 79.4 is erroneously bolded, which may lead to confusion. It is recommended that this be corrected to avoid misinterpretation of the results.

3. Secondly, it might be beneficial to include an analysis of the number of experts utilized in the method. Currently, the CLIP-MoE employs 4 experts to balance performance and computational cost. However, conducting experiments to demonstrate how varying the number of experts—either increasing or decreasing—affects both performance and computational efficiency could provide more compelling evidence for the chosen configuration.

4. Thirdly, while Multistage Contrastive Learning is described in the paper as enabling experts to learn different aspects of features—such as color, texture, and shape—this explanation seems to lack supporting experimental results. To strengthen this claim, it would be beneficial to provide concrete evidence, such as demonstrating that models trained at different stages exhibit distinct capabilities on datasets characterized by these specific features. Alternatively, offering a more nuanced explanation of the effectiveness of Multistage Contrastive Learning would help clarify its role and prevent potential misunderstandings.

5. Lastly, I think the explanation in the ablation study section to be unclear. This section mentions training a CLIP-MoE model with two experts, while Table 4 compares a fully trained CLIP-MoE with a model that has not undergone the first and second stages of MCL training. First, is the latter the CLIP-MoE model with two experts mentioned in the paper? Additionally, based on my understanding, since the fully trained CLIP-MoE in the paper includes 4 experts, it should undergo 4 stages of MCL training. The results related to the CLIP-MoE in Table 4 are also obtained after 4 stages of MCL training. Why is a model that has not gone through the first and second stages of training used for comparison?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The paper claims that compared to the standard CLIP which often encodes inputs in a very coarse-grained manner, CLIP-MoE is able to encode richer information. I am curious about CLIP-MoE's performance in fine-grained image classification tasks, such as on the Stanford Cars, FGVC-Aircraft, Flowers102 and so on.

2. From the experimental results on LLaVA-1.5, replacing the CLIP in the LLaVA model with CLIP-MoE does not significantly improve performance across the five benchmarks; in fact, there are even some cases of performance decline. Additionally, the test results of the MLLM on benchmarks like MME and POPE fluctuate considerably. Are there other benchmark results, or perhaps reasoning cases for MLLM, that can further demonstrate CLIP-MoE’s ability to encode richer information in MLLM?

3. Why did you choose 4 experts? Is there a particular reason for this choice, and is there any observed relationship between the number of experts and CLIP-MoE’s performance?

I may reconsider my score based on your response to these issues.
soundness: 3
strengths: 1. Achieved a Mixture of Experts (MoE) architecture for CLIP and proposed the Diversified Multiplet Upcycling method for CLIP.
2. Attained strong results with relatively low computational cost.
3. The paper provides extensive experimental results, evaluating CLIP-MoE’s performance in zero-shot retrieval, zero-shot image classification, and MLLM downstream tasks.
summary: This paper proposed a novel Diversified Multiplet Upcycling for CLIP. Using relatively low computational cost, it trained a CLIP model with a Mixture of Experts (MoE) structure through Multistage Contrastive Learning, enabling the model to capture more fine-grained image information. The designed CLIP-MoE performs well in zero-shot retrieval, zero-shot image classification, and MLLM downstream tasks.
weaknesses: See Questions

Meta-review: (a) Scientific Claims and Findings

The paper introduces Diversified Multiplet Upcycling (DMU), a novel method for enhancing CLIP models using a Mixture of Experts (MoE) architecture. By employing Multistage Contrastive Learning (MCL), DMU aims to improve the model's capacity to capture fine-grained image information without retraining from scratch. The method is evaluated on zero-shot image retrieval, classification, and MLLM understanding benchmarks, demonstrating good performance compared to existing models. Reviewers highlight the method's potential to enhance CLIP's performance across various tasks.

(b) Strengths

Reviewer ihbw notes the comprehensive range of tasks covered in the experiments, demonstrating the method's effectiveness. a1CZ appreciates the novel application of MoE to CLIP and the efficient training strategy that avoids retraining from scratch. fub1 commends the new perspective on fine-tuning CLIP with MCL and the coherent writing. nwTx highlights the strong results achieved with relatively low computational cost and the extensive experimental evaluation.

(c) Weaknesses

The reviewers identify several weaknesses. ihbw suggests expanding the range of datasets for a more comprehensive evaluation. a1CZ questions the motivation and rationale behind the method, suggesting a need for deeper analysis and more targeted comparisons. fub1 points out mislabeling in the paper and suggests analyzing the number of experts used. nwTx raises concerns about the performance in fine-grained tasks and the choice of the number of experts, suggesting that more benchmarks or reasoning cases could further demonstrate the method's effectiveness.

(d) Decision Reasons

The decision to reject the paper is based on concerns about the depth of analysis and the clarity of the method's motivation, as highlighted by reviewers a1CZ and fub1. While the paper presents a compelling approach, the lack of comprehensive dataset evaluation and the need for more detailed comparisons and analyses reduce its impact. Additionally, the questions about the choice of the number of experts and the performance in fine-grained tasks, as noted by nwTx, further weaken the case for acceptance. Despite the strengths in experimental validation and efficiency, the weaknesses in motivation, analysis, and evaluation lead to the decision to reject.

Decision: Reject