CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
  Multiplet Upcycling
Authors: name: Jihai Zhang, name: Xiaoye Qu, name: Tong Zhu, name: Yu Cheng

In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
cornerstone in multimodal intelligence. However, recent studies have identified
that the information loss in the CLIP encoding process is substantial, and CLIP
tends to capture only coarse-grained features from the input. This deficiency
significantly limits the ability of a single CLIP model to handle images rich
in visual detail. In this work, we propose a simple yet effective
model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
efficiently fine-tunes a series of CLIP models that capture different feature
spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
the Feed-Forward Network (FFN). These models can then be transformed into a
CLIP-MoE with a larger model capacity, leading to significantly enhanced
performance with minimal computational overhead. To the best of our knowledge,
Diversified Multiplet Upcycling is the first approach to introduce sparsely
activated MoE into CLIP foundation models. Extensive experiments demonstrate
the significant performance of CLIP-MoE across various zero-shot retrieval,
zero-shot image classification tasks, and downstream Multimodal Large Language
Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
without requiring further adaptation in downstream frameworks. Through
Diversified Multiplet Upcycling, we aim to provide valuable insights for future
research on developing more efficient and effective multimodal learning
systems.