Scaling Vision-Language Models with Sparse Mixture of Experts
Authors: Sheng Shen
Zhewei Yao
Chunyuan Li
Trevor Darrell
Kurt Keutzer
Yuxiong He
Venue: EMNLP 2023 Findings (2023)
Keywords: vision language
scaling

Abstract: The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These models aim to bridge the gap between text and visual information, enabling a more comprehensive understanding of multimedia data. However, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to addressing this challenge is the use of sparsely-gated mixture-of-experts (MoE) techniques, which divide the model into smaller, specialized sub-models that can jointly solve a task. In this paper, we explore the effectiveness of MoE in scaling vision-language models, demonstrating its potential to achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale vision-language models and other multimodal machine learning applications.

Scaling Vision-Language Models with Sparse Mixture of Experts
Sheng Shen i
Zhewei Yao
Chunyuan Li
Trevor Darrell' ICC Berkeley Microsoft
Kurt Keutzer! Yuxiong He
sheng. sheng..sberrkley..ooo {zheweiyao, chunyl weicrosooti.......oo.oooo
Abstract
2022; Yu et al., 2022) that leverage powerful neural net- work architectures to encode and decode multimodal information. However, oate-of-theear languae models like Flamingo-801 (Alayrac et al., 2022), BEIT- 3-1.9B (Wang et al., 2022b), and PaLI-17B (Chen et al., 2022) be computationally expensive and difficult train, which has motivated researchers to explore ways of improving their efficiency and effectiveness.
The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-lang language models (VLMs). These mod- els aim to bridge the gap between text and vi- sual information, enabling more comprehen- sive understanding of multimedia data. How- ever, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to address- ing this challenge is the use of sparsely-gated mixture-of -fxture-of--expets (MoE) techniques, which divide the model into smaller, specialized sub- models that can jointly solve task. In this paper, we explore the effectiveness of MoE in scaling lisoon-aaguaae models, demonstrating its potential to achieve state-of-the-at perfor- mance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabiliz- ing the training of MoE models, understanding the impact of MoB model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale vision-language models and other multimodal machine learning applications.
Recently, sparsely activated Mixture of Experts (MoE) models have been successfully employed to scale both vision (Riquelme et al., 2021; Lou al., 2021; Mustafa et al., 2022) and text models (Shazeer et al., 2017; Lep- ikhin et al., 2020; Zoph et al., 2022; Du et al., 2022). These models are motivated by the need increase model parameters while controlling compute costs. In addition, these models provide other advantages, includ- ing sparsity that can mitigate catastrophic forgetting in continual learningg (Collier et al.. 2020; Komatsuzaki et al., 2022), and an inductive bias that can enhance performance in multitask learningg (Ma et al., 2018; Kudugunta et al., 2021: Kim et al., 2021b). Overall, the use of MoEs has proven to a promising strategy for scaling deep learning models across various domains.
Building on the success of Možs in individual do- mains and applying the intuition that sparse models may better handle different tasks versus dense counter- parts, we investigate the potential of MoEs for vision- language modeling. To this end, we take the first step in this direction and explore models that can process both images and text for vision-language tasks. One simi- lar effort has been studied in LIMoE (Mustafa al.. 2022), where the authors proposed a modal-agnoste CLIP-style (Radford al., 2021) multimodal MoEs ar- chitecture, but their focus is mainly the contrastive pre-training objective and vision-only downstream tasks. There are two limitations in this setting: (1) The increas- ing model capacity of MoEs under the the simple con- trastive objective can easily lead to over fitting issues. (2) The vision-only benchmarking does not reveal the full power of scaling up multimodal models. Alterna- tively, our goal to demonstrate the effectiveness of MoEs under generative modeling for lasion-l tasks and provide a more comprehensive foundation for future research this area.
1 Introduction
The ability to understand and generate natural language from visual information is critical component of many real-world applications, including visual question an- swering (VQA), visual reasoning, and multimodal in- formation retrieval. In recent years, the success of deep learning in natural language processing (NIP) has led to the development of large-scale vision-language models (VLMs) (Tan and Bansal, 2019; Chen et al., 2020; Li et al., 2021b; Gan et al., 2020; Kim et al., 2021a; Alayrac al., 2022; Wang al., 2022c; Shen et al., 2022b; Li et al., a:; Shen et al., 2022a; Jia et al., 2021; Li et al.,
* cqual contribution; & work initiated during an intern- ship at Microsoft. Code is available at hstps: / /vl moe. github. ic.
Specifically, propose novel VLM architecture that employs MoE to scale both the text-based and

---

VPPA
1-PPN
Switch V-PFN
Switch I-PNN
Switct VL-FFN
Hull-Hoal Sollrin
Hulti Seff-kttetiion
Villi Sall Altertttiionn
Verrod
IoMee
0-0
Swftch l-Mol
[--)
Swnch to T-Maf
Saln
TCT
Hult-itaad SaPrrmmntion on
NutioHead Sell-tteettinn
Paas- Salt detedimantinn
dog ball
(a) Encode Image Only
(b) Encode Text Only
(c) EEccde Image- Pair
(d) V-MoE & T-Moo
Figure 1: The encoding process of VL-MoE for various modality inputs, for which gray and colored blocks indicate non-activated and activated modules, respectively. (a) For image input only, the encoding process switches to V-MoE or V-FFN (b) For text input only, the encoding process switches T-MoE or T-FFN. (c) For image-Text Pair input, the encoding process switches, V-MoE & T-MoE and VL-FFN. (d) For the early layers, we scale the V-FFN and T-FFN with Sparse Mixture-ff Expeets as V-MoE and T-MoE, respectively. VL-MoË will utilize conditional computation to allocate tokens in modality-specifft fashion. V/T-MoE converts multiple V/T-FFNs as experts, where the image/text input will be conditionally routed by V/T-Router Network.
vision-based feed-forward networks (T-FFN and V-FFN, respectively) in a unified framework. Our approach di- vides the model into multiple sub-models, each of which is responsible for processing a modal--specing subset of the input data. The text and vision input representa- tions are then aligned via three mask data modeling objectives (Wang et al., 2022b).
Alayrac et al., 2022; Yu et al., 2022; Wang et al., 2022b; Li et al., 2022; Chen et al., 2022; Radford et al., 2021;
Jia et al., 2021; Shen et al., 2022b,a; Yuan et al., 2021; Singh et al., 2021; Liu et al., 2023b) involves developing model architecture and pretraining objectives to learn effective multimodal representations from large-scale image-text pairs. Two main approaches are encoding distinct modalities separately with different encoders.
We train a range of VL-MoE models and evaluate the model on vision-language classification, vision- language retrieval, vision-only and language-only tasks, Our experiments demonstrate that MoE can significantly improve the efficiency and effectiveness of VLMs, en- abling them to handle large-scale, real-world multime- dia data. We scale BASE-size model up to 1.8B pa- rameter VI.-MoF parameters per token and achieves competitive perfor- mance with dense models that make use of similar or more pre-training image-text pair data and apply 3-4x more parameters per token.
For model architecture, there two main designs. The first design, utilized by models such as (Radford ct al., 2021; Jia et al., 2021; Yuan et al., 2021) separately encodes each modality with different encoders. While this approach performs well for image-text retrieval tasks, it struggles with complex vision-language tasks like visual reasoning. The second design, employed by models like (Tan and Bansal, 2019; L.i et al., 2021a; a; Lu al., 2019; Li et al., 2019; Kim al., 2021a; Chen et al., 2022; Alayrac et al., 2022). uses complex fusion mod- ule with cross-modal attention to combine modalities. However, this design sacrifices efficiency for improved performance. Recently, a new design has emerged with the MOME Transformer used both VL.O and BEIT- 3. This design unifies the dual-encoder and fusion- encoder models by introducing a mniture-oof-oddlle experts technique. With MOME, various modalities are encoded within a shared Transformer block, allowing for improved scalability and achieving state-of state-of-the-are th-thh--rt art performance vision-language tasks. There creasing interest to grow the VL model capacity with an affordable compute budget, including MoE (Mustafa al., 2022) and the injection of new trainable modules on pre-trained models (Alayrac et al., 2022: Shen et al., 2022a: Liu et al., 2023b; Li al., 2023d,b; Koh et al.. 2023); the former remains less studied.
which only applies 560M
In summary, our contributions are as follows:
We propose VL-MoE, the first large-scale genera- tive MoEs multimodal models for vision/langauge- only, as well as vision-and- lnd--uuue- tasks.
We explore various scaling strategies, including in- creasing dense model size, increasing expert num- bers, and scaling either T-FFN or V-FFN alone, investigate the trade-off's between model com- plexity and performance on various downstream tasks.
in-
We present ablations to understand VL-MoE model's behavior, interpretability, and our design choices.
2 Related Work
For pretraining objectives, multiple cross-modal pre- training objectives have been studied. They can be cate- gorized into classes: (1) Discriminative modeling. including image-text contrastive learning (Radford et al.. 2021; Jia et al., 202 I), image-text matching (Tan and Bansal, 2019; Kim et al., 2021a; Li et al., 2021a; Bao et al., 2022b) and word-patch/region alignment (Chen
Vision-Language Modeling. Vision-language pre- training (Tan and Bansal, 2019; Lu et al., 2019; Su et al., 2020; Zhang et al., 2021; Radford et al., 2021; Li et al., 2020; Kim et al., 2021a; Li et al.. 2021a; Wang et al., 2022c; Bao et al., 2022b; Wang et al., 2022a;

---

et al., 2020; Kim et al., 2021a); (2) Generative mod- eling, including masked language modeling (Tan and Bansal, 2019; Su al., 2020; Kim al., 202 or pre- fix language modeling (Wang et al.. 2022c), masked region modeling (Tan and Bansal, 2019), multimodal prefix language modeling (Wang et al., 2022c). Re- cently, BEIT-3 shows strong scaling results by unifying the generative multimodal pretraining objective with masked data modeling, which comprises masked im- age modeling and masked language modeling on the monomodal encoders and masked multimodal modeling on the multimodal encoder. In this paper, we perform MoE study, by adopting the MOME Transformer as the backbone dense network and generative (masked data) modeling as pretraining objectives given its simplicity and scaling ability.
While many works aim to improve the gating mecha- nism itself (Hazimeh et al., 2021: Lewis et al., 2021; Roller et al., 2021; Zhou et al., 2022), Moï models have also been studied for multitask learning (Hazimeh et al., 2021; Kudugunta et al., 2021) with per-task routers (Ma et al., 2018), although shared pool of experts is typi- cally used.
MoE models have heen explored for multimodal learning as well, with LIMoE (Mustafa et al., 2022) and Uni-MoB (Zhu et al., 2022) being most relevant to our work. However, LIMoE considers the CLIP-style con- trast as the pre-training objective, and vision/retrieval tasks as the downstream evaluation. Uni-MoE focuses on routing decisions with limited experts and evaluates on
tasks. To the best of our knowledge, the proposed VL-MoE is the first the MoE scaling study to consider the generalized genera- tive modeling objective in the VL pre-training, and we evaluate its scaling performance in a more comprehen- sive manner, including vision/language-onll as well vision-and and and-language tasks.
More recently, with the introduce of LLaMA (Tou- vron et al., 2023), PaLI's research (Chen et al., 2022) focused on the scaling of V&L components, while PaLM-E explored the embodied domain more deeply. BLIP-2 (Li et al., 2023c) introduced the innovative Q- former to bridge image and language encoders, and this was further enhanced by InstructBLIP (Dai et al., 2023). Otter (Li et al., 2023a) augmented the instruction- following capabilities of OpenFlamingo (Laurençon et al., 2023; Alayrac et al.; Awadalla et al., 2023). Both MiniGPT-4 (Zhu et al., 2023) and LLaVA (Liu et al., 2023a; Sun et al., 2023) draw inspiration from GPT4's capabilities but place emphasis on the efficiency and integration of visual and linguistic models. In a fresh approach, mPLUG-Owl (Ye et al., 2023) first aligns vi- sual features and subsequently fine-tunes the language model using LoRA. Shikra (Chen et al., 2023) and Kosmos (Peng et al., 2023) leverage grounded image- text pairs during their training process. Lastly, QWen- VL (Bai al., 2023) made notable strides in scaling LMM pre-training.
Method
We first describe the masked data modeling pretrain- ing objectives. We next discuss MoEs, sparse MoEs and present how apply sparse MoEs methodology to vision-language models, before explaining our design choices for the routing algorithm and the implementa- tion of VL-MoF.
3.1 Vision-Language Masked Data Modeling
We utilized unified masked data modeling objec- tive (Wang et al., 2022b) to pretrain VL-MoB on monomodal (i.c., images and texts) and multimodal data (i.e., image-text pairs). This approach has been demonstrated to be scaling-frrendly with small batch- sizes. Our pretraining process involved masked image modeling on monomodal image data, masked language modeling on monomodal text data, and masked vision- language modeling on multimodal image-text pairs.
Sparse Mixture of Experts models. We build upon the concept of deep sparse MoEs, which have been stud- ied independently in both Computer Vision (Riquelme et al., 2021; Lou et al., 2021; Mustafa et al., 2022) and Natural Language Processing (Riquelme et al., 2021; Lou et al., 2021: Mustafa et al., 2022; Shazeer et al., 2017: Lepikhin et al., 2020; Fedus et al., 2021; Du et al.. 2022; Zoph et al., 2022; Clark et al., 2022; Zhou et al.,
Masked Language Modeling We use masked lan- guage modeling (MLM) to leam language representa- tions from large-scale text-only data. For MLM, 15% of tokens in monomodal text data are randomly masked, and the model is trained to recover the masked tokens from the corrupted input text. Masked tokens are re- placed by a [MASK] token 80% of the time, a random token 10% of the time, and kept the original tokens 10% of the time, following BERT (Devlin et al., 2019).
2022; Komatsuzaki et al., 2022; Kudugunta et al., 2021; Shen et al., 2023) in the context of conditional computa- tion. The goal of conditional computation is to increase the number of model parameters without a proportional increase in computational cost, which is achieved by selectively activating only relevant parts of the model based on input-dependent factors (Bengio, 2013; Chen et al., 1999; Davis and Arel, 2013). MoE models use a learned gating mechanism that activates only subset of lc experts out of E > k for given input, allowing an input to select either all experts (Eigen et al., 2013) only sparse mixture thereof, as in recent massive language models (Fedus et al., 2021; Du et al., 2022).
Masked Image Modeling In addition to masked lan- guage modeling, VL-MoE uses masked image modeling (MIM) to learn vision representations from large-scale image data. For MIM, block-wise wise masking is applied to 40% of image patches, and the pretraining objective is to reconstruct the discrete visual tokens of masked patches, following BEiT (Bao et al., 2022a). The im-

---

Donseumat
McEarsett
MoEymantzs
MoE
Dons0gas
MoE
Denseurge
MoEurgeus
1.30
0.00
0.85
LaM (04
0.92
1.40
1.17
1.36
&
1.30
g
1.00
1:25
0.00
1.30
25
46 Grass
6k
25
46
OK
2X
4K
64
Tian SPios
Tral GPiiss
Tron GPias
(a) Total Validation Loss
(b) MLM Validation Loss
(c) VLM Validation Loss
(d) MIM Validation Loss
Figure 2: Effect of VL-MoB scaling on three mask language modeling (MLM), mask image modeling (MIM), and masked vision-language modeling (VLM) pre-training tasks across training flops.
age tokenizer of BETTv2 (Peng al., 2022) is used to obtain the discrete tokens as the reconstructed targets.
H HIW:HE
Backbone Network. The dense backbone network of VL-MoE is shared multimodal Transformer, illus- trated in Figure 1. To encode different modalities, we utilize a -ooiiret---mmlllty--eee former(Bao et al., 2022b; Wang al., 2022b), which takes image and text representations of monomodal data, as well as representations of image-text pairs as input. The MoME Transformer comprises multiple layers of blocks, each consisting of a multi-head self-attention layer and feed-forwart expert layer. While the self- attention module is shared across modalities. each feed- forward expert layer contains a pool of mpdality-seecific experts (V-FFN, T-FFN, VL-FFN) that act as sub- stitute for the feed-forward network in standard Trans- formers. This allows for hard routing over the pool of feed-forward networks based on the modality of the input tokens.
Masked Vision-Language Modeling To learn aligned vision-language representation, we use masked vision-language modeling (VLM), which extends masked language modeling and masked image model- ing to multimodal data. The task aims at recovering masked image patches and text tokens based on visual and linguistic clues. In VLM, text tokens (with 50% mask ratio) are randomly masked as in MLM, and the model is trained to recover the masked text tokens based on the joint image-text representations. Image patches are also masked with the same ratio as MIM, and the corresponding visual tokens are predicted based on the image-text pair. The VLM task further encourages the model to learn alignments between image and text pairs.
(MOME) Trans-
3.2 VL-MoE Architecture
Conditional Computation with MoEs. The concept of conditional computation involves selectively activat- ing different parts of a neural network based on the input (Bengio, 2013). One specific approach is to use mixture-of of-eeerrss- (MoE) model, where different "ex- perts" handle different regions of the input space (Jacobs et al., 1991). In this paper, we adopt the MoE layer pro- posed in (Shazeer et al., 2017), which consists of E experts and is defined McE(x: Σ g(x); ex(x). Here, æ is the input to the layer, e; R + RP is the function computed hy expert i, and g RDD + R& the "routing" function that determines the dependett weights for the experts. Both e; and are implemented as neural networks. Although this formulation still in- volves dense network, it can be made sparse by restrict- ing g to assign only lo & E non-zero weights, thereby eliminating the computation of unused experts. This approach allows for super- linear scaling of the number of model parameters both training and inference.
Input Representation. To obtain text representations, the input text is tokenized and projected onto word embeddings [ 1 where is the length the tokenized text sequence. Two special tokens, start- of-sequence token ( ([T CLS)) ) and a special boundary token ([T_SEPJ) ), are added to the sequence. Text rep- resentations are obtained by summing the word em- beddings and text position embeddings, resulting in H [w cLs], wi...., , Was, Wir sEP;] ]e:] T pos- For image representations, the input 2D image IR XC is split and reshaped into N = HW/p2 patches v² E IR NP(C),) where C is the number of channels, (H, W) is height and width of the input im- age, and P is the patch size. These patches are then flat- tened into vectors and linearly projected to obtain patch embeddings following vision Transformers (Dosovit- skiy et al., 2020; Touvron et al., 2020; Bao et al., 2022a). We prepend a learnable special token [ICCLS] to the sequence. The resulting image input representations are given by H" = [M[T,CTs] .i...., un) + V Vpn.. where H E R((+D)XD V E ()(")) is linear projec- tion, Vpos (()+ are learnable 1D position em- beddings.
We apply sparse MoE to vision- languagee models the context of the MOME. As illustrated in Figure 1, inputs from different modalities are routed to V-FFN and T-FFN in the first (L - F) layers. and V-FEN, T-FFN, or VL-FFNN in the last F lay- To form image-text input representations, con- catenate image and text input vectors, resulting in
VL-MoE.
To avoid instability due to modality input im- balance when applying MoEs to modal-agnostic VL-

---

modules in V-MOE (Riquelme et al., 2021), we only use MoE for V-FEN and T-FEN in the first (L- - F) layers. V-FEN and T-FFN have two layers and a GeLU (Hendrycks and Gimpel, 2016) non-linearity: V/T-FFN(x) = W2 (gst((Waz) For VL-MoE, we replace a subset of V-FFN and T-FEN with V-MoE and T-MoE layers, where each expert is an FFN with the same architecture (x) = FFN,, (a:) but differ- ent weights 8, (Wi, W.). This design pattern is similar to that of GShard (Lepikhin et al., 2020) and V-MoE (Riquelme al., 2021) models. In V-MoE and T-MoE layers, each token æ E R2d is processed sparsely by I out of E available experts. To select which one, a lightweight V/T-Router predicts gating weights per token: g(x)= = so fttaa(W ()) E R². where Wg E Dx learned. The activated experts' outputs are combined linearly according to the gating weights: Mc5(x) = Σ g(x).. (ar)
described in BEiT (Bao et al., 2022a). The image reso- lution is set to 224 x 224, and the patch size is 16 x 16. The maximum sequence length for text is 96. We use batch size of 6, 144 and train the model from scratch for 200k steps, which is equivalent to 40 epochs of the image-text pairs. Each batch contains 2, 048 images, 2, 048 texts, and 2, 048 image-text pairs. We perform image augmentation using random resized cropping. horizontal flipping, and color jittering, following the same method as BEiT (Bao et al., 2022a). The text data tokenized using a SentencePiece (Kudo and Richard- son, 2018) tokenizer with a vocabulary size of 64k. We use the Adam optimizer (Kingma and Ba, 2015) with 31 = 0.9 and _8 0.999 to optimize the model. The peak learning rate is 2e-3, and we use linear warmup for the first 10, 000 steps and cosine learning rate decay. The weight decay is 0.05, and we disable dropout and use stochastic depth (Huang et al., 2016) with rate of 0.1. The three pretrain losses are equally weighted as in BETT-3 (Wang et al., 2022b).
To ensure computational e efficiency and implemen- tation constraints, each expert in VI.-MoE has a fixed buffer capacity, which determines the maximum number of tokens it can process. The assumption is that tokens are approximately balanced across experts. In case the capacity is exceeded, some tokens are not processed by the expert and are dropped, leading to a decrease in the success rate. This rate is vital indicator of balanced routing and training stability. To mitigate this problem. we employ Batch Priority Routing (BPR) (Riquelme et al., 2021; Mustafa et al., 2022), which selectively skips tokens based on their routing weights. BPR pri- oritizes tokens with larger routing weights, as they are deemed more informative. Our results show that BPR is crucial for stable training of VL-MoE. We further an- alyze token routing decisions in Section 5 and dropped tokens in Appendix.
MoE Setting, For the default setting of MoEs in VL- MoEgasE/I6E, we use E experts for T-FFN and V-FEN, respectively. All VL-MoEs activate k = 1 ex- pert per token, similar to Switch Transformer (Fedus et al., 2021) and LIMoE (Mustafa et al., 2022). We replace every second dense T-FFN or V-FFN sublayer with MoE sublayer following GShard (Lepikhin et al.. 2020) and Switch Transformer (Fedus et al., 2021). We use BPR for stability in V-MoE (Riquelme et al., 2021). For auxiliary loss, we use loading loss in (Shazeer et al., 2017) for T-FEN's MoE and averaged loading loss and importance loss in V-MoE (Riquelme et al., 2021) for V- FFN's MoE. The combination ratio for auxiliary loss is set as 0.01 in all our experiments We use 32 expert par- allelism and TUTEL (Hwang et al., 2022) for fast routing and computation. All the models are based on Deep- Speed (Rasley et al., 2020). Pre-training experiments are done on 32 Nvidia Tesla V100-32GB GPUs Follow- ing ST-MolE (Zoph al., 2022), we freeee all the MoE. modules (router and expert network) during finetuning process. The capacity factor C' is set to be 1.05 during training and during inference following (Riquelme et al., 2021).
4 Experiment
4.1 Pretraining Setup
Pretraining Data. Our pretraining process uses both monomodal and multimodal data. The monomodal data comprises ImageNet-22K for images and English Wikipedia and BookCorpus (Zhu et al., 2015) for text. The multimodal data combines four datasets of image- text pairs: Conceptual Captions (Sharma al., 2018), SBU Captions (Ordonez et al., 1), COCO (Lin et al., 2014), and Visual Genome (Krishna al., 2017), con- taining total of 4 million images and 10 million image- text pairs.
VL-MoE in Pretraining. We present the validation performance of VL-MoE on the three pretraining tasks across different scales. The results show that the cost- performance tradeoff of VL-MoE in terms of pretraining flops dominates the dense models by a wide margin, in- dicating that VL-MoE offers significant improvements across all scales, from SMALL/8E to LARGE/16E. We also provide a wall-clock time versus validation perfor- mance figure in the Appendix, which shows a similar scaling trend of VL-MoE. Thanks to careful kernel opti- mization and expert parallelism in DeepSpeed (Rasley et al., 2020), the maximum wall-clock overhead VI.- MoE compared to dense counterparts can be reduced to only 13%.
Pretraining Setting. For the large-size model, we em- play a 24-layer Transformer network with 1024 hidden size and 24 attention heads, following VIT (Dosovitskiy et al., 2020), BEiT (Bao et al., 2022a), and VLMO (Bao et al., 2022b). The use of VL-FFN starts at 20th layer. The base/small-steee size model is an 12/8-layer Transformer network with 768/384 hidden size and 12/6 attention heads, where VL-FFN is used in 10/8th layer. We ran- domly initialize the model parameters using the method

---

Expert (eyes)
Expert 7 (words)
Expert (ff&v)
ON
64
Ierpastiin par
HAD
ST
af
OK
mer
6.8
(a) Vision Token Routing Decisions.
(b) Language Token Routing Decisions.
Figure 3: Token routing decisions on COCO, Examples of vision tokens routing decisions and breakdown of language token routing decisions at V/T-MoE layer placed the 6-th encoder block -i.e. middle of the network- for VL-MoE
Model
Preirained images
Pretrained Steps
Params per tolen
YQA test-std
NLVR2
COCO IR
Flickr30K TR
test-dev
dev
test. P
TR
IR
Base site models pretrained
settings
UNITER (Chen et al., 2020)
4M
200k
86M
72.70
72.91
77.18
77,85 79.30
64.4
50.3
K5.9
72.5
VILLAses (Gan al., 2020)
4M
200k
86M
73.59
73.67
78.39
86.6
74.7
UNIMORASE (Li et 2021b)
4M
500K
120M
73.79
74.02
89.7
74.7
ViLT (Kim et al.,, 2021a)
4M
200k
120M
71.26
75.70
76.13
61.5
42.7
K3.S
64.4
REFass
2021a)
4M
240k
210M
74.54
34.70
80.24
73.1
56.8
94.3
VLMOuss (Bao ot 20226)
4M
200K
180M
76.64
76.89
82.77
34
74.8
57.2
92.3
79.3
BEIT-33AsH
4M
200K
180M
76.21
76.75
84.93
85.76
78.7
60.3
95.3
83.8
VI.-Moaasseir
4M
180M
78.21
78.63
85.52
79.4
84.9
Preteined with move aggressive oest, including compute/ / data potta/mmdee moddll
(Bao VLMOLANGL
1022b)
4M
200k
560M
79.94
79.98
85 64
86.86
78.2
60.6
95.3
84.5
ALBEFosse al., 2021a)
210M
77.6
60.7
95.9
BLIP Lance (Li al., 2022)
129M
1,26M
427M
78.24
78.17
82.48
83.08
81.9
64.3
97.3
87.3
SIMVLM (Wang
2022c)
1.80
IM
230M
77.87
78.14
81.72
81.77
SIMVLMMEEE (Wang
2022c)
DEIT-3mmoz (Wang et al., 2022b)
21M
IM
1,9B
84.19
84.03
91.51
92.58
84.8
67.2
98.0
90.3
PALluree (Wang
2022b)
160
IM
17B
$4.30
84.30
BLIP2a (Li al., 20236)
129M
250k
4.1B
81.55
81.60
85.4
68.3
97.6
89.7
DE/T-3LARCH
4M
200K
560M
78.14
78.23
85.23
86.15
79.2
61.4
95.7
84.1
ABGRINE
4M
86.28
79.9
62.3
85.3
Table 1: Finetuning results of different models on vision-language classification tasks and image-text retrieval tasks. We report vqa-score on VQA test-dev and test-standard split, accuracy for NLVR2 development and public test set (test-P) and top-1 recall for image retrieval (IR) and text retrieval (TR). (" denotes the model that is reproduced by us and trained with the same setting as VL.Moo..)
4.2 Vision-and-Languun Downstream Tasks
description is true about a pair of images. We use NLVR2 (Suhr et al., 2019) dataset for evaluation. Fol- lowing OSCAR (Li al., 2020), VInVL. (Zhang al., 2021) and VLMO (Bao et al., 2022b), we reformulate triplet input into two image-text pairs, each contain- ing the text description and one image. We use VL-MoE as fusion network to jointly encode the image and text. The concatenated final vector of [T_CLS] token from the two pairs is then fed into classification layer to predict the label.
In study, we explore the performance of VL-MoE on vision-and-language downstream tasks through fine- uning experiments on three standard tasks: visual ques- tion answering (Goyal et al., 2017), natural language for visual reasoning (Suhr et al., 2019), and image-text retrieval (Plummer al., 2015; Lin et al., 2014). Fol- lowing BEIT-3, we use 480 x 480 image resolution for VQA fine-tuning and 384 x 384 for the other tasks.
Visual Question Answering (VQA). For VQA, the task is to generate/choose the correct answer given a natural image and a question. Following previous work (Kim et al., 2021at Bao et al., 2022b; Wang et al., 2022b), we utilize the VQA 2.0 dataset (Goyal et al., 2017) and formulate as a classification problem with 3, 129 most frequent answers. We finetune VI.-MoE as a fusion network to encode both the image and question. We the final encoding vector of the [T_CIS] token as the representation of the image-questin pair, and feed that into classifier layer to predict the label.
Image- Text Retrieval. For image-text retrieval, it con- tains both image-to-text retrieval and text-to-image re- trieval different target modalities. We use the widely used COCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015) datasets to evaluate the model, and adopt the Karpathy split (Karpathy and Fei-Fei, 2015) follow- ing common practices. Noted that in the architecture of VL-MoB and BETT-3 (Wang et al., 2022b), does not involve the image-text matching module existing in CLIP (Radford et al., 2021). To enable image-text matching, we further fine-tune VI.-MoE jointly with image-tes contrastive and image-text matching with hard negative mining objectives as in VLMo (Bao et al.,
Natural Language for Visual Reasoning (NLVR2). Visual reasoning task aims to predict whether a text

---

Pretraining
Tasks
Models
1.34
Modal
Imagra
Strps
ImageNet MNLIIm
Train Loss (Total)
1.32
Densecoal
Vision Preiriining
ViTEns
300M
500k
83.6
1.30
MoE_mites-vlossmm
e Joss
BIS
1.2M
500k
85.2
MoE
V-MoEgnG.IE
300M
500k
85.3
1.26
Vision Language Pretraining
1.24
SIMVLMB.s..
1.8B
1M
80.6
64.4
1.22
BEST-3
4M
200k
83.2
67.0
1.20
VL-MoEnssz/15R
4M
200k
84.5
68.1
1.18
25
50
75
100
125
150
175
Table 2: Results of base-size models on image classifi- cation (ImageNet-1K3 and natural language inference (MNLI-m). We report top-1 accuracy for both.
Number of Steps (k)
Figure 4: Effect of auxiliary loss on training stability.
2022b) and BEIT-3. During inference, VL-MoE is used to encode images and text separately and compute the matching scores by the dot product of image and text vectors to obtain the top-k candidates.
4.3 Vision/Language-Onk Downstream Tasks
Image Classification. We use the image classification task to evaluate the model the vision-only down- stream task, where the objective of this task is to cat- egorize an input image into its corresponding class. We employ the ILSVRC-2012 ImageNet dataset (Rus- sakovsky et al., 2015), which consists of 1.3M images with IK classes. Following BEIT (Bao et al., 2022a) and VLMo (Bao al., 2022b), perform average pooling over the final vectors and feed the resulting vector into linear classifier layer to predict the label.
Table presents the results of our vision-language model on classification and retrieval tasks, including VQA, NLVR2, COCO, and Flickr30K. To ensure a fair comparison, we provide details the amount of' pretraining image-text pair data, pretraining steps, and the number of parameters per input token. Following LIMOE (Mustafa et al.. 2022), we define the number parameters per imput token as the number of param- eters that the model applies each image-text token pair. Notably, VI--Mo contains 2 billion pa- rameters in total, but only applies 560 million param- eters per token. Additionally, all routers combined ac- count for less than 0.5 million parameters. Our model outperforms previous large/base-size models on VQA, NLVR2. COCO, and Flickr30K by significant mar- gin, particularly when compared to reproduced BErT- 3 (Wang al., 2022b), which pretrained using the same settings as VL-MoE. Moreover, to the best of our knowledge, VI-MoE the first to demonstrate that a mixture-of-experts architecture successfully scale with a comparably modest architecture size and training counts, while achieving generalization performance on range of tasks in the context of vision-language tasks. Interestingly, Switch Transformer (Fedus et al., 2021) struggles with generalization for language MoE, while V-MoE (Riquelme et al., 2021) and LIMoE (Mustafa et al., 2022) only evaluate on downstream vision tasks. Additionally, VL-MoE and ALBEF, which are pretrained with more image- text pair data and initialized from pretrained models, on COCO and Flickr30K and achieves competitive perfor- mance on VQA and NLVR2. We assume that this may due to the fact that the capacity of VL-FFN has not been scaled in VL-MoE, as reflected in the pretraining plot in Figure 2 (the difference of VLM loss between VL-MoE and dense BErT-3 model is smaller compared to that of MLM and MIM loss). We leave the scale of future work, considering the increasing instability in modal-agnostic MoE architec- tures demonstrated in LIMOE (Mustafa et al., 2022).
Natural Language Inference. We use the natural language inference task to evaluate the model on the language-only downstream task. The task involves de- termining the relationship between two pieces of text. In this task, model is given a premise sentence and a hypothesis sentence, and it needs determine whether the hypothesis is true. false, undetermined based the information provided in the premise. We use Multi- Genre Natural Language Inference (MNLI) (Williams et al., 2018) dataset, which contains 433k sentence pairs annotated with textual entailment information. We eval- uate on matched (MLM-m) setting only.
As shown Table we compare VL-MoE with two base-size vision Transformers and V-MOE-B//16E.16 on image classification. For BETT, BEIT-3BASH and VL-MoEgasE/IE,
perform intermediate finetuning on ImageNet-22k to compare with VIT pretrained on ImageNet-22k. The model performs competitively with previous state-of-the-an supervised and supe-supe models on ImageNet-Ik. Besides the dense counterpart BITT-3 BASD. VL-MoE also outperforms other strong vasion-language models (SIMVLM) pretrained with more data and more steps on MNLI-m.
outperforms VLMOLARGE
Discussions
We conduct ablation studics to analyze the contributions of tixture-of-Expper module used in VL-MoE from different perspectives. We evaluate the models on visual reasoning (NLVR2), image-text retrieval (Flickr30k). image classification (ImageNet-Ik) 1k) and natural lan- guage inference (MNLI-m).
the VL.-FFN module

---

Scaling Strategy
NLVR2
Flickr30k
ImageNet Acc@1
MNLI-m
Avg.
T-MoE
V-MoE
dev
test-P
TR R@!
IR @
Acc
[1]
x
X
67.42
68.21
80.4
61.7
67.2
54.3
66.5
[2]
V
X
72.42
72.73
83.2
64.7
67.8
58.3
69.9
[3]
X
71.19
72.23
82.9
64.5
69.2
55.2
69.2
[4]
72.98
73.34
84.7
65.3
69.0
58.1
70.6
Table 3: Ablation studies of scaling strategies (all the results are based on VL-MoESMALL/11 models). All the *-MoE uses 16 experts (where T/V stands for applying MoE the T/V-FFN).
Scaling Strategy. In addition to scaling both T-FFN and V-FEN, we have also explored different scaling strategies by applying Mixture-of-Fxppene (MoFs) mod- ules for either T-FEN or V-FFN alone. The results of our experiments presented in Table 3. Our findings indicate that scaling a single modality can improve the downstream performance on the corresponding modality well as overall vision-language tasks. However, observed that scaling both vision and language modali- ties leads to the most balanced performing model with 70.6% averaged performance. This may be attributed to the fact that we employ three different pretraining objectives for each modality, and scaling each modality contributes to better optimization of the specific modal- ity pretraining loss as well as the VLM loss. For further evidence, we include the pre- pre-training loss in Appendix.
Average NLVR2 and Flickr30k
90.0
87.5
85.0
small
82.5
base
77.5
75.0
Number of Experts
Figure 5: Effect of Experts Number.
Size
Methods EP
Efficiency
Val
Models
ParameMten
RN
TPS
Speedup
Loss
BETT
180M/030 3G
1002.3
4,51
VL-MoE
16
180M/1.C 6.6G
450.5
x0.9
4.49
MoE
180M/0.3G
X
685.0
x1.4
4,50
Number of Experts. The optimal number of experts in
VL-MoE
180M/0.3G
VL-MoE 8
180M/0.3G
911.5
x1.4
4.51
(MoEs) is still topic of dehate, as there is no agreement on the ideal number, Previous NLP research has experimented with a wide range of expert numbers, ranging from thousands in early stud- ies (Shazeer et al., 2017; Fedus et al., 2021), to as low as 32 or 64 more recent research (Zoph et al., 2022: Du et al., 2022; Zhou et al., 2022), which has become the standard for vision models (Riquelme et al., 2021; Mustafa et al., 2022). In Figure we investigate this further with VL-MoE, and our findings suggest that larger expert pools consistently yield performance im- provements.
VL-MoE
16
105M/0.3G
1211.3
x1.3
4.50
Table 4: Efficiency results of base-size VL-MoE models with different optimizations.
z-loss (Zoph et al., 2022)). We present the results on "L-MOESMALLJEI in Figure 4, which suggest that Z-loss presents to hurt the vision-and-lee aan-langg pretraininig of VL-MoE and using loading balance loss only will introduce unstable training and underperforming mod- els. The "vloss" turns out to lead to most stable training. which consistent with V-MoE (Riquelme et al., 202 )) and LIMOE (Mustafa al., 2022). BPR also helps in stablizing training.
Effects of the Auxiliary Losses. As previously men- tioned, experts in MoEs have fixed buffer capac- ity, and without intervention, top-k MoEs tend to col- lapse, leading to poor performance as most tokens are dropped (Shazeer al., 2017; Zhou et al., 2022). To pre- vent this, prior research has employed auxiliary losses to promote balanced routing (Riquelme al., 2021; Zoph et al., 2022; Zhou et al., 2022; Mustafa et al., 2022). However, as shown in LIMOE (Mustafa et al., 2022), in multimodal settings, new challenges emerge, such as modality misbalance, where one data type may be more prevalent than the other. We design VL-MoE in a modal-specific fashion prevent the instability caused by imbalance of multimodal data and experiment with different auxiliary losses for V-MoE: loading balance loss (Shazeer et al., 2017), averaged loading balance and important loss ("vloss") (Riquelme et al., 2021),
Token Routing Examples VI.-MoE. In Figure 3, provide qualitative analysis of token routing deci- sions on COCO. For vision tokens, their specialization clear, as they are routed specific experts such as food and vegetable experts, eyes experts. OCR experts,
etc. On the other hand, language tokens show signs of syntax specialization, with some experts processing mostly padding tokens, while others focus nouns and adjectives (and some padding), excluding prepositions, determiners, verbs.
'We find that the T-MoE is quite stable using different aux- iliary losses, and resort to the most common loading balance loss in (Shazeer et al., 2017) for T-MoF. We detail the formula of each auxiliary loss in the Appendix.

---

Efficiency In Table 4, we use one x 16 node for benchmarking the efficiency of VL-MoË with various optimizations. The EP stands for the expert parallelism provided in DeepSpeed library and KN denotes the spe- cialized kernel fusing operation we implemented (ex- pert dispatch well bias gelu fusion). From the table, we see that the throughput for the BEIT model is 1002.3 sample/s, while the optimized VL-MoE with EP and Kernel has throughput of 887.5 sample/s with the same parameters per token, which add around 11 % over- head. Despite the latter being more complex model, its throughput doesn't It fall too short of the simpler BEIT. The Speedup column also suggests that with our opti- mizations, VL-MoE can even surpass BEIT to reach the same level of validation loss in terms of speed, given the same parameter per token size. It's also valuable to note that the naive implementation of VL-MoE without any optimization indeed incurs a wall-clock time loss and significant memory cost, as seen from the throughput value of 450.5 sample/s and around 5x memory.
Size
INOshot
Models
# Param
#
50k
75k
100k
Contrastive
Pretraining
DENSE
105M
50.3
63.2
67.5
LIMoE
105M
8
53.7
62.9
62.0
LIMoE
105M
16
54.6
63.1
62.1
VL-MoE
105M
8
55.2
64.2
68.3
VL-MoR
105M
16
57.2
65.1
69.0
DENSE
180M
60.1
70.3
78.2
LIMoE
180M
8
61.5
70.4
68.2
LIMOE
180M
61.2
69.3
67.5
VL-MoE
180M
8
62.5
71.7
78.9
VL-MoE
180M
16
63.2
72.4
79.5
Table 5: Comparision between VL-MoE and LIMOE using contrastive loss.
indicate that, at this data scale, the additional capac- ity introduced by more experts isn't n't effectively utilized. However, VI.-MoE, with a higher number of experts, shows better performance progression with increasing steps, suggesting more efficient use of the additional capacity. VL-MoE consistently outperforms LIMOE in most settings, especially
Comparision with LIMOE. In LIMOE (Mustafa et al., 2022), the single-modality MoB architecture and the employed contrastive loss are the two main building blocks. To directly compare the two components of mul- timodal LIMoE under our setting, we thoroughly exper- imented with optimizing either the single-modality MoE architecture or VL-MoE with contrastive or masked data modeling (MDM) loss. However, we found that the models fail to converge when optimizing the LIMoE architecture with the MDM loss, likely due to the fact that the MDM losses consist of three losses aiming for different modalities, which may exacerbate the modality imbalance problem and make difficult to optimize MoEs even equipped with the entropy balancing loss in (Mustafa et al., 2022).
increase the number of training steps. This could be attributed to inherent archi- tectural advantages or better synergy with the training objective.
Conclusion
In this paper, we have explored the use of Mixture-of- Experts (MoE) for scaling vision-language models. Our experiments demonstrate that MoE can be a promising technique for improving the efficiency and effectiveness of vision-language models. Specifically, we have shown that dividing a large vision-language model into smaller, specialized sub-modek through MoE can achieve state- of-the-art performance on several benchmarks while re- ducing computational costs. Our experiments have also shown that larger expert pools yield consistent perfor- mance improvements. Furthermore, we have explored the impact of MoE model interpretability and found it can improve the interpretability of vision- language models by providing better insights into how the model processes different inputs.
Therefore, we focused on optimizing VI.MMoE and LIMOE with the contrastive loss, as it yielded more stable results. However, it should be noted that while LIMOE uses 1.8B image-text pairs, our setting only has 4M. We then report the training and validation loss across steps by optimizing VL-MoE or LIMOE with the contrastive loss in Figure 8. The batch size is set to be 2k. From the zero-shot validation results, it can be seen that both models quickly overfit to the 4M image- text pairs, but the single modality MoE architecture in LIMoE inherits more instability.
In conclusion, our findings suggest that MoE is valuable technique for scaling vision-language models, enabling them to handle large-scale, real-world multi- media data. Our work opens up new research directions for exploring the effectiveness of MoEs in other vision- language tasks, such as visual question answering, vi- sual reasoning and image-text retrieval, and we hope our findings will inspire further investigations into this research area.
Furthermore, we use 4M data to enrich the experi- ments using contrastive loss with different model set- tings in Table 5. We can see that L.MOE seems to ex- hibit a trend where performance doesn't It improve much or even decreases the number of training steps in- creases (from 75k to 100k), especially in the 105M parameter setting. This could be a sign of overfitting, where the model is starting to fit the training data more closely but is not generalizing as well to the valida- tion/test data. Increasing the number of experts for LIMOE does not lead significant performance gains, especially in the 105M parameter setting. This might
References
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel

---

Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et Flamingo: visual language model for few-shot learning. In Advances in Neural Infor- mation Processing Systems.
representation learning. In Computer Vision FCCVV 2020 16th European Conference, Glasgow; UK, Au- gust 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 104 124. Springer.
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican. Malcolm Reynolds. Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, An- drew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learn- ing. CoRR, abs/2204, 14198.
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bog- dan Damoc, Blake Hechtman, Trevor Cai, Sebas- tian Borgeaud, et al. 2022. Unified scaling laws for routed language models. In ICML. pages 4057-4086. PMLR.
Mark Collier, Efi Kokiopoulou, Andrea Gesmundo, and Jesse Berent. 2020. Routing networks with co-training for continual learning. arXiv preprint arXiv:2009.04381.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hes- sel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint 00X1::2308.01390
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung. and Steven Hoi. 2023. In- structblip: Towards general purpooe vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, 2023. Qwen-vl: A frontier large Usioo-language model with versatile abilities. arXiv preprint 2r0iv:2308. 12966.
Andrew Davis and Itamar Arel. 2013. Low-rank ap- proximations for conditional feedforward compu- tation in deep neural networks. arXiv preprint arXiv:1312.4461.
Hangbo Bao. Li Dong, Songhao Piao, and Furu Wei. 2022a. BEiT: BERT pre-training of image transform- ers. In ICIR.
Jacob Devlin. Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Pa- pers), pages 4171-4186. Association for Computa- tional Linguistics.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subho- jit Som, Songhao Piao, and Furu Wei. 2022b. Vimo: Unified vision-language pre-training with mixture-of- moxelity-experts.. In Advances Neural Information Processing Systems.
Yoshua Bengio. 2013. Deep learning of representa- tions: Looking forward. In Statistical Language and First International Conference, SL.P 2013, Tarragona, Spain, July 29-31, 2013. Pro- ceedings 1, pages 1-37. Springer.
Speech Processing:
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. 2020. An image is worth 16x 16 words: Trans- formers for image recognition at scale. preprint wrXiv:2010.11929
Ke Chen, Lei Xu, and Huisheng Chi. 1999. Improved learning algorithms for mixture of experts in mul- ticlass classification. Neural networks, 12(9):1229- 1252.
Nan Du Yanping Huang. Andrew M Dai, Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts In ICML, pages 5547-5569. PMIR.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang. Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal I1 Im's referential dialogue magic. arXiv preprint 2rXiv:2306.15/22
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. 2022. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint m.Xiv:2209.067..
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in deep mixture of experts. arXiv preprint arXiv:1312.4314,
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, 1bs/2101.03961.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: universal image-text

---

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adver- sarial training for vision-and-Bel lannnnagg representation learning. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. December 12, 2020, virtual.
Wonjae Kim, Bokyung Son, and Iodoo Kim. 202 la. ViLT: ssion-and-languaee transformer without con- volution or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594, PMLR
Yash Goyal. Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding visual question answering. In 2017 IEEE Conference Computer Vision and Putlern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 6325- 6334, IEEE Computer Society.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. 202 Scalable and effi- cient moe training for multitask multilingual models. arXiv preprint arXiv: 104.055.0
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdh- cry, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi. 2021. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances in Neural Information P Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021. virtual.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to im- ages for multimodal generation. arXiv preprint arXiv:2301. 138223
Aran Komatsuzaki, Joan Puigcerver, James Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghami, and Neil Houlsby. 2022. Sparse upeycling: Training mixture- -ffuurr--o---eerttt from dense checkpoints. arXiv preprint arXiv:2212.0500
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (GELUs), arXiv preprint arXiv:1606.08415.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. 2016. Deep networks with stochastic depth. In Computer Vision ECCV 2016- 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis. Li-Jia Li, David A. Shamma. Michael S. Bernstein, and Li Fei-Fei. 2017. Vi- sual genome: Connecting language and vision us- ing crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32-33.
Springer.
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang. Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, al. 2022. Tutel: Adap- tive mixture-of-experts at scale. arXiv preprint arXiv:2206.00 033882........
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. As- sociation for Computational Linguistics.
Robert Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79-87. 87.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Lu- ong, and Orhan Firat. 2021. Beyond distillation: Tesk-level mixture-of-experr for efficient inference. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3577-3599.
Chao Jia. Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and visionelanguage representation learning with noisy text supervision. In Proceedings of the 38th In- lernational Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904 4916. PMLR.
Hugo Laurençon, Lucile Saulnier. Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. 2023. Obelisc: An open web- scale filtered dataset of interleaved image-text docu- ments. arXiv preprint arXiv:2336. 16527
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic aliguments for generating image descrip- tions. In IEEF Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 3128-3137. IEEE Computer Society.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020.

---

Gshard: Scaling giant models with conditional com- putation and automatic sharding. arXiv preprint MiXiv:2006.1660
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE lay- ers: Simplifying training of large, sparse models. In ICML. PMLR.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu, 2023a. Otter: multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning.
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jian- feng Gao, Yong Jae Lee, and Chunyuan Li. 2023b. Learning customized visual models with retrieval- augmented knowledge. CVPR.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. 22ip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint 131.5:230..12597
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. 2021. Cross-token modeling with conditional computation, arXiv preprint arXiv:2 109.02088.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023c. Blip-2: Bootstrapping language-iman pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:230I.I 12597.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining lask-agnostic visiolin- guistic representations for assionaand-aanguaee tasks. In Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13-23.
Junnan Li, Dongxu Li, Caiming, Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision- languala understand- ing and generation. In ICML, pages 12888 -19000. PMLR.
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare. Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021a. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in neural information processing sys- tems, volume 34, pages 9694-9705.
Jiaqi Ma. Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling task relation- ships in multi-task learning with multi-gate mixture- of of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. ACM.
Liunian Harold Li, Mark Yatskar, Da Yin. Cho-Jui Hsieh, and Kai-Wet Wei Chang. 2019. Visualhert: A sim- ple and performant baseline for vision and language. CoRR, 0bss/1908.0355
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint S.Xiv:2006.027700
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021b. UNIMO: towards unified-modal understanding and generation via cross-modal modal contrastive learning. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021. (Volume 1: Long P,pers), Virtual Event, August 1-6, 2021, pages 2592- 2607. Association for Computational Linguistics.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using million captioned photographs. In Advances Neural Infor- mation Processing Systems 24: 25th Anmual Confer- ence on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 1143-1151.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantic aligned pre-training for vision-language tasks. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, vol- ume 12375 of Lecture Notes in Computer Science, pages 121-137. Springer.
Zhiliang Peng. Li Dong. Hangbo Bao, Qixiang Ye, and Furu Wei. 2022. Beit v2: Masked image model- ing with vector-quantized visual tokenizers. ArXiv, Abs/2208.06366
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint MrXiv:230611484.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023d. Gligen: Open-set grounded text-to-image generation. CVPR.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting

---

region-lo-phrase correspondences for richer image- to-sentence models. In 2015 IEEE International Con- ference on Computer Vision, ICCV 2015, Sanliago, Chile, December 7-13, 2015, pages 2641-2649 IEEE Computer Society.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jian- wei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et al. 2022a. K-lite: Learning trans- ferable visual models with external knowledge. In Advances in Neural Information Processing Systems.
Sheng Shen, Liumian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022b. How much can clip benefit vision-and-languaee tasks? In ICLR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. 8763. PMLR.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami. Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. 2021. FLAVA: A foun- dational language and vision alignment model. CoRR. abs/21 12.04482.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020, VL-BERT; pre- training of generic visual-linguisti representations. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506.
Alaae Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- ume 1: Long Papers. pages 6418 1-88-648..22 Association for Computational Linguistics.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto. Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. Ad- vances in Neural Information Processing Systems.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 14, 2021, virtual.
Zhiqing Sun, Sheng Shen. Shengcao Cao, Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Xu-Xiong Wang. Yiming Yang, et al. 2023. Aligning large multimodal models with factually aug- mented rlhf. arXiv preprint arXiv:2309. 14525.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C Berg, and Li Fei-Fei. 2015. Imagenet large scale visual recognition challenge. IJCV.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross- modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-JJNNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5099-5110. Association for Computational Linguistics.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernmeed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2556-2565. As- sociation for Computational Linguistics.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolies, and Hervé Jégou. 2020. Training data-efficient image trans- formers & distillation through attention. preprint arXiv:2012.12877
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le. Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gatee gated mixture-of of ofpeeersss layer. In ICLR. OpenReview.net.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du Shayne Longpre, Jason Wei, Hyung Won Chung. Barret Zoph. William Fedus, Xinyun Chen, et al. 2023. Mixture- of-experts meets instruction tuning: A winning com- bination for large language models. arXiv preprint arXiv:2305.144 14705.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jin- gren Zhou, and Hongxia Yang. 2022a. Unifying ar- chitectures, tasks, and modalities through a simple peuuence-to-squeeee learning framework. CoRR. abs/2202.003522.

---

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022b. Image a foreign language: Beit pretraining for all vision and iision-languae tasks, arXiv preprint 2208 10442.
Yukun Zhu, Ryan Kiros, Rich 7emel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE inter- national conference on computer vision, pages 19-27.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022c. SimVLM: Simple visual language model pretraining with weak supervision. In ICLR.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du. Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint 200.2200..8806.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A covad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, pages 1 1112- 1122, New Orleans, Louisiana.
A
Appendix
A.I Further Analyses
"Dropped" Tokens. In MoE training, the issue of "Dropped Tokens" is inherited (Lepikhin et al., 2020; Shazeer al., 2017: Mustafa et al., 2022; Riquelme et al., 2021; Zhou et al., 2022) and caused by the lim- ited capacity of each MoE expert, which can lead to instability. To provide detailed analysis of this issue, we present Figure 6, which illustrates the distribution of dropped tokens in VL-MoEWSSHIGR across different pre-training tasks. The figure shows that MLM and MIM tasks exhibit a more balanced distribution of to- kens compared to VLM task, which may explain the improved performance of using MoEs in the former two pre-training tasks, as depicted in Figure 2. Addi- tionally. the problem of dropped imag tokens is more severe compared to dropped text tokens, which aligns with the results of different scaling strategies presented in Section 5 and the findings in (Mustafa al., 2022; Riquelme et al., 2021).
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang, An- wen Hu, Pengcheng Shi. Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large lan- guage models with multimodality. arXiv preprint 1xXi:2334414177.
Jiahui Yu. Zirui Wang. Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text founda- tion models. CORR, abs/2205.01917.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang. Boxin Li. Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. 2021. Florence: A new foundation model for com- puter vision. CoRR, abs/21 11.111111432.
Pretrain Losses for Different Scaling Strategies. We additionaly report the effect of different scaling strat- egy in Section 5 for VI--Mo scaling on three mask language modeling (MLM), mask image modeling (MIM), and masked vision-language modeling (VLM) pre-training tasks across training steps in Figure 7. The results support our hypothesis that using three distinct pretraining objectives for each modality and scaling each modality leads to improved optimization of both the specific modality pretraining loss and the VLM loss.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. VinVL: Revisiting visual representa- tions in vlsion-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579-5588. Computer Vision Foundation IREE.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew M Dai, Zhifeng Chen, Quoe V Le, and James Laudon. 2022. Mixture- of-experts with expert choice routing. In Advances in Neural Information Processing Systems.
We conduct experiments using COCO captions following (Wang et al., 2022b), where VI. achieves 139.2 for CIDEr and 23.1 for SPICE, which outperforms the BEIT-3 with 137.5 for CIDer and 22.7 for SPICE using base-size. We also observe interesting routing specialization when generating the final word "cake" considering the T-MoE in VL-MoE in Figure 3. "NN: lady" and "NN: slicing" route experts 1 and 13 respectively. "DT: A, a" both route to expert 1. "IJ: hairnet, big" route to expert 7. These routings under- score the inherent nature of expert specialization in the VL-MoE model, potentially highlighting its advantages.
Additional Results
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing rision-language understanding with advanced large language models. arXiv preprint a30. 105522.
Jinguo Zhu, Xizhou Zhu, Wenhai Wang. Xiaohua Wang. Hongsheng Li, Xiaogang Wang, and Jifeng Dai. 2022. -ni-perceiver-mo::: Learning sparse gener- alist models with conditional moes. arXiv preprint #rXiv:2206.04674

---

dropped text token
text token
dropped imag token
imag token
nustee taban
Expart 1D
Expert 1D
Expart 1D
Expert
(a) MLM T-MoE Layer 2
(b) MLM T-MoE Layer 4
(c) MLM
T-MoE MLM
Layer 6
(d) MLM T-MoE Layer 8
No quaceal RURAE auaaerroon solan
Expert
(c) MIM V-MoE Layer
( MIM V-MoE Luyer
MIM V-MoE Layer 6
(h) MIM V-MoE Layer 8
of overal roundd
VL. T-MoF. T-MoR..ayer 2
VL.M T-MoF Layer
(k) VLM T-MoR Layer 6
i VLM T-MoF Layer
overat bulte
ouerra muubd Eaae
(m) VLM V-MoE Layer
(n) VLM V-MoE Layer
(0) VLM V-MoE Layer
(p) VLM V-MoE Layer
Figure 6: "Dropped" Token analyses for VL-MoB Argmin with three mask language modeling (MLM), mask
image modeling (MIM), and masked vision- lanigage modeling (VLM) pre-training tasks. Above dashed line denotes the ratio of tokens that exceed the expert capacity and will be dropped.
Denseamal
MoEumattFLe
MoEsmanrre--ee
MoE
1.30
0.80
0.900
3.50
1.28
1d Lose (M84)
1.26
0.70
0.850
0.825
2.46
0.60
0.56
0.775
3.42
1.18
0.45
1.40
Train
Train
(a) Total Validation Loss
(b) MLM Validation Loss
VLM Validation Loss
(d) MIM Validation Loss
Figure 7: Effect of different scaling strategy in Section for VL-MoE scaling on three mask language modeling (MLM), mask image modeling (MIM), and masked vision-language modeling (VLM) pre-training tasks across training steps.
A.2 Hyperparameter
batch size. The peak learning rate is 3e-5. Following VLMO (Bao et al., 2022b), the input image resolution 480 480.
Visual Question Answering (VQA). We fine-tune the base/large-size models for 10 epochs with 128

---

1.75
70
1.70
Densegrut
Densearal
Ochet
65
1.65
VL-Mo
1.70
VL-MoE
60
1.60
LiMoE
1.65
LIMoE
Loss
1.50
1.50
1.55
50
45
Den5ermal
1.40
1.45
40
LIMoE meetteess
35
100K
100K
50K
100K
Steps
Steps
Number Steps
(a) Contrastive Train Loss
(b) Contrastive Validation Loss
(c) ImageNet Oshot Acc
Figure Comparision of Dense, VL-MoE, and LIMoE on contrastive pre-training task across training steps.
Natural Language for Visual Reasoning (NLVR2).
and the loss 2imp is defined via the squared coefficient of variation for imp(X (X) = f/imp (Xp (x)__
For results of Table I, the base/large-size models fine-tuned for 10 epochs with 128 batch size. The peak learning rate of the base-size models is set to 5e-5. The input image resolution is 384 384. For ablation ex- periments, we fine-tune the models for 10 epochs with 128 batch size, and choose learning rates from (5e-5, le-4). The input image resolution is 224 x 224. All the ablation results NLVR2 are averaged over runs.
(imp (Mx) x))
Dimp (X)
mean
Load loss. Like previously, we follow (Riquelme al., 2021). We assume the gating weights Juoisy (ar) are obtained by perturbing the routing function with noise, i.e., Jnoisy (æ) so softmax(War + e) with NO0 o I) and th largest entry of Wæ + c. The importance loss Qimp aims balance the selection probability experts by focusing on the likelihood of choosing them, as assign- ing tasks to experts is a discrete process. The load loss R complements this by striving to out the ber of assignments among the experts. To calculate the selection probability, the expert E fl,.... EE} is as- sumed to be among the top-k even when resampling only the noise as
COCO. We fine-tum tune the base/large-sie model for 20 epochs with 2048 batch size. The peak learning rate is 2e-5 and the input image resolution is 384 x 384.
1/E. We denote yn the k-
Flickr30K. For results of Table the base/large size models are fine-tuned for 40 epochs with batch size 2048 and a peak learning rate of le-5, We use the fine-tuned model COCO as the initialization. The input image resolution is 384 384. For all ablation experiments, we fine-tune the models for 10 epochs with 1024 batch size. The peak learning rate is set 5e-5, and the input image resolution is 224 x 224.
1/6 (Wa)e
(x) =
ImageNet-Ik. We fine-tune the base-size VL-MoE with V-MoE and V-FEN only for 15 epochs with 2048 batch size. The peak learning rate is 3e-5 and the input image resolution is 384 384.
with d cumulative distribution function of Gaus- sian distribution. The load loss Peodd is eventually de- fined by
sta(load(
We fine-tune the base-size VL-MoE with T- MoE and T-FFN only for 10 epochs with 32 batch size. The peak learning rate is 3.-5.
MNLI.
Pead x) = mean (load( X)) )
where load( (X) x)= = (load. (X)
Joad{(X)
Σ (x)) 26XX
A.3 Formula of Auxiliary Loss
Given a token E RR_ we denote by g(æ) softmax( (x) ) E F the gating weights across the experts, with W Ex being the routing parameters.
Z-loss. The z-loss Ostoes introduced in (Zoph al.. 2022) aims controlling the maximum magnitude of the router activations A = (W ) E RaxE with entries at,, = (W xx)... The loss is defined by
When we deal with a batch of multiple tokens (æ. Pi we use the notation X B
Importance loss. We follow from (Riquelme et al., 2021; Mustafa et al., 2022). The importance loss Qamp ensures that the gating weights are evenly distributed among the experts, maintaining a balanced profile. For any expert E {1,..., have
definition
Patres (X)
log Σ exp (ar.))
T2
E....,E), E). we
v-loss. The notation "v-loss" we used in Section 5 is essentially the final employed loss V-MOB (Riquelme al., 2021), where Potess (X) 0.5 * Pmmp (X) ) 0.5 Ppsad ())
imp.(X) Σ g(
X


Reviews:
[{"review_id": "XRAZ33VAWg", "reviewer": "Reviewer_htd9", "summary": "", "questions": "1. What if the total parameter size (not parameters per token) of VL-MoE becomes the same as the BEIT-3? For example, when the total parameter size of VL-MoE is 560M, what is the performance gap between the VL-MoE and BEit3? \n2. ﻿I cannot find a wall-clock time versus validation performance figure in the Appendix, moreover, In lines 450-455, is this mean that VL-MoE (LARGE /16E) is faster than its dense model (BEiT-Large)? Since I’m not familiar with DeepSpeed,  I'm uncertain how this could be possible. Based on my understanding, when the parameter per token size is the same, one would expect the training time to be comparable, although the total memory requirements for the model with more parameters would be larger. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your questions. We would like to respond to each question in detail!\n\nResponse to ***Q1: What if the total parameter size (not parameters per token) of VL-MoE becomes the same as the BEIT-3? For example, when the total parameter size of VL-MoE is 560M, what is the performance gap between the VL-MoE and BEit3?***\n\nThank you for your insightful question. We conduct experiments to illustrate the impact of varying total parameter sizes and model size on VL-MoE. To manipulate the pretraining scale, we explore different expert numbers in the pretraining phase. Our evaluation includes NLVR2. Flickr30k, ImageNet 0shot as well as MNLI across various modality understanding tasks. The summarized results are presented below:\n| Model  | Total Param | Param Per Token | Expert Number | NLVR dev | Flickr30k | IN 0shot | MNLI-m |\n|--------|-------------|-----------------|---------------|----------|-----------|----------|--------|\n| BEiT3  | 560M        | 560M            | -             | 85.23    | 95.7      | 84.3     | 67.9   |\n| VL-MoE | 480M        | 180M            | 8             | 85.37    | 95.9      | 84.4     | 67.9   |\n| VL-MoE | 800M        | 180M            | 16            | 85.52    | 96.1      | 84.5     | 68.1   |\n| VL-MoE | 530M        | 105M            | 16            | 83.98    | 94.5      | 83.2     | 66.3   |\n\nIt's evident that when the total parameter size of VL-MoE is approximately 560M (similar to BEiT3), its performance is comparable to, if not slightly better than BEiT3 across the evaluated benchmarks. Specifically, the VL-MoE model with 480M total parameters and 180M parameters per token, utilizing 8 experts, surpasses BEiT3 on NLVR dev, Flickr30k, and IN 0shot while achieving a similar performance on MNLI-m. \n\nA key observation is that the effective model size of VL-MoE, as represented by parameters per token, significantly influences its final performance. When the effective parameter size is set to 105M, the performance lags behind its dense counterparts, even with a higher number of experts. This indicates that to optimally scale VL-MoE, one should prioritize scaling the parameters per token before other factors.\n\nWe also want to emphasize that with proper framework, the training cost of MoE could be comparable to a dense with similar parameters per token (See response to Q2), which makes the comparison fairer; \n\nResponse to ***Q2: ﻿I cannot find a wall-clock time versus validation performance figure in the Appendix, moreover, In lines 450-455, does this mean that VL-MoE (LARGE /16E) is faster than its dense model (BEiT-Large)? Since I’m not familiar with DeepSpeed, I'm uncertain how this could be possible. Based on my understanding, when the parameter per token size is the same, one would expect the training time to be comparable, although the total memory requirements for the model with more parameters would be larger.***\n\n| Model  | Method        | Memory (GB) | Expert Num | Param Per Token | Throughput /s | Val Loss | Speedup |\n|--------|---------------|-------------|------------|-----------------|---------------|----------|---------|\n| BEiT3  | -             | 0.3         | -          | 180M            | 1002.3        | 4.51     | -       |\n| VL-MoE | -             | 1.6         | 16         | 180M            | 450.5         | 4.49     | x0.9    |\n|        | + EP          | 0.3         | 16         | 180M            | 685           | 4.50     | x1.37   |\n|        | + EP + Kernel | 0.3         | 16         | 180M            | 887.5         | 4.48     | x1.77   |\n| VL-MoE | + EP + Kernel | 0.3         | 8          | 180M            | 911.5         | 4.51     | x1.4    |\n| VL-MoE | + EP + Kernel | 0.2         | 16         | 105M            | 1211.3        | 4.50     | x1.27   |\n\nThis is indeed a key question. To illustrate this clearly, we conduct a throughout wall-clock time benchmark with using 1 V100x16 node for training. The speedup denotes the wall-clock time speedup when VL-MoE achieves the same or lower loss as BEiT3. The EP stands for the expert parallelism provided in DeepSpeed library and Kernel denotes the specialized kernel fusing operation we implemented (expert dispatch as well as bias gelu fusion). \n\nFrom the table, we see that the throughput (indicative of speed) for the BEiT3 model is 1002.3/s, while the optimized VL-MoE with EP and Kernel has a throughput of 887.5/s with the same parameters per token, which add around 11% overhead as shown in the paper lines 450-455. Despite the latter being a more complex model, its throughput doesn't fall too short of the simpler BEiT3. The Speedup column also suggests that with our optimizations, VL-MoE can even surpass BEiT3 to reach the same level of validation loss in terms of speed, given the same parameter per token size.\n\nWhile reducing the parameters-per-token size, it's the distributed nature of the computations in MoE, combined with these optimizations, that might allow it to achieve comparable, or in some scenarios, even better throughput than dense models. \n\nIt's also valuable to note that the naive implementation of VL-MoE without any optimization indeed incurs a wall-clock time loss and significant memory cost, as seen from the throughput value of 450.5/s and around 5x memory footprint. As we optimize further, we can notice a considerable improvement in wall-clock times, emphasizing the significance of optimization techniques.\n\n\nResponse to ***Q3: The main difference between the proposed method and LIMOE should be more clearly described.***\n\nThanks for the question! We expanded and summarized the difference between VL-MoE and LIMOE in Appendix A.1 as follows:\n- The scope of LIMOE mainly focuses on CLIP-style contrast as the pre-training objective, and vision/retrieval\ntasks as the downstream evaluation, while we present the MoE scaling study to consider the generalized generative modeling objective in the VL pre-training, and we evaluate its scaling performance in a more comprehensive manner, including vision/language-only, as well as vision-and-language tasks.\n\n- We found the simple extension of LIMOE with MDM loss fails to converge even equipped with the entropy balancing loss when using 8/16 experts in small/base settings. We found token dropping issues as well as the modality imbalance problem in LIMOE are exacerbated given the three losses in MDM losses aiming for different modalities. \n\n- In our experiments using LIMOE with a contrastive loss, we observed overfitting issues, particularly when working with smaller datasets (4M in our tests) compared to the larger 3.6B dataset used in LIMOE. The overfitting was evident across various model sizes. Below, we present our experimental results categorized by different model size settings:\n| Model | Param Per Tok | Expert Num |  | IN0shot | |\n| --- |---| ---| ---| --- | --- |\n| |  |  | 50k | 75k | 100k |\n| Dense | 105M | - | 50.3 | 63.2 | 67.5 |\n| LIMOE | 105M | 8 | 53.7 | 62.9 | 62.0 |\n| LIMOE | 105M | 16 | 54.6 | 63.1 | 62.1 |\n| VLMoE | 105M | 8 | 55.2 | 64.2 | 68.3 |\n| VLMoE | 105M | 16 | 57.2 | 65.1 | 69.0 |\n| Dense | 180M | - | 60.1 | 70.3 | 78.2 |\n| LIMOE | 180M | 8 | 61.5 | 70.4 | 68.2 |\n| LIMOE | 180M | 16 | 61.2 | 69.3 | 67.5 |\n| VLMoE | 180M | 8 | 62.5 | 71.7 | 78.9 |\n| VLMoE | 180M | 16 | 63.2 | 72.4 | 79.5 |\n\t\n\t- Overfitting with Small Data: LIMOE seems to exhibit a trend where performance doesn't improve much or even decreases as the number of training steps increases (from 75k to 100k), especially in the 105M parameter setting. This could be a sign of overfitting, where the model is starting to fit the training data more closely but is not generalizing as well to the validation/test data.\n    - Model Complexity vs. Data Size: The dense models with 180M parameters per token significantly outperform the 105M counterparts. However, LIMOE doesn't show the same extent of improvement when scaling up, suggesting that perhaps the complexity of the MoE architecture needs a larger data scale for efficient training, as opposed to denser models.\n    - Effect of Expert Number: Increasing the number of experts for LIMOE does not lead to significant performance gains, especially in the 105M parameter setting. This might indicate that, at this data scale, the additional capacity introduced by more experts isn't effectively utilized. However, VLMoE, with a higher number of experts, shows a better performance progression with increasing steps, suggesting a more efficient use of the additional capacity.\n    - VLMoE vs. LIMOE: VLMoE consistently outperforms LIMOE in most settings, especially as we increase the number of training steps. This could be attributed to inherent architectural advantages or better synergy with the training objective.\n    - The model checkpoints and framework for LIMOE remain inaccessible to the research community, posing challenges for result reproducibility. However, we are committed to transparency and collaboration. Upon acceptance, we plan to release our codebase, including kernel optimizations and various MoE loss schemes, to foster further exploration and verification in the field.\n\n\nResponse to ***Q4: Limitation of the proposed method is absent.***\n\n- The overall memory usage of VLMoE is notably higher than that of Dense models. Furthermore, without leveraging 16/8 GPUs, expert parallelism and proper optimization, the throughput and memory consumption of VLMoE could exceed those of Dense models.\n\n- Our framework, built on DeepSpeed and Pytorch, achieves its efficiency through substantial engineering efforts. However, directly transitioning from our framework to others, such as JAX, might pose challenges.\n\n- It's worth noting that as we increase the number of experts in VLMoE, we witness diminishing returns. This phenomenon, also observed in prior studies like LIMOE/V-MoE, could potentially constrain the optimal scaling of VLMoE.\n\nResponse to ***Q5: Moreover, BEIT-3 (Base) already outperforms ALBEF (14M) and VLMO (Large) in the COCO and Flickr. Since the contribution of VL-MoE can be exaggerated, the comparison with BEIT-3 would be suitable.***\nThanks for pointing this out; We will highlight the comparison with BEiT-3 only in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed VL-MoE provides a feasible scaling strategy for improving the performance of vision and language pre-training.\n2. This paper presents a comprehensive experimental evaluation of multiple datasets, which demonstrates the improved performance of downstream vision and language tasks by VL-MoE\n3. The writing is clear and well-organized.", "reasons_to_reject": "1. Despite strong experimental results, the contributions seem marginal since their idea and method are almost similar to the LIMOE. The primary differentiation between the two approaches lies solely in their applications: the former adopts a dual uni-modal architecture and utilizes contrastive learning losses, while the latter expands to a multi-modal architecture that incorporates generative losses  (BEIT-3).   In particular, it is unclear why the simple application of LIMOE's MoE to the BEIT3 is not possible.\n(I am somewhat concerned that the proposed method may be a straightforward extension of LIMOE.)\nThus, the main difference between the proposed method and LIMOE should be more clearly described.\n2. Some points are unclear in writing (question-2) \n3. The comparison with ALBEF (14M) and VLMO (Large) seems inappropriate (lines 532-536) since their architecture and pre-training losses are different from VL-MoE. Moreover, BEIT-3 (Base) already outperforms ALBEF (14M) and VLMO (Large) in the COCO and Flickr. Since the contribution of VL-MoE can be exaggerated, the comparison with BEIT-3  would be suitable. \n4. Limitation of the proposed method is absent. ﻿ ﻿ ﻿ ﻿For example, if the parameters per token are the same as BEIT-3, the training time would be similar to the dense model, but the requirement of memory for the VL-MoE would be much bigger.  ", "questions_to_authors": "1. What if the total parameter size (not parameters per token) of VL-MoE becomes the same as the BEIT-3? For example, when the total parameter size of VL-MoE is 560M, what is the performance gap between the VL-MoE and BEit3? \n2. ﻿I cannot find a wall-clock time versus validation performance figure in the Appendix, moreover, In lines 450-455, is this mean that VL-MoE (LARGE /16E) is faster than its dense model (BEiT-Large)? Since I’m not familiar with DeepSpeed,  I'm uncertain how this could be possible. Based on my understanding, when the parameter per token size is the same, one would expect the training time to be comparable, although the total memory requirements for the model with more parameters would be larger. ", "paper_topic_and_main_contributions": "Upon the MoMe transformer from BEIT3 and VLMO, \nauthors propose the scaling strategy that both text-based and vision-based feed-forward networks (V-FFN / T-FFN)\nis partially substituted with the sparsely activated Mixture of Experts (MoE), dubbed as V-MoE, T-MoE.\nTheir main concept is similar to LIMoE, which appends MoE to the dual uni-modal encoders while adopting the contrastive learning loss as objectives.\nHowever, the authors extend it to the more generalized multi-modal encoder (MoMe transformer) architecture and generative modeling losses.\nThrough extensive experiments, authors show that MoEs effectively scale the VLP models. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LSpX9kbfGH", "reviewer": "Reviewer_sswr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to ***Q1: Could you show the results on image captioning tasks and provide corresponding qualitative results for understanding how MoEs work?***\n\nThank you for the suggestion. It indeed provides valuable insights for our research. We proceeded with preliminary experiments using COCO captions and the results are tabulated below:\n| Model | Param Per Tok | Expert Num | B@4 | M | C | S |\n| --- |---| ---| ---| --- | --- |--- |\n| Dense | 180M | - | 40.6 | 29.8 | 137.5 | 22.7 |\n| VLMoE | 180M | 16 | 40.9 | 30.1 | 139.2 | 23.1 |\n\nThe results reveal a consistent performance enhancement with VLMoE when compared to its Dense counterparts. We are committed to providing a more detailed analysis and extended results in the final version, pending acceptance.\n\nQualitative examples: \nIn Figure 3(b), when generating the final word \"cake\" and considering the first T-MoE of VLMoE$_\\text{B/E16}$, we observe the following routing:\n\n\"NN: lady\" and \"NN: slicing\" route to experts 1 and 13 respectively.\n\"DT: A, a\" both route to expert 1.\n\"JJ: hairnet, big\" route to expert 7.\nThese routings underscore the inherent nature of expert specialization in the VLMoE model, potentially highlighting its advantages.\n    \nResponse to ***Q2: Technical Contributions.***\n\nThank you for highlighting this aspect! We're acutely aware of the prior propositions regarding MoE losses, BPR, and several stabilization techniques. However, there seems to be a gap when it comes to investigating multimodal MoE with generative loss. In addition, there's a noticeable absence of public frameworks and implementations that cater to efficient multimodal MoE training and inference. As mentioned in our response to ***Q2 to reviewer htd9*** , without rigorous engineering, training can become inefficient and unstable. We are committed to open-sourcing our framework upon acceptance, hoping it will significantly benefit the research community at large.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Achieve state-of-the-art performance on a range of related vision-language benchmarks\n- Comprehensive analysis and experiments are conducted, exploration about various scaling strategies is interesting.\n- The paper is well-written.", "reasons_to_reject": "- No obvious weakness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose a MOE multimodal models for vision-and-language tasks, and explore various scaling strategies to provide insights into stabilizing the training of MoE models for v-l tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ePKcFzLuX5", "reviewer": "Reviewer_rELj", "summary": "", "questions": "Could you show the results on image captioning tasks and provide corresponding qualitative results for understanding how MoEs work?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. We're glad to hear that the reviewer recognizes the state-of-the-art performance we achieved on various vision-language benchmarks. We endeavored to provide a thorough analysis and experimental setup to ensure clarity and depth in our findings. The reviewer’s acknowledgment of our exploration of scaling strategies is particularly encouraging. We also appreciate your compliment on the writing quality of the paper. We will continue to work diligently to maintain and elevate the standard of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors extend the MoE's techniques for multimodal models that can be applied to vision, language, and VL tasks.\n\n2) The findings from various experiments reveal diverse scaling methodologies, and qualitative analyses provide insights into the mechanisms of MoE.", "reasons_to_reject": "1) The novelty of the technical contributions is somewhat limited as the losses, designed MoE architecture, and some tricks are based on earlier works, including (Wang et al., 2022b, Bao et al., 2022b, Wang et al., 2022b, Mustafa et al., 2022) and (Riquelme et al., 2021).\n\n2) To comprehensively showcase the potential of the proposed vision-language generative MoEs model, the authors should undertake experiments involving more VL tasks, including image captioning.", "questions_to_authors": "Could you show the results on image captioning tasks and provide corresponding qualitative results for understanding how MoEs work?", "paper_topic_and_main_contributions": "This work aims to reduce costs associated with the training and deployment of vision-language models through sparsely-gated mixture-of-experts techniques. The contribution in this work is primarily twofold. Firstly, the researchers have successfully implemented large-scale generative MoEs multimodal models for vision, language, and VL tasks. Secondly, a comprehensive series of experiments have been conducted, offering valuable insights into scaling strategies and VL-MoE's behavior.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 060: learningg ->  learning"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the effectiveness of MOE models in vision-language pretraining and VL tasks. MOE has been shown success in both vision and language models and thus it is natural and valuable to also explore its application in VL models.

Pros:
1. This paper presents a comprehensive study of MOE, and strong experimental results are sufficient to demonstrate the effectiveness of MOE. These experimental results and observations are potentially helpful and inspiring for the field.

Cons:
1. As pointed out by reviewers, the novelty and technical contribution of this work is somewhat limited. Many designs are based on existing works and thus it is less exciting in that sense.
2. Besides, some reviewers also have some concerns about the experimental setting and the model is not large enough compared to state-of-the-art language/vision models (with billions of parameters), despite the word "scaling" in the title.
Decision: Accept-Findings