Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
Authors: Basil Mustafa
Carlos Riquelme Ruiz
Joan Puigcerver
Rodolphe Jenatton
Neil Houlsby
Venue: NeurIPS 2022 Accept (2022)
Keywords: machine learning
computer vision
natural language processing
multimodal
mixture of experts
conditional computation
sparse computation
contrastive learning
zeroshot classification

Abstract: Large sparsely-activated models have obtained excellent performance in multiple domains.
However, such models are typically trained on a single modality at a time.
We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning.
LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss.
MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.
However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme.
Across multiple scales, we demonstrate performance improvement over dense models of equivalent computational cost.
LIMoE-L/16 trained comparably to CLIP-L/14 achieves 77.9% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 83.8%, approaching state-of-the-art methods which use custom per-modality backbones and pre-training schemes.
We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the emergence of modality-specific experts.

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
Basil Mustafa", Carlos Riquelme,, .oan Puigcerver', Rodolphe Jenatton, Neil Houlsby Google Brain
(basilm, rikel, jpuigcerver, rjenatton, neilhoulsbby)@goglee.co
Abstract
Large activvly-ctivatee models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using contrastive loss. MoEs for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme. Across multiple scales,
natural fit
demonstrate remarkable performance improvement over dense models of equivalent computational cost. LIMoE-L/16 trained comparably CLIP-L/14 achieves 78. 88.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 84. %, comparable to tftte-oftthe-art methods which use larger custom per-modality backbones and pre-training schemes. We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts.
Introduction
Sparsely activated mixture of expert (MoE) models have recently been used with great effect to scale up both vision 2 and text models 3 The primary motivation for using MoEs is scale model parameters while keeping compute costs under control. These models however have other benefits; for example, the sparsity protects against catastrophic forgetting in continual learning (5) and can improve performance for multitask learning (6) by offering a convenient inductive bias.
Given success in each individual domain, and the intuition that sparse models may better handle distinet tasks, we explore the application MoEs to multimodal modelling. We take the first step in this direction, and study models that process both images and text. In particular, train single multimodal architecture that aligns image and text representations via contrastive learning 7
When using setup proposed in prior unimodal models [. find that feeding multiple modalities to single architecture leads to new failure modes unique to MoEs. To overcome these, we present set of entropy based regularisers which stabilise training and improve performance. We call the resulting model LIMoE (Language-Image MoE).
We train range of LIMoE models which significantly outperform compute-matched dense baselines. We scale this up large 5.6B parameter LIMoE- H/14, which applies 675M parameters per token. When evaluated zero-shot 7 on ImageNet-2012 9 it achieves an accuracy of 84. 1%, competitive with two-tower models that make use of modality pre-training and feature extractors, and apply 3-4x more parameters per token.
Authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Zt
In summary, our contributions are as follows.
AvgPool
Contrastive
Loss
We propose LIMoE, the first large-scale multimodal mix- ture experts models.
MoE Encoder Layer
Dense Encoder Layer
We demonstrate in detail how prior approaches to regular- ising mixture of experts models fall short for multimodal learning, and propose a new entropy-based regularisation scheme to stabilise training.
MLP
MIL
MLP
MUP
MLP
MoE
Encoder
show that LIMoE generalises across architecture scales, with relative improvements in zero-shot ImageNet accuracy ranging from 7% to 13% over equivalent dense models. Scaled further, LIMoE-H/ 1/ achieves 84.1% zero- shot ImageNet accuracy, comparable SOTA contrastive models with per-modality backbones and pre-training.
Router
Layer
SelfAttn
Dense Encoder Layer
Lookup embed
embed
Lastly, we present ablations and analysis understand the model's behavior and our design decisions.
Text
Images
Figure LIMoE, sparsely activated multimodal model. processes both im- ages and texts, utilising conditional computation to allocate computations in a modality fashion.
2 Multimodal Mixture of Experts
Multimodal contrastive learning typically works with inde- pendenl per-modality encodings [7] That is, separate models J' are trained to provide a final representation for every input from the corresponding modality, m. In the case of some image and text inputs, and f., we have 24 = Simage (i) and & = (t.. For contrastive learning with images and text, this approach results one-tower setup instead, where single model is shared for all modalities, as shown in Figure [ The one-tower design offers increased generality and scalability, and the potential for cross-modal and cross-task knowledge transfer. We next describe the LIMoE architecture and training routine.
"two-tower" architecture, one for each modality. We study
2.1 Multimodal contrastive learning
Given n pairs of images and text captions {{1, t;)J the model learns representations 2, = {(2, Ze, )) such that those corresponding to paired inputs are closer in feature space than I[]. with learned temperature T, is:
those of unpaired inputs. The contrastive training objective
1 log Σ 7 TT
( (e 3/T
(2(
log
2
Ek=1 (0 /TT 1/T
image-to-text loss
text-to-image
2.2 The LIMoE Architecture
We use a single Transformer-based architecture for both image and text modalities. The model uses linear layer per modality project the intrinsic data dimension the desired width: for text, standard one-hot sentencepiece encoding and learned vocabulary [12] and for images, ViT--tyle patch-based embeddings [3]. Then all tokens are processed by shared transformer encoder, which is not explicitly conditioned on modality. The token representations from the final layer are average- pooled to produce a single representation vector zm for each modality. To compute the training loss in (T), the paired image and text representations are then lincarly projected using per-modality weight matrices W,, and C2 is applied to {(W Wiiaaee& Zi. W text Zt.) )
This one-tower setup can be implemented with a standard dense Transformer (and we train many such models baselines). Next, we describe how introduce MoEs to this setup for LIMOE.
Sparse MoE backbone: Sparse MoE layers are introduced following the architectural design of 13 The experts-panl of the model activated in input-dependent fashion-an are MIPss. LIMoE contains multiple MoE layers. In those layers, each token x TRD processed sparsely by K out of E available experts. To choose which K, a lightweight router predicts the gating weights per token:

---

Expert 2 (Plants]
Expert [Eyas]
Expert 19 (Wheels)
Expert (Hands]
Expert iStriped Textures)
(Solid
Expert Sky!
OU
CUSEE
AI TR
Figure Token routing examples for Coco. Image examples of how patches are routed the MoE layer placed in the 18-th encoder block -i... middle of the network- for the LIMoE-H// model.
g(x) sof tmax W xX) RE with learned W E RDDE The outputs of the K activated experts are linearly combined according to the gating weights: MoE(x) = > g(x)e MLP.(x).
Note that, for computational efficiency and implementation constraints, experts have afieed buffer capacity. The number of tokens each expert can process is fixed advance, and typically assumes that tokens are roughly balanced across experts. If capacity is exceeded, some tokens are "dropped":; they are not processed by the expert, and the expert output all zeros for those tokens. The rate which tokens are successfully processed (that is, not dropped) is referred to as the "success rate". It is an important indicator of healthy and balanced routing and often indicative of training stability.
We discovered that routing with tokens from multiple modalities introduces new failure modes; in the next sections we demonstrate this phenomenon, and describe our techniques address it.
2.2.1 Challenges for multimodal MoEs
As mentioned, experts have a fixed buffer capacity. Without intervention, Top-K MoEs tend to "collapse", thus using only one expert. This causes most tokens to be dropped and leads to poor performance 14 Prior works therefore use auxiliary losses encourage balanced routing
In multimodal settings, new challenges arise; one modality misbalance. In realistic setups, there will likely be more of one data type than another. Accordingly. we do not assume or enforce balanced data across modalities, and our experiments have 3 17x more image tokens than text tokens.
Modality -peeciiifiic experts tend to emerge naturally. In this imbalanced context, this leads to scenario where all of the tokens from the minority modality get assigned single expert, which runs out of capacity. On global level, routing still appears balanced: tokens from the majority modality are nicely distributed across experts, thereby satisfying modality-agnostic auxiliary losses. For example, in our standard B/16 setup, the router can optimize the importance loss [14] to within 0.5% of its minimum value by perfectly balancing image tokens but dropping all text tokens. This however leads to unstable training and unperforming models.
2.2.2 Auxiliary losses
We refer to auxiliary losses used in V-MoE [ the classic auxiliary losses. We find that they do not yield stable and performant multimodal MoE models. Therefore, we introduce two new losses: the local entropy loss and the global entropy loss, which applied combine these losses with the classic losses; see Appendix B for a summary of all auxiliary losses.
per-modality basis. We
Definition, In each MoE layer, for each modality the router computes gating matrix G. E R X. Each row of G, represents the probability distribution over experts for one of the fy tokens of that modality in the batch. For a token that corresponding row is pm (experts(x) E

---

this later dictates which experts process x. The local and global entropy losses are defined by:
1
Sucaal (m)
Hlym (expmexxxtt))))) and Rglohal (Gm):= - (m (expertx)))))
(2)
where pm (experts) = 1 L
ii Pm Pm (experts())) is the expert probability distribution averaged mm
over the tokens and H(p) = - Pe lpg( denotes the entropy. Note that Pm (experts) R Pm Pm (expperts) since approximate the true marginal from the tokens in the batch. We use the terminology local vs. global to emphasise the fact that Mocaa applies the entropy locally for each token while Nglobal applies the entropy globally after having marginalized out the tokens.
Effects of the losses. Figure 3 shows why these losses are necessary. With the default losses, speecic experts naturally emerge, but the router often changes its preference. This results in unstable training and poor success rate, particularly for the text modality. The local entropy loss encourages concentrated router weights (Peereexs exxerrs x2) have low entropy), but the expense of the diversity of the text experts: the same expert used for all text tokens (the marginal Preu (experts) also has low entropy), leading to dropping. In this setup, many layers have poor text success rates.
To address this, Ogiotal encourages maximization the marginal entropy, thus pushing Prest (experts) towards a more uniform expert distribution. The result is diverse expert usage, stable and confident routing, and high success rates. These are consequently the most performant models.
Intuitively, it is desirable for text tokens to use multiple experts. but not all of them. In order to allow flexibility, we threshold the global entropy loss as global (m.)) max (0, + D2aloball (G (G,,), }} such that the model is encouraged to have a certain minimum entropy, but after exceeding that, the loss is not applied. This avoids distributional collapse but does not apply overly restrictive priors on the routing distribution, as there are many optimal solutions. This can be thought of as "soft minimum" S. With r (S) the model must use at least S experts to minimize the loss (either a uniform distribution across experts -with entropy log(S)- or non-un mon-uniform distribution using more than S). Figure shows the latter occurs; the empirical effect of these thresholds is analysed in Section 4.1
Performance
A: Image routing success rate
B: Text routing success rate
C: Layer 5 text specialisations C3
50%
A1
300%
1 chhassi
3 22344
10%
clasio
clastc beal
global
40%
60%
L13
1.7
60%
40%
1.7
25%
45%
2: chosic + bocal
100%
300%
230%
40%
80%
20%
toxt booore
65%
30%
200
60%
60%
40%-
25.0%
3: ciassie can local global
A3
100%
C3
55%
10.0%
60%
10%
56%-
3.0%
25%
0.0%
2.5
50t
75k
300G
40%
40%
0%.
25c
50c
100c
Stope
Stras
(a) Performance wirk aux losses.
(b) Analysing routing behaviour of the auxiliary losses. First colomo: Average
image layers 177111. Second
colmmn: Same, for text. colauu: some experts layer 5, what fraction tokens go experts
Figure 3: What necessitates entropy losses? Classic refers to the standard formulation (importance + load losses [I.. We add the local entropy loss to text tokens (middle row), followed by the global entropy loss (bottom row). Left: The "classic" setting is low-performing and unstable. Right: Analyzing the entropies shows us why: Without the local loss, the model is prone to unstable changes in expert preferences (CI), and routing success rates are low (A1, B1). The local loss fixes this but causes distributional collapse for one modality (C2), with all text tokens going to one expert (expert 11); this causes even poorer text success rates (B2). This is addressed by the global loss, which has stable expert allocations (C3) and consistently high success rates (A3, B3).

---

Connection with mutual information. The sum Mreall (Gm) + Ogglobal (m)) corresponds to the (negative) mutual information [[] between experts and tokens, conditioned on the modality m, which we write - ,m (epprrs;; x). For each modality taken separately, we are effectively encouraging the knowledge of the token representation to reduce the uncertainty about the experts selection. We also tried other variants of the losses which exploit this connection, such as the mutual information between the experts and modalities, -MI(experter m). obtained by first marginalizing the tokens.
2.2.3 Priority routing
With Top-K routing, some token dropping is virtually inevitable. Batch Priority Routing (BPR) actively decides which tokens to skip based on their routing weights. It assumes that tokens with a large routing weight are likely be informative, and should be favored. BPR was mostly used inference time in [.] allowing for smaller expert capacity buffers. In this setup, one must take care not systematically favor one modality over the other, for instance, by determining which token to drop based on their rank in the batch, which are usually grouped according to the token modality. BPR provides an essential stabilisation effect during training (Figure ( we show that does not trivially rank one modality over another, and it cannot be replaced by other methods of re-ordering the batch. In the appendix we further show how routing priorities compare across text and images.
3 Experiments
We study LIMoE in the context of multimodal contrastive learning. We first perform controlled comparison of LIMoE to an equivalent "standard" dense Transformer, across a range of model sizes. We then show that when scaled up LIMoE can reach a high level of performance. Finally, we ablate the various design decisions leading to LIMoE in Section
Training data. By default, all models are trained on paired image-text data used in [16]. consisting of 3.6B images and alt-texts scraped from the web. For large LIMoE-H/I /4 experiment, we also co-train with JFT-4B [17]. We construct artificial text captions from JFT by comma-delimited concatenation of the class names [8] Appendix A contains full details of our training setup.
Evaluation. Our main evaluation is "zero-shot": the model uses its text representations of the classes to make predictions on a new task without extra training data 19 7 We focus on image classification accuracy ImageNet [9] and cross cross-modal retrieval MS IS-COCO (20), following the protocol in [16]. We also evaluate LIMoE's ess image representations via a linear adaptation protocol [3], and report 10-shot accuracy ImageNet accuracy accordingly. Where ranges are given, they report 95% confidence intervals across three trials.
3.1 Controlled study across scales
We train a range of LIMoE models at batch size 16k for 781k steps. This matches the number of training examples used for CLIP (7]. Due to use of different training data and additional tricks, direct comparison is difficult; we therefore train dense one-tower models as baselines. All models activate k = 1 experts per token, similar to Switch Transformer [8]
Figure shows the performance of each model (dense and sparse) against forward-pase FLOPs (for step times and further discussion on compute costs, see Appendix[D.2 The cost cost-performance Pareto frontier for LIMoE dominates the dense models by wide margin, indicating that LIMoE offers strong improvements across all scales from S/32 up to L/16. The effect is particularly large on zero-shot and 10-shot ImageNet classification, with absolute performance improvements of 10.1 % and 12.2 % on average. For text-to-image retrieval on COCO, LIMoE offers a strong boost at small scales, while larger scales the gains are more modest but still significant.
3.2 Scaling up LIMoE
We increase the architecture size, training duration, and data size to assess the performance of LIMoE the large-scale regime. In particular, we train a 32-layer LIMoE-H/14 with 12 expert layers; these are non-uniformly distributed, with 32 experts per layer, and K = 1 activated per token. It was trained at a batch size of 21k, introducing 25% JFT-4B images [1 into each batch (with class names as texts). We average checkpoints towards the end of training 21 refer to Appendix A.3 for details.
5

---

Zero-shot ImageNet
10-shot ImageNet
Coco T2I Recall@
80.0%
40.0%
Zero-shot ImageNet
10-shot ImageNet
70.0%
T2) Recalli31
70.0%
35.0%
60.0%
30.0%
60.0%
25.0%
40.0%
20.0%
10a
10a
10a
10a
10a
10a
FLOPS (x109)
FLOPS (x10P)
FLOPS (x10P)
Arch: S/32
S/16
B/32
B/16
L/32
L/16
Type:
LIMoE
Dense
Figure LIMoE scales well to large models, with consistent performance improvements.
The model contains 5.6B parameters total, but only applies 675M parameters per token. All routers combined account for less than 0.5M parameters. shoos its performance alongside current state-of-the-an contrastive models. LIMoE achieves 84.1% zero-shot ImageNet classification accuracy with comparably modest architecture size and training counts. LIMoE fully trained from scratch, without any pre-trane ire-trained components, and is the first competitive model with shared backbone.
In light of its modality agnostic approach, this result is surprisingly strong. Large models handling dozens of distinct tasks are increasingly popular [22]. but do not yet approach the state-of-the-ad in these tasks. We believe the ability to build a generalist model with specialist components, which can decide how different modalities or tasks should interact, will be key to creating truly multimodal multitask models which excel everything they do. LIMoE is promising first step in that direction.
Table 1: Comparing state the zero-shot classification models. At a relatively modest scale, LIMoE-H/14 is comparable with the best two-tower models, and it is the first performant one-tower model at this scale. T-x refers to Transformer [23] with the equivalent parameters of ViT-x [13
Rey: Pretrained
Exoupies nm during preteiiuin
Uses FuRes
Other non-contrassiie voiining objertiv
Architecture
Batch
Examples seen
Parameters per token
ImageNet top-1
Image
Text
size
Test
V2
R
A
COCA 25
ViT-g
T-g
65k
32.8B
1.1B
86.3
80.7
96.5
90.2
BASIC
18
CoAtNet-7
T-H"
65k
19.7B
+32.8B
1.5B
85.7
80.6
95.7
85.6
LIT 16
ViT-g"
T-g
32k
25.8B + -1.2BB
1.1B
84.5
78.7
93.9
79.4
ALIGN 10
EffNet-L22
T-L.
16k
19.8B
410M
76.4
70.1
92.2
75.8
CLIP 7
ViT-L/14
T-B
32k
12.8B
200M
76.2
70.1
88.9
77.2
LIMoE
H/14
21k
23.3B
675M
84.1
77.7
94.9
78.7
4
Ablations
We use a smaller setup to study various aspects of LIMOE. We train B/16 models at batch size 8096 for 100,000 steps (see Appendix (A.2 for further details). Table shows the average three trials of this setting alongside dense one-tower and two-tower baselines. LIMoE greatly outperforms both dense models ImageNet ()- and 10-shot, while confidence intervals overlap for retrieval with two towers. The two-tower model is twice as large and expensive, and still falls behind the sparse one.
4.1 Routing and auxiliary losses
Choice of auxiliary losses. With the introduction of the entropy based losses in addition to classic ones, there are 7 possible auxiliary losses. We aimed to find the simplest combination of these which obtains good performance. To study this, we performed large sweep of auxiliary losses: for N E 12.... 5]. we considered all (7) possible loss combinations. Table 3 shows, for each loss, the highest performing model with and without that loss. Some conclusions stand out: Both entropy losses are important for text, but for images, the global loss is not impactful and the local

---

Table Baselines for ablations: B/16 with batch size 8096 trained for for 100,000 steps. Oshot and 10shot columns show accuracy (%), t2i and i2t show recall @ (%).
Model
ilk Oshot
ilk 10shot
cocO 12i
CocO i2t
dense one-tower
49.8
43.8 43.3
23.7
36.7
40.2
25.4
34.0
dense two-tower
54.7
10.2
47.1
17.6
26.6
27.1
41.3
12:0
LIMoE
56.9 56.7
50.5 50.2
25.6
39.7
42.2
25.8
Table 3: Across 121 combinations, each row shows the best accuracy (%) of all combinations that included the auxiliary loss (V) vs. those that did not (X). Bold auxiliary losses indicate they are in LIMoE. Validation accuracy is the average contrastive accuracy a minibatch of size 1024.
Validation /
Oshot
10shot
Auxiliary loss
X
X
X
Importance
70.5
70.6
55.4
56.2
51.1
51.3
Load
70.3
70.6
56.2
55.7
51.3
51.1
Z-Loss
70.3
70.6
55,8
56.2
50.5
51.3
Global Ent Image
70.6
70.5
56.0
56.2
50.8
51.3
Global Ent Text
69.1
70.6
54.3
56.2
51.1
51.3
Local Ent Image
70.6
68.7
56,2
53.5
51.3
47.5
Local Ent Text
67.2
70.6
53.3
56.2
47.5
51.3
loss is harmful. The final combination of losses was chosen based on validation accuracy alongside qualitative observations around training stability and routing success rate.
Threshold for global entropy losses. In Section 2.2.2 introduced a threshold encourage balanced expert distributions without forcing all modalities to use all experts. To understand the importance of this threshold, we sweep over it for both the image and text global entropy losses. Appendix B contains full analysis; the most important conclusions are:
Timage did not affect the number experts used for images, as global entropy was always high. Aside from these threshold experiments with very high Timage this loss is usually inactive. It was used our main experiments, but can likely be removed future work.
soft minimum for text experts: Sweeping Tiex, we typically observed approximately = e text experts.
The threshold Tiex behaved exactly
Performance robust to different values of Tixxt provided not too low. A low Trext can be useful to limit the number of text experts, for later pruning, see Appendix F.4
Mutualiinformation auxiliary loss. In Section 2.2.2 we discussed an alternative loss, namely - (xxpertsss m). based on the mutual information between experts and modalities. While it has the advantage of merging the local and global entropy losses for both the text and image modalities into single term, without threshold parameters, it leads to slightly worse results: in comparable setup, it had 1.5% 5% and 0. % worse zero-shot and 10-shot performance compared to Table 2
classic (dashed) vs. entropy (solid)
The effect of modality balancing. Our models use a text sequence length of 16. but image sequence lengths from 49 to 400 (for these ablations, 196).
73%
Ling = ingoxe00.61
w9%
King 16
53%
King -
Our ablations reveal that the entropy losses are important when applied to the text tokens. This leads to hypothesis that these are only necessary or use- ful in the imbalanced case. To test this, we vary the modality balance of LIMoE-B/16 by varying the patch size; this enables us to control the number of image tokens, and hence image:text balance, without changing the information content in the data. Fig-
83%
Ingrou5.111
33%
ing
Lug - 196
20%
ingooo- 12.23
13%
ims:00020.21 20.213
Figure 5: Entropy losses are not just ad- dressing modality imbalance. With dif- ferent image:text balancing. including com- pletely balanced, the entropy losses substan- tially improves over the classic setting.
7

---

ure 5 shows the results. First, we observe that, with entropy routing, a longer image sequence length always better. This shows that entropy routing can effectively handle highly imbalanced setups, and mirrors the observation that for classical Vision Transformers: longer sequence is better. Importantly, entropy routing is always far superior to the classical setup with growing gaps, even when the modalities are balanced 1:1 (Timg 16). This experiment also confirms the robustness of entropy routing to different setups.
Batch priority routing as a training stabilizer. Figure shows the effect of BPR during training. BPR not only ameliorates against token dropping, but also improves training stability. Models with no dispatch order intervention ffrst-in-firste first text first. These routers have low success rate. Randomly shuffling tokens (i.e. deciding which tokens to drop at random when an expert becomes full) partially ameliorates this, but its performance still much worse than that of models trained with BPR. We further analyse BPR in Appendix F.5 and show that it does not simply rank one modality above another.
perform extremely poorly, whether we route images
Image success
Text success
1.00%
5.%
% succestuu routis
pettor moance
80%
80%
60%
80%
20%
20%
20%
0%
0%
0%
Wik odhee
cera E2
caro itt
100K
25k
stop
stup
metri:
lexi First
image Grrst
sbu/Te
hpp
Figure BPR stabilizies training and enables performant models; the first figure shows different performance metrics. The last two show success rates for the MoE router in Layer 9.
4.2
Other ablations
We summarize our other ablations here due to space constraints; details can be found in Appendix E
Router structure (Appendix F.3 Our router is modality agnostic; we experiment with per-modality routers, and separate pools of per-modality experts. We find they all perform comparably generic, modality agnostic setup, but that separate pools of experts by design is more stable and does not require auxiliary losses for regularisation- harder scale to many modalities and tasks.
Increasing selected experts per token (Appendix E.7y the local auxiliary loss to generalise to K > 1; by doing We propose modifications to BPR and steadily increase performance by increasing K, e.g. from 55.5% zero-shot accuracy with K to 61.0% with = 5.
Total number experts (Appendix E.2). We show that increasing the pool of available experts at fixed K improves performance (unlike what was observed for vision-only tasks I
Expert pruning (Appendix E.4). We show using simple heuristics we can prune down to modality- specific experts for unimodal forward passes, thus avoiding expert collapse under unimodal batches.
Training on public data (Appendix E.6) The majority of LIMoE models were trained on propri- etary data [T6]. We show that LIMoE works similarly well publically available data, retaining performance improvements against a comparable dense model.
5
Model Analysis
In this section, explore some of the internal workings of LIMoE. We use simple B/32 and B/16 models with experts, and the large H/14 with 32. See Appendix For further details and experiments.
Multimodal experts arise (Appendix F./ Aside from encouraging diversity, we do not explicitly enforce experts to specialize. Nonetheless, we observe the emergence of both modality-specinf experts, and multimodal experts which process both images and texts (per-expert distributions in F.]
8

---

We analyse some example data and show a clear emergence of semantically meaningful experts. With images for instance, some experts specialize on lower level features (colours, lines) while others on more complex features (faces and text), see Figure
Qualitative analysis (Appendix F.2).
BPR ranking (Appendix (.5). The local loss encourages high max-routing weights for text, and BPR ranks according to this. We show however that this does not mean text is always prioritised first: Especially in later layers, the model often prioritises important image patches over text.
6
Related work
Unimodal, task-specifie neural networks have long been researched, with increasing convergence towards Transformer-bat based architectures [23 [6] for both NLP [27 and Computer Vision 28 29
Multimodal models aim to process multiple types of data using single neural network.
tackle inherently multimodal tasks. LIMoE is more similar to approaches which do not do that, and still operate as unimodal feature extractors. Some co-train on distinct tasks [34 35 36 22) without aligning fusing cepresentations-effccciiiv
Many approaches "fuse" modalities 30 31
32
33
sharing weights across whereas others include both unimodal aspects and fused multimodal aspects for functionality in both contexts
37
We build on deep Sparse Mixture of Experts models, which have been studied independently in Computer Vision [ 2 and NLP [4
3 8. typically in the context of transfer learning. These models use a learned gating mechanism whereby only a subset of K experts out of > K are activated for given input. Many works aim to improve the gating mechanism itself, by making differentiable [38] reformulating as a linear assignment task [39]] or even swapping it out for a simple hashing algorithm [40]. MoE models have also been studied for multitask learning (38), with per-task routers [6] but a shared pool of experts. To our knowledge, sparse models have not been explored for multimodal learning.
but also in supervised regimes [42). Multimodal contrastive learning trains aligned data from multiple modalities. Originally studied for medical images and reports [11]. it was recently scaled to noisy web data [0]]. where strong image- alignments enabled performant image classification and cross-modal image-text retrieval without finetuning on downstream data, Follow up works improved upon this significantly by scaling up and using pretrained models [18] [6] and multitask training with generative modelling (25) or other vision tasks 43). These works use unimodal models which separately process image and text data; we are not aware of previous research using single model to process both images and texts for contrastive learning, neither with dense nor with sparse models.
large body of research exists on contrastive learning, usually in self-supervised
7 Conclusions and Future Work
We have presented LIMOE, the first multimodal sparse mixture of experts model. We uncovered new failure modes specific to this setup and proposed entropy based auxiliary losses which stabilises training and results in highly performant models. It works across many model scales, with average improvements over FLOP matched dense baselines of +10.2% vero-shot accuracy. When scaled to large H/14 model, we achieve 84. accuracy, competitive with current SOTA approaches.
Societal impact and limitations: The potential harms of large scale models [44), contrastive mod- els 7 and web-scale multimodal data (45) also carry over here, LIMoE does explicitly address them. On the other hand, it has been shown that pruming models tends to cause low-resource groups to be forgotten [46), causing performance disproportionally drop for some subgroups. This would be worth considering for our ppperttpruning experiments, but by analogue, the ability to scale models with experts that can specialize deeply may result in better performance on underrepresented groups.
Environmentally speaking, training large models is costly, though efforts are made to use efficient datacenters and offset emitted CO2. Prior works however show that most environmental impact occurs during model inference, and that MoEs are significantly more efficient in that regard [47]; LIMOE is naturally good candidate for efficient, large-scale multimodal foundation models.
Future work: There are many interesting directions from here. The routing interference with multiple modalities still is not fully understood. In general, conclusions from applications of MoEs

---

to NLP have not carried over perfectly to Vision, and vice-versa, and here we see again different behaviour between images and text. Naturally, extensions more modalities should be explored; even with only two we see fascinating interactions between different data types and the routing algorithms, and that will only get more difficult, and interesting, with more modalities.
There are always more modalities to learn, and larger models to build: sparse models provide a very natural way scale up while juggling very different tasks and data, and look forward to seeing more research in this area.
8 Acknowledgements
We first thank Andreas Steiner, Xiao Wang and Xiaohua Zhai, who led early explorations into dense single-tower models for contrastive multimodal learning, and also were instrumental in providing data access. We also thank Andreas Steiner, and Douglas Eck, for early feedback on the paper. We thank André Susano Pinto, Maxim Neumann, Barret Zoph, Liam Fedus, Wei Han and Josip Djolonga for useful discussions, and Erica Moreira and Victor Gomes for help scaling up to LIMoE-H/14.
10

---

References
[1] Carlos Riquelme, Joan Puigeerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems. Curran Associates, Inc., 2021.
[2] Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Cross-token modeling with conditional computation, 2022.
[3] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[4] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus, St-moe: Designing stable and transferable sparse expert models, 2022.
[5] Mark Collier, Et Kokiopoulou, Andrea Gesmundo, and Jesse Berent. Routing networks with co co-training for continual learning, 2020.
[6] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi- task learning with multi-gate mixture-of -fxeeetts....e....... In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018. ACM, 2018
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision. In /CML, Proceedings Machine Learning Research. PMLR, 2021.
[8] William Fedus, Barret 7oph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 23(120), 2022.
[9] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image datahase. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA. IEEE Computer Society, 2009.
[10] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-languan representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings Machine Learning Research. PMLR, 2021.
[11] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning. and Curtis P. Langlotz. Contrastive lcarning of medical visual representations from paired images and text. CoRR, 2020.
[12] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: Syxtem Demonstrations, Brussels, Belgium, October November 4, 2018, Association for Computational Linguistics, 2018
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
[14] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The carrelyyggtted mixture-of-expertt layer. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[15] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[16] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. CVPR, 2021.
[17] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CVPR, 2021.
[18] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu. Yi- Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le. Combined scaling for open- vecablocab image classification, 2022.

---

[19] Richard Socher, Milind Ganjoo. Christopher D Manning. and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2013.
[20] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll' ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, Lecture Notes in Computer Science. Springer, 2014.
[21] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblithh, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. CoRR, 2022.
[22] Scott Reed. Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikow, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky. Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
[24] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herv'e J'egou, Fixing the train-test resolution discrepancy. In Advances Neural Information Processing Systems (NeurlPS), 2019.
[25] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022.
[26] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: bidirectional transformers for language understanding. In NAACL, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Pre- trainian of deep
[27] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Sure. 2022.
[28] Liu, Yutong I lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[29] Kishaan Jeeveswaran, Senthilkumar Kathiresan, Arnav Varma, Omar Magdy, Bahram Zonooz, and Elahe Arani. comprehensive study of vision transformers dense prediction tasks. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2022, Volume 4: VISAPP, Online Streaming, February 6-8, 2022. SCITEPRESS. 2022.
[30] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
[31] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2020.
[32] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining aask-agnostic visiolinguistic representations for lannauugeee tasks. In NeurlPS. Curran Associates, Inc., 2019.
[33] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. CoRR, 2019.
[34] Valerii Likhosherstov, Anurag Amah, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. CoRR, 2021.
[35] Hassan Akbari, Liangzhe Yuan, Rui Qian, Woi-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021.
[36] Qing Li, Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du Ming Hsuan Yang, and Matthew Brown. Towards a unified foundation model: Jointly pre-training transformers on unpaired images and text. arXiv preprint arXiv:21 2021.
12

---

[37] Jianfeng Wang, Xiaowei Hu. Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for vision-language representation learning. CoRR, 2021.
[38] Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong. and Ed H. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021. virtual, 2021.
[39] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simpli- fying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 Jul 2021, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2021.
[40] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, 2021.
[41] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2020.
[42] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[43] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang. Jianfeng Wang. Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng. Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. CoRR, 2021.
[44] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel. Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus. Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky. Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and al. On the opportunities and risks of foundation models. CoRR, 2021.
[45] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, porno- raphy, and malignant stereotypes. CORR, 2021.
[46] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily L. Denton. Characterising bias in compressed models. ArXiv, abs/2010.03058, 2020.
[47] David A. Patterson, Joseph Gonzalez, Quoe V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. CORR, 2021.
[48] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.
[49] Christoph Schuhmana, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk. Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, LAION-400M, open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114 04,14,2220011 2021.
[50] Steven Bird, Edward Loper, and Ewan Klein. NLTK: the natural language toolkit. In ACL 2006, 21st International Conference Computational Linguistics and 44th Annual Meeting the Association for Computational Linguistics. The Association for Computer Linguisties, 2006.
13


Reviews:
[{"review_id": "sIMwbn_zz63", "reviewer": "Reviewer_g9Jj", "summary": "The paper addresses an important problem, in which multimodal data is required for a model, by designing a unified architecture that accepts images and text concurrently with the use of the contrastive learning method for multimodal representation, trained on a paired Image-Text dataset. In addition, the paper addresses load balancing, which is one of the most important problems in the MoE setting, with an entropy-based regularization technique. ", "questions": "I am not sure if I miss/misunderstand something including the Strengths/Weaknesses above. Still, it is not clear to me how the routing for text and images is done.  Using image-text pairs, it is said that it’s possible that all text can go to a single expert while image tokens are distributed almost equally. So they are routed not in pairs but individually (which is somewhat not similar to the illustration in Figure 1). Please explain why and why not are you separate the 2 modalities at routing? \n\n\nAnother somewhat related question is that for contrastive training, how do you identify the negative examples? \n", "limitations": "\nMaybe not very relevant since the paper addresses the system-related level and thus is hard to judge those impacts. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper is well written. \n2. The paper offers many studies that are useful for understanding many aspects it mentions.\n \nWeakness:\n1. The paper claims to be the first multimodal sparse MoE model, which is probably incorrect/too strong (line 299), e.g. [0] was first submitted in Nov 2021. \n2. The fact that this paper centers around image-text data leads to the need of showing results of image-text benchmarks e.g. in [0], while the paper focuses on zero-shot. Even so why the main results in Table 1 doesn’t have other results for other scales of the model (likewise, if you have an even larger model having comparable parameters, would it beat the performances of those baselines?). For tasks other than Imagenet classification, it would be more convincing if the paper presents the comparison with some current baselines, in addition to the ablations. \n3. Some acronyms might need further explanation such as the ones in Figure 3 for different models, e.g. B.16, S.16, … \n4. No code is provided. \n\n\n[0] VLMO: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2111.02358.pdf (arxiv.org)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TyKsUg5SFtl", "reviewer": "Reviewer_Lcgn", "summary": "This paper proposes a new multimodal contrastive learning framework named as LIMoE. This is the first large-scale mixture-of-experts model for multiple modalities. Specifically, this paper designs a modality agnostic model which is not explicitly conditioned on modality. To address the training stability issue and to balance expert utilization, two entropy-based regularization losses are introduced. The effectiveness of this proposed model is quantitatively and qualitatively evaluated across multiple scales and network architectures.", "questions": "1. Table 2: the caption contains two “for” which should be a typo\n2. It is suggested to use “zero-shot” rather than “0shot”\n", "limitations": "Not applicable", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\n\nOriginality: the design of the multimodal mixture-of-experts (MoE) model is novel. To the best of my knowledge, previous MoE models are designed for single modality. The two regularization losses are widely used in previous studies, but the combination with MoE is valuable.\n\n---\n\nQuality: \n\nStrengths: this work is technically sound. The claims are well supported by comprehensive empirical studies. The proposed regularization losses are reasonable and are expected to stabilize the training, and in turn bring performance improvement. \n\nWeakness: The two proposed losses (as shown in Equation 2), which are also known as information maximization loss, are widely used in previous studies. It is quite straightforward to use this loss to mitigate the collapse issue in expert utilization. Therefore, this work is more like an experimental study.\n\n---\n\nClarity: this paper is well written and well organized. It is easy to follow. \n\nSignificance: this work is a good effort in introducing MoE into multimodal contrastive learning. Although the proposed solution is trivial, it brings some insights to the community. \n\n---", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ArHoIMwFjFG", "reviewer": "Reviewer_Ycx8", "summary": "Along the line of mixute of experts that add expert layers to the Transformer architecture (e.g., ST-MoE in NLP and Vision MoE for vision), this paper introduces Language+Image MoE. Similar to other MoE, the single FeedForwardNetwork is replaced by an expert layer that contains many parallel FFNs, each of which is an expert. The difference is that given a sequence of tokens to process, a simple router learns to predict which experts should handle which tokens. ", "questions": "> 1) Why In an imbalanced context all of the tokens from the minority modality get assigned to a single expert?\n>2) some effects such as the behavior of global entropy threshold (text and image) are attributed to the imbalance dataset. How different would it be in the case of a balanced dataset?", "limitations": "-", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Model size can increase while keeping computational cost constant (compared to BASIC the number of params is almost half for LIMoE H/14, while number of params per token- for inference- is less than 50%)\n- As a sparse model, LIMoE avoids negative interference and catastrophic forgetting.\n- Proposed new auxiliary loss(local/global entropy loss) and routing prioritization to prevent sending all tokens [from different modalities] to the same expert.\n- Studing the scaling of LIMoE and ablations for various design decisions, e.g. loss, router architecture, number of experts, etc. \n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept