Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Along the line of mixute of experts that add expert layers to the Transformer architecture (e.g., ST-MoE in NLP and Vision MoE for vision), this paper introduces Language+Image MoE. Similar to other MoE, the single FeedForwardNetwork is replaced by an expert layer that contains many parallel FFNs, each of which is an expert. The difference is that given a sequence of tokens to process, a simple router learns to predict which experts should handle which tokens. 
strengths_and_weaknesses: Pros:
- Model size can increase while keeping computational cost constant (compared to BASIC the number of params is almost half for LIMoE H/14, while number of params per token- for inference- is less than 50%)
- As a sparse model, LIMoE avoids negative interference and catastrophic forgetting.
- Proposed new auxiliary loss(local/global entropy loss) and routing prioritization to prevent sending all tokens [from different modalities] to the same expert.
- Studing the scaling of LIMoE and ablations for various design decisions, e.g. loss, router architecture, number of experts, etc. 





questions: > 1) Why In an imbalanced context all of the tokens from the minority modality get assigned to a single expert?
>2) some effects such as the behavior of global entropy threshold (text and image) are attributed to the imbalance dataset. How different would it be in the case of a balanced dataset?
limitations: -
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses an important problem, in which multimodal data is required for a model, by designing a unified architecture that accepts images and text concurrently with the use of the contrastive learning method for multimodal representation, trained on a paired Image-Text dataset. In addition, the paper addresses load balancing, which is one of the most important problems in the MoE setting, with an entropy-based regularization technique. 
strengths_and_weaknesses: Strength: 
1. The paper is well written. 
2. The paper offers many studies that are useful for understanding many aspects it mentions.
 
Weakness:
1. The paper claims to be the first multimodal sparse MoE model, which is probably incorrect/too strong (line 299), e.g. [0] was first submitted in Nov 2021. 
2. The fact that this paper centers around image-text data leads to the need of showing results of image-text benchmarks e.g. in [0], while the paper focuses on zero-shot. Even so why the main results in Table 1 doesn’t have other results for other scales of the model (likewise, if you have an even larger model having comparable parameters, would it beat the performances of those baselines?). For tasks other than Imagenet classification, it would be more convincing if the paper presents the comparison with some current baselines, in addition to the ablations. 
3. Some acronyms might need further explanation such as the ones in Figure 3 for different models, e.g. B.16, S.16, … 
4. No code is provided. 


[0] VLMO: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2111.02358.pdf (arxiv.org)

questions: I am not sure if I miss/misunderstand something including the Strengths/Weaknesses above. Still, it is not clear to me how the routing for text and images is done.  Using image-text pairs, it is said that it’s possible that all text can go to a single expert while image tokens are distributed almost equally. So they are routed not in pairs but individually (which is somewhat not similar to the illustration in Figure 1). Please explain why and why not are you separate the 2 modalities at routing? 


Another somewhat related question is that for contrastive training, how do you identify the negative examples? 

limitations: 
Maybe not very relevant since the paper addresses the system-related level and thus is hard to judge those impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new multimodal contrastive learning framework named as LIMoE. This is the first large-scale mixture-of-experts model for multiple modalities. Specifically, this paper designs a modality agnostic model which is not explicitly conditioned on modality. To address the training stability issue and to balance expert utilization, two entropy-based regularization losses are introduced. The effectiveness of this proposed model is quantitatively and qualitatively evaluated across multiple scales and network architectures.
strengths_and_weaknesses: ---

Originality: the design of the multimodal mixture-of-experts (MoE) model is novel. To the best of my knowledge, previous MoE models are designed for single modality. The two regularization losses are widely used in previous studies, but the combination with MoE is valuable.

---

Quality: 

Strengths: this work is technically sound. The claims are well supported by comprehensive empirical studies. The proposed regularization losses are reasonable and are expected to stabilize the training, and in turn bring performance improvement. 

Weakness: The two proposed losses (as shown in Equation 2), which are also known as information maximization loss, are widely used in previous studies. It is quite straightforward to use this loss to mitigate the collapse issue in expert utilization. Therefore, this work is more like an experimental study.

---

Clarity: this paper is well written and well organized. It is easy to follow. 

Significance: this work is a good effort in introducing MoE into multimodal contrastive learning. Although the proposed solution is trivial, it brings some insights to the community. 

---
questions: 1. Table 2: the caption contains two “for” which should be a typo
2. It is suggested to use “zero-shot” rather than “0shot”

limitations: Not applicable
review_title: null

Meta-review: metareview: The authors use a mixture-of-experts model in a multimodal setting.  the reviewers consider the work technically strong and interesting; the AC concurs.
confidence: Certain
Decision: Accept