Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a scaling strategy for vision transformer using a widely-used approach, Mixture of Experts (MoEs) on Feed-Forward Networks. The simple scaling strategy could achieve a high performance of 90.2% top-1 accuracy with a 15B parameter vision transformer. Interestingly, V-MoE merges FFN-MoE with a spatial dynamic computation algorithm, to decide whether each patch would be computed in each layer and reduce the computation in the inference stage with few performance drop.
main_review: For originality, this paper extends the existing techniques (MoEs) in NLP to Vision Transformer, and also studies how to scale up vision transformer in an empirical manner. The combination of MoE and spatial dynamic computation is interesting and smart. The experimental study of this paper is valuable to the community.

The quality of this paper is good. From an experimental perspective, this paper contains sufficient experimental results, verifies important claims, and achieves state-of-the-art performance on ImageNet even with a significantly large number of parameters.

Concerns:
1. Some study has found many of the labels of ImageNet validation set are incorrect (“Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks”). I wonder whether the 90.2% accuracy is truly correct or somewhat overfit the validation set. Or it would be better to see the real upper bound of the accuracy on the ImageNet validation set or how the models perform on ImageNet-V2 validation set.
2. Data leaking of the Imagenet-1K validation data is also a potential issue since the de-duplication method is not so reliable when the data number is extremely large (e.g. 3B).
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper trained the current largest vision model through MoE techniques. It achieved strong top-1 accuracy on ImageNet-1K image recognition, at 90.25%. It also propose batch prioritized routing. When working together with a C << 1, it can further largely reduce the computation.


main_review: Originality

-- Sparse MoE has proved working well in NLP, and so it is not strongly original. Nevertheless, I see the achievement of successfully adapting this model to the field of computer vision as adequate originality. To the technical part itself, batch prioritized routing is novel, and seems generalizable to other problems. It will be interesting to see its performance in NLP tasks.

Quality

-- The designs seem reasonable and make sense to me. The experiments are also rich. It is a submission of high-quality.

Clarity

-- very clear and well written.

Significance

-- Strong as I can tell. The difficulty of deployment may block its higher interest to the community. Nevertheless, this is a problem of the sparse MoE technique, but not refers to the specific vision applications.
limitations_and_societal_impact: -- The sparse MoE algorithm is efficient in inference only when run in batch mode. When there is only one sample or a few samples, it will be unpractical. This could prevent its further prevalence.

-- In fig.3, it seems meet overfitting problem when the model is as large as H/14. Can the authors explain more on it please?

-- In Line 86-88, it gives a impression that the routing after softmax is novel. However, Gshard and Switch Transformer have used this technique. It is better to refer to Gshard and Switch Transformer when introducing this.

- The expert number is fixed as 32. What is the performance at more experts, e.g. 64, 128, 256, or less experts, e.g. 16, 8 and etc.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper extends the sparsely-gated mixture of experts networks for NLP to the applications in the computer vision. A batch prioritized routing technique is proposed to discard the least useful patches to reduce the buffer size and computational cost. From the experimental results, the proposed method has shown improvements in large vision models and representation learning.
main_review: 1. Originality:  The paper is an extension of the works of "sparsely-gated mixture-of-experts layer", "switch transformers”， and "scaling giant models". A batch prioritized routing technique is proposed to improve the performance. The work is more technically sound than the novelty.

2. Quality: The work is technically sound. The experimental results are sufficient and shown improvements in large vision models and representation learning.

3. Clarity: The paper is well written and organized. The paper is easy to follow.

4. Significance: The result has referential significance. It is not a new idea, but the work is technically sound.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an approach to scale vision transformers by using sparse mixture of experts. The idea is simple and straightforward---using a subset of experts defined by a topk operation. The authors conducted sufficient experiments on ImageNet and the experiments demontraste the approach can achieve a nice tradeoff between accuracy and comptutational cost.
main_review: Strengthes:
1. The idea is simple and straightforward.
2. Sufficient experiments are analysis are provided.

Weaknesses:
1. The presentation of this paper could be further improved. For instance, Figure 1 is not intuitive and straightforward. In L81, the authors defined the rounting function $g(x) = Top_k(softmax(Wx + \epsilon))$. What does $x$ mean here? a patch representation? In 208, the routing function is defined to operate on $X$. So the notations throughout the paper are rather confusing.

2. How do you optimize top_k since this would be a non-differentiable operation?

3. In figure 1, what does device mean? different gpus? different computation nodes?

4. The authors mentioned that "The difference between previous formulations [38] is that we apply TOPk after the softmax over experts weights, instead of before". If you select top k experts and only use them for computation, you can save computation. I am not sure how could you save computation if you use all experts first, and then select topk experts?

5. It would be better to explictly discuss wow the MoE layer is different from a standard MLP, e.g., describing their weight dimensions.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: All the reviewers agree that this submission made a significant contribution to the community. The combination of MoE and spatial dynamic computation is interesting and smart. The experimental study of this paper is valuable to the community. Although combining MoE to transformer has been studied in the NLP community and thus the originality of this submission is kind of limited, the submission still makes valuable contributions as mentioned by the reviewers. AC has read the submission, reviews and discussion and agrees with the reviewers on their recommendation. 
Decision: Accept (Poster)