Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper initiates an interesting exploration of MAE with mixture of experts (MoE). The method is quite well-motivated, with interesting and fairness-in-mind designs, and a good amount of experiments devoted to it. The final results are reported using a suite of 11 downstream classification tasks, typically used to evaluate self-supervised learning methods. The proposed method is shown to significantly improve the overall accuracy, while maintaining efficiency.
strength_and_weaknesses: Strengths:
- This is one of the first works that I am aware to study MAE and MoE jointly. The exploration on this direction is interesting and of significance.
- The paper motivates the approach by pointing to an empirical result of MAE suffering from the negative transfer phenomenon, and the final approach (guided by the motivation) is able to significantly outperform the MAE baseline. Overall it looks like a healthy research cycle to me.

Weaknesses:
- One major concern I am having with MoE-kind approach is about larger models. While used in a sparse way, MoE is still having a lot of parameters, and if directly comparing against MAE of the same backbone, it does not sound too fair to me. So I would like to see two things: 1) with a larger MAE model that roughly has the same number of parameters as a model with MoE, what's the comparison? and 2) Whether the improvements of the current model can transfer to larger models. Related: in table 3, I am not able to find the column for MAE. All I find are columns for TokenMoE and MoCE, while the caption says MAE is under comparison.
- Related to the above, I am not sure the "negative transfer" phenomenon still exists with larger and larger MAE models? The hypothesis here is that maybe with larger models, it can capture both the man-made objects and the natural ones from ImageNet?
- Maybe it is demanding too much computation overhead, but I noticed that the method in the paper is built from pre-trained MAE, and not training from scratch.  So I am wondering what would MoCE be like when pre-trained from scratch?
- For the distillation loss, I am not sure why it still maintains an efficiency advantage over TokenMoE because the "whole" network is used as a "teacher", and presumably it can be costly to forward through all the parameters.
- (minor as I don't know where to put it in the review) Why does MoCo v3 achieve so good results on Table 1? Given that MoCo v3 is better than MAE, isn't it more reasonable to start MoE explorations from MoCo v3 given the results listed in the table? In the same table, I am also curious to see what the supervised ViT will achieve on these downstream datasets. 

clarity,_quality,_novelty_and_reproducibility: Clarity: Overall the paper is a good read, and it is reasonably clear. One thing not clear to me: for data clustering, what features do it use? I am thinking the features are important, and if they are computed online, their FLOPs should also be counted? One could compute it offline, but it is still a computation budget over there if we want to compare overall pre-training cost fairly. Another (potentially minor) thing: for PSNR comparison in Fig 4, is it evaluating the reconstruction quality of the pre-training (auto-encoding) task? Or is it something else?

Quality: I think this is a useful exploration marrying MAE and MoE. The paper has good illustrations, nice storyline, and sufficient amount of experiments. One pity is that the MAE w/ MoE model is not pre-trained from scratch, and this may cause some noise in the signals, but the presented work is done with high-quality.

Novelty: While the techniques in the paper (how to do clustering, how to do gating in MoE) do appear to be existing ones, I think the work has directional novelty in exploring MAE w/ MoEs.

Reproducibility: If there is no difficult constraints, I hope the code of the paper is released to facilitate reproducing the results.
summary_of_the_review: I would be on the acceptance side for the paper given the pros and cons listed above. I hope the authors can address the clarification questions I have, and try to tackle the empirical study (e.g. with larger models) as best as they can.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the negative transfer phenomenon of MAE pre-trained models. Authors first experimentally show the existence of negative transfer and then show that natively using MoE can not solve the problem. Then, a clustering-based method is proposed to solve the problem.
strength_and_weaknesses: Generally, the paper is well-organized and easy to follow. The research topic is relatively new and interesting. The proposed MoCE seems correct and is well supported by the experimental results. 
However, I also have the following concerns:
1. In the abstract and introduction, the authors position this work as a pretraining method. However, MoCE is based on a pre-trained MAE. Considering the extremely heavy cost of training foundation models, it would not be feasible. 
2. Based on 1, the cost of training MoCE would be twice of MAE. To make a fair comparison, what if we train MAE with a dataset of twice scale or train MAE with more epochs?
3. If we focus on the negative transfer phenomenon, not combining MoE, there would be other transfer learning and model adaptation methods (e.g., prompt tuning) that could ease the negative transfer phenomenon. Could you compare it with some of them?
4. Since the input of Eq.5 is cluster centroid and there are only m clusters, is it enough to learn W_g well?
5. For Eq.6, is the loss implemented over all samples in the dataset or a batch? 
In the experiment, only 11 small-scale classification tasks are evaluated, how about other more important tasks, e.g., object detection, semantic and instance segmentation?
clarity,_quality,_novelty_and_reproducibility: Generally, the paper is presented clearly. The method should be original.
summary_of_the_review: I like the idea but believe the current experiments are not sufficient. If my concerns are solved, I will be willing to raise the score.


------ After rebuttal --------
In general, some of my concerns have been solved, e.g., training from scratch, and transferring to det and seg tasks. Thus, I would like to raise my score to borderline accept considering the interesting design of MOE.

However, I still have some concerns and hope the authors can address them in the final version. First, the added supervised pertaining results remind me that comparing MoCE with [1] would be necessary, which shows that simple MLPs can significantly improve the transferring performance of supervised pretraining.
Second, as you claim/argue the method is pretraining not transferring, it would be better to update all the results of MoCE with training from scratch instead of based on MAE to avoid confusion. This issue has also been pointed out by other reviewers (I believe it's feasible for the authors to train MoCE 1600 epochs from scratch since they already show some results about 200 epochs).

[1] Revisiting the Transferability of Supervised Pretraining: an MLP Perspective, CVPR 2022
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates practical techniques to apply the Mixture of Experts (MoE) model to the MAE-based pre-training paradigm. 

Taking a supervised MoE model for ViT as a baseline, the authors introduced data clustering as a strong baseline for experts' partition and assignment. 

Several additional terms, such as imbalance regularization and distill loss, are also proposed in the self-supervised MoE design. 

On 11 downstream tasks, the proposed method outperforms MAE, supervised TokenMoE and several recent self-supervised single model methods (non-MoE). 

strength_and_weaknesses: # Strength
- This paper proposed practical techniques to address the expert collapse issue in TokenMoE. 
- The proposed MoCE improves the baseline performance on 11 downstream tasks, showing the effectiveness of the proposed method. 
- Reasonable visualization of the expert routing and expert visualization is shown in the experiment. 
- Several ablation studies are conducted in Table 4-7. 


# Weaknesses
- Although better performance is achieved on the benchmark dataset, the technical contribution is kind of incremental. The main algorithm re-uses multiple design components of (Liu et al., 2022) and applies them to a slightly different scenario, i.e. MAE self-supervised training with a Mixture of Experts. Compared with (Liu et al., 2022), the improvements are (1) exploring the new self-supervised paradigm MAE, (2) applying to the Transformer backbone, and (3) different routing mechanisms with MoE. Both (1) and (2) are straightforward extensions to new technologies. As to (3), it offers some insights and benefits on self-supervised pre-training. 
- As mentioned above, the proposed method heavily relies on design modules from (Liu et al., 2022), e.g. Data Clustering (Eq. (3)), Distillation term. However, most of the methodology and experimental comparison are conducted with respect to TokenMoE, a specific supervised MoE model. As also verified by the authors in Figure 3(a), TokenMoE is not specifically designed for the setting of self-supervised pre-training. Therefore, more comparisons and discussions should be made w.r.t. (Liu et al., 2022) rather than TokenMoE. 
- TokenMoE used a load loss to balance the expert assignment, so it looks a little strange in Figure 3(a) the expert collapsing. 
- In Table 3, the results for MAE are missing. 
- It seems the results of DeepClusterV2 and BYOL in Table 1 are different from the results from (Liu et al., 2022), both on Resnet 50. 
- Other minors:   
> "decades(Jacobs et al.," and "processing(Lepikhin et al., 2020", missing space before left parentheses.   
> Equ. 5 is not so common, usually Eqn. 5 or Eq. 5.   
> In Eq. (3), $Tr(Q^TA), Q\in \mathbb{R}^{n\times m}, A\in \mathbb{R}^{n\times m}$, please check if this is correct. A more common notation for Sinkhorn is using matrix product $<Q, A>$.   

clarity,_quality,_novelty_and_reproducibility: This paper is based on two previous works, i.e. (Liu et al., 2022) and TokenMoE. It offers some insights into the Mixture of Experts on MAE pre-training. The overall technical novelty is kind of incremental. 

There is no code attached to this paper, but it will be possible to reproduce the results based on the descriptions and experimental details. 



summary_of_the_review: Although this paper obtains improved results over MAE and other self-supervised methods, the technical contribution is kind of incremental w.r.t (Liu et al. 2022) and TokenMoE.   
And there is a lack of discussion and comparison w.r.t (Liu et al. 2022)   

Overall, this paper is below the acceptance threshold. 

------------------------------
After Rebuttal:   
The authors addressed most of my concerns and made significant efforts. I would like to see this paper accepted and increased the overall score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
The paper has a good motivation of transferring a pretrained model into diverse tasks by tackling the problem of negative transfer. The solution is having an abstraction layer of semantic, where input is grouped into different clusters based on their semantics before routing to experts. Experiments show a very good improvement of 2.54% on ImageNet. 

strength_and_weaknesses: Strength: 1/ Having a good motivation on an important application, where we should take advantage of pretrained models to save cost.  2/ The testing efficiency is really good. 

Some Weaknesses

1/ About cost saving: the proposed solution has a non-negligible overhead of data clustering, which is itself an optimization problem. My understanding is this overhead is not included in the training analysis (please correct if I am wrong). 

2/ It’s not entirely clear about the location and number of MoCE layers chosen even with Fig 4b and Table 6,  e.g. the intuition as to why placing them on, say, 10th and 12th layers are good but not before that? This is also along with the claim of hierarchical learning by the model concerning Fig 4b. Why don’t we try a couple of other patterns such as interleaving odd and even layers, or early in the layers while the input is already arranged in clusters, which is somewhat more intuitive? 

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and easy to follow. 

It’s somewhat surprising to see TokenMoE is much lower than MAE (Table 2). Do you have any intuition as to why? Taking a look at, e.g. Fig 3a, something might happen with TokenMoE, leading to imbalancing. 

Any reason why not using, say, t-SNE to visualize the clusters? 

No code or appendix is provided so it’s hard to judge the reproducibility. 

summary_of_the_review: 
The paper motivates a good problem in transferring a vision task and proposes a good solution in which semantics is used to train the mixture-of-expert model, and perform well with diverse tasks. Although there are some parts that need more clarification, generally it’s a good and well written paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper improves the Masked Auto-encoders (MAE) with a Mixture of Cluster-conditional Experts (MoCE) approach. The key finding of this paper is that the semantically irrelevant pre-training information might result in negative transfer, even for larger models pre-trained with bigger datasets. Note that negative transfer is a very important problem for pre-trained models, and this paper tackles this problem with a reasonable solution -- introducing the clustering information into the MoE-based backbone. Evaluation shows significant empirical benefit compared with the close baselines MAE and TokenMAE.

The paper initially received four insightful and constructive review reports, though of mixed ratings. Authors made a big effort in the rebuttal phase, by providing updated version with most requested results (larger models, fair model size, ablations, etc.) augmented to the paper and the response letters. Reviewers actively interacted with the authors, acknowledged the updates, and converged finally to unanimous positive recommendation. 

AC concurs with the four reviewers' opinion on the paper. Authors are requested to incorporate all rebuttal material into the future version, in particular, add train-from-scratch results which were not provided in the rebuttal phase due to time and resource constraint.
justification_for_why_not_higher_score: The current paper addresses an important problem, negative transfer of pre-trained models to downstream tasks. The paper follows a healthy cycle of research, as mentioned by one of the senior reviewers, that pinpoints true research problem followed by a systematic study based on well-established baselines and benchmarks. Therefore, AC believes this paper shall be highlighted as a spotlight at the conference. However, an oral is not appropriate, not only because of the limited quota of orals, but also because the current methodology is mainly built upon MAE and MoE, which is not so pronounced compared to MAE, MoCo, and SimCLR.
justification_for_why_not_lower_score: All reviewers agree to accept this paper unanimously, some with favorable thoughts of the paper. To make a fair decision, AC finds no evidence to assign a lower score for this good paper.
Decision: Accept: notable-top-25%