Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies efficient sparsely activated transformers. The authors expand the search space of transformers with a few mixture-of-experts (MoE) design choices: i.e., the number and size of experts after each transformer block. Then, the authors apply weight-sharing super network training and evolutionary search to explore the best sparsely activated MoE architecture under the given resource constraints. The proposed AutoMoE has achieved moderate #FLOPs reduction compared with the dense transformer.
strength_and_weaknesses: ---
Strengths:
- The paper is very well-written and easy to follow.
- The proposed framework is technically sound and performs well on machine translation benchmarks.
- The paper provides sufficient implementation details (e.g., design space configuration, and search and training hyperparameters). The submission also comes with a code implementation, which will facilitate other researchers' reproduction.

---
Weaknesses:
- The technical novelty of this paper is a bit limited. The core contribution lies in the search space design since the other components (i.e., super network training and evolutionary search) are standard (used in most neural architecture search papers). However, the authors adopt most of their search space designs from HAT except those related to MoE. The support for elastic MoE also seems straightforward (basically the same as FFN). Though I like the idea of incorporating MoE into the transformer's design space, I am not confident whether these technical contributions are sufficient for publication.
- The experimental evaluation needs improvement:
  - Despite mentioning the baseline with homogeneous experts in the text, the authors have not provided the corresponding results in Table 4.
  - The authors have only presented #FLOPs in Table 4. However, #FLOPs reduction does not necessarily translate into measured speedup.
  - The authors have compared AutoMoE with manually-designed baselines in Table 1. However, I suspect the shallower encoder contributes the most to the latency and #FLOPs reduction.

---
clarity,_quality,_novelty_and_reproducibility: This paper has good clarity, fair quality but limited novelty.
summary_of_the_review: My current recommendation is primarily based on the limited novelty of this paper. However, I would love to see the authors' response before making the final recommendation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on neural machine translation and develops a new NAS approach. Specifically, the authors propose a transformer-based search space by incorporating the MoE module into it. Nevertheless, the novelty seems very limited since the proposed method can be regarded as a direct application of HAT on MoE architectures. Moreover, the performance improvement over HAT is marginal.
strength_and_weaknesses: Strengths:
1. This paper develops a new NAS approach by incorporating MoE module into the search space
2. This paper provides sufficient details of the proposed search space as well as interesting analysis.


Weaknesses:

1. Compared with HAT, the novelty of this paper seems very limited. If I understand correctly, the essential difference from HAT is including the MoE module into the search space. Besides this, all the others (including supernet training, evolutionary search algorithm) remain exactly the same as HAT.

2. How do the authors evaluate latency in Table 1? If the latency is measured based on a single sample, the practical latency for a mini-batch may be very slow since different sub-networks are activated for different inputs. Thus, it would be better to provide the comparisons of latency on both single sample and a mini-batch.

3. The performance improvement over HAT seems very marginal in Table 4.

4. In NAS papers, random search is always a strong baseline. Thus, it would be better to include the comparisons with random search in Table 4.

5. In Table 5, this paper only shows the results of the proposed method. It is interesting to see the comparisons between the proposed method and HAT, which is also able to produce architectures under specific constraints. 

6. Does AutoMoE also need to train a latency predictor like HAT? If not, how to efficiently obtain the latency for candidate architectures during the search process?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written. However, the novelty is limited compared with HAT.
summary_of_the_review: The novelty is limited since the proposed method can be regarded as a direct application of HAT on MoE architectures. Moreover, the performance improvement over HAT is marginal.
Maybe I missed something important. If there are some other essential differences, I am happy to change the rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces AutoMoE, an AutoML framework for searching efficient sparsely activated models. Experiments are conducted on three machine translation benchmark datasets, including WMT'14 En-De, WMT'14 En-Fr, and WMT'19 En-De. Compared with previous models (e.g., HAT), AutoMoE achieves a better trade-off between FLOPs and BLEU scores.  
strength_and_weaknesses: Strength:
1. This paper is well-written. 
2. The practical improvements over previous methods are clear, and the search cost is affordable. 

Weakness:
1. 1. I feel this work is a straightforward application of existing NAS methods on MoE with some engineering efforts. The novelty of the proposed framework is limited. According to the experiments, the improvements are not surprising, given that the search space and the overall framework are much more complicated than previous methods. 
2. In addition to FLOPs, the GPU latency of the models should be included in Table 4. 

clarity,_quality,_novelty_and_reproducibility: I think this paper is well-written and easy to reproduce. 
summary_of_the_review: I have some concerns regarding the novelty of this work. And it is unclear to me whether AutoMoE is better than previous models when comparing their efficiency on hardware (e.g., GPU).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers find the contributions in the paper limited and straightforward application of existing methods. Further the improvements obtained by the costly NAS search seem marginal over standard baselines. Author's response didn't change reviewers mind and I also agree with reviewers that the current submission looks like a straightforward application of existing results. Overall the current submission is weak and will need a significant revision before resubmission. Hence I recommend rejection.
justification_for_why_not_higher_score: Limited contributions.
Decision: Reject