Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an improved training strategy for oneshot NAS supernetworks. The key idea is to view the training of each subnetwork as a "task", and then to apply an MAML/Reptile-style meta-learning scheme to ensure efficient cross-task adaptivity. Experiments on NAS-Bench-201 and ImageNet show improved calibration between the supernet’s predictions and the architectures' true rankings.
main_review: Strengths:

The paper is in general well-written and easy to follow. The idea is conceptually simple and part of the results (especially the ones related to Kentall’s tau) are looking promising.

Weaknesses:

Results in Table 4 are supposed to be critical as they aim to verify the effectiveness of the proposed MAML-style supernet training strategy. However, they're not yet fully convincing because some key ablation studies are missing, such as the results without any adaptation step. It would also be interesting to see whether the performance would saturate as we increase the number of adaptation steps further above 4.

I also believe the search space of NAS-Bench-201 is probably too small to make any robust conclusions. While the authors did conduct ImageNet experiments as additional evidence, results there do not appear strong as compared to existing methods. For example, the new method achieved only 0.3% improvement over FairNAS on ImageNet at the same MAdds (Table 3) despite substantially improved Kendall’s tau (Table 2). This makes me wonder whether Kendall’s tau is the right metric to measure the effectiveness of the proposed method. One hypothesis that might explain those observations is that the method indeed improved rank correlations for the vast majority of the "average" architectures but not the calibration for the small number of high-performance ones. An analysis on this direction would be helpful.

summary_of_the_review: While I'm not fully convinced by all the experimental results (see comments above), this is an overall interesting paper because the method is conceptually simple and the results are looking promising (at least in terms of rank correlations).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addressed the multi-model forgetting problem in supernet training by a supernet learning strategy based on meta-learning. The evaluation on the NAS-Bench-201 and MobileNet-based search space demonstrated an improved ranking correlation and promising performance.
main_review: Strengths:
1.	The paper is well-written and the analysis gives insight.
2.	It addressed an important problem——multi-model forgetting problem by an elegant algorithm based on meta-learning.

Weakness:
1.	The top-1 performance (Table 1 and Table 3) is not impressive although the ranking correlation seems to be improved. 
2.	The training time of supernet should also be listed for comparison.
3.	The comparisons are based on less competitive baselines (e.g., I am curious about the performance of KD based method, like BigNAS (https://arxiv.org/abs/2003.11142), Cream (https://arxiv.org/abs/2003.11142), etc.


==========================after rebuttal====================
I appreciate the authors taking the time, attempting to address the comments through more details and new experiments. After reading the authors' response and revisions, I think my concerns are almost addressed. If the authors could present the comparison of training (search) time in the main Table, that will be better. I lean to accept this work. 
summary_of_the_review: This paper is well-written and the analysis is insightful. However, some related papers are not taken in the comparison, such as BigNAS and Cream.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on improving the predictive ability of supernet in one-shot NAS, which considers a meta-learning strategy to tackle the multi-model forgetting issue. Through utilizing MAML to minimize the expectation of the loss across sub-models, the supernet can learn unbiased meta-features to improve the ranking ability.
main_review: This paper utilizes a simple while interesting meta-learning strategy to train the supernet in one-shot NAS, and the experimental results on the NAS-Bench-201 and MobileNet show the effectiveness of the proposed method. However, it seems that the proposed method is a direct application of MAML on one-shot NAS. More importantly, the experimental results can hardly support the claim. For example, although the proposed method achieves a high Kendall Tau value with 0.84 on NAS-Bench-201, we should notice that this search space is very simple, and even without any training, the zero-cost NAS [1] can achieve a similar Kendall Tau value. In addition, paper [2] also consider meta-learning for the neural architecture search.

The marginal improvements in experimental results could hardly convince me that the proposed paradigm can benefit the one-shot NAS. I am confused on the right of Figure 3, where the proposed method SUMNAS achieves 79.79+-9.43 accuracy while Table 3 only achieves 77.3. Please clearly state the difference between the two settings.

[1] Zero-cost Proxies for Lightweight NAS
 [2] M-NAS: Meta Neural Architecture Search
summary_of_the_review: The idea is not so surprising and the experiments can not convince me. I tend to reject this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work targets better ranking performance after supernet training in NAS. The authors leverage meta-learning to make the shared supernet weights adaptive to randomly sampled subnetworks. Experiments are conducted on NAS-Bench-201 and MobileNet space.
main_review: Strengths
I generally agree with the proposed idea. Naively train the supernet will introduce unfair competition among operators. Instead, the supernet's weights should be adaptive to its subnetworks. Correlation and Accuracy on NAS-Bench-201 and MobileNet space are competitive.

Questions:
1. How sensitive Algo.1 is to the inner and outer learning rate?
2. How does the index permutation work is not well explained (Line 4 in Algo.1). Why is it required? Is this for the purpose of randomness during meta-training? Will there be any gap if no such permutation?
3. Is $g$ aggregating parameter changes ($\theta - \tilde{\theta}$) from all subnetworks (Line 11 in Algo.1)? Will the method work if only part of subnetworks are sampled?
4. I understand that Reptile is efficient since the calculation of Hessian is avoided. I am curious if the authors have ever tried other meta-learning methods, such as the original MAML? If we do not consider computation cost, will the original MAML also work?
summary_of_the_review: I generally like this idea, although seems straightforward, applying meta-learning during training the supernet is a reasonable and correct strategy. The analysis and experiments are also very comprehensive. I vote to accept.
review_title: null

Meta-review: The paper proposes a supernet learning strategy for NAS based on meta-learning to tackle the knowledge forgetting issue. Forgetting happens when training a sampled sub-model to optimize the shared parameters overrides the previous knowledge learned by the other sub-models. The main idea of the paper is to consider training of each subnetwork as a task, and then apply MAML to ensure efficient cross-task adaptivity. While the reviewers found the proposed method mainly an application of the existing meta-learning strategies to one-shot NAS, additional experimental results provided by the authors mostly convinced them about the effectiveness of the proposed method.
Decision: Accept (Poster)