Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 

This paper proposes searching for an architecture generator that outputs good student architectures for a given teacher. The authors claim that by learning the parameters of the generator instead of relying directly on the search space, it is possible to explore the search space of architectures more effectively, increasing the diversity of the architectures explored. They show that this approach combined with the standard knowledge distillation loss is able to learn good student architectures requiring substantially less samples and achieving competitive performances when comparing to other knowledge distillation algorithms.

Pros:
+ The paper is clear overall. A system for combining knowledge distillation and architecture search is proposed that combines surrogate functions, multi-fidelity optimization, and neural architecture generators.
+ The results on CIFAR10 and CIFAR100 are compelling.

Cons:

- Somewhat limited conceptual innovation. The authors combine NAGO and BOHB to obtain a solution for architecture search for knowledge distillation. While the results are solid, there is little insight about the behavior of the method, e.g., no ablations are performed beyond the comparison with NAGO, so it is hard to assess the importance of the individual components. 
- No results on ImageNet.
- The results are compared with models that use other teacher architectures, so it is hard to determine if the improvements are due to an improved teacher or a better knowledge distillation method. The fact that both student architecture size and student accuracy are important metrics means that there is no easy way 
- No code is included, but I assume that this is something that the authors will address for the final version.
- The use of BOHB and the surrogate loss are insufficiently described, for example, in Algorithm 1.
- The claim that a standard neural architecture search would produce architectures sampled from the same distributions and therefore not be efficient in exploring the space of students is insufficiently explored.

Comments: 

In essence, this paper only corresponds to a difference in how the architectures for knowledge distillation are generated. The claim that not going through the architecture generator would lead to architectures that are too similar lacks substantiation. Additionally, the paper does not have information about comparisons with other architecture search algorithms for knowledge distillation or surrogate functions, therefore the introduction of a new framework may not be warranted under the claims of the authors. 

review_title: Nice addition for combining architecture search and knowledge distillation, but lacking support for some claims

final_score: rating: 3: Clear rejection
review_content: This paper applied knowledge distillation (KD) on network architecture generator optimization (NAGO) which is one of NAS. Specifically, KD has used in the procedure of searching and the parameter of KD has involved in the search space. As a result, the proposed method (AutoKD) seems to improve NAGO.

Pros)
- Extensive experiments are performed

Cons)
- Some figures are not clearly shown. Please refine the figures (e.g., figure 2) for clarity.
- Applying KD into NAGO seems to be naively done. It seems that the proposed method is incremental and the contribution is limited and the differences are not highlighted. 
- The result comparison in Table 1 looks not fair:
  - KD used in this paper used better teachers following the convention, but the competitor KD-LSR and SKD in the table are self-distillation methods, so the comparison is meaningless.
  - On CIFAR100 dataset, CRD in the original paper used WRN-40-2 as a teacher and trained the student of WRN-16-2, which has only 0.7M parameters with an accuracy of 75.64. However, this paper reports CRD used ShuffleNetV1 which have more parameters 
  - On MIT67 dataset, VID used an ImageNet-pretrained model for transfer learning, but AutoKD used the fine-tuned teacher which is much beneficial to KD in terms of performance.
  - On CIFAR10 dataset, the compared models (WRN 16-1 and two WRN 40-2s) have fewer parameters than that of NAGO for AutoKD. Therefore, it is hard to say that AutoKD outperforms them.
- Experimental results are somewhat unconvincing:
   - As weight is zero in Figure 2, the accuracies in the table should show consistent performance but are deviated w.r.t temperature. The authors should clarify this.
   - Why the accuracies of NAGO in Figure 4 look low compared to the other results in the paper? 
- Using KD on NAS leverages additional computational cost, but it is not clearly compared quantitatively


Comments)
- The method is incremental, and the novelty is limited. The experimental results comparing with other methods are biased to the proposed method, where the competitors' performances are not fairly compared, so it is hardly convincing the results and the effectiveness of the proposed method. 

review_title: Limited novelty

final_score: rating: 3: Clear rejection
review_content: 
Summary and contributions
  The paper takes advantage of NAGO and proposes to search for a family of student network architectures instead of a single architecture, aiming to be more sample efficient. This reformulation of the NAS problem makes it possible to search in an expressive searching space, at the same time, avoid to waste time in comparing similar architectures.

Strengths
  This KD-NAS approach further develops the benefit of NAGO and makes it convenient to search for a family of student architectures. The optimization objective of finding a family instead of a single architecture helps to speed up NAS process or more sample efficient.

Weaknesses
  There have already exist KD-NAS approaches and the main difference of this work is to search for a family. This objective mainly takes advantage of the generator in NAGO, so the contribution and novelty should be reduced accordingly.
  The purpose of some experiments in this paper is a little confusing. From my point of view, this paper aims to utilize NAS to benefit KD, at the same time, make NAS more efficient. Maybe you should compare AutoKD with earlier KD-NAS approaches instead of NAGO to show the impact of KD on NAS.
  The title shows the fact that the macro-structure of a network is more important than its micro-structure, which has been studied in previous work. But this fact doesn’t logically lead to idea of ‘searching for a family’. Also, I think it isn’t clearly articulated that how this family of student architectures can benefit knowledge distillation. Maybe the performance gains result from the ensemble of networks.
  Finally, I think it doesn't make much sense to have the comparisons showed in Figure 7. Besides, the logic and results shown in the visualization of Figure 6 are not clear enough to me.


[ Detailed comments]
1. In Chapter 3.2, the original meaning is unclear: ‘The hyperparameters ... that it represents.’
2. In Algorithm 1, there are many unmarked sentence endings. In addition, where are the definitions of functions f and α, and what does D refer to?
3. In Figure 6, the specific meanings of various arrows and various colors need to be marked.

review_title: A very rough method: NAS on KD directly, for what?

Meta-review: The paper proposes a new approach to knowledge distillation by searching for a family of student models instead of a specific model. The key idea is that given an optimal family of student models, any model sampled from this family is expected to perform well when trained using knowledge distillation. Overall this is an interesting idea and an important direction of research. However, the reviewers raised several concerns regarding novelty and experimental evaluation. There was a clear consensus among the reviewers that the paper is not yet ready for publication. The specific reasons for rejection include the following: (i) the proposed method is somewhat incremental, and the paper's contributions should be adjusted accordingly; (ii) the experimental results in the paper do not provide a clear/fair comparison with existing approaches, and additional baselines should be considered. The reviewers have provided detailed feedback in their reviews, and we hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject