Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper propose a teacher awared accuracy predictior which solves the issue for DaNAS that previous predictors are not aware of influence of the teacher model. The results demonstrate the efficacy of the proposed methods. 
strength_and_weaknesses: Strength:

S1: The idea is quite novel. The previous predictor-based NAS methods focus on accuracy, latency, FLOPs, and #params. The prediction of performance for a student after KD considering the given teacher and task is novel.

S2:The motivation is interesting: previous DaNAS can not generalize well to any other tasks with new combinations and teachers(not flexible), thus requiring retraining for these tasks and teachers' combination. Meta NAS enhance the fast adaption to the new task with less training cost, but none of them considers the influence of the KD. This paper proposes a method that tackles the above two obstacles: meta leaned prediction model which is 1) meta-trained (for fast adaption); 2) takes as input the teacher encoding and is guided by the accuracy of the teacher during the mete training.

However, I still have below concerns

W1.Some logic is hard to follow, for example, in Intro, conventional NAS is sub-optimal for searching students under KD because the optimal student distilled from the teacher is different from an architecture trained from scratch. I do not understand why the training pipeline difference would cause conventional NAS not work. Maybe more sentences can help like: if we want to adopt NAS to search for a suitable architecture for KD, we should consider the influence of the components from KD (teacher and source dataset, etc.). But traditional NAS is designed for searching an architecture according to its evaluations trained from scratch instead of the KD aspects.
The logic should be explained. And there are many other examples, in the abstract, the author claims that meta-learning NAS is sub-optimal for DaNAS. But from my perspective, they are designed originally for solving the generalization issue for NAS, and of course, they are not the optimal choice for DaANS. The author claims they are not optimal  just because they want to adopt meta-learning in DaNAS and find that they do not consider teacher apsect so they are not optimal. The logic needs to be reconsidered here and for the rest of the paper.

W2. typo: many sentences are too difficult to understand, eg on page 2: we propose a distillation-aware task encoding function that remaps parameters from the teacher to the student and considers the output from the embedding network of the remapped student as the embedding to estimate the teacher network’s impact on the final accuracy of the distilled model.

W2.1. The writing is really hard to follow, especially when combing a lot of new concepts. More, concepts need to be expiated clearly for example the setting of the few-shot learning under this paper's scenario.

W3.The layout is so confusing that the reference is several pages away from the table/figure and the caption is not clear to understand. For example, the main results are in Tab2, what is the unit of the results, I can not figure out them only by a few dataset names. And what are the model components: Set. PR. and Gui.? Please explain more in the caption (Tab2,3,4,5).  The figure is also confusing, what is collecting time in figure 4?

W4.Please explain more clearly in section 5 about your training and searching setting when combining Meta, KD, few-shot learning and one-shot adaption, and NAS. This is really confusing.

clarity,_quality,_novelty_and_reproducibility: Some comments on the clarity:
- typo: many sentences are too difficult to understand, eg we propose a distillation-aware task encoding function that remaps parameters from the teacher to the student and considers the output from the embedding network of the remapped student as the embedding to estimate the teacher network’s impact on the final accuracy of the distilled model.
- The writing is really hard to follow, expecially when combing a lot of new concepts.
summary_of_the_review: See previous sections for detailed comments, and I recommend the authors to revise the paper accordingly. 


# Updates after reading the response

I thank the authors for providing me a informative response that addresses nearly all of my previous concerns.  As this work proposes to combine two sub-domain of NAS, distillation aware NAS and meta-learning one, it opens a new field compared to previous works. So this work should be considered as a new baseline method in this newly proposed domain. So no matter how simple this method is, it should be considered as novel.  

The only concern left is why people should care such a sub-field. However, I do not see this is a problem as we should encorage people to step out of the previous settings in my opinion. So I raise my score to accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: NAS for knowledge distillation requires architecture search for a student network which will learn well using teacher network. The method DaNAS proposes learning meta model for accuracy prediction given a dataset, teacher and student architecture and using this to find optimal student network.
strength_and_weaknesses: Strength
1. Paper proposes a new method for fast generation of student architectures when training on new problems of knowledge distillation

Weaknesses:
1. Severely restricted architecture space: the NAS template restricts students to be smaller than teacher, further the students either have no kth layer or just copy the parameters of teacher for kth layer. While this is good to demonstrate empirical results, it will severely limit the impact in real world problems. 
2. the paper only shows results on small image datasets, I would like to see if model does well on other non image modalities and other real world image datasets - imagenet/faces
3. I am not sure how the paper prevents the model from learning 1-1 mapping - i.e. every "optimal" student is same architecture as teacher. It needs to be clearly exposition as the teacher parameters are simply being copied.
4. The idea shows good results but highly constrained - I think at least exploratory experiments showing use of LSTM etc will greately strengthen the paper
clarity,_quality,_novelty_and_reproducibility: The paper is implementable, I think the section on encoding could be clearer (this is the size of embedding for teacher, dataset - hzt and size of encoding of student - ht). Additionally its good that the code is included to reproduce results.
summary_of_the_review: Authors describe an algorithm to generate student architectures for distillation from teacher network and dataset. The show emprically promising results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for Distillation-aware Network Architecture Search (DaNAS). The main component of the method is a “distillation-aware meta accuracy prediction model” which maps a (student architecture, dataset, teacher network) tuple to a prediction of the accuracy of the student on the particular dataset when the student is trained via knowledge distillation (KD) from the teacher network.

The distillation-aware-meta-accuracy-prediction model is adapted to a target task via gradient-based one-shot adaptation from one teacher-accuracy pair (i.e., the accuracy of the teacher network on the target task).

Experimental results show that the proposed prediction model adapts to target tasks within 3.45 secs on average without direct training on the target tasks. Further, it is shown that the proposed method outperforms existing rapid NAS methods (such as meta-NAS methods and zero-cost proxies) on accuracy estimation and end-to-end DaNAS on a set of unseen tasks.

strength_and_weaknesses: ## Strengths

**S1.** The method is well-motivated and mostly presented with clarity.

**S2.** The experimental validation suggests the proposed method is effective, as compared to recent methods, in the DaNAS setting.

## Weaknesses

**W1.** My main concern with this submission (and related work) is the set of concerns raised in (Guo et al. ECCV 2020). In particular, the sets of tasks considered in the present submission come mostly from the same domain (e.g., natural images). What would one find out if the present work were to be evaluated across domains like those in (Guo et al. ECCV 2020)?

**W2.** Table 5 shows a large performance gap between the teacher network and the result of DaNAS. What does this gap mean about the performance of the entire pipeline?

**W3.** The experiments are limited to tiny images and the choice of architecture parameters (for teachers and students) were not discussed. In my view, the practical value of the proposed approach is not clear.

**W4.** The remapping of parameters from the teacher network to a given student candidate is not clear to me. In particular, eqs. (6) and (7) do not seem to account for any dimensionality reduction.

## References

Guo et al. A Broader Study of Cross-Domain Few-Shot Learning. ECCV 2020.

clarity,_quality,_novelty_and_reproducibility: Clarity is quite good.

Quality is comparable to recently published related work.

Originality is limited as the proposed predictor mainly adds one element to prior predictors (namely, the teacher network) and employs priorly proposed gradient-based adaptation.

summary_of_the_review: Similar to related papers, this submission is incremental. However, I am not sure it is practically useful (e.g., tiny images, tiny models -- I think, though this was not clearly specified) and I believe it ignores not-so-recent developments on (the issues of work in) meta-learning.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to make NAS distillation-aware by searching for a student architecture that performs best when distilling from a given teacher, with a meta-learning component that allows generalization to new datasets.
During the discussion, the authors were able to produce several new results to satisfy various concerns by the reviewers, and as a result two reviewers increased their score to 8. The third reviewer, FiPx, quoted the ICLR FAQ that new experiments should not significantly change the content, but did not take into account that these experiments were done specifically to answer the questions of reviewers (which is perfectly OK). The reviewer stated (in private message) that they would rate the submission as above the threshold if the revision is allowed. This is the case (at least in my interpretation of the guidelines, but I believe this aligns with the community's interpretation), and therefore, the effective scores are 8,8,6. 
Furthermore, meta-learning is a very understudied area of NAS, making the work very timely. I therefore would recommed the paper for a spotlight.
justification_for_why_not_higher_score: Too many of the results were produced in rapid cycles during the rebuttal period; there is thus a certain risk of them not having been as carefully checked as results that were in the original paper and had some time to settle.
Also, the authors only got so much interaction (and thus an average increase of scores of more than 2 points) because they sent 10+ reminders, which implicitly pressured the reviewers to iterate several times; while this iteration is a valuable possibility of the OpenReview rebuttal process, it is not sustainable for the community that everyone requests it, and I would not reward it with an oral.
justification_for_why_not_lower_score: I would also be fine if the PCs decide to downgrade to a poster if there are good reasons for that looking at the big picture. 
However, please don't go by the scores of 8,8,3 in the system but rather the effective scores 8,8,6 (the rejecting reviewer didn't update their score but said the paper is above the acceptance threshold for them now). So, the effective average score is 7,33, which I would expect to be in the spotlight zone.
Decision: Accept: notable-top-25%