Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets
Authors: Hayeon Lee
Sohyun An
Minseon Kim
Sung Ju Hwang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Neural Architecture Search
Meta Learning

Abstract: Distillation-aware Neural Architecture Search (DaNAS) aims to search for an optimal student architecture that obtains the best performance and/or efficiency when distilling the knowledge from a given teacher model. Previous DaNAS methods have mostly tackled the search for the neural architecture for fixed datasets and the teacher, which are not generalized well on a new task consisting of an unseen dataset and an unseen teacher, thus need to perform a costly search for any new combination of the datasets and the teachers. For standard NAS tasks without KD, meta-learning-based computationally efficient NAS methods have been proposed, which learn the generalized search process over multiple tasks (datasets) and transfer the knowledge obtained over those tasks to a new task. However, since they assume learning from scratch without KD from a teacher, they might not be ideal for DaNAS scenarios. To eliminate the excessive computational cost of DaNAS methods and the sub-optimality of rapid NAS methods, we propose a distillation-aware meta-accuracy prediction model, DaSS (Distillation-aware Student Search), which can predict a given architecture's final performances on a dataset when performing KD with a given teacher, without having actually to train it on the target task. The experimental results demonstrate that our proposed meta-prediction model successfully generalizes to multiple unseen datasets for DaNAS tasks, largely outperforming existing meta-NAS methods and rapid NAS baselines. Code is available at https://github.com/CownowAn/DaSS.

Published as a conference paper at ICLR 2023
META-PREDICTION MODEL FOR DISTILLATION-AWART NAS ON UNSEEN DATASETS
Hayeon Lee; Sohyun An", Minseon Kim, Sung Ju Hwang
KAIST, South Korea
Thayeon92,, sohyunan, minseonkim, s sjhwang82) skaist..c.
ABSTRACT
Distillation-awar Neural Architecture Search (DaNAS) aims to search for optimal student architecture that obtains the best performance and/or efficiency when distilling the knowledge from a given teacher model. Previous DaNAS methods have mostly tackled the search for the neural architecture for fixed datasets and the teacher, which are not generalized well unseen dataset and an unseen teacher, thus need to perform a costly search for any new combination of the datasets and the teachers. For standard NAS tasks without KD, meta-learning based computationally efficient NAS methods have been proposed, which learn the generalized search process over multiple tasks (datasets) and transfer the knowledge obtained over those tasks to new task. However, since they assume learning from scratch without KD from teacher, they might not be ideal for DaNAS scenarios. To eliminate the excessive computational cost of DaNAS methods and the sub-optimality of rapid NAS methods, we propose distillation-awar
new task consisting of an
meta accuracy prediction model, DaSS (Distillation-awe) Student Search), which can predict a given architecture's final performances on dataset when performing KD with given teacher, without having actually train it on the target task. The experimental results demonstrate that our proposed peta-prediction model successfully generalizes multiple unseen datasets for DaNAS tasks, largely outperforming existing meta-NAS methods and rapid NAS baselines. Code is available http://gghuubbcom/oowwww/////
INTRODUCTION
Distillation-wwars Neural Architecture Search (DaNAS) aims to search for an optimal student architecture that obtains the best performance and efficiency knowledge from the given teacher to it (Liu et al., 2020; Gu & Tresp, 2020; Kim et al., 2022). For the DaNAS task, need to design framework that considers the effect of Knowledge Distillation (KD), yet, conventional NAS frameworks may be sub-optimal as they do not consider KD components at all by searching for architecture according its evaluations trained from scratch. As explained in Liu et al. (2020), the sub-optimality of conventional NAS methods on DaNAS tasks results from: 1) For the same target dataset, optimal student architecture for distilling the knowledge from the teacher and an optimal student architecture for learning from scratch with only ground boound-tuuth labels may be different. Even for the same dataset, the optimal student architecture may depend on the specific teacher. To tackle such challenges, existing DaNAS methods guide the search process using the KD loss (Liu et al., 2020) or propose a proxy to evaluate distillation performance (Kim et al., 2022).
given dataset when distilling the
However, such existing DaNAS methods do not generalize to multiple tasks, require training for any combination of dataset and teachers, and may result excessive computational cost (e.g.. days with 200 TPUv2, for each task (Liu et al., 2020)). This hinders their applications to real-world scenarios since optimal student architectures may vary depending on the type of datasets, teacher, and resource budgets. Therefore, need rapid and lightweight DaNAS method that can be generalized across different settings.
For standard NAS tasks without KD, there has been some progress in the development of rapid NAS methods that are computationally efficient, such as 1) meta-learning beasening--aeeed transferable NAS
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2023
TakE
Tak
Unseon Boak
Tabi
Seen Ootneer
Sean Corneet2
Misent
Training
Cuetee O[1)
Voar
Unsen Na
Traising
Inference
Telospadti
Existing Meta-learned
Meta-learned Our Model
Prediction Model
OIN]
(a) Training Complexity
(b) Task-specific Model
(c) Meta-NAS
(d) DaSS (Ours)
Concept. To search for a student architecture optimized for a distillation task, the prediction model should estimate the final accuracy of architecture diffcrently depending on a datasct, a teacher network, a student architecture, and a distillation process. While existing meta- mecta-prediction models only support set- co-ooniiiioeed prediction, the proposed meta-prediction model, DaSS performs the
Figure 1:
prediction.
methods (Lee et al., 2021a;b; Jeong et al.. 2021) and 2) zero-cost proxies (Mellor et al., 2021; Abdelfattah al., 2021). The former meta-NAS methods learn the generalized search process over multiple tasks, allowing it to adapt to a novel unseen task by transferring the knowledge obtained over the meta-learning phase to the new task without training the NAŠ framework from scratch. To this end, meta-NAS methods generally utilize task-adaptive prediction model, which rapidly adapts to novel datasets and devices. They outperform bascline NAS methods on multiple benchmark datasets (Lee et al., 2021a) and real-world datasets (Jeong et al., 2021), as well with various devices (Lee et al., 2021b), significantly reducing the architecture search time to less than a few GPU seconds on an unseen setting. The latter, zero-cost NAS methods have proposed several proxies that can be obtained from the first mini-batch without fully training the architecture on the target dataset.
However, despite the success of such rapid NAS methods on standard NAS tasks, they may be sub-optimal for DaNAS scenarios since they assume training from scratch, not KD from a teacher, which may significantly impact the actual accuracy of the architecture retrieved from the search. Therefore, to overcome the high search cost of DaNAS methods and the sub-optimality of rapid NAS methods, we propose a rapid distillation- aware meta- meta-prediction model, DaSS (Distillation-wve) Student Search), for DaNAS tasks (Fig. 1). Following the previous works on meta- we leverage meta-learning learn a prediction model that can rapidly adapt to an unseen target task. However, our approach has two main differences.: 1) Distillation and 2) meta-learning scheme that utilizes already trained teachers, both of which are optimized for the DaNAS task. First, we propose distillation-awarl task encoding function that considers the output from the student whose parameters are remapped from the teacher to estimate the teacher's s impact on the actual performance of the distilled student network. Second, use the accuracy of the teacher to guide the gradient-base based adaptation of the meta veta--rediction model. This allows a more accurate and rapid estimation of the architecture's performance target task (dataset) with specific teacher.
design of the meta-prediction model
We meta-learn the proposed distillation-aware prediction model the subsets of TinylmageNet and neural architectures from the ResNet search space. Then we validate its prediction performance on heterogeneous unseen datasets such as CUB, Stanford Cars, DTD, Quickdraw, CropDisease, EuroSAT, ISIC, ChestX, and ImageNet-IK. The experimental results show that our meta-lcarned prediction model adapts to novel target tasks to estimate the actual performance of an architecture distilled by an unseen teacher within 3.45 (wall clock sec) on average without direct training on the target tasks. Further, the DaNAS framework with the proposed distillation-awar meta-prediction model outperforms existing meta-NAS and zero-cost proxies on the same set of datasets.
To summarize, our contributions in this work are as follows:
We propose novel meta-prediction model, DaSS, that generalizes across datasets, architec- tures, and teachers, which can accurately predict the performance of an architecture when distilling the knowledge of the given teacher.
We propose novel distillation-awar aware task encoding based on the functional embeddings of a specific teacher and parameter-remapper student architecture candidates.
We enable a rapid gradient- based one-shot adaptation of the meta- predictreiitton model on a target task by guiding it with teacher-accuracy pair.
2

---

Published as a conference paper at ICLR 2023
We experimentally validate that our meta-prediction model successfully estimates the actual accuracies of the student architectures for a given teacher on multiple unseen datasets, significantly outperforming existing methods in the correlation between prediction and GT accuracies and in the end-to-end DaNAS performances.
2 RELATED WORK
Distillation- NAS Knowledge Distillation (KD) is an efficient model compression technique under the resource budgets by transferring the knowledge from the large teacher to the smaller student network. Since distilling performance depends on architectures as verified in Liu al. (2020), it is important to search for an optimal student architecture given the dataset and teacher. Recently, while Neural Architecture Search (NAS) suggests automating designing the optimal architecture (Zoph & Le, 2017; Baker et al., 2017; Zhong et al., 2018; Real et al., 2017; 77 Liu et al., 2018; Elsken et al., 2019; Real al., 2019; Liu al., 2019) that aims to search for an optimal architecture that has high accuracy when trained with only the ground-trute label, it might be suboptimal for searching for an architecture optimized for distilling given the teacher. Liu et al. (2020) verified the assumption with the RL agent, which KD-trains sampled architecture for a few epochs to obtain a KD &Dgguided accuracy proxy. It takes 5 days with 200 TPUv2 on a single task, so it cannot be applied to real-workl world KD scenarios. Kim et al. (2022) suggested the trust region with Bayesian optimization search algorithm and a new KD-guided score. However, it cannot be generalized to multiple tasks, which makes rapid search impossible. To obtain pruned student architecture, (Gu & Tresp, 2020) specify scaling factors for all channels of a teacher and optimize that with KD-aware loss function applying a structured pruning method. As the optimization should be performed repeatedly for each task get the optimal student architecture, the total time is still too large.
Meta-learning and Meta-prediction Model Meta-learning (learning to learn) (Thrun & Pratt, 1998) allows model to quickly adapt to an unseen task, by learning generalize over the distribution of tasks and transferring the meta-knowledge to the unseen task. One of the main meta-learning approaches gradient-based meta-learning (Finn et al., 2017: Li et al., 2017), which learns an initialization parameter shared across multiple tasks to be optimal for any tasks within a few gradient-based updates from the initial parameter. Recently, meta-learning-bais based NAS (Meta-NAS) frameworks (Lee et al., 2021a; Jeong et al., 2021; Lee et al., 2021b) have been proposed, which are rapid and transferable large-scale scale unseen tasks. MetaD2A (Lee et al., 2021a) and TANS (Jeong et al., 2021) designed the accuracy prediction models and Lee et al. (2021b) proposed the latency prediction model. MetaD22 and TANS proposed the set encoder for task-adaptive accuracy prediction models. However, none of them do consider the teacher and distillation process, thereby architecture obtained by them might be suboptimal when distilling knowledge from the teacher to the architccture. Due to the high cost and time to collect architecture-accuracy pairs as samples for adaptation on unseen tasks, they do not leverage the advantage of the few-shot adaptation on a target task. On the other hand, we utilize teacher-aceuracy pair for adaptation, allowing the predictive model to do gradient-basee adaptation to new tasks. The additional required time is the time for a forward pass to obtain the accuracy of the teacher.
3 BACKGROUND
This section describes existing accuracy prediction mod- els designed for NAS and their limitations in using the
Table I: Comparison between accuracy pre- diction models for NAS and DaNAS.
models rapidly across multiple tasks of searching for op- timal student architectures to distill the knowledge from teacher networks. In Table 1, we briefly summarize dif- ferences between task-specific accuracy prediction mod- els (Task-spec.), existing meta-accuracy prediction mod-
Task spec.
MeaDZA
TANS
DaSS (Ours)
Time Compl.
O(N)
O1l)
O11)
O()
Meuslasniig
Unsees Arch
Unseon Dats.
Usseen Teach.
0
els (MetaD2A (Lee et al., 2021a), TANS (Jeong et al., 2021)), and the proposed meta-accuracy prediction model for DaSS (Ours). As the sase-fpeccfic model needs to be trained on each task, the total training time increases the number of tasks increases (O(N)). Since
Disnit sware
X
x
X
0
Fast Aclept
X
x
x
0
meta- prediction models including ours can be generalized to multiple unseen tasks (datasets) after one-time meta-training, the training time is constant even if the number of tasks increases (O(1))
3

---

Published as a conference paper at ICLR 2023
However, while ours is generalized for unseen teacher networks (Unseen Teach.) and estimate Distillation-aware accuracy (Distil-aware.), existing meta- pretiicttnn models have difficulty doing them. Furthermore, we empirically demonstrate the efficacy of the gradient-based one-shot fast adaptation with teacher-accuracy pair to improve the prediction performance of the meta-prediction model on unseen task in NAS (Fast Adapt.).
During the search process, training each architecture candidate on task until convergence to obtain its final performance, which is single-scalar value, is wasteful. Thus, many NAS methods (Luo et al., 2018 8; Wang et al., 2020; Cai et al., 2020) leverage a task-specific accuracy prediction model as a proxy to reduce the excessive search cost spent on training the models. Let us assume a task specification
Task-specific Accuracy Prediction Model
(D", S, YT where D C D is dataset, C S is a set of neural architectures, and Y C y is a set of accuracies obtained by training each architecture s S on the dataset D The task-specific accuracy prediction model (): : - R parameterized by & is trained for each task T that estimates the accuracy y" E Y of a neural architecture S for a given dataset D by minimizing empirical loss, L (e.g. mean squared error) defined on the predicted values f (s; Φ) and actual measurements y" as follows:
min (f: YY)
However, the task-specific accuracy prediction model is highly limited in its practical application to real-world setting. Since it will not generalize to a new task (dataset), we need to learn a tpec-ipccific accuracy prediction model on the new target dataset. However, this is highly time-consuming since we need to collect many pebitectuue-accuraan pairs (at least thousands of samples) by training sampled architectures on the new dataset.
Meta-accuracy Prediction Model To address such inefficiency in task-specific NAS and enable transferrable NAS on multiple tasks, Lee et al. (2021a) and Jeong al. (2021) proposed generalizable accuracy prediction models that can rapidly adapt to unseen datasets within a few GPU seconds after a single round of meta-learning. Specifically, such rapid search is made possible by meta-learning the prediction model conditioned on the given dataset: [(s, D"; Φ) S DD + R. This enables it to support multiple datasets with a single prediction model since the predicted accuracy y + (s, D*) is dependent on both the dataset type D and the architecture s. Then, they train the prediction model to generalize over the task distribution p(r) by sampling task T per iteration via amortized meta-learning follows:
min Σ =()(S)D":)) YT) T-p(()
(2)
After this one-time meta-learning, the set-conditioned accuracy prediction model f(; Φ²) with meta- learned parameters o² can be transferred to an unseen task to search for an architecture optimized for f. Note that the meta-lcarned prediction model works well on an unseen dataset D$ which has no overlap with datasets (D7} used in the meta- meta-training phase. This reduces the meta-prediction model's 's complexity from O(N) to O(1) on N tasks.
While the meta- accuracyy prediction models promising they enable rapid search on multiple tasks, directly using the existing meta-accurase prediction model for distillation-an aara NAS problems is suboptimal. This is because they do not consider the teacher and may be inaccurate for DaNAS scenarios due to the following reasons:
Linitations
1. Distillation-indeeeneent accuracy prediction. DaNAS is different from the existing NAS problems in that task 7 in the distillation-aware NAS domain includes a teacher t(:; O(T which is parameterized by ① and is trained on dataset D Thus, for the same dataset, the optimal student architectures can be different if the teachers are different. However, the existing meta-prediction model does not encode the teacher and thus will search for the same student architecture for different teachers, which might be sub-optimal. Thus, we need ceacher-conditione accuracy prediction model for DaNAS.
2. No gradient-guided task adaptation. Existing Meta-NAS frameworks aim to learn generalized accuracy prediction for multiple tasks with amortized meta-learnin conditioned the task embeddings, not leveraging the power of gradient-basee adaptation steps (Finn
4

---

Published as a conference paper at ICLR 2023
Meta-training
Meta-test
Task T2
Preshe Partererer
[++ mb)
Mess Herteedding Yook hell PV
Dataur
Teasher
Studeet
Task T1
Dataeet
Accummry
Fecher
Meta Prediction Model
Trengle diceweedge
Task 1
Teather Network
Tosk Ty
Loss
rool
Trashon
Studest Arch.
Task Ex
eslesting
Figure 2: Overview of DaSS. To search for an optimal student architecture that can distill the knowledge of given teacher t" (:: (:; 0)) well, the sampled task should be encoded in distillation-aware way. We first remap the trained parameters of the teacher to the randomly sampled student. Then, by taking fixed input the teacher and the remapped student, we can encode the teacher and student functionally a distillation- way. Along with the functional embedding, by considering the architecture configuration of the student, we encode sampled tasks in way appropriate for the DaNAS. By transferring kell-know ø learned from meta training phase, mela-prediction model [(; d²) can adapt to unseen task f quickly through the guided adaptation with almost no cost.
et al., 2017; Li et al., 2017). This was mainly done to expedite the meta-training step and reduce the adaptation time on new task since gradient-based adaptation requires obtaining few-shot architecture acccuecture-accuracy samples of the target dataset, which takes at least a few GPU min to hours and renders rapid NAS impossible.
To address the above issues, we aim to design a teacher- conditioned accuracy prediction model and the accuracy of teacher-guided meta-learning for rapid DaNAS tasks. In our work, we reveal that accuracy prediction is more accurate than set-conditioned accuracy prediction when estimating actual performances of student architectures in DaNAS scenarios. In addition, we show that it is possible to perform gradient- guided task adaptation using the accuracy of the teacher. Please refer to Fig. 2, which illustrates our distillation-awaal performance prediction model, DaSS.
METHOD
Distillation-awee aware Accuracy Prediction Model For each task r = (S, Y (T; θ-)} from a distillation- aware NAS domain, we propose distillation-awe aware accuracy meta-prediction model f(: Φ) parameterized by Φ as follows:
f(s, (6,,t((a (t:
8 X
(3)
that can estimate an accuracy of a student architecture s differently depending on the student
architecture and the teacher t" (=; DD,00 ,T)) which is trained on the target dataset, D We encode task 7 by utilizing the teacher which is trained on the target dataset and the student architecture in distillation-awaal way. To this end, we encode each sampled student architecture candidate s(:; 0)
in teacher-adaptee manner by leveraging parameter remapping 0 -g(²)) with the remapping function g from the parameters Ô of the teacher trained on the dataset D the parameters 0 of each student architecture s(:; O.), which can be done at almost no cost. More detailed descriptions about the parameter remapping function, g is in Appendix B. Next, we embed each remapped student candidate s(: θī) and the teacher (=: (T) by feeding in fixed Gaussian random noise z (Jeong et al., 2021) 1) as input as follows:
s s(z; θ = h ((2;0)) = h
(4)
In addition, as described in Appendix A, our ResNet search space is factorized hierarchical search space, and all architecture candidates in this search space are composed of 4 stages. Student architecture candidates in this search space can have various depths and channel widths at each stage. As suggested in Cai et al. (2020), the architecture configuration, such as depth and channel widths at each stage of 8, can be represented as one-hot encoding vector. Concatenating one-hot encoding vectors for all stages, The accuracy prediction model takes three embeddings has h and h as inputs and outputs the estimated accuracy, with additional nonlinear projection layers. For the detailed descriptions, please refer to Appendix B.
can obtain the embedding for the whole network architecture, he-
5

---

Published as a conference paper at ICLR 2023
Adaptation Guided with Teacher-Aceuree Pair To obtain meta-learned f(: meta-learn the proposed accuracy prediction model over task samples from the task distribution p(T) as follows:
min
Σ (",,
D
(5)
T>-((T)
where L is the Mean Square Error loss. Differently from the previous meta-accuracy prediction models (Lee et al., 2021a; Jeong et al., 2021) based on the amortized meta-learning, we further perform guided adaptation the target task with the teacher t and its accuracy ø² evaluated on
the target dataset, by conducting the inner gradient updates with them as follows:
8 (c+1) 0/)) aV si C(f(t, t (=DD,, "'D"00'" 0 (()))))))))))) for i i 1, I,...,,
(6)
where denotes the i-th inner gradient step, is the total number of inner gradient steps, and is the multi-dimensiong inner learning rate vector (Li et al., 2017). This meta-learning scheme enables the accuracy prediction model start from the knowledge learned from the meta-training tasks and rapidly adapt to the new task with guidance from the teacher- accuracy pair. Note that while there are gradient-bart based meta-learning methods (Finn et al., 2017: Li et al., 2017; Lee al., 2021b), our work is the first to train the meta-accuracy prediction model for distillation-wwarr NAS that can generalize to unseen datasets via gradient-based guided adaptation. Furthermore, by introducing the simple yet effective approach of using the teacher and its accuracy pair as few-shot training instances for task adaptation, we significantly reduce the consemonnuming collection of the student architecture-accuraayy pairs for each task, which allows efficient adaptation each task. The final meta-learning objective reflecting Eq. (5) and Eq. (6) is given as follows:
min &
Σ ((f(S.t((::) O²);; Trp(T))
("): 1(1+1)), Y")
Adapting the Meta-prediction Model on Unseen Tasks (Meta-test) In Fig. 2, we utilize the meta-learned prediction model f(: x*) on a novel distillation task f that was unseen during meta- training. Given a task f = {D", S, Y (:; 0²) }}, we first compute the accuracy o² = = (D*; 0*) of the teacher through the forward process on the validation. Then, we update the meta-learned parameters (1)) Ø" by adapting it to the unseen task by taking inner gradient steps, using Eq. (6)
to obtain ((+1)) with (=; D "T)), 0²), 1.7") Then, for each student architecture candidate s, we remap parameters and compute h and h with Eq. (4) and obtain h Finally, we concatenate h h and h and compute the estimated accuracy of a on the task f with f(h 20 h 2,07 h:; ( (1+))
5 EXPERIMENT
We first validate and analyze our meta-prediction model, DaSS, in terms of 1) a novel distillation- aware task encoding and 2) a gradient- based one-shot adaptation strategy with the teacher by pre- dicting the distillation performance of unseen student architectures on an unseen dataset with an unseen teacher in Section 5. Second, we report the time efficiency of DaSS compared with few-shot meta-prediction models in Section 5.2. Next, we demonstrate the performance of DaSS on end-to-end DaNAS tasks compared against existing rapid NAS methods in Section 5.3. In addition to this, we compared the performance of ours and the existing Meta-NAS methods through end-to-end DaNAS tasks on cross-domain datasets in Section 5.4. Lastly, we validated DaSS on ImageNet-I 1K which is one of the representative large-scale datasets with 256 x 256 pixels in Section 5.5. A detailed description of our search space throughout the experiments is in Appendix A, and we use the largest architecture in our search space as teacher architecture for all experiments.
Meta- Pata-Training Database Following Ryu et al. (2020), we use TinyImageNet (Le & Yang, 2015) as the source dataset for meta-training. We generate a heterogeneous set of tasks by class-wisely splitting the dataset into 10 splits. We use subsets consisting of and splits for meta- meta-training and meta-validation, respectively. Please refer to Appendix D for the more detailed explanation.
Generalization to Unseen Datasets We transfer the knowlede of the prediction model trained over the meta-training phase to meta-test tasks. Each meta-test task consists of an unseen

---

Published as a conference paper at ICLR 2023
Table 2: Efticacy of DaSS for DaNAS task. We report Spearman's rank correlation coefficient between ranking by actual distillation performance and ranking by predicted distillation performance of 50 unseen student architectures on an unseen dataset, with an unseen teacher. The reported values are the average over 3 random runs. Our meta- ineaa-prediction model with macher-ggudde meta-learning strategy outperforms all rapid pedd-prrdintion models.
Model Components
Spearman's 's Rank Correlation Coefficient
Method
Arch.
Set Enc. Distill-aware.
Guide
CUB
Cars
DTD
Draw
Mean
MAML (Finn et al., 2017)
0.63
0.61
0.38
0.42
0.51
Meta-SGD (Li et al., 2017)
0.18
0.49
0.19
0.63
0.37
MetaD2A (Lee et al., 2021a)
0.63
0,51
0.40
0.26
0.45
TANS (Jcong al., 2021)
0.43
0.61
-0.15
-0.17
0.18
DaSS (Ours, Non-Guide)
0.69
0.77
0.50
0.62
0.64
DaSS (Ours, Guide)
0.69
0.88
0.57
0.75
0.72
dataset, an unseen teacher trained on the unseen dataset, and multiple unseen student architecture candidates. We covered 9 datasets, including tine-grained datasets (CUB (Wah et al., 2011), Stanford Cars (Krause et al., 2013) (Cars), CropDisease (Guo et al., 2020) agriculture images), out-of- distribution datasets (DTD (Cimpoi al., 2014) texture images, Quick Draw (Jonas et al.) (Draw) black-and-whie white drawing images, EuroSAT (Guo et al., 2020) satellite images, ISIC (Guo et al., 2020)- - medical images about skin lesions, ChestX (Guo al., 2020) X-ray images), and large-scale dataset- ImageNet- IK (Russakovsky al., 2015).
5.1 EFFICACY OF THE PROPOSED META-PREDICTION MODEL ON UNSREN DATASETS
We validate whether our novel distillation- task encoding and meta-learning strategy. which is a gradient-based one-shot adaptation with the teacher, help improve the performance of the accuracy prediction model. We use fixed 50 student architectures and distill knowledge from the teacher on each unseen dataset to compute Spearman's 's rank correlation (the higher, the better) between the predicted and actual performances and report the efficacy of cach method. In Table 2, Arch. means taking architecture configuration, Set Enc. means taking dataset embedding, Distill-wware. means taking the functional embeddings of student architecture candidate and teacher, and Guide means the adaptation strategy guided by teacher-accuracy pair in meta-learning scheme.
DaSS (Ours, Guide) outperforms all rapid meta reta-prediction models (Finn et al., 2017; Li et al., 2017: Lee et al., 2021a; Jeong al., 2021) on 4 unseen datasets. MAML and Meta-SGD, which learn accuracy prediction models by taking only the architecture configuration (Arch.) of a student candidate as an input, show poor performance, as they predict the distillation performance without consideration of a target dataset and teacher. Set-conditioned (Set Enc.) prediction models (MetaD2A and TANS) also achieve significantly lower performance than ours since they search architectures in teacher-agnostic manner, which greatly impacts the performance of the student trained with distillation. On the other hand, using the proposed distillation-are -ware embeddings of a student and teacher (Distill-aware.), DaSS (Ours, Guide) significantly outperforms all baselines all 4 unseen tasks. Furthermore, we observe that adaptation to the target task by guiding our meta-prediction model with the accuracy of the teacher the target dataset largely improves the prediction performance over the non non-guided search, DaSS (Ours, Non-Guide Guide...)). In sum, the results show the clear advantage of both the design of our distillation task encoding and the reacher-guides meta-learning scheme.
5.2 EFFICIENCY OF TEACHER GUIDED ACCURACY PREDICTION
In general, guided search, such as fho- adaptation on the target task, requires the collection of few-shot acchtecture-accurayy pairs. To obtain even single pair of architecture-accurace we should train the architecture until it fully converges, which prevents rapid search at meta-test time. How- ever, the DaNAS environment is different from the conventional NAS in that we already have trained model, the teacher. There is
Doa Peacher
praun Superoo
Calleding
'Number Bemples
Number Samples
(a) CUB
(b) DTD
Figure 3: Time and Sample Efficiency.
7

---

Published as a conference paper at ICLR 2023
no need to wait to get the accuracy-aree architecture pair(s) for adaptation since can get the accuracy of the teacher with a single forward pass, In Fig. 3, we report the time efficiency of DaSS, compared pair(s).
against that of the baselines, when adapting to unseen tasks with student architecture-accuus The Collecting time refers to the time required to obtain architecture-pen performmaepe pairs (Number of samples) in the guided meta-learning strategy. Our one-shot adaptation guided search with the teacher network which is already trained under the DaNAS task is much more efficient than utilizing a few-shot adaptation with few student architecture-accurayy pair(s). Our model only needs 2.57(s) and 1.55(s) on CUB and DTD, respectively, which significantly reduces the adaptation time.
5.3 COMPARISON WITH RAPID NAS METHOSS
Table 3: The reported results are SRC between ranking by actual distillation performance and ranking by predicted distillation performance unseen student architectures on an unseen dataset, with an unseen teacher. Search Time is the time for predicting the performance of each unseen student architecture by given method. Concretely, the search time represents the total search time divided by the number of architecture candidates (e.g., search time per architecture).
Method
Spearman's 's Rank Correiation Coefficient (SRC)
Search Time / Arch. (GPU sec)
CUB
Cars
DTD
Draw
Mean
CUB
Cars
DTD
Draw
Mean
Grad Norm (Abdelfattah et al., 2021)
-0.07
0.09
0.19
-0.60
-0.09
0.98
1.00
1.11
2.11
1.30
Snip (Abdelfattab al., 2021)
0.11
0.27
0.24
-0.31
-0.04
1.05
1,03
0.97
2.14
1,30
Grisp (Ahdelfattah al., 2021)
4015
-0.06
-0.19
0.28
-0.03
1.34
1.34
2.56
1.65
Zero-cost proxies
Fisher (Abdelfittal ct al., 2021)
0.03
0.08
0.20
-0.79
-0.12
0.97
1.04
1.05
2.10
1.29
Plain (Abdelfattal al., 2021)
0.12
0.22
0,26
-0.39
0.05
0.90
1,01
1.03
2.10
1.26
Synflow (Abdelfartah
2021)
-0.10
0.13
0.17
-0.26
-0.01
0.98
0.97
2.16
NASWOT (Mellor et al., 2021)
-0.45
0,15
0.32
-0:42
-0.10
0.92
1,00
1.02
2.03
1.24
MetaD2A (Lee et al., 2021a)
0.63
0.51
0.40
0.26
0.45
0.01
0.01
0.01
0.01
0.01
Meto-lcaming
TANS (Jeong al., 2021)
0.43
0.61
-0.15
-0.17
0.01
0.01
0.01
0.01
DaSS (Ours)
0.69
0.88
0.57
0.75
0.72
0.05
0.05
0.05
0.05
0.05
Table 4: End-to-end DaNAS on Unseen Datascts The distillation performance for the student architecture searched by cach method was evaluated the target validation set. All MACs values shown in the Table are based on 64 x 64 pixels and the reported accuracies are the mean and standard deviations over runs.
Unseen Dutaset
CUB
Stanford Cars
DTD
Quick Draw
Ace. (%)
MACs (M)
Acc. (5)
MAC's (M)
Acc. (%)
MACs (M)
Acc. (%)
MACs (M)
Teacher
37.19
1459.51
43,25
1459,50
41.49
1456,47
44.14
1459.50
Grad
42.23 3:35
1029.07
1010.42
42,80 2:1
1235.98
981.49
Snip
39.55 1a5
1175.61
47.83 6:1
1150.48
43.19 a.s
1235.98
45.24 nor
981.49
Zero-cost proxies
Grasp
Fisher
39.41 =:11
39.55 1:8
1175.61
711.06
48.22 21.12
48.07 2018
1010.42
719.71
44.92 oss
43.75 1:14
1076.26
711.06
45,28 am0
45.31 w
981.49
745.29
Plain
40.85
702.63
51.75
903.76
42.98
1137.60
45,24
981.49
Synllow
39.97 1:
1251,26
46.75 6.25
1251.26
43.57 622
1251.26
45.71 noe
1251.26
NASWOT
40.94 2:
1030.65
52.03
746.44
45,47
874.63
45,48
702.72
MetaD2A
41.64 =0
708.75
56.74 2:3
734.97
45.13 o.a
912.10
46.80 aue
761.73
Meta-learning
TANS
41.74 2130
743.99
56.00
995,44
45,26
200
836.21
45.61
702.60
(Ours)
43.48 2m
792.97
789.13
47.86 2.4
789.13
714.13
In Table 3, we validate our meta-prediction model with other rapid performance prediction methods. Zero-cost proxies, which are representative rapid performance prediction methods, calculate the scores of architectures by feeding a few batches of the target dataset into randomly initialized architectures. Therefore, they efficient but cannot consistently predict the distillation performance across all unseen datasets. As shown in Table 3, ours outperforms all rapid NAS baseline models in terms of SRC, including these 6 zero-cost proxies (Abdelfattah et al., 2021; Mellor et al., 2021) and 2 meta-accuraa Additionally, the average search time of ours is only 0.05 GPU seconds, which is competitive with other rapid NAS methods. We further validate the proposed meta-prediction model by conducting end-to-end DaNAS tasks on 4 unseen datasets. For fair comparison, we randomly sample 3, 000 student architectures and select the Top-1 student architecture whose estimated performance by each method is the highest, then distill the knowledge from the same teacher trained on the target dataset to the selected student architecture with the same number of epochs. The results in Table 4 show that the student architecture found by ours distills the knowledge from the teacher network better than the student architecture found by other baselines on 4 unseen datasets.
prediction models (Lee et al., 2021a; Jeong et al., 2021) on 4 unseen datasets.
5.4 COMPARISON WITH META-ACCURACY PREDICTION MODELS ON UNSEEN DATASETS ACROSS DOMAINS
We further conducted an evaluation of DaSS on Broader Study of (BSCD-FSI) benchmark datasets (Guo et al., 2020). The benchmark datasets are from different
Few-Shot Learning
8

---

Published as a conference paper at ICLR 2023
Table End-to-end DaNAS on Unseen Datasets across Domains We compared DaSS to the meta accuracy prediction model baselines on BSCD-FSL benchmark containing datasets across domains. All MACs values are based on 64 x 64 pixels and the reported accuracies are the mean and standard deviations over 3 runs.
Unseen Dataset
CropDisense
EarOATT
ISIC
ChestX
Acc. (9)
MACs
(8)
MACs
MACS (M)
Ape. (%)
(M)
Teacher
99.66
1464.66
1464665
79.27
1464.65
40.32
1464
Metal22A
99.68 0.07
374.10
97.21
0.29
398.06
79.27 +: 1.56
374.10
39.95
1.38
393.50
TANS
99.72 0.02
398.90
97:33
0.48
384.51
79.79 +: 0.89
387.32
39:47
1.40
385.99
DaSS (Ours)
99.71 0.01
397,84
97.95 +: 0.06
378.99
81.52 +: 1.08
374.82
42.25
0.94
371.47
domains compared with the datasets that we have used to learn our meta-prediction model during the meta-training phase. As shown in Table 5, for the EuroSAT, ISIC, and ChestX of the BSCD-FSL. benchmark except for the easiest dataset, CropDisease, the student architectures obtained by DaSS are better at KD than other meta--reeicctoon model baselines. The results show that the meta-knowledge of our prediction model with distillation-aware task encoding trained over the meta-training tasks with our adaptation strategy with the teacher is effective in domminn datasets as well as in the same-domain datasets.
5.5 VALIDATION ON A LARGE-SCALE DATASET WITH DIFFERENT PIXELS AND MODELS
We validate DaSS with the larger pix- els (256 x 256) on ImageNet- IK, a representative large-scale real-world dataset, by conducting end-to-end DaNAS with the pre-trained teacher, which is the largest architecture in a ResNet search space. The search space in this Section is defined by Cai et al. (2020), and it is based on more depth and channel widths compared to previous experiments. Using our method DaSS and other meta-accuracy prediction model baselines (MetaD2A and TANS), we select an architecture with a Top-1 score in each method. Then we distill knowledge from the same pre-trained teacher to the obtained student architecture in the same training setting and report the best performance evaluated on the validation set. As shown in Table 6, DaSS successfully searched for more accurate and efficient student architecture, which shows 79.00% accuracy, outperforming the student architectures retrieved by the baseline methods. sum, the results show that DaSS is not limited to specific models and works well with larger pixels, demonstrating its practicality. More results of experiments with larger pixels are in Appendix F, and more description about the experiment setting is in Appendix E.
Table 6: MACs values this Table based on 256 256 pixels and the architecture configuration is based on Cai et al. (2020).
ImageNNet (256 256)
Acc. (%)
MACs (M)
Depth Config.
Width Conlig.
Teucher
76.73
9843.17
[4, 64 4]
[256, 512, 1024, 2048]
MctaD2A
78.43
5411.69
3,,3,5,4)
[208, 512, 816, 2048] [208. 512, 1024, 2048]
TANS
78.27
4211.43 43
3,
DaSS (Ours)
79.00
4029.64
[2,
5156,512, 664, 2048]
6 CONCLUSION
We proposed a novel meta-prediction model, DaSS, that generalizes across datasets, architectures, and teacher networks on DaNAS tasks, which can accurately predict the performance of an architecture when distilling the knowledge of the given teacher network. Existing task-independent DaNAS models require collecting a large number of architecture-accuut wew.eetttcccheecehhieeeeeee
samples the training set for each triplet. The condiiondi meta- meta-prediction models, on the other hand, can generalize to new datasets, but they are sub-optimal in that they do not consider the KD aspects. In order to devise a meta-prediction model for solving DaNAS tasks, we introduce a novel distillation- aware task encoding based on the embedding of the teacher and parameter remapping scheme. We also leverage rapid gradient-based one-shot adaptation of the meta- peeta-pediction model on a target task by guiding it with a teacher-accuracy pair, unlike existing meta-accuracy prediction methods, which do not take maner-gradien steps for task adaptation. The experimental results demonstrated that the proposed model outperforms existing rapid NAS methods, such as meta-NAS methods and zero-cost proxies, for the accuracy estimation on the heterogeneous unseen DaNAS tasks.
Acknowledgements This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSTT) (N0.2019-0-00075, 20.2022-0- 00713). This material is based upon work supported by Google Research Grant and the Google Cloud Research Credits program with the award (PMF4-4RK-MM MEEKK---N1--A M9N1-YB37)

---

Published as a conference paper at ICLR 2023
Ethics Statement This work has concerns about the ethical implications.
Reproducibility Statement First, we attached the code proposed method to help with reproducibility. Second, in the appendix document (after references), we enable the experimental results the main paper to be reproduced by providing a detailed description of the materials and elaborating the experiment setup, which is organized as follows:
the
Section A - We provide the details of the search space that we used in all experiments the main document.
Section B We describe the implementation details of functional embedding, architecture embedding. and distillation-awar student architecture embedding as the final input of our model, and meta-surrogaa performance prediction model.
Section C- We provide detailed descriptions of baselines compared against our mode the experiments of the main document.
Section D We elaborate the detailed training details, such meta-training the pro- posed prediction model and distilling knowledge from given teacher network to the student architecture, corresponding to the experiments introduced in the main document.
Section E We provide additional experimental setup about time measurements and condi- tions to sample student architecture in the search space.
Section F We provide additional experimental results.
REFERENCES
Mohamed S Abdelfattah, Abhinav Mchrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. International Conference on Learning Representations (ICLR), 2021.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In In International Conference Learning Representations (ICLR), 2017.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2020.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-360 3606,,1,,-1 2014.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. International Conference on Learning Representations (ICLR), 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning (ICML), 2017.
Jindong Gu and Volker Tresp. Search for better students to learn distilled knowledge. In ECAI, 2020.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. BCCV, 2020.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Wonyong Jeong, Hayeon Lee, Geon Park, Eunyoung Hyung, Jinheon Back, and Sung Ju Hwang. Task- adaptive neural network search with meta-contrastive learning. Advances in Neural Information Processing Systems, 34:21310-21324, 2021.
Jongejan Jonas, Rowley Henry, Kawashima Takashi, Kim Jongmin, and Fox-Gieg Nick. The quick, draw! a.i. experiment. URL httpt:ttp///t....t :/tppp::://///ieee ckdraw. wi thgoog l com/
10

---

Published as a conference paper at ICLR 2023
Tachyeon Kim, Heesoo Myeong, and Se- Young Yun. Revisiting architecture- aware knowledge distillation: Smaller models and faster search. arXiv preprint arXiv:2206.1311 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 2021a
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. Hardware-adaptive efficient latency prediction for nas via meta-leaminn. Advances in Neural Information Processing Systems, 34: 27016-27028, 2021b.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint
2017.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierar-
chical representations for efficient architecture search. In International Conference on Learning Representations (ICLR), 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In In International Conference on Learning Representations (ICLR), 2019.
Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls everywhere but not the eyes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7539-7548, 2020.
Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti.
Nsganetv2: Evolutionary multi-objective surrogate- assisted neural architecture search. In European Conference on Computer Vision, pp. 35-51. Springer, 2020.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Yan Liu. Neural architecture optimization. In Advances in neural information processing systems (NeurIPS), 2018.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning (ICMI), pp. 7588-7598. PMLR, 2021.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning (ICML., 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence (AAAT), 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-44 1-251 2015.
Jeong Un Ryu, Jaewoong Shin, Hae Beom Lee, and Sung Ju Hwang. Metaperturb: Transferable regularizer for heterogeneous tasks and architectures. Advances in Neural Information Processing Systems, 33:1 1501-11111122 2020.
Sebastian Thrun and Lorien Pratt (eds.). Learning Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems (NIPS), 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
11

---

Published as a conference paper at ICLR 2023
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, 2020.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2423-2432, 2018.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.
12

---

Published as a conference paper at ICLR 2023
A SEARCH SPACE
Our ResNet (He al., 2016) search space factorized hierarchical search space; thus, architecture candidates in this search space can have variable depth and channel widths in each stage, which means a sub-module that divides the entire network into smaller units. More specifically. our ResNet search space in Section 5 consists of 4 stages. Similarly to (Cai et al., 2020; Lu et al., 2020), we perform depth-wise search to select the number convolutional neural layers for each stage and then perform channel-wiss search to select the number of channel widths for each layer. To further explain the architecture configuration notation in Section 5, Detph Config. means how many blocks exist in each stage respectively, and Width Config. means the number of channels in the ResNet block convolutional layers at each stage that are obtained by multiplying a channel width shrink ratio by base channel widths predefined for each stage. In Section except for Section 5.5, Depth Config. can be chosen from {1, 3, 4} for each stage, the base channel width is [32, 64, 128, 256]. and the channel shrink ratio for each layer can be chosen from {0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0). In the case of Section 5.5, Depth Config. can be chosen from {0. 2 and it indicates how much more depth is added the existing base depth each stage. Additionally, the base channel width is [256, 512, 1024, 2048), and the channel shrink ratio for each layer can be chosen from (0.65, 0.8, 1.0). For all architectures in the search space, the channel shrink ratio of the last layer each stage is fixed to in order to make the dimension of the output feature the same for each stage.
B IMPLMENTATION DETAILS
B. PARAMETER REMAPPING
We consider network is a composition of M stages, where each i-th stage corresponds to B, By doing so, we can represent the teacher network and a randomly sampled student architecture in our search space as:
t (-: θT) BB ..... B1 (1:
s (::6T) BM ... (=:
(8)
In our search space, the depth and channel widths of the sampled student architecture are different from those of the teacher network )T). Also, we assume that in the distillation process, the depth and channel widths for each layer in each stage of a given teacher network are always greater than equal to the student. When performing distillation-aware task encoding, encode each sampled student architecture in a teacher-adaptive manner with a remapping function, g. Since the corresponding architecture configuration of the teacher network and sampled student architecture is different, the remapping process is conducted hierarchically for each stage m, by first performing depth-level remapping followed by subsequent width-level remapping. For the sampled student architecture, which has d depth and k1, f2. ka channel width for each depth at stage m. remapped student parameters, 0-- = (( ;r) is as follows:
σ- s,m.o 0 Ô for 0= I,...,d.
(9)
θ- o.m.o.k m.akk for k = 1,..., Ko.
θ
(10)
B.2 FUNCTIONAL EMBEDDING
With remapped student candidate, s(: Qz), is also important get functional embedding that has task-agnostic shape. The spatial dimension of final output logit for the given network varies across tasks and the depth and channel width within the same stage are also different for each sampled student architecture s. However, the output dimension of the feature map at each stage is the same across all tasks from the task pool p(T), as described Appendix A. By feeding fixed Gaussian random input tensor into the remapped student candidate, we can obtain teacher-conditioned network output feature map, T E HC at the last stage of the student, By while satisfying the condition of getting a task-agnostic shape. The output, T, can be used as the teacher-adaptive functional representation of the corresponding student candidate with respect to the target DaNAS task. By feeding in the teacher-adaptive functional representation tensor, T, into the functional projection layer, 9,, we obtain the functional embedding vector, df (T). Since we leverage the parameters of given teacher network, (T) can be teacher-adaptive.
13

---

Published as a conference paper at ICLR 2023
B.3 ARCHITECTURE EMBEDDING
Our prediction model takes an architecture configuration of sampled student architecture which contains information such depth and channel widths. As our ResNet search space is composed of 4 stages and layers for each stage, the total number of layers is 20. For each layer, the channel shrink ratio should be chosen from {0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0] except for Section 5.5. Similar to the one-hot encoding scheme used in (Cai et al., 2020), we can construct one-hot architectural encoding, a. Then, by passing the one-hot architecture encoding vector the architectural projection layer qa- we can get an architecture embedding vector, Qu (a). For the case of Section 5.5, we obtain the architecture embedding similar way.
B.4 DISTILLATION-AWAN
TASK ENCODING
With ga (a) and gf (T), which represents architecture embedding and teacher-adaptive functional embedding of sampled student architecture respectively, we concatenate those two embedding vectors and embedding of the teacher and learn another non-linear projection layer, o, which outputs distillationawwar network embedding.
C BASELINES
In this section, we describe the rapid NAS baselines which we compared with our meta- meta-prediction model. The baseline methods can be divided into two categories, 1) Zero-cost proxies, 2) Meta- prediction model.
C.1 ZERO-COST PROXIES
Abdelfattah et al. (2021) proposed some metrics based on pruning techniques that can quickly find architectures by leveraging few batches of target datasets with random state architectures. metric that measures how neural network's activations correlate when it is exposed to various inputs over few batches of the target dataset was also proposed by Mellor et al. (2021). Unlike other methods that require training sampled architecture at least few epochs proxy task, they have received lot of attention as very rapid NAS becomes possible.
C.2 META- PREDICTION MODEL
MAML (Finn al., 2017) and Meta-SGD (Li al., 2017) are representative gradient- based meta- learning models to learn to generalize to few-shot regression tasks. These baselines perform set- and distillation-igg inneeiionnnddeeeeeennt prediction on each task. MetaD2A (Lee et al., 2021a) and TANS (Jeong et al., 2021), which are the most closely related to our work, learn dataset-conditioned accuracy prediction models that can generalize to unseen datasets.
D TRAINING DETAILS
In this section, we first describe the meta-training database. Then, we describe the details of two main training schemes used in our experiments as follows: deta--taining of the proposed meta- prediction model and 2) Knowledge distillation from the given teacher network to the student architecture.
D.1 META-TRAINING DATABASE
To learn the proposed meta- meta- prediction model, collect multiple DaNAS tasks. Each task consists of a dataset and teacher trained on the dataset, a set of student architectures, and actual accuracies of student architectures after distilling knowledge from the teacher network. As described in Section we use Tiny Imageet (Le & Yang, 2015) as the source dataset for meta-training and we generate a heterogeneous set of tasks by class-wisely splitting the dataset into 10 splits. We use subsets consisting of 8 and 2 splits for meta-training and meta-validation, respectively. We train teacher on each sub-dataset for 180 epochs. Then we randomly sample 2000 student architectures and, for each dataset-teache network pair, collect 200 student aichtecture-accuraee pairs after distilling knowledge from the teacher network. For each task, we consider the lataset-teacher pair while randomly selecting distilled student architectur-accuraay pair train meta-prediction models. Note
14

---

Published as a conference paper at ICLR 2023
that the collection of the meta- meta-training database and meta-training done only once for each search space, as the meta-trained prediction model can transferred to any DaNAS tasks with different unseen datasets and teacher networks at almost cost (forward cost get teacher network accuracy).
D.2 META-TRAINING THE PROPOSED PREDICTION MODEL.
We learn our meta-predictton model over task distribution via episodic training (Vinyals al., 2016). For each episode, we sample a meta-batche containing multiple tasks for each iteration. More specifically, the number of meta-batch is 8; in other words, randomly sample 8 tasks from the meta-training database for each iteration. Each task consists of a specific teacher network trained on specific dataset, the accuracy of the teacher network measured on the dataset, 50 student architecture candidates, and their actual accuracy obtained by distilling the knowledge from the teacher network each student architecture.
For each given task, the meta meta-prediction model 1) Takes a student architecture and teacher network, 2) Encodes the task proposed distillation-awe aware way; we leverage the embedding of the teacher, architecttre embedding of a student, and functional embedding of the student whose parameters are remapped from the teacher, and 3) Predict the distillation performance of the given student architecture. By minimizing the predicted and actual accuracy of the student, we learn the proposed meta-prediction model.
As mentioned in Eq. (6), we utilize the teacher network-accuracy pair for one-shot adaptation. When conducting the inner gradient updates with the teacher-aceuracy pair, the total number of inner gradient steps is set to 1. After the one-shot teacher-guided adaptation, the outer loss is calculated with randomly sampled student architecture-distllaatie network and dataset. During meta- eeta-training, adopt meta- reta-prediction model with the meta- validation database After fixing our meta-prediction model, a meta-test phase is conducted. As in the meta-learning phase, the one-shot adaptation strategy with the teacher network-accuracy pair was used in the meta-test phase.
performance pairs for the given teacher
D.3 KNOWLEDGE DISTILLATION
To distill the knowledge of the teacher network to the sampled student architecture, we need to minimize the KD-divergence between the probabilities of the teacher and student network. The probabilities of teacher and student network is as follows:
T exp(ze Per
Per exp(zxr,1/T)
(11)
Ej exp(2e s/T)'
Σ, exp(2s'
where žgr,, and & are ouputs before the last softmax layer, which is referred as logits, and T is temperature dyper-paramedd which means the degree soften logits. We use T as 6 for all experiments in ours. With H, which is cross-entropy loss, the KD loss LKD is as follows:
CKD 3 PP DT y) = a. H(y. +1
H(P. .. H(P P
(12)
For every knowledge distillation process, we distill the knowledge of the teacher network to the student by leveraging the KD-aware loss, LKD.a, for 50 epochs with learning rate of 5e-2 2 and we set the value of a as 0.5.
E EXPERIMENTAL SETUP
E. EXPERIMENTAL SETUP IN SECTION 5.3
KD is one of the representative model compression techniques to obtain small network to be used under resource-limited environments real-time applications. Considering the real-world KD scenarios, natural to assume that there resource budget given environment, and the optimal student architecture must be found within it. In Section 5.3, the teacher network is network with 7, 7.46M parameters, and MACs constraint on the DaNAS task is as follows: MACs < 1300M when the MACs of the teacher network is about 1450M. All the experimental values shown in this Section are based on 64 x 64 pixels.
15

---

Published as a conference paper at ICLR 2023
E.2 EXPERIMENTAL SETUP SECTION 5.5
We validate our method with the larger pixels (256 x 256) on ImageNet-IK, a representative large- scale real-world dataset, by conducting end-to-end DaNAS with the largest ResNet defined by Cai et al. (2020) as the teacher in Section 5.5. We first randomly sample 1, 0.0 architecture candidates; the size them in terms of MACs is 0.9x smaller than that of the teacher, from the ResNet search space defined by Cai et al. (2020). We predict the performance of the 1, 000 architectures and select the architecture with Top-1 predicted performances computed by each method. Then we distill knowledge from the same pre-trained ResNet50 teacher to the obtained student architecture with the same 50 epochs and 256 256 pixels and report the best performance evaluated on the validation set.
E.3 TIME MEASUREMENT FOR ADAPTATION AND NEURAL ARCHITECTURE SEARCH
In Section 5.2, we measure the time for collecting the few-shot student architecture-accuracy pair(s) and the time for conducting the single forward pass to get a teacher network network-accuracy pair. In all cases, the time taken measured we also measure the time required to find an optimal student architecture in DaNAS settings. We randomly sample 3, 000 student architectures and select the Top-1 student architecture found to be most optimal by each method. For every method, after obtaining the total time for scoring for 3, 000 samples first, the average search time per architecture was calculated. As in the previous case, all the time required for searching was measured on a single NVIDIA's RTX 2080 Ti GPU.
single NVIDIA's RTX 2080 Ti GPU. As shown in Table 3,
F VALIDATION ON LARGE- LARGE-SIZE PIXELS
To demonstrate the practical value of the proposed approach, conducted an experiment on a large-scale real-world dataset, which is ImageNet-IK, with large- size pixels of 256 x 256 in Section 5.5. experi-
Table 7: Validation on large-size pixels MACs values in this Table is based 224 224 pixels and the reported accracies are the mean and standard deviations over runs.
Furthermore, conducted
ChestX (224 x 224) Depth Coufie
ment similarly on ChestX which is the hardest cross-domain dataset with large- size pixels of 224 x 224 in Table 7. As shown in Table and Table 7, our meta- prediction model is robust to various im- age sizes, increasing practicality. We ob- served that the proposed method successfully searched for more accurate and efficient student architecture, which shows 79.00% and 44.34% distillation performances on ImageNet-IKK and ChestX, respectively, outperforming other meta-accuracy prediction baseline methods.
Ape. (9)
MAGs (M)
Witth Config.
Toucher
5,,5,
[37, 128, 256]
MasDDA
2,
[20, 104, 128]
TANS
41.79 0.68
4707.01
[1.2.5. 1]
[32, 44, 72, 256]
Ours
44.86 0.58
4532.10
[32, 64, 06. 128]
16


Reviews:
[{"review_id": "zOTvHrkIlXB", "reviewer": "Reviewer_FiPx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for Distillation-aware Network Architecture Search (DaNAS). The main component of the method is a “distillation-aware meta accuracy prediction model” which maps a (student architecture, dataset, teacher network) tuple to a prediction of the accuracy of the student on the particular dataset when the student is trained via knowledge distillation (KD) from the teacher network.\n\nThe distillation-aware-meta-accuracy-prediction model is adapted to a target task via gradient-based one-shot adaptation from one teacher-accuracy pair (i.e., the accuracy of the teacher network on the target task).\n\nExperimental results show that the proposed prediction model adapts to target tasks within 3.45 secs on average without direct training on the target tasks. Further, it is shown that the proposed method outperforms existing rapid NAS methods (such as meta-NAS methods and zero-cost proxies) on accuracy estimation and end-to-end DaNAS on a set of unseen tasks.\n", "summary_of_the_review": "Similar to related papers, this submission is incremental. However, I am not sure it is practically useful (e.g., tiny images, tiny models -- I think, though this was not clearly specified) and I believe it ignores not-so-recent developments on (the issues of work in) meta-learning.", "strength_and_weaknesses": "## Strengths\n\n**S1.** The method is well-motivated and mostly presented with clarity.\n\n**S2.** The experimental validation suggests the proposed method is effective, as compared to recent methods, in the DaNAS setting.\n\n## Weaknesses\n\n**W1.** My main concern with this submission (and related work) is the set of concerns raised in (Guo et al. ECCV 2020). In particular, the sets of tasks considered in the present submission come mostly from the same domain (e.g., natural images). What would one find out if the present work were to be evaluated across domains like those in (Guo et al. ECCV 2020)?\n\n**W2.** Table 5 shows a large performance gap between the teacher network and the result of DaNAS. What does this gap mean about the performance of the entire pipeline?\n\n**W3.** The experiments are limited to tiny images and the choice of architecture parameters (for teachers and students) were not discussed. In my view, the practical value of the proposed approach is not clear.\n\n**W4.** The remapping of parameters from the teacher network to a given student candidate is not clear to me. In particular, eqs. (6) and (7) do not seem to account for any dimensionality reduction.\n\n## References\n\nGuo et al. A Broader Study of Cross-Domain Few-Shot Learning. ECCV 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Yix2ebkuTt", "reviewer": "Reviewer_L5tN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "NAS for knowledge distillation requires architecture search for a student network which will learn well using teacher network. The method DaNAS proposes learning meta model for accuracy prediction given a dataset, teacher and student architecture and using this to find optimal student network.", "summary_of_the_review": "Authors describe an algorithm to generate student architectures for distillation from teacher network and dataset. The show emprically promising results.", "strength_and_weaknesses": "Strength\n1. Paper proposes a new method for fast generation of student architectures when training on new problems of knowledge distillation\n\nWeaknesses:\n1. Severely restricted architecture space: the NAS template restricts students to be smaller than teacher, further the students either have no kth layer or just copy the parameters of teacher for kth layer. While this is good to demonstrate empirical results, it will severely limit the impact in real world problems. \n2. the paper only shows results on small image datasets, I would like to see if model does well on other non image modalities and other real world image datasets - imagenet/faces\n3. I am not sure how the paper prevents the model from learning 1-1 mapping - i.e. every \"optimal\" student is same architecture as teacher. It needs to be clearly exposition as the teacher parameters are simply being copied.\n4. The idea shows good results but highly constrained - I think at least exploratory experiments showing use of LSTM etc will greately strengthen the paper", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "31cvTpHZvq", "reviewer": "Reviewer_T2PL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a teacher awared accuracy predictior which solves the issue for DaNAS that previous predictors are not aware of influence of the teacher model. The results demonstrate the efficacy of the proposed methods. ", "summary_of_the_review": "See previous sections for detailed comments, and I recommend the authors to revise the paper accordingly. \n\n\n# Updates after reading the response\n\nI thank the authors for providing me a informative response that addresses nearly all of my previous concerns.  As this work proposes to combine two sub-domain of NAS, distillation aware NAS and meta-learning one, it opens a new field compared to previous works. So this work should be considered as a new baseline method in this newly proposed domain. So no matter how simple this method is, it should be considered as novel.  \n\nThe only concern left is why people should care such a sub-field. However, I do not see this is a problem as we should encorage people to step out of the previous settings in my opinion. So I raise my score to accept. ", "strength_and_weaknesses": "Strength:\n\nS1: The idea is quite novel. The previous predictor-based NAS methods focus on accuracy, latency, FLOPs, and #params. The prediction of performance for a student after KD considering the given teacher and task is novel.\n\nS2:The motivation is interesting: previous DaNAS can not generalize well to any other tasks with new combinations and teachers(not flexible), thus requiring retraining for these tasks and teachers' combination. Meta NAS enhance the fast adaption to the new task with less training cost, but none of them considers the influence of the KD. This paper proposes a method that tackles the above two obstacles: meta leaned prediction model which is 1) meta-trained (for fast adaption); 2) takes as input the teacher encoding and is guided by the accuracy of the teacher during the mete training.\n\nHowever, I still have below concerns\n\nW1.Some logic is hard to follow, for example, in Intro, conventional NAS is sub-optimal for searching students under KD because the optimal student distilled from the teacher is different from an architecture trained from scratch. I do not understand why the training pipeline difference would cause conventional NAS not work. Maybe more sentences can help like: if we want to adopt NAS to search for a suitable architecture for KD, we should consider the influence of the components from KD (teacher and source dataset, etc.). But traditional NAS is designed for searching an architecture according to its evaluations trained from scratch instead of the KD aspects.\nThe logic should be explained. And there are many other examples, in the abstract, the author claims that meta-learning NAS is sub-optimal for DaNAS. But from my perspective, they are designed originally for solving the generalization issue for NAS, and of course, they are not the optimal choice for DaANS. The author claims they are not optimal  just because they want to adopt meta-learning in DaNAS and find that they do not consider teacher apsect so they are not optimal. The logic needs to be reconsidered here and for the rest of the paper.\n\nW2. typo: many sentences are too difficult to understand, eg on page 2: we propose a distillation-aware task encoding function that remaps parameters from the teacher to the student and considers the output from the embedding network of the remapped student as the embedding to estimate the teacher network’s impact on the final accuracy of the distilled model.\n\nW2.1. The writing is really hard to follow, especially when combing a lot of new concepts. More, concepts need to be expiated clearly for example the setting of the few-shot learning under this paper's scenario.\n\nW3.The layout is so confusing that the reference is several pages away from the table/figure and the caption is not clear to understand. For example, the main results are in Tab2, what is the unit of the results, I can not figure out them only by a few dataset names. And what are the model components: Set. PR. and Gui.? Please explain more in the caption (Tab2,3,4,5).  The figure is also confusing, what is collecting time in figure 4?\n\nW4.Please explain more clearly in section 5 about your training and searching setting when combining Meta, KD, few-shot learning and one-shot adaption, and NAS. This is really confusing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%