Towards Fast Adaptation of Neural Architectures with Meta Learning
Authors: Dongze Lian
Yin Zheng
Yintao Xu
Yanxiong Lu
Leyu Lin
Peilin Zhao
Junzhou Huang
Shenghua Gao
Venue:  (2020)
Keywords: Fast adaptation
Meta learning
NAS

Abstract: Recently, Neural Architecture Search (NAS) has been successfully applied to multiple artificial intelligence areas and shows better performance compared with hand-designed networks. However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. Generally, the architecture for a new task is either searched from scratch, which is neither efficient nor flexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal. In order to tackle the transferability of NAS and conduct fast adaptation of neural architectures, we propose a novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS. T-NAS learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task. Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.

Published as a conference paper at ICLR 2020
TOWARDS FAST ADAPTATION OF NEURAL ARCHITEC- TURES WITH META LEARNING
Dongze Lian'; Yin Zheng"; Yintao Xu', Yanxiong Lu", Leyu Lin', Peilin Zhao', Junzhou Huang* Shenghua Gao
Shanghai'ecch University.
Weixin Group, Tencent,
Tencent AI Lab, University of Texas at Arlington
(liandz, xuyt, gaoshh tech edu. cn, 1yzheng3xg)8gmal com, (alanlu, goshawklin, masonzhao www.eeettten.ooo.oo.omoo... {jzhuang) aathuang/uuta.edee .ddu eddu
ABSTRACT
Recently, Neural Architecture Search (NAS) has been successfully applied to mul- tiple artificial intelligence areas and shows better performance compared with hand-designed networks. However, the existing NAS methods only target a spe- cific task. Most of them usually do well in searching task but are troublesome for multiple datasets or multiple tasks. Generally, the ar- chitecture for new task is either searched from scratch, which is neither efficient nor flexible enough for practical application scenarios, or borrowed from the ones searched on other tasks, which might be not optimal. In order to tackle the trans- ferability of NAS and conduct fast adaptation of neural architectures, we propose novel Transferable Neural Architecture Search method based on meta-learning in this paper, which is termed as T-NAS. T-NAS learns meta- veta-architecturg that is able adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task. Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.
architecture for single
INTRODUCTION
Deep neural networks have achieved huge successes in many machine learning tasks GGishhick 2015 He et al. 2016 [Sutskever et 2014 Zheng et al. 2015b Lian et 2019 Cheng et al. 2219
Zheng al. [015a]] Lauly et al.] Diang et al. 2017 Zheng et al. (016)) Behind their successes,
the design of network architecture plays an important role, and the hand-designed networks (e.g.,
ResNet (He et al. 2016), DenseNet (Huang et al. 2017 have provided strong baselines in many tasks.
Neural Architecture Search (NAS) (Pham et al.) 2018 Liu al. 2018b; Guo et al. 2019) is pro- posed to automatically search network structure for alleviating the complicated network design and heavy dependence prior knowledge. More importantly, NAS has been proved to be effective and obtained the remarkable performance in image classification (Pham et al. object detection (Ghiasi et al.) (2019) and semantic segmentation (Chen al. 2018 Liu al. 2019)
2018 Liu et al. 2018b),
However, the existing NAS methods only target a specific task. Most of them usually do well in searching an architecture for single task but troublesome for multiple datasets multiple tasks. As shown in Figure [] we get the architecture-00 - on a given dataset using NAS method. Now, what if there exists new task? This drives us to ask: how to get suitable architecture for a new task in NAS? Generally, there exist two simple solutions in handling multiple tasks. One of them (S1) is to search an architecture for a new task from scratch but application scenarios. Another solution (S2) is to borrow architecture from the ones searched on other tasks but might be not optimal for the new task. Therefore, it is urgently needed to study the transferability of NAS for large-scale model deployment in practical application. It should be
inefficient and not flexible for practical
"quual contribution. this work is done when Dongze Lian works as an intern in Tencent Al Lab. "orresponning author.

---

Published as a conference paper at ICLR 2020
Daceet
Daleset
Inwiicient
X
Borrowed Iram
Howis guck
areit Rectare- 6
Anee
his optiral
X
Archfactun3 for
get archioccure now tosk?
Gurasiution
Figure 1: Left: how to search the network architecture when given new task? Middle: two sim- plc solutions that are inefficient or not optimal. Right: we propose T-NAS method to get meta- architecture, which is able to adapt to different tasks easily and quickly.
more desirable to learn transferable architecture that can adapt to some new unseen tasks easily and quickly according to the previous knowledge.
To this end, we propose a novel Transferable Neural Architecture Search (T-NAS) method (the bottom of Figure The starting point of T-NAS is inspired by recent meta-learning methods (Finn) et al., 2017: Antoniou et al. 2019 Sun et al., 2019), especially Model-Agnostic Meta-Learning (MAML) (Finn al.] 2017). where model learns the meta-weight weights that are able to adapt to new task through a few gradient steps. Push it forward, it is also possible to find a good initial point of network architecture for NAS. Therefore, the T-NAS learns meta-architecture (transferable architecture) that is able to adapt to a new task quickly through a few gradient steps, which is more flexible than other NAS methods. Similar MAML, such good initial meta-architecture for adaptation should be more sensitive to changes in different tasks such that it can be easily transferred. is worth mentioning that this is not the first work on the transferability of neural architecture. There are also some recent works that attempt to utilize the knowledge on neural architectures learned from previous tasks, such as Wong et al. (2018); Shaw et al. (2018). Specifically, |Wong et al.) (2018) proposes to transfer the architecture knowledge under a multi-task learning perspective, where the number of tasks fixed during training phase, and it cannot do fast adaption for a new task. In contrast, model is able to make the adaption fast and the number of tasks is unlimited during training. The difference between our model and [haw et al. (2018) is also obvious, where Shha et al. (2018) is based on Bayesian inference but model is based on gradient-based veta-learning. The quantitative comparison with Shaw et al. 2018) can be found Table 3
Generally, architecture structure cannot be trained independently regardless of network weights (Liu et al. 20186 Pham et al.) 2018). Analogously, the training of meta-architecture is also associated with meta-weights. Therefore, the meta-architecture and meta-weights need be optimized jointly across different tasks, which is a typical bilevel optimization problem (Liu et al.) (2018)). In order solve the costly bilevel optimization in T-NAS, we propose an efficient first-order approximation algorithm to update meta-architecture and meta-weights together. After the whole model is opti- mized, given new task, we can get the network architecture structure suitable for the specific task with a few gradient steps from meta- aeeh-erchitecture and meta-weights. At last, the decoded discrete architecture used for the final architecture evaluation.
To demonstrate the effectiveness of T-NAS, we conduct extensive experiments task-level prob- lems due to amounts tasks. Specifically, we split the experiments into two parts: few-shot learn- ing setting and supervised lcarning setting. For few-shot learning, T-NAS achieves state-of-the-an performance on multiple datasets (Omniglot, Mini-Imagenet, Newshot-CIFARIOO previous methods and other NAS-based methods. As for supervised learning. a 200-shot 50-query 10-way experiment setting is designed on the Mini-Imagenet dataset. Compared with the searched architectures from scratch for given tasks, T-NAS achieves comparable performance but with 50x less searching cost.
compared with
Our main contributions summarized as follows:
2

---

Published as a conference paper at ICLR 2020
We propose novel Transferable Neural Architecture Search (T-NAS). T-NAS can learn meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which is more flexible than other NAS methods.
We give the formulation of T-NAS and analyze the difference between T-NAS and other NAS methods. Further, to solve the bilevel optimization, we propose an efficient first-order approximation algorithm to optimize the whole search network based on gradient descent.
Extensive experiments show that T-NAS achieves state-of-the-arr performance in few-shot learning and comparable performance in supervised learning but with 50x less searching cost, which demonstrates the effectiveness of our method.
2 RELATED WORK
2.1 NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) designs network architectures automatically instead of hand- designed ones. Generally, NAS strategies are divided into three categories reinforcement learning, evolutionary algorithm and gradient-based methods. Some other strategies can refer to the survey paper (Elsken al. 2019). Reinforcement learning (RL) based methods (Zoph & Le 2016 Zoph ct al. 2018) utilize a controller to generate the network structure and operations. For efficient search- ing. ENAS (Pham et al. 2018) shares parameters among child models and achieves state-of fif-the-nd art performance with only one GPU day. Evolutionary algorithm based methods (Real et al. 2018) evolve neural architectures and also achieve comparable results with RL based methods.
Unlike reinforcement learning and evolutionary algorithm, gradient-based methods (Liu et al, 2018b Cai al., 2019) continuously relax the discrete architecture with all possible operations, which makes it possible to jointly optimize the architecture structure and network weights based on gradient descent. Not limited image classification problems, recent works also introduce NAS to object detection (Ghiasi et al. 2019) and semantic image segmentation (Chen et al. 2018
2018)Liu Liu et al.,
2019 More recently, NAS also applied to the generative model, such as AutoGAN (Gong al.,
2019
These NAS methods show that the searched networks outperform the hand-designed ones.
However, in these methods, only fixed architecture is searched for a specific task, which makes hard to be transferred to other tasks. In order to obtain more flexible network, InstaNAS (Cheng al., 2018) proposed to search the network architecture structure for each instance according to different objectives, such as accuracy latency. Different from Cheng et al. (2018), we in- corporate the ideas from meta-learning based methods and extend NAS to T-NAS, which learns a anchtarchitecctre that is able to adapt to different tasks.
2.2 FEW-SHOT META-LEARNIH
Recently, most few-shot learning problems can be cast into the meta-learning field, where a model trained quickly adapt new task given only a few samples (Finn et al. (017). Such few-shot
meta-lcarning methods can be categorized into metric learning (Vinyals et al. 2016 Sung et al.) Munkhdalai
2018 Snell al. [2017), memory network (Santoro et al., 2016; Oreshkin et 2018
et al. 2018 Mishra et al. 2018) and gradient-based methods (Finn et al. 2017 Zhang et al. 2018
Sun al. 2019).
Here, we only focus on the gradient-baase based methods, which contain base-learner and a meta-learner. MAMI. (Finn al., 2017) is one of the typical gradient-based methods for fast adaptation, which consists of meta-train and meta-test stages. In the meta-train stage, the model extracts general knowledge (mita-weights) from amounts of tasks such that can be utilized for fast adaptation in the meta-test stage. The latest variant of MAML is MAML+ (Antoniou et al., 2019), which analyzes the shortcoming of MAML and proposes some tips on how to train MAML to promote the performance. We extend the adaptation of weights in MAML to the adaptation of architectures that also based on MAML, and propose to automatically learn a meta-are architeccee, which is able to adapt different tasks quickly.
3

---

Published as a conference paper at ICLR 2020
3 PRELIMINARY
To introduce T-NAS, we briefly review the knowledge about meta- for fast adaptation (Finn al. 2017, Antoniou al. 2019) and DARTS for NAS (Liu al. 2018b) in this section, which helpful to understand the concept of T-NAS.
3.1 META-LEARNING
The whole dataset, meta-train and meta-test dataset are denoted D, meta--rain and D meetn-test spectively. In meta-train stage, set of tasks {7} (are also called episodes) are sampled from the task distribution p(T) in D meets-tain- Note that in the i-th task T,, there K samples from each class and classes in total, which is typically formulated as a N-way, K-shot problem. The train- ing split samples in T; used optimize the base-learner
called support set, denoted as T, and test split samples used to optimize the meta-learner are called query set, which T," The main idea of MAML (Finn et al. 2017) is learn good initialized weights w for all tasks [7], such that the network can obtain high performance in D meta-test after a few gradient descent steps from w. The base-learner is optimized according the following rule:
u t w Qinner Vwnn ((f(T,";("))
(1)
where Oinner the inner (base) learning rate of weights w and m represents the inner step. f the parametrized function with network weights and L is the loss function. In the base-leamnr process, T,, used to compute the loss and we update weights w from w to u. m=+ for the i-th task (w) w) After M steps, L(F(T, () in Ta computed for the meta-learner update, which can be formulated as:
W w
Olouter V
Σ LU(TT:W)))
T (PP)
where Oouter is the outer (meta) learning rate of meta- weigh-s w. Finally, the model learns the good initialized meta-weights w when it converges. Such meta-weights are sensitive enough so that it can adapt each task in meetastss after few gradient descent steps.
3.2 DARTS
2018b) is to continuously relax the discrete architecture with all The core of DARTS (Liu et al. possible operations and jointly optimize the architecture structure and network weights based on gradient descent. Consider a set of candidate operations, where each operation proposal represented with Given the input the output is the weighted sum of all possible operations o(r):
exp((.) Edeo I exp(O.)) o(x)
σ(x) = E NEO
where 0 is the vector to represent the coefficients of different operation branches. When decoding, the operation o* arg max.eo 9,. Therefore, 0 is also the encoding of the architecture.
To solve such bi-level optimization problem, two-step update algorithm is applicd:
{
0 = 0 BVo[(w <V C(w 0),
w = w - V, C(w,
where the loss function and is the learning rate of inner optimization. In this paper, we use the first-order optimization of DARTS (g 0) for efficiency.
APPROACH
In this section, we first introduce Transferable Neural Architecture Search (T-NAS) and give its for- mulation. After that, analyze and illustrate the difference between T-NAS and NAS. Finally, the first-order approximation algorithm proposed for the optimization T-NAS, and the adaptation and decoding process also described in detail.
4

---

Published as a conference paper at ICLR 2020
4.1 THE FORMULATION OF T-NAS
To make searched network architecture flexible, focus the transferability of NAS. As shown in Sec. 3, MAML. is trained to learn meta-weights w for fast adaptation in a new task. Similarly, T-NAS devotes itself to leam meta-architecture 9 that is able to adapt to a new task through few steps. In this work, 0 and 0 are defined as the encoding of the architecture and transferable architecture, which are represented as matrices following DARTS (Liu et al. 2018b).
To make the searched architecture transferable, we utilize the meta-learning based strategy to learn task-sensitive meta-architecture 0. However, similar other NAS methods (Pham et al. 2018 Liu et al. 2018b). where the architecture usually cannot be trained independently regardless of network weights the training of meta-architecture 0 is also associated with meta-weights W. In this work, 0 and W are optimized jointly across different tasks in T-NAS.
As shown in Sec. 3, there exist two learners for the learning of weighweights W, ie.. Eq. 1 used to update the base-learner and Eq. (2) is used to update the meta-leamer. Similarly, T-NAS consists of two searchers: base-searcher and meta-searcher. In the base-searcher, 0 and w are optimized jointly to search architecture for the specific task T,", which be optimized with:
m++ w;;" Orinner V WT L(g 8Thh w(")) W; 0m = 0Thh Binner V C(g(T;"; 0.hh w. w)++)
where Sinner is the inner (base) learning rate of architecture f. g the parametrized function with the architecture 0 and network weights w (6 0, u w) After M steps, and are also updated to get good initial point for architecture adaptation in the meta-searcher, where L[)(T,;; A.M w.)) in 7. computed. The formulation can be represented as:
W Olouter V
Σ
L(g(T,:0," ..)0 )(
TPmp((T
(6)
δ Ū- Bouter V
Σ OM, w/
T.
where Bouter is the outer (meta) learning rate of the meta-architecture 0. When the meta-searcher converges, the optimal meta-an arch-tccchee / and meta-weights こ can be obtained. We argue that such a can quickly adapt to new task. The complete algorithm of T-NAS is as shown in Alg.
Algorithm 1: T-NAS: Transferable Neural Architecture Search
Input: Mleaa-rrain dataset Dmeta-train- learning rate Ciinner Olouters Sunnr and Bouser- Randomly initialize architecture parameter 0 and network weights w.
while not done do Sample batch of tasks (T) in tranta
for T E {T} do
Get datapoints T
Compute C(g(T,";;0) w""") according to the standard cross-entropy loss; Alternatively update with and 8.mh with Eq. ( for M steps;
Get datapoints T," for meta-searcher;
end
10
Alternatively update vi and 0 with Eq. 6
11 end
It is worth noting that the transferability of architecture is generalized concept, which is not limited to the representation of architecture. T-NAS employs DARTS (Liu al.) (2018h) for NAS but other representations of architectures such ENAS (Pham al. 2018) also he adopted.

---

Published as a conference paper at ICLR 2020
Table 1: The main differences among NAS, Solution1 (S1), Solution2 (S2) and T-NAS.
Methods
Task(s)
Transferability
Characteristic
troublesome for
NAS
single
no no (search from scratch)
multiple tasks inefficient & time-consuming
multiple
borrows from
S2
multiple searched architecture
not optimal
T-NAS
multiple
adaptation
fiexible
4.2 T-NAS VS. NAS
As mentioned before, the previous NAS methods usually do well in searching an architecture for single task but are troublesome for multiple datasets or multiple tasks. So we focus on the trans- ferability of NAS across multiple tasks in this paper. Two simple solutions (S1 and S2) have been pointed in Figure but they are either inefficient or not optimal. T-NAS aims to learn a transfer- able and flexible architecture that can adapt a new task easily. Table [ lists the main differences among NAS, two simple solutions (S1 and S2) and T-NAS. S1 does not study the transferability of NAS and searches architectures for different tasks (e.g., 0,, 02,,...,,) from scratch. S2 borrows from searched architecture directly such that all tasks share the same architecture (e.g., 0). Differ- ently, T-NAS searches the meta-architecture 0, which is able to adapt to different tasks quickly (e.g., Ø + 01, 02,0....... .... 0,). The experimental results show that method achieves better performance than the S2 and comparable performance with S1 but with the less searching cost.
It is worth mentioning that if directly apply NAS to few-shot meta-learning, egg., MAML (Finn) let al. 2017), we will search a good network architecture for MAMI., which is named Auto-MAMI. In fact, Auto-MAML is special case of $2 in Figure [ where all tasks share the same architecture searched with a meta-learning method. In the experiments in few-shot learning, we also introduce Auto-MAML as a baseline. However, such shared architecture is not suitable for each task. Auto- MAML can outperform MAML but is inferior to T-NAS. The specific algorithm and experimental settings of Auto-MAML are provided in the supplementary material.
The core of T-NAS is based on MAML (Finn et al. 2017). which is kind of gradient-based meta- learning method. Recently, MAML++ is proposed by Antoniou al. (2019), which introduces several techniques to improve the performance of MAML. These techniques can also be utilized by T-NAS, which is termed as T-NAS++ in this paper. The experiments in Section 5 confirm that T-NAS++ can further improve the performance of T-NAS.
4.3 OPTIMIZATION
Although the formulation of T-NAS is proposed, the model is hard to be optimized directly according to Alg. On one hand, updating 0 and é introduces the high-order derivative in Eq. (6). On the other hand, the continuous relaxation of architecture makes amounts of memory occupied. At the first glance, such a problem might be solved by the first-order approximation in Liu et al. (0)8b), however, there still exists a lot of time overhead, even the experiments cannot be carried out when step M is large in Eq. (6). To tackle this problem, we transform the alternative update strategy of w and in Eq. (5) into simultaneous update, which means the w and are treated equally as the parameters of function g. Such a replacement can update parameters (w and ()) by only backpropagating once instead of twice. The Eq.
(5) can be modified to:
[w 0m++11 [w. 0m7) Tinner (wr ,099 ((g(T") : 0.Hh w"")),
(7)
where Vinner [Qinner Pinner]- In addition, to avoid the high-order derivative, we also utilize the first-order approximation to compute the derivation of wi'h and 0.M instead of w and as follows:
Tw: 0
[ 0
Mouter
(wn oN C(g(TT OM wx()
Temp(T)
These techniques include cosine annealing of the meta-optimizer learning rate, the adding of inner steps etc..

---

Published as a conference paper at ICLR 2020
where Mouter [Cooterr Bouter] Such modifications while maintaining comparable performance. Thus, we can use the Eq. ( and Eq. (8) to replace the Eq. (5) and Eq. (6) in line and line 10 of Alg. to update 8 and in the implementation.
than half of the search time and memory
4.4 ADAPTATION AND DECODING
Once 0 and obtained by training the base-searcher and the meta-searcher with the first-order approximation of Alg. I we can adapt them to the i-th task and get the task-specific architecture 0 for the specific task T, according to the following Alg. 2
Algorithm 2:
Adaptation and decoding
Input: Meta-test dataset mett-test, learning rate Otinner and Finer- The task task-specific architecture 0e for the i-th task Tr. Obtain the specific task T; from Deetu-test
Output:
Update wIth and 0 for M step with Eq. 7 and get 0.M:
Decoding 0.M to task-specific architecture 0f by following the method in Liu et al.
2018b)
Following previous NAS methods (Zoph & Le 2016 Zoph et al. 2018 Pham et al. 2018 Liu et al, 2018b), after getting 0f. we evaluate the task-specific architecture by training in the task T, from scratch. As shown in Sec. 5, the T-NAS achieves state-of-the-at performance in few-shot learning and comparable performance in supervised learning but with less searching cost.
5 EXPERIMENTS
We evaluate the effectiveness of T-NAS in both few-shot and supervised learning settings, as well as multiple datasets. For each dataset, conduct experiments containing architecture search and architecture evaluation. In the architecture search stage, we use T-NAS to search for a meta- architecture. In the architecture evaluation stage, we evaluate the transferred task-specific archi- tectures by training them from scratch and compare their performance with previous methods. S1 and S2 in the following sections mean two simple solutions in Figure except for the specific in- structions. Code is available
5.1 DATASETS
Omniglot is a handwritten character recognition dataset proposed in Lake et al. (2011). which con- tains 1623 characters with 20 samples for each class. We randomly split 1200 characters for training and the remaining for testing, and augment the Omniglot dataset by randomly rotating multiples of 90 degrees following (Santoro et al. 2016).
Mini- Imaieet dataset is sampled from the original ImageNet (Deng al. 2009) classes in total with 600 images for each class. All images are down-sampled to 84 84 pixels and the whole dataset consists of 64 training classes, 16 validation classes and 20 test classes.
There are 100
Newshot-CIFARIOR (FC100) dataset is proposed in Oreshkin et al. 2018). which is based on a popular image classification dataset CIFAR100 It is more challenging than the Mini- Imageneteeet due to the low resolution. Following Oreshkin et al. (2018), FC100 divided into 60 classes belonging 12 superclasses for training, 20 classes belonging 4 superclasses for validation and testing.
5.2 T-NAS FOR FEW-SHOT LEARNING
5.2.1
ARCHITECTURE SEARCH.
We first get the meta-architecture by optimizing the search network with first-order approximation of Alg. [ In the architecture search stage, employ the same operations as Liu et al. (2018b):
3 x 3 and separable convolutions, 3 and 5 x dilated separable convolutions, 3 max

---

Published as a conference paper at ICLR 2020
0 normal
Breduce
normal
reduue
0 normal
reduce
Architecture (Onormal: freducee searched with Auto-MAMI. (left), meta-architecture (Onormal, rrdduce) searched with T-NAS (middle), and the transferred architecture (0th (onemal) Oc Preduce) for the specific task Ti (right). The experiments are conducted in 5-way, 5-shot setting of Mini- Imagenet.
Figure 2:
pooling, 3 x average pooling, identity and zero. ReLU-Conv-BB order used for convolutional operations and each separable convolution is applied twice following (Liu et al.) 2018a b). For all datasets, we only use one (normal + reduction) cell for efficiency and preventing overfitting, thus the meta-architecture is determined by (Onoemal, Oreduce). Once 0 is obtained using T-NAS, we can obtain the optimal architecture 0 for the specific task T, from Alg. 2
We utilize the training and validation data of dataset for architecture search. In N-way, K-shot setting, we firstly randomly sample classes from the training classes, and then randomly sample K images for each class to get a task. Thus, there are N x K images in each task. On the Mini- imagenet dataset, One (normal + reduction] cell is trained for 10 epochs with 5000 independent tasks for each epoch and the initial channel is SGD to optimize the network weights wi"h and architecture parameter 0Th with inner learning rate O'nner 0.1 and Amner = 30. The inner step M is set as 5 for the trade-off between accuracy and efficiency. For the meta-searcher, we use the Adam (Kingma & Ba 2014 to optimize the meta- architecture 0 and network weights w with outer learning rate Clouter 10-3 and Bouter 10-3 All search and evaluation experiments are performed using NVIDIA P40 GPUs. The whole search process takes about 2 GPU days.
16. For the base-searcher,
use the vanilla
In addition, we also conduct Auto experiments where all tasks share the same searched ar- chitecture. Auto-MAML is special case of S2 of Figure where all tasks share the same architec- ture searched with meta-learning method. the practical algorithm, it is similar to T-NAS, which behaved removing the update for in the meta-searcher stage. However, in Auto-MAMI, can divide the whole dataset into two splits for the updates of and ã following the recent gradient- based NAS methods (Pham al. 2018 Liu et al. (018b). Here, the Dmettattain is divided into two 1. The specific algorithm for meta-train and meta-test and searched architecture structure can be found in the supplementary material.
independent splits Pusin-sppict and D spain-spliz with
To show the transfcrability of meta-architecture, we visualize the (encoding of) architecture searched with Auto-MAML, meta-architecture 0 searched with T-NAS, and transferred architec- ture 0f for a specific task Te in Figure 2 It is worth noting that the architecture encoding matrix (Onormal, Broduce) searched with T-NAS is smoother than that with Auto-MAML, which implies that (0 (Onormal, Oreduce) is easier to adapt to the specific task (0 - 0¹) than Auto-MAMI, thus the meta- architecture searched with T-NAS is more flexible.
5.2.2 ARCHITECTURE EVALUATION.
After getting the architecture structure 0 for task Tú, we evaluate 87 by training it from seratch. In architecture evaluation, we train the task-specific architecture for 20 epochs with 15000 indepen- dent tasks for each epoch. Note that different from Liu et al. (2018b), we directly use the searched network structure evaluate performance without any modification (e.g. g. the number of channels or layers). We optimize the network weights w with Qinner = 0.1 and = 5. We use Adam Kingma & Ba 2014) to optimize the meta-weights 인 with outer learning rate Crouter 10 The experimental results on Omniglot, Mini- Imagenet and FC100 are shown in Table. [2] Table. and Ta- ble. respectively, where T-NAS is based on first-order MAMI.. Specifically, T-NAS outperforms
8

---

Published as a conference paper at ICLR 2020
Table 2: 5-way accuracy results the Omniglot dataset.
Methods
1-shot
5-shot
Siamese Nets Koch et al. 2015)
97.3%
98.4%
Matching nets (Vinyals et al.
20T6
98. 1
98.9%
Neural statistician (Edwards & Storkey 2017)
98.1% %
99.5%
Memory Mod. Kaiser
2017
98.4%
99.6%
Meta-SGD LI el al.
2017
99.53 + 0.26 %
99.93 + 0.09%
MAML (Iin et al.
2017
98.7 1 0.4%
99.9 0.1%
MAML++ (Antoniou al. 2019)
99.47%
99.93%
Auto-MAMI. (ours)
98.95 + 0.38%
99.91
0.09%
T-NAS (ours)
99.16 + 0.34%
99.93 + 0.07%
T-NAS+ (ours)
99.35 + 0.32%
99.93 + 0.07 %
Table 3: 5-way accuracy results on Mini- Mini-Imagenet.
Methods
Arch.
#Param.
1-shot
5-shot
Matching nets (Vinyals
2016)
4CONV
32.9K
43.44 1 0.77%
55.31 + 0.73%
ProtoNets (Snell et al (017)
4CONV
32.9K
49.42 1 0.78%
68.20 + 0.66%
Meta-LSTM, (Ravi & Larochelle
2017)
4CONV
32.9K
43.56 + 0.84%
60.60 + 0.71 %
Bilevel Francescht al. 2018
4CONV
32.9K
50.54 + 0.85%
64.53 = 0.68%
CompareNets (Sung ct al.
2018
4CONV
32.9K
50.44 + 0.82%
65.32 = 0.70%
LLAMA Grant al. 2018
4CONV
32.9K
49.40 +:: 1.83%
MAML (Finn et al 2201
4CONV
32.9K
48.70 + 1.84%
63.11 = 0.92%
MAML (first-order) (Finn al. 2207
4CONV
32.9K
48.07 + 1.75%
63.15 = 0.91 %
MAML++ (Antoniou al. 2019)
4CONV
32.9K
52.15 + 0.26%
68.32 = 0.44%
Auto-Meta (small) Kim
2018
Cell
28/28 K
49.58 + 0.20%
65.09 + 0.24%
Auto-Meta (large) Kim ct al.
2018
Cell
98.7/94.0 K
51.16 + 0.17%
69.18 + 0.14%
BASE (Softmax) (Shaw et al. 2018
Cell
1200K
65.40 + 0.74%
BASE (Gumbel-Softmax, Shaw
2018
Cell 1200K
66.20 = 0.70%
Auto-MAMI. (ours)
Cell 23.2/26.1 K
51.23 + 1.76%
64.10 - 1.12%
T-NAS (ours)
Cell 24.3/26.5 K'
52.84 + 1.41% %
67.88 + 0.92%
T-NAS++ (ours)
Cell
24.3/26.5 K"
54.11 + 1.35% %
69.59 + 0.85 %
means the average parameters of architectures for evaluation.
MAML and Auto-MAML (52.84% vs. 48.70%, 51.23%), which validates advantage T-NAS. It also achieves better performance than other architecture transfer methods (e.g., BASE (Shaw et al. 2018)). Actually, since the advantage of T-NAS is that the architeccit could adapt to new task rather than using fixed architecture like MAMI. and Auto-MAMI, it usually has additional time cost for the adaption. Usually, the adaptation procedure costs about 1.5 seconds (1-shot) and 7.8 seconds (5-shot), which is negligible compared with the improvement of accuracy. Moreover, can also see that T-NAS++, which is an improved version of T-NAS described in Sec.4.2, achieves the best performance among all the baselines.
5.3 T-NAS FOR SUPERVISED LEARNING
Besides few-shot learning classification, we also conduct experiments on Mini-Imagena for general supervised learning. Different from few-shot learning, the architecture can be searched and trained for each task due to the sufficient samples, which can be regarded as S1 in Figure [1 Due to the lack of baselines in the supervised learning setting, we choose 10 tasks with 200-shot 50-query 10-way for each task based the Mini-Imagen dataset for meaningful experiments.
In the experiments of supervised learning, we follow the same setting as few-shot learning for trans- ferable architecture search. The difference is that we can train each task independently from scratch in architecture evaluation. For tasks in supervised learning, we train the task task-specific architecture for 200 epochs with cosine schedule, where the initial learning rate is 0.05. We the SGD with momentum 0.9 to optimize the network weights and crop the original image and flip randomly for data argumentation.

---

Published as a conference paper at ICLR 2020
Table 4: 5-way accuracy results FC100.
Methods
1-shot
5-shot
10-shot
MAML (Finn et al. 2017
38.1 + 1.7%
50.4 + 1.0%
56.2 + 0.8%
MAML+ (Antoniou et al. 2019)
38.7 + 0.4%
52.9 + 0.4%
58.8 + 0.4%
Auto- Auto-MAMIL MAML (ours)
38.8 + 1.8%
52.2 + 1.2%
57.5 = 0.8%
T-NAS (ours)
39.7 + 1.4%
53.1 + 1.0%
58.9 0.7%
T-NAS+ (ours)
40.4 + 1.2%
54.6 = 0.9%
60.2 0.7 %
Table 5: 200-shot, 50-query, 10-way accuracy results of supervised learning on Mini- -imi-emmgeeet.
Methods
200-shot
Time
Random
61.20 0.09%
N/A
$1
64.84 = 0.04%
266 min
S2
62.99 0.05%
N/A
T-NAS (ours)
64.23 + 0.05 %
5 min
The experimental results in the supervised learning setting are shown in Table. 5 In S1, we search the architecture for each of 10 tasks from scratch and evaluate them. For S2, directly five architectures searched respectively in five different tasks (sampled with 200-shot 50-query 10-way for each task in the meta-train dataset) for the evaluation in 10 tasks. For fair comparison, also pick five architectures randomly from search space for each task, evaluate them in the specific task, and report their average results. It is worth noting that it does not consume searching time to randomly generate architectures or directly use the prepared architectures searched in other tasks. Thus, the time of Random and Method2 in Table. is not applicable. Our T-NAS can learn meta- architecture 9 and get the task-specific architecture by only updating several steps from á instead of the shared architecture. Thus, T-NAS obtains better performance than random architectures and $2 (64.23% vs. 61.20% 62.99%). In addition, T-NAS achieves competitive performance with S1 but with 50x less time cost min vs. 266 min). The fact that the performance of S1 is superior to that of T-NAS slightly is because SI directly searches network architecture for different tasks from scratch, which is laborious well time-consuming. On the contrary, T-NAS can adapt to different tasks quickly by finding good initial point 0, which avoids laborious searching for many tasks and saves lot of time.
Finally, it is interesting that although the architectures searched with SI and those transferred from meta-architecture searched with T-NAS are different for the specific tasks, their final evaluation performance is very close and is better than that of the random architectures. Such observation implies that some subspaces in architecture search space might be suitable for specific task and T-NAS is able adapt architecture initialized with the subspaces.
CONCLUSION AND FUTURE WORK
In this paper, focus on the transferability of Neural Architecture Search, that is to say, how to get a suitable architecture for a new task in NAS? The two simple solutions are either inefficient or optimal. To tackle this problem, we propose novel Transferable Neural Architecture Search (T-NAS) for fast adaptation of architectures. Specifically, T-NAS learns meta-architectuue that is able to adapt to new task easily and quickly through few gradient steps, which is more flexible than the existing NAS methods. In addition, to optimize the whole search network, we propose an efficient first-order approximation algorithm. Extensive experiments show that T-NAS achieves state-of-the-art performance in few-shot learning setting. As for the supervised learning setting, T- NAS achieves comparable performance with other baselines but the scarching cost decreased by 50x, which demonstrates the effectiveness of our method.
For future work, we can study the transferability of NAS for those tasks from different task dis- tributions, where some transfer learning methods might be helpful. We hope that this work can provide some insights on the transferability of NAS, which might potentially benefit the roal-world applications.
10

---

Published as a conference paper at ICLR 2020
Acknowledgement. The work is supported by the National Key R&D Program of China (2018AAA0100704) and the National Natural Science Foundation of China (NSFC) under Grants No. 61932020. We would like thank Jiaxing Wang for some meaningful discussions.
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In JCIR, 2019.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
Liang-CCiehh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Advances in Neural Information Processing Systems, pp. 8699-8710, 2018.
An-Chieh Cheng, Chieh Hubert Lin, Da-Cheng Juan, Wei Wei, and Min Sun. Instanas: Instance- aware neural architecture search. arXiv preprint arXiv:1811.1000) 2018.
Hao Cheng, Dongze Lian, Bowen Deng. Shenghua Gao, Tao Tan, and Yanlin Geng. Local global learning: Gradually adding classes for training deep neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. ICML,, 2018
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpm: Learning scalable feature pyramid architec- ture for object detection. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 7036-7045, 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. arXiv preprint urXiv:1908) 03833, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In ICLR, 2018.
Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. Nat: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems, pp. 735-747, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
11

---

Published as a conference paper at ICLR 2020
Zhuxi Jiang, Yin Zheng. Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In International Joint Con- ference Artificial Intelligence, pp. 1965-1972, 2017.
Eukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning remember rare events. In ICLR, 2017.
Jaehong Kim, Sangyeul Lee, Sungwan Kim, Moonsu Cha, Jung Kwon Lee, Youngduck Choi, Yongseok Choi, Dong- Yeon Cho, and Jiwon Kim. Auto-meta: Automated gradient based meta learner search. arXiv preprint arXiv: 1806. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
Brenden Lake, Ruslan Salakhutdinov. Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Stanislas Lauly, Yin Zheng, Alexandre Allauzen, and Hugo Larochelle. Document neural autore- gressive distribution estimation. The Journal of Machine Learning Research, 18(1):4046-400) 2017.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 7707009835 2017.
Dongze Lian, Jing Li, Jia Zheng, Weixin Luo, and Shenghua Gao. Density map regression guided detection network for rgb-d crowd counting and localization. In The IEEE Conference on Com- puler Vision and Pattern Recognition (C (CVPR), June 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. arXiv preprini arXiv:1901 190.009800.0085 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint drXiv:1806.00055. 2018b.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. In ICLR, 2018
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In ICML., 2018
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv: /802.00000..1/11 088800/448, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with ing, pp. 1842-1850. 2016.
neural networks. In International conference on machine learn-
12

---

In Auto-MAML. we search good network architecture for MAML. In fact, Auto-MAML is spe- cial case of Method2 of Figure I in this paper, where all tasks share the same architecture searched with meta-learning method. In the practical algorithm, it is similar to T-NAS, which is behaved as removing the update for 0 in the meta-searcher stage. However, Auto-MAMI, we can divide the whole dataset into two splits for the updates of 0 and w following the recent gradient-based NAS methods (Pham et 2018 Liu et al. 2018b). Here, the Dmeta-train is divided into two independent The specific algorithm for meta-train and meta-test is Published as a conference paper at ICLR 2020
Albert Shaw, Bo Dai, Weiyang Liu, and Le Song. Bayesian meta-network architecture learning.
CoRR, abs/ 2018. URL hallhtt::/ httt:://arxive wwtp.i...gg/a////11/11111 09584
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Puttern Recognition, pp. 1199-1208, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks.
In Advances neural information processing systems, pp. 3104-3112, 12, 2014.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer learning with neural automl. In Advances in Neural Information Processing Systems, pp.
2018.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. In Advances in Neural Information Processing Systems, pp. 2365-2374 2018.
Yin Zheng, Richard S Zemel, Yu-Jin Zhang, and Hugo Larochelle. A neural autoregressive approach to attention-based recognition. International Journal of Computer Vision, 13(())67779, 201 5a.
Yin Zheng, Yu-Jin Zhang, and Hugo Larochelle. A deep and autoregressive approach for topic modeling of multimodal data. IEFE Transactions on Putlern Analysis and Machine Intelligence, 3866):05561106, 2015b.
Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hamning Zhou, A neural autoregressive approach to collaborative filtering. In International Conference on Machine Learming, pp. 764-773, 2016.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0157.. 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018
A THE EXPERIMENTS OF AUTO-MAML
splits spain-split and upain-spha2 with shown in Alg. 3
We follow the same definition for architecture search as T-NAS and we also use one {normal reduction] cell for Auto-MAML. The searched architecture 6* is shared by all tasks. We utilize the two splits of training data for architecture search. The search model is trained for 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16. For base-searcher, we use the vanilla SGD to optimize the network weight w with inner learning rate Qinner = 0.01. The inner step M is set as 5 for the trade-off between accuracy and time. For the meta-update, we use the Adam to optimize the network weights w and architecture with outer learning rate Quaner = 10-3 and 8 = 3 10 4. The hyperparameter setting for network evaluation is the same T-NAS. Here, we visualize some discrete architecture structure searched with Auto-MAML on the Mini- Amni-emaeenet dataset in Figure 3 and Figure
13

---

Published as a conference paper at ICLR 2020
My cour
pool 3x3
< 3k-3
[.13]
por
peep
aquarees
0g pol
skp
di core 3u5
(a) Normal cell
(b) Reduction cell
Figure 3: Architecture searched with Auto-MAML in 5-way 1-shot setting Mini-imagenet.
pool.Se Set
oVg pool
(K:1)
aap post
51d max pool 3nd someome
AM
(a) Normal cell
(b) Reduction cell
Figure 4: Architecture searched with Auto-MAML in 5-way 5-shot setting of Mini-imagenet.
B
ARCHITECTURES
The aim of this paper learn transferable architecture that is able adapt a new task through few gradient steps. Therefore, it is meaningless that directly decoding for the searched meta- architecture i without regard to the specific tasks. Here, we visualize the (encoding of) transferable architecture 0 [ searched with T-NAS and task-specific architecture 01 02, 0 in Figure 5 The matrix (Onormat: Broduce) searched with T-NAS is smoother than the task-specific architecture matrices (Onormats Of Fredyya), which shows that the meta-architecture is flexible and easy adapt to these specific task - 00,02, 0²))
C COMPLETE EXPERIMENTAL COMPARISON
In this section, we show the complete experimental comparison of our method with those methods using pretrained model in Table Some methods (Oreshkin et al. 2018 [Sun al. 2019) get better performance by employing more complex networks and pretrained model.
D PERFORMANCE COMPARISON ON CIFAR- 10 AND IMAGENET
To evaluate the transferability of our method, we also conduct the experiments CIFAR-10 and ImageNet. Firstly, we construct a larger dataset from ImageNet to learn the meta-architecture. and then adapt the meta-architecture on CIFAR- to decode the final architecture. We test the perfor- mance of the final architecture 7 and Table g From these two tables, we can see that the learned arttattcchit from T-NAS can quickly adapt new tasks and achieve favorable performance. For example, given the learned vetta-rrhiteectre from T-NAS, only takes 0.042 GPU days to derive an architecture that achieves test error of 2.98% CIFAR-10 10 and 27.2% on ImageNet. In contrast, searching for architecture that achieves similar performance from scratch on CIFAR-10 by DARTS (first order) would cost
CIFAR- 10 and ImageNet and report the performance in Table
'I is represented with matrix Liu et al. 2018b
14

---

Published as a conference paper at ICLR 2020
Algorithm 3: Auto-MAMI.
Input:
Dataset Dirain-splitt, D spliis inner learning rate Cinner. outer learning rate Outeer and architecture learning rate 3.
Output: The searched architecture 0*.
% Meta-train:
while not done do
% Update
Sample batch of tasks {7] in DunDaiii Ppain-split11 for Ti E {T) do
Get datapoints T.";;
Compute Vug" (TT 0, w(())) according the standard cross-entropy loss;
Update w with mo+++ = " - C'inner V uu ((," 0, )" for M steps;
Get datapoints 7.¹ for meta-update;
end 10
11
Update w with w = ã Qoouter V
LT,
12
% Update A
13
Sample batch of tasks (T} in spliz:
for T: E {T} do
15
Get datapoints T,";
Compute V W (Tg"((((T)))))) u(")) according the standard cross-yntropy loss;
Update w with wi++1 w O'ineer V (Tg(T w(')) ) for M steps;
17
Get datapoints 7," meta-update;
18
19
end
Update with VV ETiPp(T) ((g(T,"; wM)):
29
21 end
22 % Meta-test:
23
Sample tasks (T} in ppaiinsplit:;
24 for Te E (T) do
Update w!? with w m+++1 w.. O'inner V w L(g(T;;;, w:")) for M steps:
25
26
Compute test accuracy Acc; in T,":
27 end
28 Return architecture 0* according to the best average accuracy of [Acc].
Table 6: 5-way accuracy results on Mini-Imagenet.
Methods
Architectures
Parameters
1-shot
5-shot
Pretrained
TADAM Oreshkin et
2018
ResNar12
2039.11
58.5 + 0.3%
76.7 + 0.3%
Y
MTL Son ef
ResNet12
2039.22
61.2 1.8%
75.5
Matching neis
Vinyals
2016
4CONV
32.9K
43.44 0.77%
55.31 0.73%
ProtoNets Snell et ai 2017
4CONV
32.9K
49.42 0.78%
68.20 - 0.66%
N
Meta.1.SPP
helle
Bilevel (Fanceschi di A1
4CONV
32.9K
50.54 0.85%
64.53 + 0.68%
N
ComparcNets SSil al
2018
4CONV
32.9K
50.44 0.82%
65.32 0.70%
LLAMA Grasl
2018
4CONV
32,9K
49.40 1.83%
N
MAML Finn et 2017)
4CONV
32.9K
48,70 + 1.84%
63.11 = 0.92%
MAML (lirat-oruer)
4CONV
1.75%
MAML++ Antonion et
019
4CONV
32.9K
52.15
0.26%
68.32 0.44%
Auto-Meta (smalll
Kundel
2018
Cell
28/28
49.58
0.20% $1.16
65.09 0.24%
Auto Meta (large).
Cell
98.7/940 K
0.17%
69.18 0.14%
BASE. (Softmax)
Shaw al
2113)
Cell
1200K
65.40 0.74%
BASE (Gumbel-Softmans (Shaw
Cell
1200K
66.20 0.70%
Auto MAML.
Cell
23,2/26. 1 K
$1.23 1 1.76%
64.10 1 1.12%
24.3/26.5 K
1.41%
0.92%
T-NAS+
Cell
24.3/26.5 K"
54.11
1.5%%
69.59 0.85%
means the average parameters of architectures for evaluation.
1.5 days, which is about 36 times longer than that of T-NAS. This result confirms the advantage of T-NAS and also indicates that is possible to apply T-NAS to practical scenarios.
15

---

Published as a conference paper at ICLR 2020
38
10
92
42
80
12-
50
normal
2 reduuce
0 normal
01
reduce
10
10
300
0.5
02
2
1.3
4-
6
1-
3
2
normal
reduce
B
normal
0 reduce
Figure 5: architeccie matrix (Onormal, Oreduce) searched with T-NAS and three task-specific architecture matrices (8) (fhormat: 8r reduuee) The search experiments are conducted in 5-way, 5-shot setting of Mini-Imagenet dataset.
Table 7: Comparisons with state-of-the-art image classifiers CIFAR-10.
Test Error
#Param.
Search Cost
Methods
(%) )
(M)
(GUU days)
Random search baseline + cutout
3.29 = 0.15
3.2
NASNet-A + cutout (Zoph et al. 2018)
2.65
3.3
180
AmoebaNet-A cutout (Real
2018
3.34
3.2
3150
AmoebaNet-B + cutout (Real et al. 2018
2.55 + 0.05
2.8
3150
PNAS (Liu et al. 2018a)
3.41 + 0.09
3.2
225
ENAS+cutout (Pham et al. 2018
2.89
4.6
0,5
DARTS (first-order) cutout (Liu al.
2018b
3.00 0.14
3.3
1.5
DARTS (second-order cutout (Liu et al.
(018b)
2.76 0.09
3,37
4
Ours (first-order) cutout
2.98 + 0.12
3.4
0.043
Table 8: Comparisons with state-of-the-an image classifiers ImageNet in the mobile setting.
Test Error (%)
Search Cost (GPU days)
Methods
#Params (M)
top-1
top-5
NASNet-A Zoph et al.
2018)
26.0
8.4
5.3
1800
NASNet-B
Zoph et al.
2018
27.2
8.7
5.3
1800
NASNet-C
Zoph al.
2018
27.5
9.0
4.9
1800
AmoebaNet-A (Real et al.
20088
25.5
5.1
3150
AmoebaNet-B
(Ral el al.
2018)
27.2
8.7
5.3
3150
AmoebaNet-C RRal ct al.
2018)
27.5
9.0
4.9
3150
PNAS LLu et al.
201 Sa)
25.8
8.1
5.1
255
DARTS [Liu et al.
220 8b)
26.9
9.0
4.9
4
Ours
27.3
9.0
4.9
0.043
16


Reviews:
[{"review_id": "HklZo4iptr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the author propose a combination of the neural architecture search method DARTS and the meta-learning method MAML. DARTS relaxed the search space and jointly learns the structural parameters and the model parameters. T-NAS, the method proposed by the authors, applies MAML to the DARTS model and learns both the meta-parameters and meta-architecture. The method is evaluated for the few-shot learning and supervised classification.\nThe idea is an incremental extension of MAML and DARTS. However, the idea is creative and potentially important. The paper and method are well described. The experimental results indicate a clear improvement over MAML and MAML++. A search time improvement at the cost of a small drop in accuracy is showns in Table 5. The experiments conducted follow closely the setups from the original MAML paper. Some more focus to the standard benchmarks for NAS would have been great. A comparison to the state-of-the-art in NAS on CIFAR-10 or ImageNet is missing.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkeug-RptS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, which is named T(Transferable)-NAS, for neural network architecture search (NAS) by leveraging the gradient based meta learning approach. The state of the art NAS algorithms search an architecture for a single task and a new architecture is searched from scratch for each new tasks. T-NAS learns a meta-architecture in order to adapt to the new tasks through a few gradient steps. This works uses MAML (Finn et al 2017) and some of its variants for gradient based meta learning. But instead of learning the weights (w), they learn a task sensitive meta architecture (\\theta). To decode the learned \\theta to task-specific architectures, they follow the method proposed by Liu et al 2018b. In that case, different architectures is used for different tasks as opposed to the baseline algorithms.\n \nThis paper proposes an incremental approach which is a combination of the existing algorithms. The most important contribution of this paper is providing Equation (5) that is used to update w and \\theta parameters together by only backpropagating once. In this way, you don’t need the high-order derivatives and so you need less memory and search time. They compare T-NAS with state-of-the-art few-shot learning methods. I like the extensive empirical work of this paper. The prediction accuracy is mostly comparable with the baselines even some of the baselines are needed to be trained on many-shot classification tasks or use more complex architectures.\n \nI think it would be good to add the method of Liu et al 2018b as a background. Because it is used as a part of the proposed algorithm and it is important to know the method to understand how the learned parameter \\theta is adapted to the task-specific architectures. Besides that it is only mentioned once in the experiments section that T-NAS++ is based on MAML++ (which is an improved version of MAML, that investigates how to train MAML to promote the performance). If T-NAS++ is superior to the other baselines, including T-NAS, the difference should be clarified and emphasized more. \n \nTable-3 presents the experiment results in a confusing way. In the current version, it is not very clear which method performs the best. For example I prefer to see the best results in bold. In addition, maybe it could be better to present the results separately for methods that uses pretrained models and do not use pretrained models. The reason why some methods perform better than the proposed method could be mentioned in the text instead of mentioning it as a footnote. Finally, it would be better for the readers to see the time comparisons explicitly given that one of the most important contribution of this work is to increase the efficiency.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByexFlX9tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nCurrent neural architecture search (NAS) methods work in the mode of what is called S1 in this paper: given a dataset, search for a new architecture from scratch for that particular dataset. This paper proposes using MAML-style metalearning for learning meta-architectures across many meta-training tasks so that given a new test task (dataset) the meta-architecture is a few search steps away from a near-optimal one. Hence the name Transferable-NAS (T-NAS). \n\nThe main method pretty much follows what one would expect for a MAML-style approach. During meta-training a number of tasks (datasets) are used to learn a meta-architecture and corresponding meta-weights (Algorithm 1) and then during meta-testing time, given a new task (dataset) use any NAS search technique starting from the meta-architecture and corresponding meta-weights. (The authors use DARTS as the NAS technique which by itself is the same bilevel optimization as MAML, but for NAS. This sets up a 4-level optimization problem during meta-training time!).\n\nExperiments on two settings are presented: Few-shot learning and the more traditional supervised learning in NAS literature. \n\nComments:\n\n- I really like the premise of the paper. While there have been papers trying to leverage dataset-level features for NAS transfer via bayesian approaches, computing dataset level features is always a bit difficult. The nice part about the MAML is that the transfer is embedded into the representation itself. But I have a bunch of clarification questions which is quite possible is mostly due to my misunderstandings. So please bear with me:\n\n1. Section 5.2.1: \"On the Mini-imagenet\ndataset, One{normal + reduction} cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16.\" What are the 5000 independent tasks? In 5.1 it is said that 64 training classes, 16 validation classes and 20 test classes are present. I took that to mean that tasks are different classes of images in the few-shot setting. Clearly that is not the case. What precisely is a task in the few-shot learning case?\n\n2. Secton 5.3: \"...we choose 10 tasks with 200-shot, 50-query, 10-way for each task....\" Again, what precisely is a task in the supervised learning setting?\n\n3. One curious question I had is how much does architecture search matter as opposed to just taking the meta-architecture found by Alg 1 and just pretraining it on large image datasets like ImageNet or bigger and then just finetuning on downstream tasks. Specifically if you look at Table 3 ResNet12 pretrained has really good performance (although its size is ~100 times that of the others in the table). Perhaps S2 or AutoMAML with pretraining is the best option? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces T-NAS, a neural architecture search (NAS) method that can quickly adapt architectures to new datasets based on gradient-based meta-learning. It is a combination of the NAS method DARTS and the meta-learning method MAML.

All reviewers had some questions and minor criticisms that the authors replied to, and in the private discussion of reviewers and AC all reviewers were happy with the authors' answers. There was unanimous agreement that this is a solid poster. 

Therefore, I recommend acceptance as a poster.
Decision: Accept (Poster)