Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method, which is named T(Transferable)-NAS, for neural network architecture search (NAS) by leveraging the gradient based meta learning approach. The state of the art NAS algorithms search an architecture for a single task and a new architecture is searched from scratch for each new tasks. T-NAS learns a meta-architecture in order to adapt to the new tasks through a few gradient steps. This works uses MAML (Finn et al 2017) and some of its variants for gradient based meta learning. But instead of learning the weights (w), they learn a task sensitive meta architecture (\theta). To decode the learned \theta to task-specific architectures, they follow the method proposed by Liu et al 2018b. In that case, different architectures is used for different tasks as opposed to the baseline algorithms.
 
This paper proposes an incremental approach which is a combination of the existing algorithms. The most important contribution of this paper is providing Equation (5) that is used to update w and \theta parameters together by only backpropagating once. In this way, you don’t need the high-order derivatives and so you need less memory and search time. They compare T-NAS with state-of-the-art few-shot learning methods. I like the extensive empirical work of this paper. The prediction accuracy is mostly comparable with the baselines even some of the baselines are needed to be trained on many-shot classification tasks or use more complex architectures.
 
I think it would be good to add the method of Liu et al 2018b as a background. Because it is used as a part of the proposed algorithm and it is important to know the method to understand how the learned parameter \theta is adapted to the task-specific architectures. Besides that it is only mentioned once in the experiments section that T-NAS++ is based on MAML++ (which is an improved version of MAML, that investigates how to train MAML to promote the performance). If T-NAS++ is superior to the other baselines, including T-NAS, the difference should be clarified and emphasized more. 
 
Table-3 presents the experiment results in a confusing way. In the current version, it is not very clear which method performs the best. For example I prefer to see the best results in bold. In addition, maybe it could be better to present the results separately for methods that uses pretrained models and do not use pretrained models. The reason why some methods perform better than the proposed method could be mentioned in the text instead of mentioning it as a footnote. Finally, it would be better for the readers to see the time comparisons explicitly given that one of the most important contribution of this work is to increase the efficiency.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper the author propose a combination of the neural architecture search method DARTS and the meta-learning method MAML. DARTS relaxed the search space and jointly learns the structural parameters and the model parameters. T-NAS, the method proposed by the authors, applies MAML to the DARTS model and learns both the meta-parameters and meta-architecture. The method is evaluated for the few-shot learning and supervised classification.
The idea is an incremental extension of MAML and DARTS. However, the idea is creative and potentially important. The paper and method are well described. The experimental results indicate a clear improvement over MAML and MAML++. A search time improvement at the cost of a small drop in accuracy is showns in Table 5. The experiments conducted follow closely the setups from the original MAML paper. Some more focus to the standard benchmarks for NAS would have been great. A comparison to the state-of-the-art in NAS on CIFAR-10 or ImageNet is missing.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:

Current neural architecture search (NAS) methods work in the mode of what is called S1 in this paper: given a dataset, search for a new architecture from scratch for that particular dataset. This paper proposes using MAML-style metalearning for learning meta-architectures across many meta-training tasks so that given a new test task (dataset) the meta-architecture is a few search steps away from a near-optimal one. Hence the name Transferable-NAS (T-NAS). 

The main method pretty much follows what one would expect for a MAML-style approach. During meta-training a number of tasks (datasets) are used to learn a meta-architecture and corresponding meta-weights (Algorithm 1) and then during meta-testing time, given a new task (dataset) use any NAS search technique starting from the meta-architecture and corresponding meta-weights. (The authors use DARTS as the NAS technique which by itself is the same bilevel optimization as MAML, but for NAS. This sets up a 4-level optimization problem during meta-training time!).

Experiments on two settings are presented: Few-shot learning and the more traditional supervised learning in NAS literature. 

Comments:

- I really like the premise of the paper. While there have been papers trying to leverage dataset-level features for NAS transfer via bayesian approaches, computing dataset level features is always a bit difficult. The nice part about the MAML is that the transfer is embedded into the representation itself. But I have a bunch of clarification questions which is quite possible is mostly due to my misunderstandings. So please bear with me:

1. Section 5.2.1: "On the Mini-imagenet
dataset, One{normal + reduction} cell is trained 10 epochs with 5000 independent tasks for each epoch and the initial channel is set as 16." What are the 5000 independent tasks? In 5.1 it is said that 64 training classes, 16 validation classes and 20 test classes are present. I took that to mean that tasks are different classes of images in the few-shot setting. Clearly that is not the case. What precisely is a task in the few-shot learning case?

2. Secton 5.3: "...we choose 10 tasks with 200-shot, 50-query, 10-way for each task...." Again, what precisely is a task in the supervised learning setting?

3. One curious question I had is how much does architecture search matter as opposed to just taking the meta-architecture found by Alg 1 and just pretraining it on large image datasets like ImageNet or bigger and then just finetuning on downstream tasks. Specifically if you look at Table 3 ResNet12 pretrained has really good performance (although its size is ~100 times that of the others in the table). Perhaps S2 or AutoMAML with pretraining is the best option? 
review_title: Official Blind Review #1

Meta-review: This paper introduces T-NAS, a neural architecture search (NAS) method that can quickly adapt architectures to new datasets based on gradient-based meta-learning. It is a combination of the NAS method DARTS and the meta-learning method MAML.

All reviewers had some questions and minor criticisms that the authors replied to, and in the private discussion of reviewers and AC all reviewers were happy with the authors' answers. There was unanimous agreement that this is a solid poster. 

Therefore, I recommend acceptance as a poster.
Decision: Accept (Poster)