Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors address neural architecture search (NAS) scenarios. In particular, a framework, MetaD2A, is proposed, which yields a neural architecture for a new dataset. In a nutshell, the framework learns a "dataset-to-neural-network-architecture" transformation using a database of datasets and architectures. Each dataset is encoded via a "set encode" and the architecutres are obtained via a "graph decoder". The experiments demonstrate the usefullness of the approach and its improvements over conventual NAS approaches. The approach could be described 

Positive:
- The results look very solid and indiciate improvements (time/prediction performance) over existing approaches
- The paper is well written and structured
- Additional details (e.g., implementation details) are provided in the appendix

Negative:
- The authors claim that NAS with meta learning has only been done with small datasets in the past. However, the authors do not really use big datasets as well (see Appendix C; the ImageNet subset considered is small as well, if I understand this correctly)

To sum up, I think the work might be a suitable candidate for being accepted at ICLR. I have to admit that this is not precisely an area of my expertise, so I might be missing something.

review_title: Solid paper, weak accept

final_score: rating: 4: Ok but not good enough - rejection
review_content: Pros:
1.	This paper proposes new scene, where a meta model is pre-trained on some datasets, and transfer the learned meta feature onto other datasets to do fast adoption. This scene can be applied in variety of domains.
2.	The experiment shows this method can fast adapt NAS from one image dataset to others and achieve SOTA performance. 
3.	The paper is well-organized, the paper structure is clear.

Cons:
1.	The three parts of your model are of little novelty. The dataset encoding part is just borrowed and there is no improvement to adapt NAS tasks. Similar graph decoder is proposed in previous NAS works [1] and performance predictor are proposed more times. It seems that the proposed framework is just putting existing models together. 
2.	No comparing to other methods on fast adaptation by NAS such as [2]. Besides, meta-learning methods may also be compared.
3.	According to Figure 8 and Figure 9, it is likely that your graph decoder can only generate one type of edge connections. Your graph decoder may fail. Since your framework needs other methods (GDAS / NAS-Bench-201) to provide training material. These materials are all good architectures. It is possible that your framework just gives architectures the same as those good architectures rather than using meta features. Your experiment should prove that your model can generate variety of architectures.

Overall Review:
This paper proposes a new scene of fast adaption of NAS, which may be a good direction of NAS & meta-learning. The paper proposes a framework to generate good architectures according to the datasets. However, the model may need to improved and more experiment need to be done to solve the problems mentioned above.

[1]Does unsupervised architecture representation learning help neural architecture search? NeurIPS 20’
[2]Fast neural network adaptation via parameter remapping and architecture search ICLR 20’ 

review_title: This paper proposes a framework to learn meta knowledge which helps to transfer architecture search task among different datasets. This is a new attempt to adopt meta NAS on such a scene. This work encode dataset into embedding space, then sample a vector from the space and decode it into an architecture. A predictor is also used to find the optimal architecture. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a novel framework named MetaD2A. Their motivation is interesting to me and they clarify their difference compared with traditional metaNAS in Figure 1. There are mainly three components in MetaD2A: a set encoder, a graph decoder and a meta-performance predictor. I think the main contribution of this paper is their intuition that performing neural architecture search rapidly from the datasets, while the components are all proposed before in different NAS scenarios. In summary, the paper has following drawbacks that need to be further concerned:

1. Since the predictor consists of two linear layers, we cannot take architecture with different nodes as input. Thus, it will limit the generalization ability of the whole algorithm. Moreover, there are also different kinds of performance predictors in NAS field like LSTM and GCN predictor [1]. And I prefer to see the effect of predictor part with different predictors.  

2. In Table 1, MetaD2A is pretrained on Meta-ImageNet while other baselines are trained from the scratch. I think this is unfair to compare them. It's better to compare MetaD2A with some MetaNAS methods listed in related work part such as [2]. In my view, I think MetaD2A pays more emphasis on meta-learning in NAS field.

2. In ablation study section, they only compare MetaD2A with random, which is a somewhat weak baseline. However, the improvement of their method is not significant. I think there are some other perspectives to perform ablation study. For instance, they can replace hierarchical set pooling with flatten set pooling and then see the importance of different component.

[1] Shi, Han, et al. "Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS." Neural Information Processing Systems. 2020.

[2] Lian, Dongze, et al. "Towards fast adaptation of neural architectures with meta learning." International Conference on Learning Representations. 2019.
review_title: The authors present a novel framework to learn architecture efficiently from datasets. Even though some components have been proposed before, the way of combination is interesting to me. However, their experiment results are not convincing.

Meta-review: The authors proposed a meta learning framework for NAS, namely MetaD2A (Meta Dataset-to-Architecture), that can stochastically generate graphs (architectures) from a given set (dataset) via a dataset-architecture latent space learned with amortized meta-learning. Each dataset is encoded via a set encoder and the architecutres are obtained via a graph decoder. MetaD2A is trained once on a database consisting of datasets and pretrained networks and can rapidly search a neural architecture for a novel dataset. While the set encoder and graph decoder for NAS have been introduced by existing work, the main contribution of the paper is to show that the meta-learning of a "dataset-conditioned architecture generation" framework can enable fast generation of a good architecture without training on the target dataset. The proposed method is interesting and effective, however it requires an existing pool of good architectures for a given task, which may limit its applicability to a diverse set of real-world problems. I strongly encourage the authors to include experiments on a larger pool of architectures than the NAS-Bench-201 search space to show the strength of their proposed method in generating good architectures. While training MetaD2A with pairs of MetaImageNet and randomly sampled graph shows that the proposed framework can generate graphs with different types of edges, it doesn't show that it can successfully meta-learn to produce better architectures for a new task from an existing pool of good architectures. 

We believe that many of the reviewers comments were addressed in the rebuttal, so while the scores are low, they do not reflect neither the contribution nor the reviewers opinion well (e.g., R3, in his last post, seems to suggest that his review should be updated but it has not happened).
Decision: Accept (Poster)