Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper set out to solve the domain discrepancy issue between source architecture space and the target space in Neural Architecture Predictors. The paper proposed a progressive subspace adaptation strategy to train a Cross-Domain Predictor.
strengths_and_weaknesses: Strengths: 
(1) The proposed CDP method is simple and intuitive, this idea of transfering knowledge from NASBenches and leveraging LMMD loss from transfer learning/domain adaption sounds novel to me.
(2) The performance of transfer learning in NAS is impressive to me, though the search cost comparison is a bit misleading as stated in the questions. 
(3) The paper is overall well-written and well-presented.

Weakness: 
(1) The 0.5306 KTau in Table 5 is not impressive, the author should consider reporting the TopK% (or Top-50/100 depending on search space size) Kendall-Tau correlation of predictor, since NAS only output the BEST predicted architecture, that's where different predictor-based NAS methods would make a real difference.
(2) The comparison methods (NAO, Neural Predictor, SemiNAS) in Table 1 seem to be outdated, a comparison with more recent Predictor-based NAS methods [1][2] should also be included: 
[1] BRP-NAS: Prediction-based NAS using GCNs, NeurIPS 2020
[2] Stronger NAS with Weaker Predictors. NeurIPS 2021

questions: (1) I'm curious how would the performance be, if we only include NAS-Bench-101 or NAS-Bench-201 as source domain?
(2) In Table 1/2, would that be unfair that other methods do not have any transferred knowledge, while CDP has prior knowledge from NAS-Bench with the training time of NAS-Bench not included.
limitations: (a) Some of the up-to-date methods are not compared as stated in the Weakness part.
(b) Some additional ablation is needed as stated in Questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Recent advances in Neural Architecture Search (NAS) investigate the neural predictor, which directly estimates the performance of neural architectures, to alleviate the costs of performance evaluation. However, the neural predictor still requires a set of annotated architectures for training. To alleviate this issue, this paper proposes a cross-domain adaptation strategy (CDP; Cross-Domain Predictor), which mitigates such training costs for annotation by utilizing existing labeled datasets.
strengths_and_weaknesses: Although this paper borrows existing concepts from domain adaptation, it does not hurt the significance of the work; I believe that converting key ideas across sub-fields (in this case, from domain adaptation into neural architecture search) is still valuable for academia.

Overall, the paper is well-written; it clearly presents the existing problems for training neural predictors and how the proposed method alleviates that. Applying domain adaptation methodologies to utilize existing labeled architecture space for another space seems reasonable. Moreover, NAS-specific improvements make the paper have its own contribution. I especially like the ablation study in Section 4.3, which measures the quality of the neural predictor for various settings of the proposed cross-domain adaptation strategies.

*Caveat: For the other experimental results in the paper (including Sections 4.1 and 4.2), I would defer to the other reviewers; since I am not familiar with NAS experiments.
questions: - How does this paper measure GPU hours? While the seminal work of DARTS [Liu et al., 2019] stated that “All of our experiments were performed using NVIDIA GTX 1080Ti GPUs.”, I cannot find such statements in this paper.
- Using Kendall’s Tau (KTau; Sen, 1968), in Section 4.3, the paper demonstrates the improvements from the proposed components for CDP (i.e., cosine scheduling, medium-sized assistant space, and LMMD) in terms of the quality of neural predictors. However, there are no comparisons with existing methods for training neural predictors, which utilize annotated datasets for the target space (e.g., Wen et al., 2020). It will be nice to check whether the proposed method produces high-quality neural predictors comparable to the previous approaches without such annotations.
---
- [Liu et al., 2019] DARTS: Differentiable Architecture Search. 2019.
- [Sen, 1968] Estimates of the Regression Coefficient Based on Kendall's Tau. 1968.
- [Wen et al., 2020] Neural Predictor for Neural Architecture Search. 2020.
limitations: This paper did not include both limitations and societal impacts of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops a cross-domain predictor, called CDP, which does not require training neural architectures in the particular search space. It leverages the existing NAS benchmark datasets and adopts domain adaptation to adapt from the source space of NAS benchmark to the target space. Therefore, it does not require to train architectures in the target space and largely reduces the cost.
strengths_and_weaknesses: Strengths:
1. The motivation is clear to leverage the existing NAS benchmarks for training-free NAS.
2. The proposed method is sound with proof.
3. Improved experimental performances against other works on ImageNet and CIFAR-10.

Weaknesses:
1. The method relies on existing NAS benchmarks. Therefore, it is hard to apply to some domain/tasks that do not have collection of architecture-performance pairs.
2. The design of assistant space could be impractical for other domains/complex spaces. The NAS-Bench space and DARTS space are still simple.
questions: 1. How does CDP perform if the two domains are for different tasks? For example, from image classification to NLP task?
2. How is the kernel function and mapping function designed?


limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work targets on leveraging leverage existing NAS Bench to predict out-of-domain architecture’s performance. The author propose to learn a predictor on network's performance, via closing the domain feature gap between source and target architecture space.
strengths_and_weaknesses: 1. The targeted question quite makes sense: to use prepared NAS benchmarks to learn the performance predictor.

2. The proposed method also sounds novel: to train the predictor by minimizing the distribution distance between the network's feature space and the label distribution.

3. The authors provide good ablation studies.
questions: 1. Eq. 9, the scheduler for the Subspace Partition, sounds empirical. The authors may consider down-play its contribution, or compare Eq. 9 with other possible schedulers to show its importance.

2. I can understand the importance of the Assistant Architecture Space. However, is there any possible principles in designing it? Otherwise it will hurt the generality of the proposed method.

3. d_H in Eq. 10 is not defined or discussed, or did I miss anything? I suppose it is a kind of distance measure between two domains?

4. I think Thm 2 is too general to be helpful to the main purpose of this (NAS) work. As far as I understand, the most insight from Thm. 2 is that larger m and m’ would give us a lower error bound. But this conclusion is very general and is not leveraged in this work (thus sounds less important).
limitations: I think in general the target problem and the method of this work is novel. However, some designs in the method (Eq. 9, and the assistant space) may be empirical and may hurt its practical use.
review_title: null

Meta-review: metareview: This paper introduces ideas from domain adaptation to improve NAS: leveraging leverage existing NAS Bench to predict out-of-domain architecture’s performance. This is an important question that has been overlooked, and the authors propose to learn a predictor via closing the domain feature gap between source and target architecture space. Both the problem setting and the proposed methods are novel. The authors also present good ablation studies, and the rebuttal was able to address a few clarification questions. After the rebuttal, all reviewers seem to be positive about this work, and the AC sides with them.
confidence: Certain
Decision: Accept