Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an unsupervised (UDA) performance predictor called USPP, which aims to mitigate the architecture performance prediction gap between source and target domain. 
strength_and_weaknesses: Pros:
1. The paper is well written and easy to follow.
2. The proposed method is technically sound. 
3. The idea of using UDA and adversarial training to address the domain gap issue in NAS is smart.

Cons:
1. The name of the paper is a little bit confusing at the first glance. I thought it is a method related to unsupervised learning, but turns out to be related to Unsupervised Domain Adaptation.
2. Missing recent baselines in the result table[1][2].
3. The author uses GPU day as a metric to evaluate searching speed. My question is that are the compared methods using the same GPU? If not, then they are not comparable.
4. The author only report one domain transfer result (NAS101 --> DARTS). I would expect to see more domain transfer case, like NAS101 --> NAS102,NAS102 -->DARTS etc.

[1] Mellor, J., Turner, J., Storkey, A., & Crowley, E. J. (2021, July). Neural architecture search without training. In International Conference on Machine Learning (pp. 7588-7598). PMLR.
[2] hen, W., Gong, X., & Wang, Z. (2021). Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535.
clarity,_quality,_novelty_and_reproducibility: 1. The paper is well-written and easy to follow, while the authors forgot to report what type of GPU are they using to report the searching speed.
2. The author has provided code.
3. The proposed method is somehow novel to me.
summary_of_the_review: The overall quality of this paper is good. The proposed method has promising results on the setting of transferring NAS101 to DARTS search space. I would expect the author to conduct more transfer settings to validate the effectiveness of the proposed method. I would like to recommend this paper to score 6 temporarily.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unsupervised performance predictor called USPP to reduce the training/search cost of NAS. To bridge the source and target search spaces, the authors develop a progressive domain-invariant feature extraction method to obtain domain-invariant features of architectures. Moreover, this paper provides sufficient ablation studies for each module/element of the proposed method. Nevertheless, both the novelty of the learning operation embedding and the application scenario of the proposed method are very limited.
strength_and_weaknesses: Strengths:
1. This paper develops a new NAS method that trains a NAS model and a regressor model in an adversarial scheme.
2. The proposed progressive domain-invariant feature extraction method is interesting.


Weaknesses:

1. There is a typo in the abstract. The accuracy on ImageNet should be 76.50% instead of 96.50%.

2. As highlighted in the paper, one of the contributions is the learnable operation embedding. However, it is not novel because almost all Reinforcement Learning (RL) based NAS methods exploit a learnable embedding for each operation, such as ENAS. Please further clarify it if there are some other differences from what is used in ENAS.

3. The motivation for reducing the divergence between source and target search space seems questionable. For example, in Figure 1, the architectures in source and target domains may share exactly the same or similar architecture. In other words, the discriminator would definitely fail in this case since it cannot distinguish between two identical/similar architectures. Instead, what should be bridged/distinguished between two spaces is the accuracy y instead of the architecture x.

4. The application scenario of the proposed method seems very limited. According to the paper, this method can be only used on the search space of DARTS. Given another search space, e.g., MobileNet based space used in OFA, is the proposed method still applicable in this case?


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow.
summary_of_the_review: The application scenario of the proposed method seems very limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a surrogate model that predicts the accuracy of an architecture.  The surrogate model has access to the architectures and their corresponding accuracies from the source search space. It is tasked with predicting the accuracies of architectures that are sampled from the target search space. It uses adversarial training for unsupervised domain adaptation. A Graph convolution network is used to extract the latent features from the architectures and serves as a generator. The discriminator in turn tries to differentiate the domain of the architecture.  A fundament layer F models the entire feature space.  A bridge layer B captures the domain specific features. The difference between the $F(x_{i})$ and $B(x_{i})$ gives the domain invariant features. The domain invariant features are fed to a predictor layer which in turn predicts the accuracy. 

The overall loss function is a combination of the domain loss from the discriminator, the bridge layer loss and the regression loss from the predict layer. The input to the surrogate model is an adjacency matrix. It learns an embedding for the operations of the architecture. 
strength_and_weaknesses: **Strengths**:
1. The paper is able to learn a surrogate model on the target search space without any training data

**Questions**:
1. Could you evaluate the quality of the domain invariant features that are learnt? For example, use the source domain as nasbench-201 and target as Darts, source domain as nasbench-101 and target as nasbench-201 and compute the distance (KL-Divergence) or similarity (cosine similarity) between the learnt domain invariant features using source domain 1 and source domain 2.
2. Could you also include a table for the regression error and the domain classification  accuracy?
3. Is USPP DANN + bridge layer. If so, in the ablation study, when the bridge layer is removed, why is the kendall tau different from DANN? For table 5, are you using operation embeddings for all the baselines?
4. For table 5, rather than training the architectures for darts search space, why not use NDS-Darts [1] benchmark? Evaluate it on 100 architectures sampled from that search space.  Similarly, for table 2, if you could actually use NDS-Darts to find the best architecture and report the numbers for other algorithms also from the same benchmark, it would be a fair comparison.
5. Please include darts-pt [2] also in table 1 and 2.

[1] On Network Design Spaces for Visual Recognition, Radosavovic et al.
[2] Rethinking Architecture Selection in Differentiable NAS, Wang et al.
clarity,_quality,_novelty_and_reproducibility: This paper is written very clearly. But the novelty of this paper is limited. The idea is heavily borrowed from the paper "Gradually Vanishing Bridge for Adversarial Domain Adaptation" by Cui et al. That paper uses a bridge layer for both generator and discriminator and use the domain invariant features to compute the classification loss and adversarial loss.


summary_of_the_review: Domain adaptation to learn a surrogate model for NAS is very useful for practical purposes. But the paper is mainly adapted from "Gradually Vanishing Bridge for Adversarial Domain Adaptation". Using operations for embeddings has also been done in the past. So the novelty is limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an unsupervised performance predictor for NAS which uses a progressive domain-invariant feature extraction method to obtain domain-invariant features of architectures. Although the topic is interesting and can potentially reduce the NAS search cost, all the reviewers agree that the current experiments are not sufficient to show that the proposed method works in general cases. To justify the use cases of the proposed method, we suggest the authors conduct experiments on other sources and target domains, beyond transferring from NAS-bench to DARTS space. 
justification_for_why_not_higher_score: Although the direction studied in the paper is interesting, all the reviewers think the current experiments are not sufficient to justify the use cases of the proposed method.
Decision: Reject