Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This article introduces a novel low-cost regression proxy task, which has a high correlation with the test performance of the final model. Its contributions include the following:

1. Introducing a novel low-cost regression agent task, which greatly reduces the cost of evaluation.
2. This proxy task is basically independent of downstream tasks, and can make the searched model achieve approximately consistent results in multiple search spaces and datasets.
3. Competitive results can be achieved in End-to-End NAS.
main_review: + **Originality**: This article designs a very interesting proxy task, which allows the network to fit a set of hand-designed bases through regression to determine whether the network performance is good or bad.
+ **Quality**:  There are a few questions:
1. In the proxy task, why the input of CNN is unlabeled real data, but the input of RNN is random data?
2. The image does contain information of different frequencies, so the basis of the design looks more reasonable for the image. But for RNN, is it reasonable to choose the same base?

+ **Clarity**: This article is basically written in a well-organized manner. From the introduction of proxy task, to why such a proxy task was designed, to experimental verification and proxy task improvement (proxy task search), the whole logical line is very clear.

+ **Significance**: Designing a low-cost proxy task is very important for NAS. The proxy task designed in this article uses a set of bases that have nothing to do with downstream data to evaluate the performance of the network at a very low cost. This is relatively novel. Previous work focused on network structure and data. This work introduces a new direction.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a generic NAS method named GenNAS, which uses regression as the self-supervised task. Moreover, the proposed method is task-agnostic and highly efficient. Experiments demonstrate the effectiveness of the proposed method. My detailed comments are as follows. 

Positive points:
1.	The authors proposed a new regression-based NAS method. 
2.	The results from experiments show the proposed method achieves promising performance.

main_review: Negative points:
1. In section 3.1.1, it is not clear how to divide a network into N stages. More details should be provided.
2. In the last line of Table 3, cls@ep10 method yields better performance than GenNAS-N. More explanations are required. Moreover, it seems that GenNAS should not be highlighted.
3. In section 4.4, line 352, it’s not clear that how to obtain L^*=4.36.
4. Several state-of-the-art NAS methods [1,2] should be compared in the paper.
5. Kendall Tau (KTau) is a widely used metric to measure the ranking ability. It would be stronger to use KTau to compare different methods.
6. Since this paper is a NAS paper, the results of the final searched architectures on CIFAR and ImageNet should be reported.

References:
[1] Proxylessnas: Direct neural architecture search on target task and hardware, ICLR 2019.
[2] Once-for-all: Train one network and specialize it for efficient deployment, ICLR 2020.

--Post Rebuttal--

I am happy to raise my score based on the additional results. I hope that all the results can be included in the paper as promised by the authors.

limitations_and_societal_impact: The limitations and potential negative societal impact of their work should be clarified.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This work made an interesting attempt to solve the generic neural architecture search (NAS) based on regression. A self-supervised regression task based on synthetic signal is proposed to evaluate the performance of a neural network architecture without the true labels of the images as in most previous NAS algorithms, leading to an efficient and effective search scheme. The ranking correlation coefficient, i.e., Spearman coefficient is taken to quantitatively evaluate the efficacy of this novel scheme on extensive search space.
main_review: Generally speaking, the whole idea seems a good attempt for the hot topic of NAS, leading to possible new insight to task agnostic NAS. Comprehensive tests across 13 CNN search space and 1 NLP space illustrate a promising ranking correlation between the ground truth and the network evaluated based on regression tasks.  However, some big concerns remain for its current form, which are shown in the below section.
limitations_and_societal_impact: 1.  Though for one-shot NAS, the ranking correlation seems to be a good metric for its performance evaluation. There is still some performance gap between the searched one and the stand-alone model. So the authors should better provide more justification on this aspect, that is, why the ranking correlation is enough to validate the effectiveness of their NAS scheme.

2. No strong analysis or no insight on why such a synthetic signal works effectively for evaluation of a network architecture.

3. The authors claimed that the proposed NAS is task agnostic, however, in all the experiments, only classification related experiments are provided to show the performance. As such, this seems not convincing to support such a claim.  More tasks such as detection, segmentation should be provided on this point.

4. Only CIFAR-10 results are reported for end-to-end NAS comparisons, which is not convincing to show the efficacy of the proposed scheme. ImageNet based results should be provided to further support this point.

5. Representation of this manuscript needs a lot of polish.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work designed synthetic signals as proxy tasks for Neural Architecture Search. The author claimed that self-supervised regression tasks on the synthetic signals can deliver task-agnositc NAS, and make searched results more highly correlated with ground truth accuracy of architectures.
main_review: 1. "thus" in the first item in the contribution list on page 2: I don't think it is the "switching from classification to regression in NAS" that leads to being "agnostic to the speciﬁc downstream tasks". This description could be misleading.
2. Making NAS agnostic to tasks is a reasonable motivation, but this is irrelevant using regression or classification as the loss function, but more related to whether or not using supervision signals. Many places (e.g. line 56-57, 139) make a reader feel that regression is vital, but I disagree. Introducing regression tasks in NAS is not a novelty. Properly designing proxy tasks is the root.
3. I could not see the necessity of using manually designed signals. If this is true, we should use large amount of manually designed signals for contrastive learning, to learn meaningful representations before downstream fine-tuning. Actually self-supervised learning on large-scale datasets like ImageNet, JFT can work very well recently [1,2]. Not to say the effort and uncertainty of manually designing these signals.
4. I think the correlation comparison in Table 2 is not fair. GenNAS can use combo of signals, but all other methods only measure the correlation on a single dataset. Maybe using combination of CIFAR and ImageNet-16 can further boost their correlation.
5. Another efficient & w.o. proxy task NAS work is not considered [3].

[1] Improved Baselines with Momentum Contrastive Learning

[2] A simple framework for contrastive learning of visual representations

[3] Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
limitations_and_societal_impact: My main concern is the usage of the regression tasks on manually designed signals, which may not widely adoptable (than unlabeld natural images) and further cost design efforts.
review_title: null

Meta-review: This paper proposes a novel approach to ranking architectures using only one minibatch of data by constructing unsupervised regression tasks for CNNs and RNNs via passing the inputs through manually designed feature extractors. The authors have shown the efficacy of the results on a large number of search spaces and baselines.

During discussion period a number of extra results and nuances have been elicited. The authors are strongly recommended to include these in the manuscript. Examples include the precision@K bin of architectures, Kendall-Tau scores and also properly reporting the GenNAS-N gpu hours since it otherwise hides the time for the extra task-specific search.

This ranking method can be quite valuable to the community, so it is strongy recommended that the authors  release high-quality code such that subsequent papers on the topic can utilize this very well.
Decision: Accept (Spotlight)