Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work the Authors introduce the total path count (TPC) of a network as a zero-cost measure of network performance that can be used to rank CNN architectures very efficiently.

They further combine the TPC as a performance predictor with an evolutionary algorithm as a search strategy to introduce a new architecture search method (TPC-NAS). 

TPC-NAS is then used to find high-performing architectures with hardware constraints on image classification, object detection and super-resolution tasks. 
strength_and_weaknesses: This paper introduces the TPC as an extremely efficient architecture performance predictor that depends only on the architecture topology. This can significantly speed up NAS algorithms when combined with different black box search strategies.

However, the Kendall tau analysis in section 3.4 is very limited, and the search space is not clearly described. Extending the experiments would be useful, in order to have a better assessment of how TPC performs in ranking architectures. One can for example sample architectures from CNN NAS search spaces such as NAS-Bench-201 whose architecture test accuracies can be queried as well. 

Furthermore, in the experiments section it is not clearly stated whether the search spaces are the same for all methods in table.1. In general, in section 4, further experiments could be useful to clarify whether it is the TPC scoring method, the evolutionary search strategy, or the chosen search space which plays the most important role in achieving high performance on different tasks. 

(Also, the mutation operation in algorithm 2 is not defined, and in the main text there is no reference to the search space defined in the appendix.)
clarity,_quality,_novelty_and_reproducibility: The paper is overall well written and easy to follow. But, some experiments need further clarifications, as described above. The TPC scoring method, and application to object detection and super-resolution seem to be the main contributions of this work. Code is also provided for reproducibility. 
summary_of_the_review: The efficient architecture performance predictor introduced in this paper is interesting and can in principle be very useful for neural architecture search if it is proven to be effective. However, experiments in section 3.4 are not extensive enough to support this fact. Moreover, some confusions need to be addressed in the experiments section. I would therefore rate this paper as marginally below acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposed an efficient neural architecture search (NAS) algorithm for computer vision tasks including 2D image classification, object detection, and super-resolution. The proposed algorithm introduces the total path count (TPC) score as an accuracy predictor. The TPC score is computed based on the shape of convolution kernels to indicate models’ expressivity. And its computation can be efficiently achieved within few minutes. Moreover, the experimental results validate the proposed algorithm with several datasets of different computer vision tasks. All experiments use convolutional neural network (CNN) based search space.
strength_and_weaknesses: <Strength>

•	The paper is well-organized and well-written.

•	The experimental results supports the claims made in the manuscripts.

•	The submitted content is related to the application of fundamental tasks in computer vision, which is highly relevant to the ICLR audience.

<Weakness>

•	The novelties of the proposed algorithm is limited. Computing the TPC score is similar to study the effective receptive field of CNN models. It is a fact that large receptive field helps models’ expressivity because of long-range dependencies between image pixels.

•	The impact of the proposed approaches could be limited. Because it is applied to CNN models only and within a relative small search space.

•	Discussion or ablation studies are not sufficient (see below).

<Comments>

1.	How to extend the TPC score to the popular neural networks like various vision transformers or models with self-attention layers?

2.	Does the training recipe matter for the NAS outcome? Does different training recipes or with different random seeds result in different Kendall rank correlation coefficients?

3.	The manuscript claimed that the TPC score can be computed within a few microseconds. Please further clarify the device for such computation (e.g., what type of CPU or GPU)?

4.	Does the TPC score correlate with the accuracy in a higher order? For instance, does large TPC score gap mean larger accuracy gap?

5.	What are the (approximated) Kendall rank correlation coefficients for object detection and super-resolution?

6.	It is important to get a high ranking correlation for the model candidates with top performance because they matters more for final architecture selection. It would be better to compute the Kendall ranking coefficients for the models with top performance (e.g., top 5~10% accuracies).

7.	What are the typical cases for good models with low TPC scores? What are the reasons causing the low scores for the models?

8.	Can the TPC score be used for comparison of different CNN models (e.g., ResNet versus Inception network)?

9.	Typo: Page 4, “RELU” => “ReLU”.

clarity,_quality,_novelty_and_reproducibility: The manuscript is of moderate quality, with good clarity and limited novelties.
summary_of_the_review: My concerns about the manuscript are about its novelties and practical impacts. The novelties of the proposed algorithm is limited because the TPC score concept is very similar to the model’s receptive field. And the proposed algorithm may not be practical as it is limited for CNN models only and applied within a small search space.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the total path count (TPC) score as an efficient accuracy predictor for neural architecture search. Particularly, the TPC score is defined as the number of paths of the ”vanilla” convolutional neural network, which is very simple to come by. In experiments, the proposed approach was evaluated on three vision tasks: image classification, object detection, and super-resolution.
strength_and_weaknesses: Strength 

- proposes the total path count (TPC) score as an efficient accuracy predictor for neural architecture search

- outperforms the most relevant hand-crafted and NAS-discovered architectures on three applications

Weaknesses

- Clarity needs to be improved. Some important sections/functions are not well explained. For example, there is not a detailed explanation regarding the MUTATE function. It would be better to include an explanation to make the paper to be self-contained.

- Experiments are lacking. There are no ablations on the hyperparameters, like Search Space S, hardware constraints K, maximal depth D, number of iterations M, population size N, and initial structure F0. The paper only compares the best results with the state-of-the-art approaches but does not explain the sensitivity to the hyperparameters. It is hard to justify the superior performance of the proposed approaches without giving the hyperparameters.

clarity,_quality,_novelty_and_reproducibility: An anonymous GitHub link is included in the paper for reproducibility.
summary_of_the_review: This paper proposes the total path count (TPC) score as an efficient accuracy predictor for neural architecture search and outperforms the most relevant hand-crafted and NAS-discovered architectures on three vision applications. But clarity needs to be improved and experiments are lacking.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a training free algorithm for Neural Architecture Search (NAS). The proxy metric (TPC) used is the number of paths from the input to the output layer. To generalize it for very large networks, the network can be viewed as a graph and one could compute the outdegree of a node. This score is then used to rank all the architectures. TPC-Score was used a proxy instead of validation accuracy in  an evolutionary algorithm and neural architecture search was performed.  For image classification, they used ResNet search space where they constrained the number of model parameters to 1M. In addition to this, TPC-NAS was used to search for object detection architecture in the YOLOv4 search space. This metrics is very faster compared to other training-free proxies.
strength_and_weaknesses: **Strengths**
1. It is almost instantaneous to compute the metric for a given architecture
2. The experiments demonstrated that TPC-NAS is able to find well performing networks in ResNet search space to solve image classification and object detection.

**Weakness**
1. In Figure 3, the Kendall tau is computed on only 20 architectures. We need at least 100 architectures for the correlation to be significant.
2. NASWOT and TE-NAS are computed on NasBench 101 and 201. So please demonstrate how effective your metric is on those 2 search spaces for at least 2 datasets. Also, please don't restrict the number of parameters
3. As mentioned in Section 5.1, when the dataset is not taken into consideration while computing the proxy metric, then the architectures would be ranked the same on all the datasets. The authors suggest that TPC score could be used to eliminate bad candidate architectures early on. But for a good proxy metric, one must take into consideration the dataset characteristics. As mentioned in 2, please compute Kendall Tau for two datasets on the same search space and compare them against the other baselines. 
4. TPC Score only takes CONV layers into consideration. What about the impact of dropout, pool or skip connections? Initialization also plays a crucial role in the final performance of the network.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and is novel. The authors provided the code. So it is reproducible.
summary_of_the_review: This paper devised a training-free NAS algorithm, where the metric is the number of paths from the input to the output layer. But its performance needs to be tested on NASBench-101 and 201 to be fair to other baselines. It is also important to understand if it adapts its score to a given dataset as they do not consider any dataset characteristics.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new zero-cost proxy for NAS: the total path count (TPC) and uses it with an evolutionary algorithm to find strong architectures for image classification, object detection and super resolution.
While these applications are interesting, the reviewers prominently criticized the evaluation of the core contribution of TPC on just 20 architectures from the Zen-NAS paper. Other points of criticism included novelty, clarity, comparability of methods, and the lack of ablation studies. 
Due to these points all reviewers' scores for this paper are marginally below the acceptance threshold, and I follow the reviewers' suggestion. 
I would, however, like to strongly encourage the authors to continue this work and to strengthen the empirical assessment of the core TPC proxy on common benchmarks. A potentially easy way to do this on many benchmarks could be to use the contemporary work on NAS-Bench-Suite Zero that was just presented in the NeurIPS datasets and benchmark track (https://openreview.net/forum?id=yWhuIjIjH8k), which contains the performance of 13 zero cost proxies on 28 tasks. This benchmark may allow a quick way to scientifically study for which types of NAS benchmarks TPC works well and for which it does not work, as well as to understand when one should use it over other zero cost proxies. 
justification_for_why_not_higher_score: Criticisms by reviewers included novelty, clarity, comparability of methods, and the lack of ablation studies. All reviewers give rejection scores.
Decision: Reject