Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: The authors provide a deeper understanding of various zero-cost proxies by interpreting them based on a pruning perspective, and they propose a new combination of zero-cost proxies which achieve state-of-the-art performance. More specifically, they introduce a new gradient-based metric called SSNR and use layer-wise naswot as an activation-based metric to propose a combined proxy called T-CET. And the superiority of the proposed proxy is confirmed through various experiments.
overall_review: The strengths of this paper lie in the analysis of various zero-cost proxies in the pruning literature, the proposal of a new proxy that combines existing proxies, and the demonstration of high performance. The theoretical and experimental analysis of the paper is expected to be of great help to future research in the zero-cost NAS field. However, the lack of detailed descriptions of the experimental environment and the brief analysis of the experimental results serve as drawbacks.
review_summary: This paper is technically correct and the proposed method consistently beat comparing methods on various search spaces and datasets. Also, the authors provide a novel view to analyze zero-cost proxies. Therefore, my recommendation is a strong accept.
actions_required_to_increase_overall_recommendation: It would be better if more information about the experimental setup could be added. Additionally, although it seems to be due to the page limit, the size of Table 2 is too small to be easily read on the typical A4 size.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The authors of this paper claim two main contributions, as follows:
- revisiting and unifying the understanding of metrics used to predict architecture performance in neural architecture search (NAS)
- proposing novel metrics that outperform existing ones from a theoretical and practical perspective
overall_review: This paper revisits and unifies the understanding of metrics used to predict architecture performance in neural architecture search (NAS). In addition, authors also propose novel metrics that they claim outperform existing ones from a theoretical and practical perspective. The topic is relevant and, if contributions were achieved, the paper would have significant impact on the field, since it brings a better and broader understanding of existing techniques. In addition, given the relevant of metrics to predict architecture performance in NAS, improved metrics are very welcome and would be readily adopted by the community. Furthermore, the paper is generally well-written, though the assessment part of the paper is considerably brief. Since the theoretical analysis section was rather long and verbose, there is clearly room to better balance the different parts of the paper. More importantly, the experimental assessment is very limited for contributions to be validated. Nonetheless, since the theoretical part of the paper escapes my area of expertise, I evaluate it as a borderline paper.
review_summary: This paper addresses metrics used to predict architecture performance in neural architecture search (NAS). In detail, authors revisit, unify, and propose novel metrics. The topic is relevant, since it brings a better and broader understanding of existing techniques. Furthermore, the paper is generally well-written, though the assessment part of the paper is considerably brief. More importantly, the experimental assessment is very limited for contributions to be validated. Since the theoretical part of the paper escapes my area of expertise, I evaluate it as a borderline paper.
actions_required_to_increase_overall_recommendation: - a thorough experimental assessment of the metrics considered and proposed
- a revision of the paper, making the theoretical discussion more straightforward and substantiating the paper with experimental results
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: In this paper, the authors identify some existing metrics and classify them into gradient property based and activation pattern based. They identify the factors that contribute to the success of ZiCo and reformulate NASWOT to perform better. They also produce a state of the art zero cost proxy.
overall_review: The paper is well written, well motivated and easy to understand. However, the resulting scoring function (T-CET) does not deliver sufficiently improved KT/SPR to have a more profound impact on the community. Focusing on identifying more zero cost proxies and classifying them as done in Figure 1 and drawing insights to develop better zero cost proxies may help solidify the contributions of the paper, as well as more results (NDS search space can be easily initialized from their open source code for generating KT/SPR, as well as more search results demonstrating superior Top1-accuracy) would help.
review_summary: The paper is well written, and takes an interesting approach towards improving existing zero cost proxies. Unfortunately, the lack of a significant improvement over existing zero cost proxies, as well as lack of random search based results on more architecture spaces limits the impact of this work in practice. 
actions_required_to_increase_overall_recommendation: 1. More 'random search' results to demonstrate effectiveness of T-CET in practice.
2. NDS search space correlation scores.
3. More indepth classification of different zero cost proxies as done in Figure 1 would certainly be interesting.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The article discusses Neural Architecture Search (NAS) and how it has been successful in discovering high-performance neural network architectures. However, current NAS approaches require extensive evaluation or training, so zero-cost proxies have been proposed to efficiently predict architecture performance. The article argues that existing proxies should be revisited and analyzed through the lens of network compressibility, trainability, and expressivity. By doing so, a better understanding of the high-level relationship between different proxies can be built and some can be refined. The article proposes a novel saliency and metric aggregation method informed by compressibility, orthogonality, and network topology that yields state-of-the-art results in popular NAS benchmarks.
overall_review: In general, this paper will be interesting to people working in NAS and AutoML.
I have voiced my concerns above.
review_summary: This paper provides a concrete analysis of the new proxy it proposes (which is very interesting), and conducts comprehensive experiments.
I vote weak accept. I hope the author could address my concerns above.
actions_required_to_increase_overall_recommendation: I am open to further discussions, and willing to adjust my score if the author could respond my questions above.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper discusses Neural Architecture Search (NAS) to discover high-performance neural network architectures. However, current NAS approaches can be costly, requiring extensive evaluations or training of large super networks. Zero-cost proxies have been proposed to predict performance efficiently to reduce search costs, but little attention has been given to their mutual effects. The paper analyzes existing proxies through network compressibility, trainability, and expressivity and designs a new saliency and metric aggregation method informed by these factors, leading to state-of-the-art results across popular NAS benchmarks.
overall_review: The proposed method in the paper appears to be a unique approach to combining different zero-cost NAS (ZC-NAS) methods for evaluating neural architectures. However, the results achieved are only marginally better or sometimes worse than those of single ZC-NAS methods, which raises some concern. Additionally, the paper mentions TE-NAS in the related work section but does not provide further details about it in other parts of the paper, which could be confusing.

It is also worth noting that the paper "Generic Neural Architecture Search via Regression" presents a near zero-cost method with significantly better results. This suggests that while the approach in the current paper is interesting, there may be other more effective methods for zero-cost NAS that the authors could consider exploring.

In general, I believe this is a well-written paper. However, if the authors could improve the significance of their results, I would be inclined to give it a higher rating.

review_summary: Please refer to the above section.
actions_required_to_increase_overall_recommendation: Please refer to the above section.
review_title: null

Decision: Accept(Main Track)