Crafting Zero-Cost Proxy Metrics for Neural Architecture Search via Symbolic Regression

Authors: Quan Minh Phan, Ngoc Hoang Luong
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: neural architecture search, symbolic regression, zero-cost proxy metrics, genetic programming

Abstract:
Using zero-cost (ZC) metrics to estimate network performance instead of relying on expensive training processes has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a significant limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Additionally, the design of current ZC metrics is manual, which is a lengthy trial-and-error process and requires expert knowledge to develop ZC metrics effectively. These challenges raise two questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a better one? In this study, we propose a framework based on Symbolic Regression to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to network performance across multiple problems within just a few minutes. Extensive experiments on 13 problems in NAS-Bench-Suite-Zero, covering various search spaces and tasks, demonstrate the superiority of our automatically designed proxies over hand-crafted ones. By integrating our proxy metrics into an evolutionary algorithm, we could identify a network architecture with comparable performance on the CIFAR-10 dataset within 15 minutes using a single GeForce RTX 3090 GPU.

Decision: reject
Reviewer scores: 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729028315135

thread_type:
official_review

<thread_object>
creator: reviewer_ubay

<content>
summary:
This work proposes an extensible framework for the automatic discovery of ZC metrics, where SR is used to guide the production of new metrics.

strengths:
The author clearly expresses the proposed method and demonstrates the advantages of the proposed method through numerous experiments.
It makes sense that many new ZC methods (e.g., ZiCo and MeCo) are integrated into the SR model.

weaknesses:
I think testing on Imagenet using the DARTS search space is necessary.
The proposed method is not advantageous compared to AZ-NAS.

questions:
Regarding the automatic design of ZC agents, some works have been left out [r1-r3].
[r1] Robustifying and Boosting Training-Free Neural Architecture Search
[r2] AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
[r3] Dynamic ensemble of low-fidelity experts: Mitigating nas “cold-start”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730462384171

thread_type:
official_review

<thread_object>
creator: reviewer_c1zs

<content>
summary:
This paper proposes a higher-level zero-cost metric for neural architecture search based on symbolic regression of hand-designed zero-cost metrics. The proposed framework is extensible, consistent, and achieves a high positive rank correlation across multiple problems. Results are reported on NAS-Bench-101/201/301 and TransNAS-Bench-101-Micro/Macro as well as NAS on CIFAR-10 demonstrating competitive rank correlation.

strengths:
+ The use of rank correlation across multiple ZC metrics and problems is interesting and addresses some of the common criticism of ZC metrics.

weaknesses:
- Limited Novelty. Most contributions, e.g., the use of symbolic regression or the use of high-level ZC metrics, were proposed in earlier work. Their combination is somewhat novel but in light of the following points may not be sufficient.
- The claim "Our framework can synthesize a new ZC metric within only 10 minutes" is incorrect because the framework requires the calculation of the high-level ZC metrics which serve as input features, and therefore, the actual total time is the time required by each high-level ZC metric plus 10 mins. The same comment applies to the results in Table 4.
- The proposed ZC metric, while using existing ZC metrics, doesn't outright outperform these metrics (see Fig. 6), and in NAS, is outperformed by a number of existing NAS methods based on ZC metrics (see Table 4)
- No ImageNet results reported.

questions:
- Why was symbolic regression chosen?
- Why was the proposed ZC metric (SR-NAS) not evaluated on ImageNet?
- If the proposed ZC on average ranks better than existing ZC metrics, why does it not outperform these metrics in NAS?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699164329

thread_type:
official_review

<thread_object>
creator: reviewer_yygb

<content>
summary:
The authors proposed a framework based on Symbolic Regression to automate the design of ZC metrics, whch is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to network performance across multiple problems within just a few minutes.

strengths:
1.The approach of using symbolic regression to combine hand-crafted metrics into a superior proxy metric is innovative, setting it apart from previous methods, such as those requiring manual crafting of each metric.

2.The study have comprehensive experiments on multiple NAS benchmarks and comparisons with state-of-the-art ZC metrics.

3. The paper is generally well-structured and clear, with a logical flow from problem statement to methodology, experiments, and conclusions.

4. The proposed approach can evaluate network architectures quickly and accurately without intensive computational resources.

weaknesses:
1. The innovation of the paper is limited，the framework is a common symbolic regression to search the best mathematical expressions of ZC metrics.

2. The paper focuses on the performance of the designed ZC metrics but does not provide insights into why certain combinations of metrics work better than others.

3. While the paper claims generalizability across various NAS problems, the evaluation is primarily based on a limited set of benchmarks.

questions:
1. the Eq.1 is essentially a normalization, It also looks like giving each question the same weight.

2. Why specific metrics (e.g., FLOPs, Snip, ZiCo, etc.) that frequently appear in high-performing combinations? Are there theoretical or empirical reasons?

3. What is the multiple problems in dataset? Are they different tasks?

4. There seems to be a typo in the annotation of Figure 2. NWOT should be MeCo.

5. In Figure 4, what do SS^i and T^j in the data set represent?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732503636801

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of updates in the revised manuscript

comment:
We sincerely thank reviewers yYgB, c1zS, and ubay for their insightful comments. Your valuable feedback has significantly improved our manuscript. In response, we have made the following adjustments in the revised version:

- Corrected the typo in the caption of Figure 2.
- Clarified the definitions of SS_i​ and T_j ​ in the caption of Figure 4.
- Based on the collective suggestion to evaluate the performance on ImageNet, we conducted additional experiments within the Once-For-All search space and included the new results in Section 4.3.
- To address Reviewer ubay's concern regarding the comparison of our Symbolic Regression framework with related works such as AZ-NAS and RoBoT, we conducted a performance comparison and added the results in Appendix J.
- In response to Reviewer yYgB's comments, we have included valuable insights in Appendix K from analyzing the synthesized ZC metric and the search process of our Symbolic Regression framework.

We greatly appreciate the reviewers' constructive feedback, which has helped us enhance the quality and the scope of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel framework utilizing symbolic regression for the automatic synthesis of zero-cost (ZC) proxy metrics for neural architecture search (NAS). The reviewers acknowledged the relevance of the problem and the potential of the framework to streamline ZC metric design. However, concerns were raised about the limited novelty, the framework’s performance in specific benchmarks, and the lack of clear theoretical insights into why certain combinations of metrics outperform others. Despite the authors’ rebuttal addressing several points, such as adding ImageNet results and clarifying the framework’s contributions, these efforts only partially alleviated the concerns. With a majority of the reviewers rating the paper below the acceptance threshold, the consensus leans towards rejection.