Title: Efficient One-Shot Neural Architecture Search With Progressive Choice Freezing Evolutionary Search
Authors: Chen Zhang
Qiyu Wan
Lening Wang
Mingsong Chen
Jingweijia Tan
Kaige Yan
Xin Fu
Venue: ICLR 2023

Abstract: Neural Architecture Search (NAS) is a fast-developing research field to promote automatic machine learning. Among the recently populated NAS methods, One-Shot NAS has attracted significant attention since it greatly reduces the training cost compared with the previous NAS methods. In One-Shot NAS, the best network architecture is searched within a supernet, which is trained only once. In practice, the search process involves numerous inference processes for each user case, which causes high overhead in terms of latency and energy consumption. To tackle this problem, we first observe that the choices of the first few blocks that belong to different candidate networks will become similar at the early search stage. Furthermore, these choices are already close to the optimal choices obtained at the end of the search. Leveraging this interesting feature, we propose a Progressive Choice Freezing Evolutionary Search (PCF-ES) method that gradually freezes block choices for all subnets at different search generations. This approach gives us an opportunity to reuse intermediate data produced by the frozen block instead of re-computing them. The experiment results show that the proposed PCF-ES provides up to 55\% speedup and reduces energy consumption by 51\% during the searching stage.


Review discussion:
Strengths: important, under-explored aspect of NAS, clear description of the work
Weaknesses: lack of baselines (see next section for actionable feedback), hyperparameter sensitivity (discuss seriousness, find ways to overcome it)
Overall, the paper is well-written and it is an important problem. The presented idea is relatively simple but there has been little prior work in that direction making this quite original work.
There is a general lack of comparisons against reasonable baselines.
First of all, a comparison against NASA is missing that was correctly identified as related work.
However, other simpler ideas are not explored either:
- Cache computations on disk
- Reduce the number of generations in EA
- Reduce the validation dataset size
- Reduce the population size
Admittedly, each of these methods may have their own problems (additional hyperparameters, additional disk space, drop in predictive performance).
The hyperparameter sensitivity is a critical issue in my opinion. We neither have a theoretical nor empirical evidence that we would be able to choose the right hyperparameters for the presented method. Given that the method is only twice as fast as the hyperparameter-free baseline, retrying with a different set of hyperparameters makes this method pointless. I would like to see a strategy how to set the hyperparameters efficiently otherwise there is simply no point in using the method in the first place.
To me it is not clear why we need the importance sampling step. There is no need to keep all data in GPU memory.
However, reducing the validation data itself will obviously reduce the overall time. This can be combined with the baseline as well. It is unclear, how much this contributes to the solution.
NO.
Thank you for your careful and valuable comments. We will explain your concerned weakness point by point.
Q1: There is a general lack of comparisons against reasonable baselines.
First of all, a comparison against NASA is missing that was correctly identified as related work.
However, other simpler ideas are not explored either:
Cache computations on disk;
Reduce the number of generations in EA;
Reduce the validation dataset size;
Reduce the population size;
A: NASA proposed a hardware accelerator based on network fusion within a generation.
On the contrary, our method improves the efficiency from a higher level which explores data reuse from both inter-generation and intra-generation. Our method is orthogonal to their method, and our method can employ their accelerator at the hardware level if needed.
So the comparison between these two methods is unnecessary.
These methods such as cache computation, reduce the generation, reduce the number of generations and population size is orthogonal to our work. Our method can be integrated with these methods. Reducing the validation data directly might hurt the search accuracy.
Q2: The hyperparameter sensitivity is a critical issue in my opinion. We neither have a theoretical nor empirical evidence that we would be able to choose the right hyperparameters for the presented method. Given that the method is only twice as fast as the hyperparameter-free baseline, retrying with a different set of hyperparameters makes this method pointless. I would like to see a strategy how to set the hyperparameters efficiently otherwise there is simply no point in using the method in the first place.
A: The experiment results of different hyperparameters are shown in Figure 9, 10 and 11. For each supernet, the hyperparameter tuning needs to be performed only once. For a frequently employed model searched for numerous scenarios, the time of hyperparameter tuning is negligible compared to the overall searching time.
Q3: To me it is not clear why we need the importance sampling step. There is no need to keep all data in GPU memory.
However, reducing the validation data itself will obviously reduce the overall time. This can be combined with the baseline as well. It is unclear, how much this contributes to the solution.
A: Transferring data from CPU memory to GPU is very time-consuming and energy-consuming compared to transferring from GPU local memory. The frequent CPU-GPU data transfer impairs search efficiency significantly. Reducing the validation data directly hurt the search accuracy. Our importance sampling method with PCF-ES achieves both high efficiency and negligible accuracy loss.
Strengths:
- The paper idea is intuitive and well worked out
- The description of the idea is well presented making the idea clear to the reader
Weaknesses:
- The paper has no clear related work section and it is hence unclear how the authors feel their work relates to prior work save for the discussion of work they base themselves on.
- The conclusion is very short and provides little in the way of insights
- The only results are on ImageNet - it would be interesting to see how the approach works on datasets not over-evaluated with NAS approaches.
The paper is clearly presented and well written. For this reviewer it was easy to follow the though process in the work.
The proposed idea is quite novel, but not earth-shattering.
It would be possible to reproduce something similar to the work performed here, though some guesswork for parts would be needed.
NO.
Thank you for your careful and valuable comments. We will explain your concerned weakness point by point.
Q1: The paper has no clear related work section and it is hence unclear how the authors feel their work relates to prior work save for the discussion of work they base themselves on.
A: Although there is no stand-alone related work section in our paper, related works on which our work is based are reviewed and discussed in Section 2.
Q2: The conclusion is very short and provides little in the way of insights
A: We improved our conclusion in Section 5 and added more details.
Q3: The only results are on ImageNet - it would be interesting to see how the approach works on datasets not over-evaluated with NAS approaches.
A: We conducted our experiment on multiple datasets. Some of the experiment results on CIFAR-10 and CIFAR-100 dataset at SPOS are shown as follows:
|     Dataset      |     Method          |     Top1    |     Top5     |     GPU   Hour    |
|------------------|---------------------|-------------|--------------|-------------------|
|     CIFAR-10     |     Evolutionary    |     96.5    |     99.9    |     5.4           |
|                  |     CF-ES           |     96.5    |     99.9     |     4.6           |
|                  |     PCF-ES          |     96.4    |     99.9     |     3.0           |
|     CIFAR-100    |     Evolutionary    |     75.1    |     92.4     |     5.4           |
|                  |     CF-ES           |     75.0    |     92.1     |     4.4           |
|                  |     PCF-ES          |     74.9    |     92.3     |     2.6           |
These experiment results are added in the Appendix A.3. And observation of majority choice on more datasets are added in Appendix A.1.
Q4: Some more specific comments.
A: Section 2 is relatively long because we reviewed one-shot NAS and discussed the related work of our method in the same section.
Blue lines are employed to highlight the comparison between block choice percentages at the same generation.
The mentioned sentence is revised as advised.
# Strength
Energy saving is becoming a crucial recently. The proposed approach aims at contributing to this point.
# Weaknesses
Single Dataset: Evaluation is only done on a single dataset. Its generality is not discussed.
Observation: The observation in Section 3.1 seems to be observed on a single dataset as well. The experimental settings are not provided for this observation.
Comparison: The proposed approach is compared only with SPOS variants. It has not been compared with other one-shot NAS approaches such as DSNAS, which does the weight training and architecture search at once. Because of the lack of the evaluation it is not clear whether the proposed approach is useful among other one-show NAS approaches.
Statistical Evaluation: No statistical evaluation has been performed. It is not even clear what are the numbers in Table 1 and Figures. Are they average values?
The algorithm is clearly described and well-motivated. The idea of progressively freezing choices may be a reasonable design choice.
However, the detail of the experiments in Section 3.1 (observation) and Section 4 are not provided. Because of the lack of statistical evaluation, reproducibility is low.
NO.
Thank you for your careful and valuable comments. We will explain your concerned weakness point by point.
Q1: Single Dataset.
A: We conducted our experiment on multiple datasets. Some of the experiment results on CIFAR-10 and CIFAR-100 dataset at SPOS are shown as follows:
|     Dataset      |     Method          |     Top1    |     Top5     |     GPU   Hour    |
|------------------|---------------------|-------------|--------------|-------------------|
|     CIFAR-10     |     Evolutionary    |     96.5    |     99.9    |     5.4           |
|                  |     CF-ES           |     96.5    |     99.9     |     4.6           |
|                  |     PCF-ES          |     96.4    |     99.9     |     3.0           |
|     CIFAR-100    |     Evolutionary    |     75.1    |     92.4     |     5.4           |
|                  |     CF-ES           |     75.0    |     92.1     |     4.4           |
|                  |     PCF-ES          |     74.9    |     92.3     |     2.6           |
These experiment results are added in the Appendix A.3.
Q2: Observation.
A: More observation data with different dataset of Section 3.1 is shown in Appendix A.1. The experiment setup is illustrated in section 4.1.
Q3: Comparison.
A: The DSNAS is a conventional one-stage framework that conducts the training and searching at once.
However, our method focuses on improving the efficiency of the one-shot NAS searching process. The comparison between one-stage NAS and one-shot NAS is out of the scope of our method. We have employed our method at multiple one-shot benchmarks and shows the efficiency among all approaches evidently.
Q4: Statistical Evaluation.
A: The numbers in Table 1 are averaged values from 3 experiment. The experiment setup is discussed in Section 4.1.
Thank you for the response. I still have some concerns.
What if the GPU hour is fixed for all methods? Currently reported results only show that these three methods are at different places on the Pareto front of accuracy and GPU hour.
Because the motivation is to reduce the energy saving, one should take into account the time for the weight training as well. If one consider the overall GPU hour, is the reported difference in time between the methods meaningful? If the weight training takes significant time, why don't you just take more efficient NAS approaches than SPOS from the beginning? From this perspective, one should compare the proposed approach to other approaches applying weight sharing.
Thank you very much for your further comments and advice. We will answer your questions point by point:
Q1: What if the GPU hour is fixed for all methods? Currently reported results only show that these three methods are at different places on the Pareto front of accuracy and GPU hour.
A: In our experiment, we compared the time and energy consumption at the same search generations and our method achieves significant time and energy saving with no or negligible accuracy loss. If the GPU hour is fixed at some circumstances, our method achieves much more search generation than the conventional method and a lot more search space will be explored, and a lot more candidate subnets will be evaluated.
Q2: Because the motivation is to reduce the energy saving, one should take into account the time for the weight training as well. If one consider the overall GPU hour, is the reported difference in time between the methods meaningful? If the weight training takes significant time, why don't you just take more efficient NAS approaches than SPOS from the beginning? From this perspective, one should compare the proposed approach to other approaches applying weight sharing.
A: A supernet architecture only needs to be trained once and can be reused in numerous later works, while the searching stage needs to be performed whenever a new constraint is given.
In practice, the searching cost is much higher than supernet training in a frequently employed model. Achieving significant time and energy saving in the searching stage is important.
Although there are other weight sharing methods, we chose these benchmarks because one-shot NAS is a widely accepted and widely employed approach.
**Strengths**
* An interesting observation in one-shot NAS, the first few continuous blocks of candidate architectures become similar in the early search epoch, is presented.
* The proposed PCF-ES can reduce the architecture search cost without significant performance deterioration on the ImageNet dataset.
**Weaknesses**
* The authors mention that the architecture searching cost is higher than the supernet training cost. It may be true when searching multiple architectures with different complexities by optimizing different objective functions.
However, the proposed method only targets searching for a well-performed architecture. The reviewer suspects that the search cost reported in the experiment does not include supernet training. The search cost of both supernet training and architecture search phases, i.e., total search cost, should be reported in the experiment.
* The authors treat the existing work of NASA (Ma et al. 2021) as related work on accelerating the architecture search process.
However, an empirical comparison of NASA and the proposed method is not performed.
* The experiment is conducted using only the ImageNet dataset. It is not clear whether the assumption that the first few blocks become similar among the candidate solutions will hold in other datasets and tasks. To evaluate the proposed method in various situations, it might be a possible choice to use NAS-Bench datasets for further performance evaluations.
* The authors consider that the intermediate feature maps to be reused should be stored on the GPU memory. The reviewer thinks that it is possible to store the feature maps on the CPU memory, although the data transferring cost occurs. It is not clear the motivation for storing all intermediate feature maps on the GPU memory.
* The detailed algorithm of importance sampling is unclear. How to calculate the probability distribution of $q(x)^*$ and the integral over $x$ in (2)?
Also, the probability distribution of $p(x)$ is needed to use the equation (1). How to calculate $p(x)$ which is the distribution of input data x?
* The effectiveness of importance sampling seems to be unclear. It would be better to compare the proposed sampling method to random sampling.
* In the proposed PCF-ES, two techniques, freezing the shallow blocks and importance sampling-based input sample selection, are introduced. It is not clear which techniques mainly contribute to computational cost reduction. The ablation study should be conducted. What are the computational costs of the method only using the freezing technique and the method only using the importance sampling-based sample selection? If we store the intermediate feature maps on the CPU memory, we can evaluate the method only using the freezing technique.
* The motivation and approach of the proposed method are reasonable.
* The proposed method might be novel and is based on interesting observations.
However, it is unclear whether it works well on other datasets and tasks because the experiment is conducted using only the ImageNet dataset.
* The experimental evaluation is not enough to validate the effectiveness of the proposed PCF-ES.
* Because the authors did not provide the code and the detailed experimental settings, it is hard to reproduce the experimental results.
NO.
Thank you for your careful and valuable comments. We will explain your concerned weakness point by point.
Q1: The search cost of both supernet training and architecture search phases, i.e., total search cost, should be reported in the experiment.
A: The training time of all three benchmarks are shown as follows:
|     Benchmark                     |     SPOS    |     FairNAS    |     GreedyNAS    |
|-----------------------------------|-------------|----------------|------------------|
|     Training   Cost (GPU days)    |     12      |     12         |     7            |
The supernet training is time-consuming at first sight.
However, a supernet architecture only needs to be trained once and can be reused in numerous later works while the searching stage need to be performed whenever a new constraint is given.
In practice, the searching cost is much higher than supernet training in a frequently employed model.
Q2: However, an empirical comparison of NASA and the proposed method is not performed.
A: NASA proposed a hardware accelerator based on network fusion within a generation.
On the contrary, our method improves the efficiency from a higher level which explores data reuse from both inter-generation and intra-generation. Our method is orthogonal to their method, and our method can employ their accelerator at the hardware level if needed.
So the comparison between these two methods is unnecessary.
Q3: The experiment is conducted using only the ImageNet dataset.
A: We conducted our experiment on multiple datasets. Some of the experiment results on CIFAR-10 and CIFAR-100 dataset at SPOS are shown as follows:
|     Dataset      |     Method          |     Top1    |     Top5     |     GPU   Hour    |
|------------------|---------------------|-------------|--------------|-------------------|
|     CIFAR-10     |     Evolutionary    |     96.5    |     99.9    |     5.4           |
|                  |     CF-ES           |     96.5    |     99.9     |     4.6           |
|                  |     PCF-ES          |     96.4    |     99.9     |     3.0           |
|     CIFAR-100    |     Evolutionary    |     75.1    |     92.4     |     5.4           |
|                  |     CF-ES           |     75.0    |     92.1     |     4.4           |
|                  |     PCF-ES          |     74.9    |     92.3     |     2.6           |
These experiment results are added in the Appendix A.3. And observation of majority choice on more datasets are added in Appendix A.1.
Q4: It is not clear the motivation for storing all intermediate feature maps on the GPU memory.
A: Transferring data from CPU memory to GPU is very time-consuming and energy-consuming compared to transferring from GPU local memory. The frequent CPU-GPU data transfer impairs search efficiency significantly.
Q5: The detailed algorithm of importance sampling is unclear.
A: In equation (1), $p(x)$ denotes the uniform distribution referred from random sampling and $f(x)$ denotes the cross-entropy loss of input $x$. In equation (2), $q(x)$* is derived from $Z$ which is calculated from the $p(x)*||f(x)||_2$ over all validation data.
Q6: It would be better to compare the proposed sampling method to random sampling.
A: The experiment comparison with importance sampling and random sampling is shown as follows:
|     Sample   Rate    |     0.9     |     0.7     |     0.5     |     0.3     |
|----------------------|-------------|-------------|-------------|-------------|
|     Random   Acc     |     73.4    |     73.5    |     73.1    |     73.1    |
|     IS   Acc         |     73.5    |     73.5    |     73.4    |     73.1    |
It can be found that our importance sampling method achieves higher or the same accuracy with compared to random sampling at all sample rates. Our method can achieve a lower sample rate while maintaining high search accuracy. These experiment results are added in the Appendix A.4.
Q7: It is not clear which techniques mainly contribute to computational cost reduction.
A: We evaluate these two methods separately, the experiment results are modified with your advice. The experiment results of employing CF-ES, PCF-ES and PCF-ES with IS are shown in Table 1.
| Supernet  | Method       | T   | MS | Top-1 | Top-5 | FLOPS(M) | GPU Hour | GPU Energy(MJ) |
|-----------|--------------|-----|----|-------|-------|----------|----------|----------------|
| SPOS      | PCF-ES w/ IS | 0.7 | 3  | 73.4  | 90.1  | 327      | 4.1      | 1.3            |
| FairNAS   | PCE-ES w/ IS | 0.4 | 2  | 71.7  | 89.5  | 325      | 5.5      | 1.6            |
| GreedyNAS | PCE-ES w/ IS | 0.4 | 3  | 72.2  | 90.0  | 325      | 7.2      | 1.9            |
And the results of applying both freezing technique and importance sampling with different sample rate are shown in Figure 11. More experiment results are shown in Appendix.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: This paper introduces a method for speeding up the „search phase“ of one-shot NAS methods, which takes a trained supernet as input and only needs to find the best architecture for a given set of constraints. The reviewers agree that this is an understudied problem. However, they also criticized many issues in the paper, including a lack of reasonable baselines (2x faster and only a bit worse; but is this better than a reasonable baseline of half the epochs, half the data, etc, which would also yield 2x speedups?), evaluation on too few datasets without it being clear whether there are confounding factors, lack of discussion of related work, no code availability and thus unclear reproducibility. As a result of this, three reviewers gave clear rejections scores. One reviewer gave a score of 8, but their review is not tremendously positive, either, and they did not react to my request for a clarification about their score. I am therefore following the majority of the reviewers and recommend rejection. I encourage the reviewers to rethink their choice of baselines and experimental protocol. One resource that could help with this is https://jmlr.org/papers/v21/20-056.html for some best practices in NAS experimentation. 
justification_for_why_not_higher_score: The reviewers criticized many issues in the paper, including a lack of reasonable baselines (2x faster and only a bit worse; but is this better than a reasonable baseline of half the epochs, half the data, etc, which would also yield 2x speedups?), evaluation on too few datasets without it being clear whether there are confounding factors, lack of discussion of related work, no code availability and thus unclear reproducibility.
justification_for_why_not_lower_score: N/A

Decision: Reject