Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an efficient one-shot neural architecture search (NAS) method that progressively freezes the architectures. The authors show that the first few blocks become similar among the candidate solutions during the evolutionary architecture search and then develop an acceleration method of architecture search by exploiting this phenomenon. In addition, the input sampling method is introduced to reduce memory consumption when reusing intermediate feature maps among candidate architectures. The experimental results demonstrate that the proposed progressive choice freezing evolutionary search (PCF-ES) can speed up the architecture search compared to existing one-shot NAS methods.
strength_and_weaknesses: **Strengths**
* An interesting observation in one-shot NAS, the first few continuous blocks of candidate architectures become similar in the early search epoch, is presented.
* The proposed PCF-ES can reduce the architecture search cost without significant performance deterioration on the ImageNet dataset.

**Weaknesses**
* The authors mention that the architecture searching cost is higher than the supernet training cost. It may be true when searching multiple architectures with different complexities by optimizing different objective functions. However, the proposed method only targets searching for a well-performed architecture. The reviewer suspects that the search cost reported in the experiment does not include supernet training. The search cost of both supernet training and architecture search phases, i.e., total search cost, should be reported in the experiment.
* The authors treat the existing work of NASA (Ma et al. 2021) as related work on accelerating the architecture search process. However, an empirical comparison of NASA and the proposed method is not performed.
* The experiment is conducted using only the ImageNet dataset. It is not clear whether the assumption that the first few blocks become similar among the candidate solutions will hold in other datasets and tasks. To evaluate the proposed method in various situations, it might be a possible choice to use NAS-Bench datasets for further performance evaluations.
* The authors consider that the intermediate feature maps to be reused should be stored on the GPU memory. The reviewer thinks that it is possible to store the feature maps on the CPU memory, although the data transferring cost occurs. It is not clear the motivation for storing all intermediate feature maps on the GPU memory.
* The detailed algorithm of importance sampling is unclear. How to calculate the probability distribution of $q(x)^*$ and the integral over $x$ in (2)? Also, the probability distribution of $p(x)$ is needed to use the equation (1). How to calculate $p(x)$ which is the distribution of input data x?
* The effectiveness of importance sampling seems to be unclear. It would be better to compare the proposed sampling method to random sampling.
* In the proposed PCF-ES, two techniques, freezing the shallow blocks and importance sampling-based input sample selection, are introduced. It is not clear which techniques mainly contribute to computational cost reduction. The ablation study should be conducted. What are the computational costs of the method only using the freezing technique and the method only using the importance sampling-based sample selection? If we store the intermediate feature maps on the CPU memory, we can evaluate the method only using the freezing technique.
clarity,_quality,_novelty_and_reproducibility: * The motivation and approach of the proposed method are reasonable.
* The proposed method might be novel and is based on interesting observations. However, it is unclear whether it works well on other datasets and tasks because the experiment is conducted using only the ImageNet dataset.
* The experimental evaluation is not enough to validate the effectiveness of the proposed PCF-ES.
* Because the authors did not provide the code and the detailed experimental settings, it is hard to reproduce the experimental results.
summary_of_the_review: The authors treat an interesting topic in one-shot NAS. However, as described in the section on Strength And Weaknesses, the reviewer feels that the weaknesses of this paper outweigh the strength.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focused on accelerating one-shot neural architecture search (NAS). In particular, the authors aims to accelerate the architecture search phase based on evolutionary approach. The authors first discuss their observation that the evolutionary approaches tend to select the same shallow blocks from the early stage of the search. Then, an approach is proposed that freezes the selected blocks in shallow part of the super-network. This approach has an advantage that the features computed in all population in the evolutionary approach are the same because they share the same blocks, leading to saving some computational time. Moreover, to further speedup the search process, a mechanism to subsample validation dataset to approximate the validation accuracy with low computational time. The proposed approach, PCF-ES, have be compared with a baseline evolutionary approach and a variant of the proposed approach on three supernet variants on ImageNet. Without compromising the performance significantly, saving 50% of GPU hours and GPU energy are reported.
strength_and_weaknesses: # Strength

Energy saving is becoming a crucial recently. The proposed approach aims at contributing to this point.

# Weaknesses

Single Dataset: Evaluation is only done on a single dataset. Its generality is not discussed.

Observation: The observation in Section 3.1 seems to be observed on a single dataset as well. The experimental settings are not provided for this observation.

Comparison: The proposed approach is compared only with SPOS variants. It has not been compared with other one-shot NAS approaches such as DSNAS, which does the weight training and architecture search at once. Because of the lack of the evaluation it is not clear whether the proposed approach is useful among other one-show NAS approaches.

Statistical Evaluation: No statistical evaluation has been performed. It is not even clear what are the numbers in Table 1 and Figures. Are they average values?


clarity,_quality,_novelty_and_reproducibility: The algorithm is clearly described and well-motivated. The idea of progressively freezing choices may be a reasonable design choice. However, the detail of the experiments in Section 3.1 (observation) and Section 4 are not provided. Because of the lack of statistical evaluation, reproducibility is low.
summary_of_the_review: As mentioned in the strength and weaknesses section, numerical evaluation in this paper is not sufficient to support the claim of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces the idea of freezing layers within a One-Shot evolutionary NAS approach. As the lower layers quickly converge to a 'final' state the authors argue that freezing them and removing the calculations performed within them will reduce computation and hence energy. An approach is developed to identify when layers can be frozen and another approach to identify the 'significant' samples as pre-computing the computations (and re-using) can overwhelm the memory available within a GPU.
strength_and_weaknesses: Strengths:
- The paper idea is intuitive and well worked out
- The description of the idea is well presented making the idea clear to the reader

Weaknesses:
- The paper has no clear related work section and it is hence unclear how the authors feel their work relates to prior work save for the discussion of work they base themselves on.
- The conclusion is very short and provides little in the way of insights
- The only results are on ImageNet - it would be interesting to see how the approach works on datasets not over-evaluated with NAS approaches.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly presented and well written. For this reviewer it was easy to follow the though process in the work.

The proposed idea is quite novel, but not earth-shattering. 

It would be possible to reproduce something similar to the work performed here, though some guesswork for parts would be needed.
summary_of_the_review: The paper is clear and concise. The idea seems reasonably novel and is supported by good evidence.

Some more specific comments:

- The review of one-shot NAS is interesting, but probably too long for this paper. More on related work would be better to present.

- There is a bizarre blue line under the middle bottom graph in figure 2. 

- "This provide a solid" -> "This provides a solid"
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method which accelerates the architecture choice step of certain one-shot NAS methods. The method greedily freezes the first block choices during the architecture optimization phase with an evolutionary algorithm. This allows for caching the intermediate feature maps and therefore reduce the computational effort. In an experimental study on ImageNet, they show that this method saves time at no reduction in predictive performance. Furthermore, they investigate the hyperparameter sensitivity.
strength_and_weaknesses: Strengths: important, under-explored aspect of NAS, clear description of the work
Weaknesses: lack of baselines (see next section for actionable feedback), hyperparameter sensitivity (discuss seriousness, find ways to overcome it)
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and it is an important problem. The presented idea is relatively simple but there has been little prior work in that direction making this quite original work.

There is a general lack of comparisons against reasonable baselines. First of all, a comparison against NASA is missing that was correctly identified as related work. However, other simpler ideas are not explored either:
- Cache computations on disk
- Reduce the number of generations in EA
- Reduce the validation dataset size
- Reduce the population size

Admittedly, each of these methods may have their own problems (additional hyperparameters, additional disk space, drop in predictive performance).

The hyperparameter sensitivity is a critical issue in my opinion. We neither have a theoretical nor empirical evidence that we would be able to choose the right hyperparameters for the presented method. Given that the method is only twice as fast as the hyperparameter-free baseline, retrying with a different set of hyperparameters makes this method pointless. I would like to see a strategy how to set the hyperparameters efficiently otherwise there is simply no point in using the method in the first place.

To me it is not clear why we need the importance sampling step. There is no need to keep all data in GPU memory. However, reducing the validation data itself will obviously reduce the overall time. This can be combined with the baseline as well. It is unclear, how much this contributes to the solution.
summary_of_the_review: In this well-written paper, the authors present an original idea to reduce the time for finding the architecture after a one-shot NAS. The proposed method reduces time by a factor of 2 but additionally it adds several sensitive hyperparameters. The authors study this sensitivity but provide no advice how to select stable hyperparameters. This raises the question whether we are able to reduce the time or we spend additional time searching for these hyperparameters. Finally, the only existing baseline is not considered for comparison as well as other simple baselines that could accomplish the same.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a method for speeding up the „search phase“ of one-shot NAS methods, which takes a trained supernet as input and only needs to find the best architecture for a given set of constraints. The reviewers agree that this is an understudied problem. However, they also criticized many issues in the paper, including a lack of reasonable baselines (2x faster and only a bit worse; but is this better than a reasonable baseline of half the epochs, half the data, etc, which would also yield 2x speedups?), evaluation on too few datasets without it being clear whether there are confounding factors, lack of discussion of related work, no code availability and thus unclear reproducibility. As a result of this, three reviewers gave clear rejections scores. One reviewer gave a score of 8, but their review is not tremendously positive, either, and they did not react to my request for a clarification about their score. I am therefore following the majority of the reviewers and recommend rejection. I encourage the reviewers to rethink their choice of baselines and experimental protocol. One resource that could help with this is https://jmlr.org/papers/v21/20-056.html for some best practices in NAS experimentation. 
justification_for_why_not_higher_score: The reviewers criticized many issues in the paper, including a lack of reasonable baselines (2x faster and only a bit worse; but is this better than a reasonable baseline of half the epochs, half the data, etc, which would also yield 2x speedups?), evaluation on too few datasets without it being clear whether there are confounding factors, lack of discussion of related work, no code availability and thus unclear reproducibility.
Decision: Reject