Efficient One-Shot Neural Architecture Search With Progressive Choice Freezing Evolutionary Search
Authors: Chen Zhang
Qiyu Wan
Lening Wang
Mingsong Chen
Jingweijia Tan
Kaige Yan
Xin Fu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Neural Architecture Search (NAS) is a fast-developing research field to promote automatic machine learning. Among the recently populated NAS methods, One-Shot NAS has attracted significant attention since it greatly reduces the training cost compared with the previous NAS methods. In One-Shot NAS, the best network architecture is searched within a supernet, which is trained only once. In practice, the search process involves numerous inference processes for each user case, which causes high overhead in terms of latency and energy consumption. To tackle this problem, we first observe that the choices of the first few blocks that belong to different candidate networks will become similar at the early search stage. Furthermore, these choices are already close to the optimal choices obtained at the end of the search. Leveraging this interesting feature, we propose a Progressive Choice Freezing Evolutionary Search (PCF-ES) method that gradually freezes block choices for all subnets at different search generations. This approach gives us an opportunity to reuse intermediate data produced by the frozen block instead of re-computing them. The experiment results show that the proposed PCF-ES provides up to 55\% speedup and reduces energy consumption by 51\% during the searching stage.

Under review as a conference paper at ICLR 2023
EFFICIENT ONE-SHOT NEURAL ARCHITECTURE SEARCH WITH PROGRESSIVE CHOICE FREEZING EVOLUTIONARY SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) is fast-developing research field to promote automatic machine learning. Among the recently populated NAS methods, one- shot NAS has attracted significant attention since it greatly reduces the training cost compared with the previous NAS methods. one-shot NAS, the best can- didate network architecture is searched within supernet, which is trained only once. In practice, the searching process involves numerous inference processes for each user case, which causes high overhead in terms of latency and energy consumption. To tackle this problem, we first observe that the choices of the first few blocks that belong different candidate networks will become similar at the early search stage. Furthermore, these choices are already close to the optimal choices obtained at the end of the search. Leveraging this interesting feature, we propose progressive choice freezing evolutionary search (PCF-ES) method that gradually freezes block choices for all candidate networks during the searching process. This approach gives us opportunity to reuse intermediate data pro- duced by the frozen blocks instead of re-computing them. The experiment results show that the proposed PCF-ES provides up 55% speedup and reduces energy consumption by 51 % during the searching stage.
INTRODUCTION
Neural Architecture Search (NAS) has been proposed and extensively studied as an efficient tool for designing state-of-the-art neural networks (Elsken et al.) 2019 Wistuba et al. 2019 Ren et al.,
2020). NAS approaches automate the architecture design process and can achieve higher accuracy compared to desinnd architectures (Liu et al.) 2019) Xie al.) 2019 Cai al., 2019). However, the early NAS methods, such reinforcement NAS (Zoph & Le 2016), came with the price expensive computation costs since every searched architecture needs to be trained from scratch, which makes the total search time unacceptable. To reduce the search cost of earlier NAS methods, the weight sharing technique has been proposed (Yu et al.) [2020] Chen et al. (2020), among which the one-shot NAS method has attracted lot of attention recently (Bender al. 2018 al. 2020).
The one-shot NAS method is known cost-efficient it requires training supernet only once. A supernet is a stack of basic blocks, each of which contains multiple choices. A candidate net- work architecture (defined as subnet) can be formed by selecting one choice for each block in the supernet, and its corresponding weights can be inherited from the supernet. During the architecture searching stage, candidate architectures are evaluated on the validation dataset and the best architec- ture. i..., the architecture with the highest validation accuracy, is updated in every searching epoch of Evolutionary Algorithm (EA) (Real et al.) (2019). Surprisingly, although training commonly deemed as lengthy and energy conggiin task, the architecture searching stage in one-shot NAS much more costly (Cai et al., 2020) than training a supernet. The reason is that a new searching stage should be performed whenever a different searching scenario is given, e.g., different hardware constraints, learning tasks, and workloads, while the trained supernet can be reused. Hence, the numerous inferences the subnets take much longer time than training a supernet only once. According to You et al. 2020), searching can be 10 GPU days longer than supernet training when 10 different constraints/plattrrme are required.

---

Under review as a conference paper at ICLR 2023
To tackle this problem, work first makes key observation that, for the first few continuous blocks of the candidate architecture (defined as continuous shallow blocks), their optimal choices can be determined at early search epoch. Based on this observation, choices of continuous shallow blocks at the early search epoch, which means these choices will not be changed during the remaining search epochs. This strategy elaborately "creates" redundant computations in the continuous shallow blocks since all candidates will share exactly the same ar- chitecture, inherited weights, and input validation data for the shallow blocks during later search epochs. Then we leverage such redundancy and propose simple yet effective data reuse scheme to save large amounts of computations, thus further reducing time and energy cost. Specifically, we propose to reuse the last output of the continuous shallow blocks instead of re-computing it repeat- edly throughout the remaining searching stage. Interestingly, we further discover that the freezing strategy may in turn help to determine the optimal choices of the subsequent blocks earlier. Such phenomenon enables us to keep freezing the choices of blocks progressively after the initial freez- ing, which will create more redundant computations of the blocks that possess the same architecture (choice), thus more computations can be saved.
propose to freeze the
With the proposed freezing technique, the intermediate data (the last output of the continuous shal- low blocks) of a certain subnet can be stored and reused during the evaluation of other subnets. However, as the searching stage requires evaluate a subnet with large batch (e.g.. batch size 5000) of input samples, storing the intermediate data of only one subnet may cause serious mem- ory issues (3cc Inspired by the Importance Sampling technique employed in many training methods (Zeng et al. 2021), we propose to sample the "important" input data that contribute more distinguish the evaluation accuracy of the candidate subnets. More importantly, we empirically demonstrate that the important samples are shared across different subnets. Therefore, it only re- quires store the intermediate data of important samples for one certain subnet, and then reuse for all other subnets.
We evaluate the proposed method on multiple benchmarks trained with the state-of-the-art ap- proaches on the ImageNet dataset (Krizhevsky et al., (002). The experimental results indicate superb performance in improving the search efficiency while maintaining the search performance with only 0.1% searching accuracy loss. Our contributions can be summarized as follows:
We observe that, in the one-shot NAS evolutionary searching stage, the optimal architecture of shallow blocks is determined at the early searching stage.
We propose to freeze the choices of continuous shallow blocks for all candidates at the early stage, and progressively freeze the choices of the subsequent blocks in the later stage. This approach creates great amount of redundant computations. which provide us good opportunity to reuse the intermediate data and reduce the searching time.
To alleviate memory capacity issue for storing intermediate data, we leverage the concept of importance sampling and propose a distinguish-based based sampling method to reduce the size of the intermediate data.
We conduct extensive experiments on different benchmarks with our proposed methods. The evaluation results show that our method can achieve up to 55% time saving and 51% % energy saving with 0.1% accuracy loss.
2 REVIEW OF ONE-SHOT NAS
Different from the traditional neural network training that aims to optimize weights given network architecture, NAS seeks to optimize both weight and architecture at the same time. Conventional NAS methods (Zoph & Le 2016: Baker et al.) 2016 Zhong et al. 2018 Zela et al., 2018) have tried to solve these two optimization problems
nested approach. However, these methods are usually prohibitively expensive because each architecture sampled from the search space has to be trained from the scratch and evaluated separately. Recent works (Bender et al.) 2018 Pham et al. 2018 Cai et al. 2019 2018) have proposed weight sharing strategy to reduce high costs of architecture and weight searching procedure in conventional NAS. As one of the most popular weight sharing techniques, she- NAS achieves unprecedented search efficiency by decoupling the whole searching process into two stages: supernet training (Fig (a))and subnet searching (Fig (b)). One-shot NAS encodes the search space into supernel and trains it only once. Then it allows
2

---

Under review as a conference paper at ICLR 2023
Traising Stage
Searehing Stage
Choice
Shallow Blacks
Caniilaas
Candiibde
Dopp
3x3 Concolazion
Lvalsation
i-E!
5xd Consolation
Max Pooiling
Superne Cuppur
Materion mesome
Figure 1: An Overview of One-Shot NAS.
sampled architectures directly inherit the weights from the supernet. As shown in Fig | (a), the commonly used chain-structured supernet (Suganuma et al.) 2017) is a stack of blocks, and each block consists of multiple choices. Each choice can be different operations, e.g., 3x3 / 5 convolutional layer, a max max-pooling layer or an identity layer. We further define the first few blocks in the supernet shallow blocks, while the subsequent blocks are defined as deep blocks.
The second step of one-shot NAS is the searching stage which employs EA to find the subner archi- choice for
tecture with the highest validation accuracy. Each subnet can be obtained by selecting each block in the supernet, and its weights are inherited from the supernet. The overall workflow is shown in Fig (b). At each search epoch, also known as generation, there are N candidate subnets in the candidate pool sampled from the supernet and evaluated using validation dataset. Then the candidate subnets are ranked together with the previous top candidates according the validation accuracy. Candidates with Top-K validation accuracy are selected and then evolved into N new subnets by performing mutation and crossover operations (Real et al.) 2019). The above process is repeated until reaching the maximum number of search epochs F, and the subnet with the highest accuracy can be obtained. The architecture of this subnet will be regarded as the optimal architec- ture since the validation accuracy obtained by using inherited weights is highly predictive the accuracy obtained by training from scratch (Bender et al., 2018).
Recently, many studies focused on improving the efficiency and accuracy of the supernet training stage by introducing various subnet sampling methods (Guo et al.) 20200 Chu et al.) 2021; You et al., (020).. Nevertheless, searching stage can be more time-consuming compared with supernet training (Cai et al. 2020).
of them considered the efficiency of the searching stage. The
This is because the supernet only need to be trained once while numerous searching processes required to search the optimal architecture for various deployment scenarios, e.g., different hardware platforms, workloads. Therefore, is essential improve the search efficiency. NASA (Ma al.) 2021) is the first work that focuses on accelerating the search process, and NAS accelerator was proposed that utilizes network fusion based on the computation sharing and data reuse within search generation. Different from their approach, we propose an algorithm- optimization for one-shot NAS evolutionary search and exploit data sharing from both within generation and across generation levels, which achieves significant improvement on the searching efficiency.
3 METHODOLOGY
This section first introduces several key observations during the evolutionary searching phase, based on which the progressive choice freezing evolutionary search (PCF-ES) is then proposed, along with how it saves the computations during the searching stage. Finally, to mitigate the memory issue of our method, the distinguish-based importance sampling method is proposed to reduce the size of the intermediate data.
3.1 OBSERVATION: MAJORITY CHOICE IN CONTINUOUS SHALLOW BLOCKS
Figure illustrates how the choices of blocks evolve across generations during the evolutionary search in SPOS (Guo et al. 2020). The right-up subgraphs record the choice evolution in shallow blocks. Especially, we take the 1st, the 2nd, and the 3rd blocks as the examples to show their choice evolution across the search generations, respectively. The right-bottom subgraphs are the choice evolution in deep blocks, and the 15th, the 16th, and the 17th blocks are used as the examples. Each curve in the subgraph indicates the percentage of subnets among all candidate subnets in certain generation that select a certain choice for the block. As an example shown in the left part of the figure, at the i-th generation, the percentage of the blue choice in the 1st block equals to 80%
3

---

Under review as a conference paper at ICLR 2023
Magaait Clisse
50.3
1.0
Blockl
Shellon Blocks
5.0 5.6
30.4
Block2
2
202
Block3
0.0
20
10
15
20
0.0
Generation
Generation
Dipcki
Bigk2
Blpelc
Block15
Nateragge
Block16
Block17
Deep Blodi
0.0
0.0
Net
Ne1
Ne13
.*.
Now
Gene noion Black55
Genoration Buckks
15
Goneration
CaubllttPoolla 5-0 Generatioo
Bicck17
Figure 2: The percentage of different choices with generation from shallow blocks to deep blocks in SPOS.
Dist. to last Cen.
1.0
1.0
0.8
0.8
Dist. 18
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0 0.2 0.4 0.6 0.8 1.0
0.0 0.2 0.4 0.6
0.8 1.0
Dist. first Gen.
Dist.to first Gen.
(b)
Figure 3: The distance of the shallow blocks (a) before the majority choice appears and (b) after the majority choice appears to the best architecture in the first and last generation.
40 out of 50 subnets select the blue choice. As can be seen in the right-up subgraphs, there is one choice whose percentage increases dramatically and surpasses all the other choices in the early generations (i < 10, highlighted by the left of the blue dotted line). We define such choice as the majority choice. We further observed that the majority choice will be selected by more subnets and become even more dominating for the shallow blocks in the following generations. However, this is not the case for the deep blocks. In the right-bottom subgraphs. we observed that either the majority choice emerges at very later generations, or multiple curves intertwine with each other, thus there is no obvious majority choice for deep blocks. The above observations indicate that candidate subnets tend to vote majority choice in the shallow blocks at the early searching stage, while for deeper blocks the majority choice is unclear.
3.2 RELATION BETWEEN THE MAJORITY CHOICE AND THE OPTIMAL CHOICE
Majority choice the choice that most of the candidate subnets "think" it can lead to better accuracy during evolution. Intuitively, majority choice should be representative of optimal choice. To quanti- tatively analyze the relationship between the majority choice and the optimal choice, we first define a binary categorical choice distance DD where DD equals to O if two choices are the same otherwise D equals to 1. Then we calculate four types of D for the choices in the continuous shallow blocks as follows: the average distance 2) between the non-majority choices and the block choices in the first generation (i.e., initial block choices), the average distance DD between the non-majority choices and the block choices in the last generation (i.e., optimal block choices), the average distance D between the majority choices and the block choices in the first generation, and the average distance D between the majority choices and the block choices in the last generation. Here, all D denotes the averaged distance across all shallow blocks. Note that the non-majority choices are all the choices of shallow blocks when their average percentage is below an empirically predetermined threshold (0.7) at the carly searching stage. The majority choices are obtained once we observe their average percentage rising above 0.7. The results of non-majority choices and majority choices are shown in Figure 3] (a) and (b), respectively. The x-axis (y-axis) represents the distance between the non- majority/majority choices and the block choices of the first (last) generation. Each point represents an independent searching process with random seeds. As can be seen, for non-majority choices (Figure 3 (a)), the points are scattered in a symmetrical way, which means a certain non-majority choice can be either close or far away from the optimal choice. In contrast, for majority choices (Figure (b)), most points are located at the right-botton of the plane, which indicates that the majority choices very close the final optimal choices.
4

---

Under review as a conference paper at ICLR 2023
Redhced Compuration
Choice
Blocks
CRI
FM
UMI
Ciaa
FMD
Ca3
CB9
FMB
FM3
FMR
CBsa
Naco
Na50
di-1 Genenstion
Gesention
Figure 4: (a) No data reuse if the first two blocks have different choices (b) The overall workflow with choice freezing.
i-th Generation
0.9
Prozen)
0.3
Percentag
m
m++1
64
Bicck
Dict
Generation
Net1 Net2
Net3
Ne14
Not50
Figure 5: The choice percentage one subsequent block after its previous blocks are frozen.
3.3 KEY INSIGHTS: CHOICE FREEZING
The above comparative experiment shows that majority choices are very close optimal choices and can be safely used to approximate optimal choices. This provides solid foundation for us to freeze the majority choices in the shallow blocks when they appear in the early searching stage. The motivation of freezing the block choices is illustrated in Fig/4] As shown in (a), assuming there candidate subnets be evaluated sequentially the current searching epoch, the choices of the first two blocks CBI ,CB2 of Nel/ and Ne12 are (blue, red} and [yellow, red} respectively. Since the choices these two shallow blocks are totally the same (even though the choices of CB2 are the same), the intermediate feature maps (FM1, FM2) in Netl and Ner2 will be different. Hence, obtain the evaluation results for both subnets, the computations However, as shown in Figure 4 (b), the choices in CB/ and CB2 are already majority choices in the previous searching epoch and we managed to freeze them at the i-th search generation, Net/ will have the same choices for CBI and CB2 as Ne12, does Net3 to Ner50 in the current generation and all other subnets in following generations. Since one-shot NAS uses the same validation dataset input. the FMI and FM2 of these subnets will be exactly the same and the computations for CBI and CB2 will become redundant. Therefore, one can simply store the FM2 and reuse for all the subsequent evaluation of candidate subnets, with significant computation savings of all frozen blocks. Note that the stored FMs can be reused both within and across generations by every subnet evaluation process, which brings more computation savings than NASA (Ma al. 2021
all the blocks are required.
3.4 PROGRESSIVE CHOICE FREEZING EVOLUTIONARY SEARCI
To further explore the behaviour of the choices of deeper blocks, we keep monitoring the choice percentage of every subsequent block once the previous blocks are frozen in the candidate subnet. As shown Figure [5] we conduct three independent search processes where the first 3, 6, and continuous block choices are frozen according to the majority choice percentage threshold. For example, the orange curve represents the choice percentage of the 7th block B7 if we freeze the first 6 continuous blocks the 5th generation (marked by the dot line). It can be seen that, for all three blocks in the figure, the choice percentage increases drastically after previous blocks are fixed. Since higher choice percentage means higher possibility that majority choice is observed, freezing the previous block choices will help freeze the latter block choices. Based on this observation, we propose to freeze more block choices progressively when the previous
already frozen.
The overall idea is implemented in our proposed Progressive Choice Freezing Evolutionary Search (PCF-ES) algorithm in Alg. []] The population of the first generation is randomly generated (Line
5

---

Under review as a conference paper at ICLR 2023
Algorithm Progressive Choice Freezing Evolutionary Search
Input: supernet weight Wa, A. population size I', architecture constraints C, max generation T,, validation dataset
Do. choice percentage threshold mutation rate r. number of blocks N, monitor size
Output: architecture with best validation performance
I: Random initialize Po
2: c=
3: for = 1 : do
4: c 0 then
5: for = N: do
average choice percentage M, from B3 to B block ratio Bry from Bj to Bi
7:
8:
if M H and Bry >> 0.5 then
9:
TTop-K =0
1-j
L-g
10:
c=1
break
12: end
13:
end for
14: else
15:
for x = f do
average choice percentage My from By+1 to Bj+s if My > H and Bry 0.5 then
17:
18:
drpp-K K ama
(~()+a) (~(j+z)
19:
end
20:
end for
21: end if
22:
Pountatinn = Mulation (Topk.r. C))
23: Pornssaver Crossocer(Tophl C)
24: P= Pmuuation U Perossover
25: Acc Inference WA, Du, D).P)
26: end for
27: Return architecture with highest accuracy
1) and e flag to check if there are block choices that have been frozen (Line 4). When is 0 which implies block choice has been frozen, calculate and examine the average choice percentage the first j blocks at each search generation, and j starts from N (the number of blocks in subnet) to I because aim to freeze many block choices possible at the beginning (Lines 5-6). Note that the average choice percentage M, is the mean of choice percentage from block B1 to Bj- One obvious downside of the average choice percentage is that it cannot indicate the choice percentage of each block, thus the choices of some blocks with relatively low choice percentage may be mistakenly frozen when the averaged choice percentage across the investigated shallow blocks exceeds the threshold. We employ another metric, i.e., block ratio, to avoid the aggressive choice frozen. block ratio measures the ratio of blocks whose choice percentage exceeds the threshold. For example, if the average choice percentage of the first 10 blocks exceeds the threshold, among which only 7 of these blocks exceeds the threshold, then the block ratio 70%. Only when the block ratio is above the block ratio threshold, we allow the block choices to be frozen, which ensures that most of the frozen block choices reach the required choice percentage.
If both metrics, i.e., average choice percentage and block ratio, are above the thresholds (Line 8), choices of the first j blocks in all Top-K subnets are changed to the current majority choices (Line 9) and become frozen. Accordingly, rest of the searching process, these block choices will not be affected by the mutation and crossover operations. On the other hand, if any of the metrics is below the threshold and the flag 0, the algorithm jumps to population generation and generates new candidates normally (Lines 22-24). If the first j blocks can be frozen, we keep exploring chances to freeze more block choices during the later searching stage. The monitor size f is the maximum number of blocks we should monitor each generation in the latter searching stage (Line 15). The purpose of using monitor size here is to prevent aggressive block freezing. Specifically, since the search space is already remarkably narrowed down in the initial freezing step (up 8 continuous shallow blocks can be frozen time, while supernet usually has total of 20 blocks), keep freezing too many subsequent blocks (even if their average choice percentage and block ratio are above the threshold) can make the search space small that finally leads to inaccurate search results. In the progressively freezing phase, the
set flag
1 when the first blocks frozen. For the

---

Under review as a conference paper at ICLR 2023
40
10
0
100 200 300
400
Sample
Figure 6: The importance of samples for different subnets after shallow blocks frozen. average choice percentage Mf is the mean of choice percentage from block Bj+1 to Bj+s (Line 16). If the My is above the threshold, blocks B,+1 to By+s in Top- K subnets will be frozen based on the current majority choice.
After all candidate subnets are generated, they are evaluated on the validation dataset and ranked based on the validation accuracy (Line 25). We keep track the Top-K subnets and return the Top-1 subnet architecture when maximum generation is reached or all block choices are frozen.
3.5 DISTINGUISH-BASE IMPORTANCE SAMPLING
To achieve the potential computation savings, it's necessary to store the intermediate feature maps, e.g. FM2 in Fig 4 of all the input samples in the validation dataset, the size of which may exceed the memory capacity of the GPUs in some cireumstances. For instance, for each input sample, the intermediate data size of the second block of subnets in SPOS on ImageNet is approximately 0.8 MB. the validation dataset contains more than 5000 input images, the memory storage of all the intermediate feature maps could exceed a GPU's memory capacity with 4GB main memory and need to be saved CPU memory. However, transferring data from CPU memory to GPU is very time and energy consuming with compared to transferring from GPU local memory. The frequent CPU-GPU data transfer impairs the search efficiency significantly. To tackle this problem, we first leverage Importance Sampling (Alain al.) 2015) importance sampling comes from the fact that not all the input samples contribute to distinguishing the evaluation accuracy of candidate subnets. For example, for images which are easy be classified by most subnets, they barely contribute to distinguishing the performance of the searched subnets.
compress the intermediate data. The idea of
Inspired by the importance sampling during training (Zeng ciple to sample the important data for the subnets evaluation. Let æ denotes the input data, p denotes the uniform distribution referred from random sample, q denotes the distribution adopted by impor- tance sampling and f() denotes the evaluation accuracy of input data æ. The unbiased estimation of evaluation accuracy can be obtained by Eq.
2021) follow the similar prin-
Fg[[(x)]] / ((t))( = Fig p(x)
[(x) q(x))
if (1<)) > 0 whenever p(x) >
Moreover, the estimation variance the distribution y is minimized when
(())* = -=>((=))((())))) and ((=)))((=)ll2dz
In our design, we use the cross-entropy loss approximate the evaluation accuracy and then calcu- late the important sampling distribution qιx) based on ().
Although we can obtain the important samples for certain subnet using the above method, we cannot reuse the intermediate data of these important samples for other subnet's evaluation unless the important samples are shared across different subnets. Therefore, we quantitatively analyze the importance of samples for different subnets after the shallow blocks are frozen. As shown in Fig/o] we present the importance distribution of 500 important samples (x-axis) for 50 different subnets (y-axis). The color represents the importance of sample. It can be found that the sample with high importance (deep blue) for one subnet is also important for other subnets. Overall, the importance distribution of certain subnet (a certain row) is similar to the importance distribution of other subnets (other rows). As a result, it is only necessary to store the intermediate data important samples for subnet and then reuse it for other subnets.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of searched architecture w.r. different benchmarks and search methods. majority choice percentage threshold.MS. monitor size.
Supernet SPOS
Method
Top-1
Top-5
FLOFS(M)
GPU Energy(M))
Evalutionary
x
X
73.5
328
16.7
CF-ES
X
73.5
90.1
324
13.9
PCP-ES
0.7
73.4
90.1
316
7.5
2.2
FairNAS
Evolutionary
72.0
89.8
322
15.2
4.2
CF-FS
0.5
x
72.2
90.0
323
11.7
3.4
PCF-ES
0.4
2
71.9
90.1
329
8.4
2.5
GreadyNAS
Evolationary
72.5
90.1
322
17.9
4.8
CF-ES
0.4
X
72.3
90.2
327
15.1
3.9
PCF-S
72.5
90.2
328
11.1
3.0
Time (hour)
Time (houn
Time (hou)
0.8
0.8
0.6
0.6
0.66
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
15
Generation
Generation
Generation
(a)
(c)
Figure 7: The search time breakdown SPOS with (a) evolutionary search (b) CF-ES (c) PCF-ES.
EXPERIMENT
4.1 EXPERIMENT SETUP
Our experiments are conducted on widely used MeageNet(KKizzhvssk et al. 2012). of which the validation dataset contains 50000 images at 1000 categories. Both supernet training and subnet searching processes are performed on Tesla P100 GPUs with 16 GB memory. We employ the chain structure supernet adopted with three benchmarks: SPOS (Guo et al. 2020), FairNAS (Chu ct al. 202T) and GreedyNAS You et al. 2020). whose supernets contain 20, 19 and 21 blocks, respectively. The supernets trained with the parameter settings in the original papers. The maximum search generation is 20 and the population size is 50. At each generation, subnets with 10p-10 validation accuracy kept as parent networks.
4.2
EXPERIMENT RESULTS
The search results on different benchmarks and search methods are shown in Table 1. Evolution- ary represents the original evolutionary algorithm used in (Guo et al. 2020), which is regarded as the baseline in our study. Choice Frozen Evolutionary Search (CF-ES) represents the method that only freezes shallow blocks and does not monitor the deep blocks for further freezing. Thanks to the reduced computations for the frozen shallow blocks, CF-ES method achieves averagely 19% time saving and 18% energy saving while maintaining Top-1 and Top-5 accuracy compared to the baseline. Progressively Choice Freezing Evolutionary Search (PCF-ES) is our proposed method that freezes both shallow and deep blocks during the searching, which achieves much more latency reduction with little accuracy loss. Specifically, PCF-ES reduces 38% ~ 55% search latency and 37% 51% GPU energy consumption in three benchmarks. To understand where the speedup actly comes from, we further breakdown the execution time at each generation when using different searching methods. As shown in 7ig the searching time (latency) of each generation close to each other in the original evolutionary search, while CF-ES takes less time since the 6th generation because shallow blocks are frozen. The latency of PCF-ES begins to decrease from the 6th gener- ation and keeps dropping progressively until it becomes 0 in the 13th generation, during which all blocks are frozen. other words, it takes only 13th searching epochs find the optimal architecture when using PCF-ES method.
8 shows the Top-1 searching accuracy during the searching stage different approaches and benchmarks. It can be observed that, for all benchmarks, all three methods have the same accuracy at the early generations when no choices are frozen. Compare to other two methods, PCF-ES shows very little accuracy loss in SPOS and FairNAS, and can always terminates earlier.
To determine appropriate apper-parameters. i.c., choice percentage threshold, block ratio thresh- old, and monitor size, used in algorithm, conduct sensitivity analysis for all three types of thresholds. Note that the proper thresholds we obtained using one dataset are effective for different searching scenarios. We first study the effects of selecting the choice percentage threshold for dif-
8

---

Under review as a conference paper at ICLR 2023
Accura A
0.734
0.720
0.722
Accuracy
0.725
00.724
0.732
0.730
ACC
0.716
0.718
Exolucionayy
0.723
0.722
0.728
0.714
C2-25
(0.721
0
5
10
15
20
5
10
15
20
10 Generation
5
15
20
Generation
Generation
(b)
(c)
Figure 8: The Top-1 accuracy with different methods on (a) SPOS (b) FairNAS (c) GreedyNAS.
Rccuracy
0.734
0.720
0.725 G
0.718
0.724
0.732
0.730
T 0.6
0.716
T 0.3
0.723
0.722
T-0.4
T-03
0.728
= 0.8
0.714
0.712
0.721
=
20
10
15
20
10
15
20
Generation (a)
Generation (b)
Generation (c)
Figure 9: Top-1 accuracy v.s. generation with different majority choice percentage thresholds (T) on (a) SPOS (b) FairNAS (c) GreedyNAS.
ferent benchmarks are shown in Fig The monitor size and block ratio are empirically initialized to be 3 and 0.5. As can be seen, for SPOS, when choosing the smallest threshold (T = 0.6). the searching terminates the earliest (at the 10th generation) with relatively low accuracy. This is because some shallow blocks are mistakenly frozen. Similar behavior can be observed when setting threshold as 0.3 for FairNAS and GreedyNAS. To avoid the blocks being frozen too aggressively, proper threshold should be picked. For example, when setting threshold 0.7 for SPOS and 0.4 for FairNAS and GreedyNAS, the searching process maintains relatively high accuracy while ter- minating early well. These thresholds are the optimal thresholds that ensure both high accuracy and low latency at the same time. The effects of selecting different block ratios are not shown here since we observed similar results the effects of the choice percentage threshold. We empirically set block ratio as 0.5 in all three benchmarks.
0.733
0.736
Accuracy
0.5
0.6
S Sauing
0.6
Camp: Sa.5ai
Saving
20.7250
0.7275
Aer.mey
0.5
0.4
p. Saving
0.734
0.4
0.7200
0.7225
0.733
0,2
0.3
0.7175
0.7150
Como Sexing
0.7200
Comp, Sertag
0.732
1
2
Monitor Blocks
Monitor Blocks
Moniter Blocks
(a)
(c)
Figure 10: The searching accuracy and computation savings using different monitor sizes on (a) SPOS (b) FairNAS (c) GreedyNAS.
shoswws the searching accuracy and saved computations with different monitor sizes. For SPOS and GreedyNAS, the accuracy fluctuate with the increase of the monitor size. However, for FairNAS, the accuracy drops significantly when the monitor size is greater than 2. This is because too many deep blocks are frozen in this case. The saved computation increases steadily in all benchmarks as the number of monitored blocks increases.
0.735
Fig II shows the validation accuracy of the optimal ar- chitecture searched by PCF-ES with different importance sampling rate (the ratio of sampled data to total data). We observe that when the sample rate is higher than 40%, the accuracy impact is negligible. However, the accuracy drops rapidly if the sample rate further decreases.
0.734
0.733
0.732
0.731
1.0 0.8 0.6 0.4 0.2 Sample Rate
Figure 11: Validation accuracy V.s.
sample rate.
CONCLUSION
In this work, we profiled the evolutionary searching process of one-shot NAS and observed the key observations regarding the choice of blocks. Motivated by the majority choices that appears at the shallow blocks at the early searching stage, we proposed a progressively choice freezing evolution- ary search narrow the search space and reduce the searching time. As our evaluation results show, our method reduces averagely 46% of searching latency and 43% of energy consumption for all benchmarks while incurs only 0.1 % accuracy loss.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv: /51 1.06481, 2015.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint 1rX1v.661..02167, 2016.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference Machine Learn- ing. pp. 550-559. PMLR, 2018.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In International Conference on Machine Learning, pp. 678-687. PMLR, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference Learning Representations, 2019. URL https:
//a arx.v.orgg/aa//SS.. 18112 00330.033p0...
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2020. URL
www.9.
Xin Chen, Lingxi Xie, Jun Wu, Longhui Wei, Yuhui Xu, and Qi Tian. Fitting the search space of
weight-sharing nas with graph convolutional networks. arXiv preprint arXiv:2004.08423 2020.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12239-12244 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 19(1):197722017, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pp. 544-560. Springer, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. In Proceedings of the IEEF/CVF Conference on computer vision and pattern recognition, pp. 13836-1388 13845, 2020.
Chenxi Liu, Liang-Chied Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xiaohan Ma, Chang Si, Ying Wang, Cheng Liu, and Lei Zhang. Nasa: Accelerating neural net- work design with a nas processor. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 790-803, 2021. doi: 10. N0.1009/SCAA22022200100000 000067.
Hieu Pham, Melody Guan, Barret Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. pp. 4095-4104 PMLR, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. arXiv preprint arXiv:2006.02900 2020.
10

---

Under review as a conference paper at ICLR 2023
Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao, A genetic programming approach to designing convolutional neural network architectures. Proceedings of the genetic and evo- lutionary computation conference, pp. 497-504, 2017.
Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. survey neural architecture search.
arXiv preprint arXiv:1905.0133 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search.
In Proceedings of the International Conference on Learning Representations, 2019.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1999-2008, 2020.
Kaicheng Yu, Rene Ranftl, and Mathieu Salzmann. How to train your super-net: An analysis of training heuristics in weight-sharing nas. arXiv preprint (rXiv:2003.04276 2020.
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:/807 1807006906, 2018.
Xiao Zeng, Ming Yan, and Mi Zhang. Mercury: Efficient on-device distributed dnn training via stochastic importance sampling. In Proceedings of the 19th ACM Conference on Embedded Net- worked Sensor Systems, pp. 29-41, 2021.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2432,2432, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rX15:1611.01578, 2016
11

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 OBSERVATION
As introduced in Section 3.1 the majority choices of shallow blocks be determined in the early search generations while deep blocks cannot. Here we present the choice evolution of more blocks in two other one-shot NAS benchmarks: FairNAS and GreedyNAS. As shown in Fig. 12 Moreover, choice evolution of different dataset are shown in Fig [3] It can be found that the majority choices of shallow blocks appears in the early search stage and their percentage increases steadily. On the contrary, the majority choice in deep blocks can not be decided during the search process.
10
1.0
20.3
Percontage
80.8
80.8
10.22
20.1
Generation
5
10
Generation
15
Generation Shallow Blacks
Generation
0.4
0.3
0.2
0.1
10
15
10
15
15
10
15
Generation
Generation
Generation
Deep Blacks
(n)
0.08
Percentage
0.8
Ticenttgg
80.8
04
0.6
0.6
0.4
0.4
0.4
0.0
Generation
15
Generation
10
15
Generattio Shallow Blacks
10
15
Gereration
10
15
Generation
10
15
Percentos
0.25
0,20
0.20
0.25
0.15
0.10
0.05
10
15
5
10 15
10 Generation
15
10 Generation
15
Generation
Generation
Generation Deep Blocks
(b)
Figure 12: The percentage of different choices in shallow blocks and deep blocks v.s. generations on (a) FairNAS (b) GreedyNAS.
A.2 PROGRESSIVELY CHOICE FREEZING
In section 3.4.
show that in SPOS, the choice percentage of the subsequent blocks increases rapidly after the previous blocks are frozen. We further conduct the same experiments for other two benchmarks. As shown in Fig. 14 for all six searching processes in (a) and (b), the choice percentage of subsequent blocks increases dramatically after freezing the previous blocks.
A.3 EXPERIMENT RESULTS WITH MORE DATASET
We conducted our experiment on multiple datasets to support Section 4.1. Some of the experiment results on CIFAR- 10 and CIFAR- 100 dataset at SPOS are shown as Table [ It is can be found that, our CF-ES method could achieve up to 20% time saving with no accuracy loss while PCF-ES achieves up to 52% time saving with negligible accuracy loss.
A.4 THE COMPARISON BETWEEN SAMPLING METHOD
We compare the validation accuracy of the importance sampling with random sampling, as shown in Table It can be found that importance sampling method achieves higher the same accuracy compared to random sampling at all sample rates. Our method can achieve a lower sample rate while maintaining high search accuracy.
12

---

Under review as a conference paper at ICLR 2023
1.0
1.0
g0..8
50.8
Percentoge
80.8
30.6
0.06
0.6
0.6
202
0.0
5
10
15
20
5
10
15
20
5
10
35
20
0.0
10
15
20
Generation
Generation
Generation
Ceneration
Cesteration
(a)
0.6
0.5
Percentage
0.0
10
15
20
10
Generation
Generation
Generation
Generation
Generation
(b)
1.0
1.0
1.0
5.066
50.8
30.8
50.8
0.0
0.6
0.6
0,6
80.4
0.4
0.0
10
15
20
10
20
10
15
20
10 Generation
20
Goneration
(a)
00000000
0.8
0.0
Purcetaage
0.6
0.6
0.4
0.2
15
10 Generation
35
20
5
10
15
20
10 Generation
15
20
15
20
Generation
(b)
Figure 13: The percentage different choices in shallow blocks and deep blocks v.s. generations on SPOS on (a) CIFAR-10 (b) CIFAR-100.
m 3
I
6
0.7
0.8
iiakk Biock
88 aecentad 8
0.5
0.6
Book
percentage
855
Beck
0.5
0.3
0.4
0
0.2
0.3
0.3
0.2
5
15
20
10 Generation
15
0
10
15
20
Generation
Generation
(a)
III
Rescent
81
0.7
Bock
90 7
0.9
0.5
1.0
Pooce Bloc 1d
0.5
0.6
0.4
0.3
0.3
0.2
15 Generation
15 Generation
Generation
(b)
Figure 14: The choice percentage of subsequent blocks after previous blocks are frozen in (a) Fair- NAS (b) GreedyNAS.
Table 2: Comparison searched architecture w.r.t different dataset
Dataset
Method
Topl
Top5
GPU Hour
CIFAR- 10
Evolutionary
96.5
99.9
5.4
CF-ES
96.5
99.9
4.6
PCE-ES
96.4
99.9
3.0
CIFAR-100
Evolutionary
75.1
92.4
5,4
CF-ES
75.0
92.1
4.4
PCF-ES
74.9
92.3
2.6
13

---

Under review as a conference paper at ICLR 2023
Table 3: Validation accuracy with random sampling and distinguish-baser importance sampling
Sample Rate
90%
70%
50%
30%
Method
Random Sampling
73.4
73.5
73.1
73.1
Importance Sampling
73.5
73.5
73.4
73.1
Table 4: Validation accuracy with distinguish- based importance sampling and quantization.
Bit
32
16
8
Sample Rate
50%
73.4
73.4
73.1
73.0
30%
73.1
73.1
72.8
72.8
10%
73.1
73.1
73.0
72.8
A.5 THE EFFECTS OF QUANTIZATION
In an attempt to achieve more memory savings, we tested another compression technique, i.e., quantisation, our sampled dataset. Quantization is commonly used compression method that converts a floating-point value to a fixed-point value with fewer bits. Here the quantization is applied on the intermediate data before storing them into the memory. The validation accuracy with different importance sampling rate and quantization bit for SPOS benchmark is shown Table 4 It can be seen that the search accuracy maintains when the intermediate data quantized to 16 bit, but drops considerably when further quantized to lower bit. For the configuration PCF-ES method, 40% sample rate and 16-bit quantization for the best searching performance.
14


Reviews:
[{"review_id": "rTisiEqao3", "reviewer": "Reviewer_cLeE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the idea of freezing layers within a One-Shot evolutionary NAS approach. As the lower layers quickly converge to a 'final' state the authors argue that freezing them and removing the calculations performed within them will reduce computation and hence energy. An approach is developed to identify when layers can be frozen and another approach to identify the 'significant' samples as pre-computing the computations (and re-using) can overwhelm the memory available within a GPU.", "summary_of_the_review": "The paper is clear and concise. The idea seems reasonably novel and is supported by good evidence.\n\nSome more specific comments:\n\n- The review of one-shot NAS is interesting, but probably too long for this paper. More on related work would be better to present.\n\n- There is a bizarre blue line under the middle bottom graph in figure 2. \n\n- \"This provide a solid\" -> \"This provides a solid\"\n\n", "strength_and_weaknesses": "Strengths:\n- The paper idea is intuitive and well worked out\n- The description of the idea is well presented making the idea clear to the reader\n\nWeaknesses:\n- The paper has no clear related work section and it is hence unclear how the authors feel their work relates to prior work save for the discussion of work they base themselves on.\n- The conclusion is very short and provides little in the way of insights\n- The only results are on ImageNet - it would be interesting to see how the approach works on datasets not over-evaluated with NAS approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dz_fsieoBg", "reviewer": "Reviewer_zkYQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method which accelerates the architecture choice step of certain one-shot NAS methods. The method greedily freezes the first block choices during the architecture optimization phase with an evolutionary algorithm. This allows for caching the intermediate feature maps and therefore reduce the computational effort. In an experimental study on ImageNet, they show that this method saves time at no reduction in predictive performance. Furthermore, they investigate the hyperparameter sensitivity.", "summary_of_the_review": "In this well-written paper, the authors present an original idea to reduce the time for finding the architecture after a one-shot NAS. The proposed method reduces time by a factor of 2 but additionally it adds several sensitive hyperparameters. The authors study this sensitivity but provide no advice how to select stable hyperparameters. This raises the question whether we are able to reduce the time or we spend additional time searching for these hyperparameters. Finally, the only existing baseline is not considered for comparison as well as other simple baselines that could accomplish the same.", "strength_and_weaknesses": "Strengths: important, under-explored aspect of NAS, clear description of the work\nWeaknesses: lack of baselines (see next section for actionable feedback), hyperparameter sensitivity (discuss seriousness, find ways to overcome it)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qw-bwTCihL", "reviewer": "Reviewer_Y3fv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient one-shot neural architecture search (NAS) method that progressively freezes the architectures. The authors show that the first few blocks become similar among the candidate solutions during the evolutionary architecture search and then develop an acceleration method of architecture search by exploiting this phenomenon. In addition, the input sampling method is introduced to reduce memory consumption when reusing intermediate feature maps among candidate architectures. The experimental results demonstrate that the proposed progressive choice freezing evolutionary search (PCF-ES) can speed up the architecture search compared to existing one-shot NAS methods.", "summary_of_the_review": "The authors treat an interesting topic in one-shot NAS. However, as described in the section on Strength And Weaknesses, the reviewer feels that the weaknesses of this paper outweigh the strength.", "strength_and_weaknesses": "**Strengths**\n* An interesting observation in one-shot NAS, the first few continuous blocks of candidate architectures become similar in the early search epoch, is presented.\n* The proposed PCF-ES can reduce the architecture search cost without significant performance deterioration on the ImageNet dataset.\n\n**Weaknesses**\n* The authors mention that the architecture searching cost is higher than the supernet training cost. It may be true when searching multiple architectures with different complexities by optimizing different objective functions. However, the proposed method only targets searching for a well-performed architecture. The reviewer suspects that the search cost reported in the experiment does not include supernet training. The search cost of both supernet training and architecture search phases, i.e., total search cost, should be reported in the experiment.\n* The authors treat the existing work of NASA (Ma et al. 2021) as related work on accelerating the architecture search process. However, an empirical comparison of NASA and the proposed method is not performed.\n* The experiment is conducted using only the ImageNet dataset. It is not clear whether the assumption that the first few blocks become similar among the candidate solutions will hold in other datasets and tasks. To evaluate the proposed method in various situations, it might be a possible choice to use NAS-Bench datasets for further performance evaluations.\n* The authors consider that the intermediate feature maps to be reused should be stored on the GPU memory. The reviewer thinks that it is possible to store the feature maps on the CPU memory, although the data transferring cost occurs. It is not clear the motivation for storing all intermediate feature maps on the GPU memory.\n* The detailed algorithm of importance sampling is unclear. How to calculate the probability distribution of $q(x)^*$ and the integral over $x$ in (2)? Also, the probability distribution of $p(x)$ is needed to use the equation (1). How to calculate $p(x)$ which is the distribution of input data x?\n* The effectiveness of importance sampling seems to be unclear. It would be better to compare the proposed sampling method to random sampling.\n* In the proposed PCF-ES, two techniques, freezing the shallow blocks and importance sampling-based input sample selection, are introduced. It is not clear which techniques mainly contribute to computational cost reduction. The ablation study should be conducted. What are the computational costs of the method only using the freezing technique and the method only using the importance sampling-based sample selection? If we store the intermediate feature maps on the CPU memory, we can evaluate the method only using the freezing technique.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cst9o8HJMA", "reviewer": "Reviewer_Gnhw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focused on accelerating one-shot neural architecture search (NAS). In particular, the authors aims to accelerate the architecture search phase based on evolutionary approach. The authors first discuss their observation that the evolutionary approaches tend to select the same shallow blocks from the early stage of the search. Then, an approach is proposed that freezes the selected blocks in shallow part of the super-network. This approach has an advantage that the features computed in all population in the evolutionary approach are the same because they share the same blocks, leading to saving some computational time. Moreover, to further speedup the search process, a mechanism to subsample validation dataset to approximate the validation accuracy with low computational time. The proposed approach, PCF-ES, have be compared with a baseline evolutionary approach and a variant of the proposed approach on three supernet variants on ImageNet. Without compromising the performance significantly, saving 50% of GPU hours and GPU energy are reported.", "summary_of_the_review": "As mentioned in the strength and weaknesses section, numerical evaluation in this paper is not sufficient to support the claim of this paper.", "strength_and_weaknesses": "# Strength\n\nEnergy saving is becoming a crucial recently. The proposed approach aims at contributing to this point.\n\n# Weaknesses\n\nSingle Dataset: Evaluation is only done on a single dataset. Its generality is not discussed.\n\nObservation: The observation in Section 3.1 seems to be observed on a single dataset as well. The experimental settings are not provided for this observation.\n\nComparison: The proposed approach is compared only with SPOS variants. It has not been compared with other one-shot NAS approaches such as DSNAS, which does the weight training and architecture search at once. Because of the lack of the evaluation it is not clear whether the proposed approach is useful among other one-show NAS approaches.\n\nStatistical Evaluation: No statistical evaluation has been performed. It is not even clear what are the numbers in Table 1 and Figures. Are they average values?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject