Tiered Pruning for Efficient Differentialble Inference-Aware Neural Architecture Search
Authors: Slawomir Kierat
Mateusz Sieniawski
Denys Fridman
Chenhan D. Yu
Szymon Migacz
Pawel Morkisz
Alex Fit-Florea
Venue: Submitted to ICLR 2023 (2023)
Keywords: nas
dnas
neural architecture search
differentiable neural architecture search
state-of-the-art
imagenet
classification
gpunet
efficientnet
pruning
inference-aware
computer vision
object detection

Abstract: We propose three novel pruning techniques to improve the cost and results of inference-aware Differentiable Neural Architecture Search (DNAS). First, we introduce $\textbf{Prunode}$, a stochastic bi-path building block for DNAS, which can search over inner hidden dimensions with $\mathcal{O}(1)$ memory and compute complexity. Second, we present an algorithm for pruning blocks within a stochastic layer of the SuperNet during the search. Third, we describe a novel technique for pruning unnecessary stochastic layers during the search. The optimized models resulting from the search are called PruNet and establishes a new state-of-the-art Pareto frontier for NVIDIA V100 in terms of inference latency for ImageNet Top-1 image classification accuracy. PruNet as a backbone also outperforms GPUNet and EfficientNet on the COCO object detection task on inference latency relative to mean Average Precision (mAP).

Under review as a conference paper at ICLR 2023
TIERED PRUNING FOR EFFICIENT DIFFERENTIABLE INFERENCE- AWARE NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-bling review
ABSTRACT
We propose three novel pruning techniques improve the cost and results of Inference-Aware Differentiable Neural Architecture Search (DNAS). First, we introduce Prunode, stochastic bi-path building block for DNAS, which can search over inner hidden dimensions with ((1) memory and compute complexity. Second, we present an algorithm for pruning blocks within a stochastic layer of the SuperNet during the search. Third, we describe novel technique for pruning unnecessary stochastic layers during the search. The optimized models resulting from the search are called PRUNET and establishes new state-of-the-art Pareto frontier for NVIDIA V100 in terms of inference latency for ImageNet Top-1 image classification accuracy. PRUNET EfficientNet on the COCO object detection task on inference latency relative to mean Average Precision (mAP).
backbone also outperforms GPUNet and
INTRODUCTION
Neural Architecture Search (NAS) is a well- established technique in Deep Learning (DL); conceptually it is comprised of a search space of permissible neural architectures, search strat- egy to sample architectures from this space, and an evaluation method to assess the performance of the selected architectures. Because of practi- cal reasons, Inference-Aware Neural Architec- ture Search is the cornerstone of the modern Deep Learning application deployment process. Wang et al. 2022);; Wu et al. (2019); Yang et al. (2018) use NAS to directly optimize inference specific metrics (e.g., latency) targeted de- vices instead of limiting the model's 's FLOPs or other proxies. Inference-Aware NAS stream- process.
MuNNe
GPUNet cistilled
84
GIUNeeD dist illed
ImageNet
PRUNet
GPUNet
82
ResNitt
81
RegNetX
80
RegNetr
Effiilentet
79.
EfficiencNae-X
V3
TensorRT FP16 bs= Latency (ms)
Figure 1: PRUNET establishes new state-of-the- art Pareto frontier in terms of inference latency for ImageNet Top-1 image classification accuracy.
lines the develoopeentto--eellooymy New concepts NAS succeed with the ever- growing search space, increasing the dimensionality and complexity of the problem. Balancing the search-cost and quality of the search hence essential for employing NAS in practice.
Traditional NAS methods require evaluating many candidate networks to find optimized ones with respect to the desired metric. This approach can be successfully applied simple problems like CIFAR- 1 Krizhevsky et al. (2010), but for more demanding problems, these methods may turn out to be computationally prohibitive. To minimize this computational cost, recent research has focused on partial training Falkner et al. (2018), 8); Li et al. (2020a); Luo et al. (2018), performing network morphism Cai et al. (2018g 8a); Jin et al. (2019); Molchanov et al. (2021) instead training from scratch, or training many candidates at the same time by sharing the weights Pham et al. (201 8). These approaches can save computational time, but their reliability is questionable Bender et al. (2018); Xiing et al. (2021); Yu al.) (202I); Liang et al. (2019); Chen et al.] (2019); b; Zela al. (2019), i.c., the final result can still be improved. our experiments, we focus on a search space based state-of-the-art network showcase the value of our methodology. We aim to revise the

---

Under review as a conference paper at ICLR 2023
weight-sharing approach to save resources and improve the method's reliability by introducing novel pruning techniques described below.
Prunode: pruning internal structure of the block In the classical SuperNet approach, search space is defined by the initial SuperNet architecture. That means GPU memory capacity sig- nificantly limits search space size. In many practical use cases, one would limit themselves to just a few candidates per block. For example, FBNet Wu et al.] (2019) defined nine candi- dates; skip connection and Inverted Residual Blocks (IRB) with expansion, kernel, group E {(1, 3, 1), (1, 2). (1, 1), (1, 5, 2). (3, 3. 1), (3, 5, 1), (6,3, 1), (6, 5, 1)}. In particular, one can see that the expansion parameter was limited only three options: 1, 3, and 6, while more options could be considered not only larger values but also denser sampling using non-integer values. Each additional parameter increases memory and compute costs, while only promising ones can improve the search. Selecting right parameters for search space requires domain knowledge. To solve this problem, we introduce a special multi-block called Prunode, which optimizes the value of parameters, such the expansion parameter in the IRB block. The computation and memory cost of Prunode is equal to the cost of calculating two candidates. Essentially, the Prunode in each iteration emulates just two candidates, each with different number of channels in the internal structure. These candidates are modified based on the current architecture weights. The procedure encourages convergence towards optimal number of channels.
Pruning blocks within a stochastic layer In the classical SuperNet approach, all candidates are trained together throughout the search procedure, but ultimately, one or a few candidates are sampled as result of the search. Hence, large amounts of resources are devoted to training blocks that are ultimately unused. Moreover, since they are trained together, results can biased due to co-adaptation among operations Bender et al. (2018). We introduce progressive SuperNet pruning based on trained architecture weights to address this problem. This methodology removes blocks from the search space when the likelihood of the block being sampled is below linearly changing threshold. Reduction of the size of the search space saves unnecessary computation cost and reduces the co- co-adoption among operations.
Pruning unnecessary stochastic layers By default, layer-wise SuperNet approaches force all networks that be sampled from the search space to have the same number of layers, which is very limiting. That is why it is common to use skip connection as an alternative to residual blocks in order mimic shallower networks. Unfortunately, skip connections blocks' output provides biased information when averaged with the outputs of other blocks. Because of this, SuperNet may tend to sample networks that are shallower than optimal. To solve this problem, we provide novel method for skipping whole layers in SuperNet. It introduces the skip connection to the SuperNet during the procedure. Because of that, the skip connection not present in the search space the beginning of the search. Once the skip connection is added to the search space, the outputs of the remaining blocks are multiplied by coefficients.
RELATED WORKS
In NAS literature, a widely known SuperNet approach Liu al. (2018b); Wu al. (2019) constructs a stochastic network. At the end of the architecture search, the final non-stochastic network is sampled from the SuperNet using differentiable architecture parameters. The PRUNET algorithm utilizes this scheme it is based on the weight-sharing design Cai et al. (2019); Wan et al. (2020) and it relies on the Gumbel-Softmax distribution Jang et al. (2016).
The PRUNET algorithm is agnostic to one-shot NAS Liu et al. (2018b); Pham et al. (2018); Wu et al. 2019) where only one SuperNet trained or few-shot NAS Zhao et al. (2021) where multiple SuperNets were trained to improve the accuracy. In this work, we evaluate our method search space based on the state-of-the-ar GPUNet model Wang et al. (2022). We follow its structure including the number of channels, the number of layers, and the basic block types.
Other methods that incorporate NAS Dai et al. (2019); : Dong et al. 2018 : Tan et al. (2019) but
remain computationally expensive. Differentiable NAS Cai al. (2018)); Vahdat et al. (2020); Wu al. (2019) significantly reduces the training cost. MobileNets Howard et al. (2017): Sandler al. 2018) started to discuss the importance of model size and latency on embedded systems while
2

---

Under review as a conference paper at ICLR 2023
inference-aware pruning Cai al. (018b); Howard et al. (2017): Sandler et al. (0018): Wang et al. (2022) is usually focused on compressing fixed architectures Chen et al. (2018); Shen et al. (202 l);
Yang et al. (2018). Molchanov et al. (2021) further combines layer- wise knowledgeedistiilation distillation (KD)
with NAS to speedup the validation.
DNAS and weight-sharing approaches have major opposing problems, which address in this article: memory costs that bound the search space and co-adaptation among operation problem for large search spaces. We also compare our methodology other similar SuperNet pruning techniques.
Memory cost The best-known work on the memory cost problem is FB Wan al. (2020). The authors use different kinds of masking to mimic many candidates with (1) memory and compute complexity. Unfortunately. masking removes information from the feature maps, which results in bias after averaging with informative feature maps (see Section 3.2.3). Therefore, it may lead to suboptimal sampling. FBNetV2 can extend the search space even 10 times through this application. Unfortunately, such a large expansion may increase the impact of the co-adaptation among operations problem. Moreover, it can render block benchmarking computationally expensive.
Co-adaptation among operations The weight-sharing approach forces all the operations to be used in multiple contexts, losing the ability to evaluate them individually. This problem, named co-adaptation among operations, posterior fading, or curse of skip connect was noticed by many researchers Bender al. (2018); [i al. (2020b; Zhao al. (2021): Ding al. (2022). Progressive search space shrinking Li et al. (2020b); Liu et al. (2018)) in different forms was applied to speed up and improve Neural Architecture Search.
Super pruning Ideas similar to our "Pruning blocks within stochastic layer" modification (see Section 3.2.2p has already been considered in literature. Xia et (2022) removes blocks from the search space which are rarely chosen. In [C al.] (2021), the authors efficiently search over large search space with multiple SuperNet trainings. In contrast to both methods, our approach requires single training of SuperNet, which makes the procedure much faster. Ding et al. (2022) uses DARTs Liu et al. (2018b) to prune operations in cell. It assesses the importance of each operation and all operations but the two strongest are removed at the end of the training. In contrast, methodology removes blocks from the SuperNet during the training, reducing the training time.
METHODOLOGY
Our method can be applied both to layer-wise and cell-wise searches, cf. Bender et al. (2018) Li
al. (2018b); Mei et al. (2019); Xie et al. (2018). Cell-wise search aims to find a cell (building block), which is then repeated to build the whole network. However, as stated in Lin et al. (2020). different building blocks might be optimal in different network parts. That is why we focus on layer-wise search, i.e., each layer can use a different building block.
The SuperNet is assumed to have layered structure. Each layer built from high-level blocks, e.g.. IRB in computer vision self-attention blocks in natural language processing. Only one block in each layer is selected in the final architecture. Residual layers can be replaced by skip connections during the search, effectively making the network shallower.
3.1 SEARCH SPACE
Since layer-wise SuperNets must have a predetermined number of output channels for each layer, selecting the right numbers is critical. Thus, it is worth getting inspired by good baseline model and defining the search space such that models similar the baseline model can be sampled from the SuperNet. With search space defined this way, we can expect to find models uniformly better than the baseline or models with different properties, e.g., optimized accuracy for target latency or optimized latency for a target accuracy. In our main experiments, we chose GPUNet-1 Wang et al.) (2022) as a baseline network. In Table I we present the structure of our SuperNet defining the search space. Since we sample all possible expansions with channel granularity set to 32 (i.c., the number of channels forced to be a multiple of 32) and skip connections are considered for all residual layers, search space is covering 32 24 65 64 972 96 161 160 289 288 4491 1.7e39
3

---

Under review as a conference paper at ICLR 2023
Table 1: PRUNET search space inspired by GPUNet-1
Stage
Type
Stride
Kernel
# Layers
Activation
Expansion
Filters
SE
0
Conv
2
3
1
Swish
24
1
Conv
[3,5]
2
RELU
24
2
Fused-IRB
[3,5]
3
Swish
(0,8)
64
{0.1]
3
Fused-IRB
[3,5)
3
Swish
(0, 8]
{0.1]
4
IRB
[3,5]
2
Swish
(0,8)
160
{0.1]
IRB
[3,5]
RELU
(0, 8
288
{0.1)
6
IRB
[3,5)
6
RELU
(0,8)
448
(0.1)
7
Conv + Pool + FC
1
RELU
1280
candidates. For comparison, similar FBNet SuperNet, which would consume the same amount of GPU memory, would cover only 2 817 9e15 candidates. So, in this case, methodology can cover 2J 1.9e23 times more options with the same GPU memory consumption.
3.2
SEARCH METHOD
Our search algorithm similar Wu et al. (019). focus on multi-objective optimization
min min L(b,0),
(1)
where the network weights and the architecture weights. The goal is to minimize the following inference-awee aware loss function
L(t, 0) CE(,, 0) log(LAT(())".
(2)
where CE(W) 8) is the standard cross-entropy loss, and LAT(0) is the latency of the network. Coef- ficient defines the trade-off between accuracy and latency. Higher results in finding networks with lower latency; lower results in finding networks with higher accuracy. Coefficient 3 scales the magnitude of the latency. The loss function is inference-aware, meaning we optimize the latency of the networks for specific hardware.
We train the SuperNet using continuous relaxation. Output of (-th layer is equal to
Ø1+1 = Σ ai Bi,((1),,
where 2ı is the output of the layer (1 - 1), (xx) represents the output of the i-th block of the I-th layer. Every block in layer is assumed to have the same input and output tensor shapes. The coefficients ai,; come from the Gumbel-Softmax distributior,Jang et al. 2016)
(.r 9(.)/T]
ai Baabel-SSfttaxx(...00
(4)
E, exp((Or, + gt,))/7]
where 0,,; is the architecture weight of the block. gr. is sampled from Gumbel (0, 1). Parameter controls the temperature of the Gumbel-Softmax function. In contrast to Wu et al. (2019), set to have a constant value throughout the training. Architecture weights are differentiable and trained using gradient descent alongside i weights.
The total latency (AT the sum of latencies all layers in the SuperNet. Similarly, the latency of layer is sum of latencies of its blocks weighted by Gumbel-Softme coefficients'
LAT(O) ΣΣ a. -LA(BBB
(5)
Our method allows choosing the target device for inference. On chosen hardware, we pre-compute and store in the lookup table the latency LAT(B; Bt.) for every permissible block, the same way as in Wu et al. 2019). Then the search does not need to compute any further latencies and can be run arbitrary hardware (c.g., for embedded target hardware used autonomous vehicles, one could still train the network supercomputer).
4

---

Under review as a conference paper at ICLR 2023
0.5
041
0.4
0.6
0.4
10
0.2
0.5
0.1
0.2
0.5
0:2
0.15
0.25
0.35
0.45
THRESHOLD
EPOCH
esrmup
(a)
(b)
Figure 2: (a) SuperNet trimming procedure: For the first Cwarmup epochs, only the regular weights are being trained, and the architecture weights are frozen. After the warmup phase, the threshold increases linearly. If architecture weight of block is below the current threshold, is removed from the layer. If the penultimate is to be removed from the layer, it replaced by a skip connection block instead. The output of this skip connection block is multiplied by Φ, and the output of the other block is multiplied by À as stated in Section 3.2.3 At the end of the training, every layer contains exactly one block; thus, the result of the traiming (b) Procedure for channel pruning in case of an IRB: The inner tensor in IRB has the shape of W x H. where C' is the input number of channels of the block, is the maximal expansion ratio, W and H are the width and the height of the feature map. Small candidate uses seC' channels, and large candidate uses leC channels, where 8, E (0; 1] and
non-stochastic network.
The optimal candidate that we try to find (marked with the blue dashed line) uses oC channels, and it is assumed that S S l. Both candidates mask out unused channels. The weights are shared between the candidates. The proportion of channels both candidates use, denoted by and I, dynamically changes throughout the training.
The training is conducted on a smaller proxy dataset to make the SuperNet training faster. We empirically confirmed that the performance of the found architecture the proxy problem correlates with the performance on the original problem. Once the final architecture is obtained from the SuperNet, is evaluated and re-trained from scratch on the full dataset.
3.2. PRUNING INTERNAL STRUCTURE OF A BLOCK
We use a particular procedure for finding the final values of discrete inner hidden dimensions of block. It is required that the impact of discrete parameters the objective function (in case, function of latency and accuracy) is predictable, c.g., small parameter values mean a negative impact on accuracy but positive impact on latency, and large parameter values mean a positive impact on accuracy but negative impact on latency. We also require it to be regular - the impacts described above are monotonic with regard the parameter value. A good example of such parameter the expansion ratio in the IRB.
Prunode consists of two copies the same block. Unlike Wu et al. (019). both blocks share the weights Pham et al. (2018), and the only difference between them is masking. In contrast to Wan al. (2020), masks are not applied to the outputs blocks but only to a hidden dimension. The larger mask is initialized with all 1s, thus using all the channels. The smaller mask is initialized in a way to mask out half the channels. Prunodes are benchmarked with all possible inner hidden dimension values. The latency of prunode is the average of the latencies of the smaller and the larger candidates weighted by Gumbel-Sofftma coefficients.
We try to ensure that the optimal mask value is between the candidates. More precisely, if candidate with a larger mask has a higher likelihood being chosen, i.c., has larger architecture weight compared candidate with smaller mask 8, then both masks should be expanded. In the
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Prunode masking
Consiants c, max distunce, granalarity and momentum were set to 0.8, 0.6, 32, and 0.4 respectivel.
weight 0.0;
/1 Initialize architect weight used by Gumbel SofLnax with 3
update 0.0:
5- 0.5;
11 Small maak initially maska out half of the channels Large mask initially contains all
1.0:
channels
Procedure update upaeeed-euddllemmnnnkk masks (progress)
// Called after each training iteration // Update maaka until consecutive chaices are reschad // Momentum apeeda up convergence
(t x max, chunnels gramularity then update update x momentum + weight;
distance distance
progres)":// Calculate distance between masks; pregress [o, // Updata small maak
s +-x s update;
i then
weight t- 0: min(s, distance);
// Reset architecture weight if not corner case prevent premature convergence
else
<t 0:
// Make small mask non-negative
end
// Update large mask // Ensure divisibility by granularity channels gramularity); // Ensure small mask in bounds // Rasure divis by granutarity lage_maskk +- clip(large_mast mask, small mask + granularity, max chandes:: Enaure large_mask bounds
+ distance:
small small_mask mask + round(s x max, max__hammee chammels, gramulurit);
small mask clip(small mask, gramularity,
large_mast mask +- round(I x max channels, gramularity);
create masks(small mask, large_ mask);
end
opposite case, both masks should be reduced. The distance between masks d = | - s should decrease as the training progresses, and at the end of the search, the d should be close to zero. Due to the rules above, by the end of the search, we expect that the final values obtained are close to each other and should tend to an optimized solution. After the search, we sample a single candidate (from those two modified candidates), ef. [Wu et al. (2019). Fig. 2b visualizes the masks of two candidates in the case of using IRB. The exact procedure of prunode masking is presented in Algorithm
Our routine extensively searches through discrete inner hidden dimension parameter values while keeping memory usage and computation costs low. Our proposed method has a computation cost and memory usage of (1) with respect to the number of all possible values, as only two candidates evaluated every time. Further, as we prune the not promising candidates, the SuperNet architecture tends to the sampled one. As result, co-adaptation among operations is minimized.
3.2.2 PRUNING BLOCKS WITHIN A STOCHASTIC L.AYER
Each block in layer has its architecture weight initialized 1/N,, where Ni is the number of blocks in the layer I. Architecture weights are not modified for the first Ewarmop epochs. After the warmup phase is finished, the SuperNet can be pruned, i.e., when block in layer has probability of being chosen below a threshold, it is removed from the layer. This threshold changes linearly during the training and depends only on the value of the current epoch of the training. During epoch Ewarmup the threshold is equal to
e Ewarmup
t(e) Einitial + (frinal - (initial)
(6)
Etotal
Ewarmap
The initial threshold should not be higher than N/max(N)), where max((N) is the highest number of blocks in layer, so no blocks are immediately removed after the warmup phase. The final threshold should not be lower than 0.5; thus, all blocks but one removed from each layer during the training. An example of block pruning is presented in Fig. 2a
3.2.3 PRUNING UNNECESSARY STOCHASTIC LAYERS
Some layers might have the same input and output tensor sizes. For such layers, it is possible to remove them completely to obtain a shorter network. Removing layer is equivalent to choosing a skip connection block. However, adding skip connection block to the search space increases memory consumption. Moreover, using this approach, observed a bias towards finding shallower architectures than optimal. Instead of adding a skip connection block to the set possible blocks of a layer, we introduce GPU memory optimization. When the penultimate block is to be removed from

---

Under review as a conference paper at ICLR 2023
GPUNNet
Pruflet
Prufett
Prufett
Prufiet
Prufett
Pruteet
PruNet
10,
Con Solr
15
10
3,4
12,32
10
Stage
15,51
13.51
K3
15
15,11
15,31
13.51
k3.44
15.31
10
66,
Sugs Fuxed Sud-11856hh Satin
K3 4567,
15, 65,00.
03,0
0,
13, 00
10,45,
10,
X3 e6,
64 chasnes
<
13, 32
<1,665
K4 865,
K3 el
Shgg 96 chasnai
33,4435 4455,,3
15 25,33
KL #1.7,
1), Hl sl,
16,
83,6
6,,68
M.
K5,
100 durrsb
I& Sadt
K3.
1s, 15,444
K3, 666.5)
65,55..t
13, 67 2.
13. 65,5
15, ml
<), mi
6,
Kt 55,55
13,
13,
13, 65.44,
13,05
15,65 56.11
06.66,6
13, 67,
Suggs 768 thonnel:
is (1.
Kt 4108,
15, 15.46
[ 343
64.56
15, 04
64.6
18, 15,45.78, 7,,78
63,65.3
10,
85,77 70.
15,
15, c5..
15
15, (5.65.57.44
15. 56,6448 s1
IS c8,
15,
15,0788,
B1 BEUU
k3.
15, c5.57,
15. 07.7444 s1
15,0771
a7.8,
15, 05,
thoneel
45.36,33
13, 57.
45
13,45,71
13, 67 07,
05,07,
k3,
15, e57
13, e5.43,
83, 65.99
65, st
K5, (5.66.5)
15, 07 s1
65, (5.77,
(5.68 s1
15,
kS.08,
15, 67.77.
80.66
21.16
8L45
81.39
0.65ms
V103
0.48mm
0.56ms
0.66mms
0.77ms
6.91ms
1.00mm
1.20mm
Figure 3: PRUNET networks: Image input resolution is set to 288x288 for all networks. Thick
border represents block using Squeeze and Excitation. Darker shade of color represents kernel size Width proportional to expansion ratio value. Each architecture is followed by Convolution Ix1 with 1280 output channels, RELU activation function, Adaptive Average Pool, and Linear layer.
layer, it is replaced by skip connection block. From that point, the output of the skip connection block multiplied by s, and output the other remaining block is multiplied by the parameter A, where & and À are coefficients fixed for the whole training. Each layer uses the same values of these parameters. Once the skip connection block or the other block removed from the layer, the output no longer multiplied by any parameter. Selecting different À and ø values allows to reduce the bias towards shallower networks and thus gives more control to find an even better architecture.
4 EXPERIMENTS
test our methodology on Imagenet- Deng et al. (2009) as there are many good networks that potentially be tuned. In particular, recent GPUNet networks Wang et al. (2022) set up SOTA Pareto frontier in terms of latency and inference accuracy for small networks. Their architecture scheme is perfect showcase the value of our proposed method. The networks created during these experiments are called PRUNET, and Fig- presents their structure. In this section, we delve into details of how found PRUNET family and analyze the search cost. We also show that our networks transfer well to COCO object detection task.
4.1 IMAGE CLASSIHICATION ON IMAGENET
4.1.1 FINDING PRUNET NETWORK FAMILY
Inspired by GPUNet-I architecture, we define our SuperNet, as described in section 3.1 and Table [ with an image resolution of 288x288, 2D convolution with kernel size 3, stride 2, and Swish Ramachandran et al.) (2017) activation function the prologue, and then define 6 stages followed by an epilogue. At each stage, we use the same type of building block (convolution. IRB, or Fused Inverted Residual Block denoted by Fused-IRB), activation function, and a number of channels as in G. Within these constraints in stages from 2 to 6, we define stochastic layers that consist of four mulli--locke (kernel size E [3,5), SE E (True, False 1), all with maximum expansion of 8 and granularity of 32 channels. SE stands for Squeeze and Excitation block, cf. Hu et al. (2018). For stage 1, we define only two choices kernel size (3,5). Each stage contains one additional layer
7

---

Under review as a conference paper at ICLR 2023
that can be skipped during the training compared GPUNet-1. Thanks to such a defined SuperNet, we can sample GPUNet-1; hence the final result is expected to be improved.
To generate the entire Pareto frontier, we experiment with 7 different values of {0.2, 0.6, 0.8, 1.0, 1.2, 2.0} Parameter a is defined in equation 2 and changes the trade-off between latency and accuracy. Since proper layer skipping is crucial for finding good network, and the last 130 epochs are relatively inexpensive due to the progressive pruning of the search space, we test 3 variants of layer skipping for each a. On the basis of preliminary experiments, we chose E {0.4, 0.55, }.85 and o= 1.1. I, For each a value, we decided which À value was the best based on the final loss from the SuperNet search. The architecture with the best loss was then trained from scratch. Sableccomparee the PRUNET results against other baselines. For all the considered networks of comparable accuracy, PRUNET has significantly lower latency. It is worth noting that it does not necessarily have lower number of parameters or FLOPs. In particular, comparing PRUNETS and GPUNets (both among distilled and non-distilled networks), we observe that the obtained networks are uniformly better, meaning we get higher accuracy with lower latency.
4.1.2 ABLATION STUDY
Each of the optimizations has a different impact on search performance. Main advantage of the first one (3.2.1) memory efficiency. Prunode allows significant increase of the size of the search space without increasing the demand for compute and memory resources. The second optimization (3.2.2) can save lot of computational time without compromising the quality of the results. Last optimization (3.2.3 saves memory that would be needed for skip connection in standard FBNet approach and enhances the quality of the end results by reducing the impact of curse of skip connect. Table 3 visualizes the cost impact of each optimization in hard memory limit scenario. Appendix B further investigates the impact on the quality of the results.
4.2
OBJECT DETECTION ON COCO
0.40
EfficientDET different backbones
PRU/Net
GPUNet
The experiments were conducted on the
0.39
EfficienNeet-/1
COCO 2017 detection datasetLin et al.
mAP. MS COCO 2017
0.38
2014 We chose EfficientDet [Tan et al. 2020 as baseline model. We replaced the original EfficientNet backbone with GPUNet and PRUNET for broad compar- ison. All backbones were pretrained with- out distillation. Figure 4) shows that our chitectures can be successfully transferred to other Computer Vision tasks. PRUNET turned out to be faster and more accurate, similarly as it was on the image classifica- tion task.
0.37
0.36
0.35
0.34
4.0
4,2
4.4
4.0
5.0
5.2
5.4
TensorRT FP16 bsm] Latency (mms)
Figure 4: PRUNET as a backbone outperforms GPUNet and EfficientNet on COCO object detection task on infer- ence latency relative to mean Average Precision (mAP)
5 CONCLUSIONS
We introduced Prunode stochastic bi-path building block that can be used to search the inner hidden dimension of blocks any differentiable NAS with (L) cost. Together with two novel layer pruning techniques, we show that our proposed inference-aware method establishes new SOTA Pareto frontier (PRUNET) TensorRT inference latency and ImageNet- top-1 accuracy and enables fine granularity sampling of the Pareto frontier to better fit external deployment constraints. Further, our results in Table 2 confirm that FLOP and the number of parameters are not the suitable proxies for latency, highlighting the importance of the correct metric for architectural evaluation. Although we only present results within computer vision tasks, our methods can be generalized to searching models for natural language processing (NLP), speech, and recommendation system tasks.
8

---

Under review as a conference paper at ICLR 2023
Table 2: PRUNET image classification results, All latency measurements are made using batch size 1.
PRUNET Without Distillation Comparison
Top1
TensoRT Latency
#Params
HE
PRUNET
PRUNET
Models
ImageNet
FP16 V100 (ms)
(10e6)
(10e9)
Speedup
Accuracy
EfficientNet BO Tan and Le (211))
77.1
0.94
5.28
0.38
1.96x
1.3
EfficientNetX- BoteeennNeX----GPP.... BBO-GP et 2021
77.3
0.96
7.6
0.91
2x
1.1
REGNETY-1.6GF
Radossvoeis
78
2.55
11.2
1.6
5.31x
0.4
PRUNET
78.4
0.48
11.3
2.10
OFA 389 Cai 2019)
0.94
8.4
0.39
1.68x
0.5
EfficientNet--I1-
79.4
1.37
1.58
2.45x
0.2
OFA 482
79.6
1.06
9.1
0,48
1.89x
0.0
PRUNET
79.6
0.56
14.8
2.57
FBNetV3-B Dai et al. 2020
79.8
1.5
8.6.
0.46
2.27x
0.9
EfficientNetX-B2-GPP
80.0
1.46
10
2.3
2.21x
0.7
OFA 595
80.0
1.09
1.65x
0.7
EfficientNeetBB B2
80.3
1.65
9.2
2.5x
0.4
ResNet-50 He et al 2016
80.3
1.1
28.09
4.
1.67x
0.4
GPUNet-1 Wang et al. 2022
80.5
0.69
12.7
3.3
1.04x
0.2
PRUNET -2
80.7
0.66
18.6
3.31
REGNETY-32GF
81
4.97
145
32.3
6.45x
0.2
PRUNET
81.2
0.77
20.8
4.05
EffiientNetX-B3-GGRR
81.4
1.9
13.3
4.3
2.09x
0.1
PRUNET
81.5
0.91
24.0
4.25
EfficientNet-B3
81.6
2.04
1.91x
0.3
PRUNET
81.9
1.07
27.5
5.29
ResNet-101 101
He et al. (2016)
82.0
1.7
45
1.33x
0.3
FBNeIV3-F
82.1
1.97
13.9
1.18
1.54x
0.2
GPUNet-2
Wang et al.
2022
82.2
1.57
25.8
8.38
1.23x
0.1
PRUNET
82.3
1.28
31.1
7.36
PRUNET With Distillation Comparison
PRUNET -0 (distilled)
80.0
0.48
11.3
2.10
G (distilled)
80.7
0.61
11.9
3,25
1.09x
0.3
PRUNET 1 (distilled)
81.0
0.56
14.8
2.57
G (distilled)
81.9
0.69
12.7
3.3
1.04x
0.3
PRUNET -2 (distilled)
82.2
0.66
18.5
3.31
PRUNET -3 (distilled)
82.6
0.77
20.8
4.05
PRENET -4 (distilled)
83.1
0.91
24.0
4.25
PRUNET -5 (distilled)
83.4
1.07
27.5
5.29
GPUNet-2 (distilled)
83.5
1.57
25.8
8,38
1.23x
0.5
EfficientNetV2-5 Tan and 2021
83.9
2.67
22
8.8
2.09x
0.1
PRUNET -6 (distilled)
84.0
1.28
31.1
7.36
Table 3: Cost analysis in the 80 GB memory limit scenario. Train time considers searching for all PRUNET architectures node with 8 NVIDIA Tesla A100 GPUs (DGX-A100) Section 3.1 contains the calculations of the search space sizes.
FBNet
1st opt (32.1
2nd opt (3...2
3rd opt (3.2.3
All opts
Memory consumption [GB]
80
80
80
80
80
Search space size
9c15
7.1e38
9e15
1.5c17
1.7e39
Train time [h]
140
140
62
140
62

---

Under review as a conference paper at ICLR 2023
REFERENCES
G. Bender, P.-J. Kindermans, Zoph, V. Vasudevan, and Q. Le. Understanding and simplifying onc-shot architecture search. International Conference on Machine Learning, pages 550-559. PMLR, 2018.
H. Cai, J. Yang. Zhang, Han, and Path-level network transformation efficient architecture search. In International Conference on Machine Learning, pages 678-687. PMLR, 2018a.
Cai, Zhu, and Han. Proxylessnas: Direct neural architecture search on target task hardware. arXiv preprint 10X2::8812.000 2018b.
H. Cai, C. Gan, T. Wang, 7hang, and Han. Once-for-all: Train one network and specialize efficient deployment. arXiv preprint arXiv: 1908.9971, 2019.
C. Chen, E. Tung, N. Vedula, and G. Mori. Constraint-award deep neural network compression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 900-415, 2018.
X. Chen, I., Xie, J. Wu, and Q. Tian, Progressive diffcrentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF international conference computer vision, pages 1294-1303, 2019.
Y. Ci, C. Lin, M. Sun, B. Chen, H. Zhang, and W. Ouyang. Evolving search space for neural architecture search. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6659-6669, 2021.
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q V. Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv:1805.000 095011 2018
X. Dai, P. Zhang, Wu, Yin, Sun, Wang, M. Dukhan, Hu, Y. Wu, Y. Jia, et al. Chamnet: Towards efficient network design through platform aware model adaptation. Proceedings of IEEE/CVF Conference on Compuder Vision and Puttern Recognition, pages 11398-11407, 2019.
Dai, Wan, P. Zhang, Wu, He, Wei, K. Chen, Tian, Yu, P. Vajda, al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. 2020.
Deng, W. Dong, R. Socher, L..J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image databasc. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10.1 21110//CVPR..00.555000
Y. Ding, Y. Wu, C. Huang, S. Tang, F. Wu, Y. Yang, W. Zhu, and Y. Zhuang, Nap: Neural architecture search with pruning. Neurocomputingg 477:85 95, 2022.
J.-D. Dong, A.-C. Cheng, D.C. Juan, W. Wei, and M. Sun. Dpp-net: Device-aware progressive search for pareto-optimal neural architectures. In Proceedings of the European Conference Computer Vision (ECCV), pages 517-531, 2018
Falkner, A. Klein, and F. Hutter. Bohh: Robust and efficient hyperparameter optimization scale. In International Conference on Machine Learning, pages 1437-1446. PMI.R, 2018.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Hinton, Vinyals, Dean, et al. Distilling knowledge in a neural network. arXiv preprint arXiv:1503.0253) 2015.
A. G. Howard, Zhu, Chen, Kalenichenko, W. Wang, Weyand, M. Andreetto, Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint (00e::770.440000 2017.
J. Howard. Imagewang.
J. Hu, L. Shen, and G. Sun. Squeeze-and excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7144
anneezz-and-exciiaaition
2018. 2018.
E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 2016.
H. Jin, Q. Song. and X. Hu. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pages 1946-1956, 2019.
10

---

Under review as a conference paper at ICLR 2023
D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6988 2014.
A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. es. toronto. edu/kriz/cffar html, 5(4):1, 2010.
G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint 1oXiv:665..076444 2016.
L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-Tzur, M. Hardt, B. Recht, and A. Talwalkar, A system for massively parallel hyperparameler tuning. Proceedings of Machine Learning and Systems, 2:230-246, 2020a.
Li, M. Tan, R. Pang, A. Li, I.. Cheng, Q. Le, N. Jouppi. Searching for fast model families on datacenter accelerators. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
X. Li. C. Lin, C. Li, M. Sun, W. Wu, J. Yan, and W. Ouyang. Improving one-shot nas suppressing the posterior fading. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 13836-13845, 2020b.
H. Liang. S. Zhang. J. Sun, X. He, W. Huang. K. Zhuang, and Z. Li. Darts+: Improved differentiable architecture search with carly stopping. arXiv preprint arXiv: 1eXiv:9909)
2019.
M. Lin, H. Chen. X. Sun, Q Qian, H. Li, and R. Jin. Neural architecture design for gpu-efficient networks. arXiv preprint 2rXiv:2006.14000 2020.
T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. European conference on computer vision, pages 740-755. Springer, 2014.
C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pages 19-3 19-34, 2018a.
H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv:1806.09000 2018b.
R. Luo, Tian, T. Qin, Chen, and T.-Y. Liu, Neural architecture optimization. Advances in neural information processing systems, 31, 2018.
J. Mci, Y. Li, X. Lian, X. Jin, L. Yang, A. Yuille, and J. Yang. Atomnas: Fine-grained end-to-end neural architecture search. arXiv preprint arXiv:1912.09644 2019.
Molchanov, J. Hall, H. Yin, J. Kautz, N. Fusi, and Vahdat. Hant: Hardware-aware network transformation. arXiv preprint arXiv:2107. 10624, 2021.
H. Pham, M. Guan, B. Zoph, Q Lc, and J. Dean. Efficient neural architecture search via parameters sharing, In International conference on machine learning, pages 0094-4104, PMLR, 2018.
1. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10428- 10436. 2020.
Ramachandran, B. Zoph, and Q Le. Searching for activation functions. arXiv preprint 170.v.1710.05941. 2017.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C, Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
M. Shen. Yin, Molchanov, Mao, Liu, and J. Alvarez. Halp: Hardware-aware latency pruning. arXiv preprint arXiv:21 10811, 2021.
M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Muchine Learning, pages 6105-6114, PMLR, 2019.
M. Tan and Q Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pages 10096-10106. PML.R. R. 2021.
11

---

Under review as a conference paper at ICLR 2023
M. Tan, B. Chen, R. Pang. V. Vasudevan, M. Sandler, A. Howard, and Q. Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820- 2828, 2019.
M. Tan. R. Pang, and Q. V. Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 10781-10790, 2020.
Vahdat, A. Mallya, M.-Y. Liu, and J. Kautz. Unas: Differentiable architecture search meets reinforcement learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11266-1113 11275, 2020.
A. Wan, X. Dai, P. Zhang, Hc, Tian, S. Xie, B. Wu, M. Yu, Xu, K. Chen, et al. Fbnetv2: Diffcrentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12965- 12974, 2020.
L. Wang, C. Yu, S. Salian, S. Kierat, S. Migacz. and A. F. Florea. Gpunet: Searching the deployable convolution neural networks for gpus. arXiv preprint arXiv:2205.0m 2022.
R. Wightman.
Pytorch image
models.
https://github com/ twightman/
bytorch- image- models 2019.
B. Wu, X. Dai, P. Zhang. Y. Wang. F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecturc search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10744 10742, 2019.
X. Xia, X. Xiao, X. Wang, and M. Zheng. Progressive automatic design of search space for one-shot neural architecture search. In Proceddings of the IEEF/CVF Winter Conference Applications of Computer Vision, pages 2455-2464, 2022.
Xiang. L. Dudziak, S. Abdelfattah, Chau, N. Lane, H. Wen. Zero-cost proxies meet differentiable architecture search. arviv preprint arXiv:2106.061 2021.
Xic, H. Zheng, C. Liu, and Lin. Snas: stochastic neural architccture search. arXiv preprint arXiv: 1812.0996,, 2018.
T.J. Yang, A. Howard, B. Chen, Zhang, A. Go. Sandler, and H. Adam. Netadapt: Platform aware neural network adaptation for mobile applications. Proceedings of the European Conference on Computer Vision (ECCV), pages 285-300, 2018.
K. Yu, R. Ranftl, and M. Salzmann. An analysis of super-net net heuristics in weight-sharing nas. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Zela, Elsken, Saikia, Y. Marrakehi, T. Brox, and F. Hutter. Understanding and rohustifying diffcrentiable architecture search. arviv preprint 19xx::9999009556, 2019.
Y. Zhao, L. Wang, Y. Tian, R. Fonseca, and T. Guo. Few-shot neural architecture search. In International Conference on Machine Learning, pages 12707-12718, PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTS DETAILS
A.1 MACHINE AND SOFTWARE SETUP
The experiments were performed nodes equipped with 8 NVIDIA Tesla A100 GPUs (DGX- A100). Python 3.6.8 and Pytorch Image Models [Wightman (0199) v0.4. 12 were used inside Pytorch//yttorch::
9.1-cudal1. 1--cc-nnn--rrre
docker container.
By inference latency we mean Median GPU Compute Time measured via trtexec command with FP16 precision and batch size of I using TensorRT version 8.0.1.6 a single NVIDIA V100 GPU with driver version 450.5 inside nver wttp:/iiia/eeeeoottt::ll. 08 -py3 docker container.
A.2 IMAGENET EXPERIMENTS
The experiments were conducted on the Imagenet- I Deng et al. (2009) image classification dataset. consists of 1.2 million training samples and 50 thousand validation samples, which span over 1000 classes. For all experiments we used weight decay of le-5 and an AutoAugment Cubuk al. (2018) with an augmentation magnitude of 9 and standard deviation 0.5 (corresponding to the probability of applying the operation) for both architecture search and training from scratch. The experiments were performed in automatic mixed precision (AMP).
A.2.1 ARCHITECTURE SEARCH DETAILS
An architecture search was performed on 10% of randomly selected classes from the original dataset. The input images were scaled to resolution of 288 x 288. The search lasted 200 epochs, with total batch size of 256 and a cosine learning rate scheduler that had an initial value of 0.1. We used the Adam Kingma and Ba (0014) optimizer for the architecture parameters and the RMSprop optimizer with an initial learning rate of 0.002 for the weights. We divided the search into two phases. In the first phase we train only the regular weights for the first Cwammup 70 epochs, and only do it once to save computational time. In the second phase, which lasts the remaining 130 epochs, in each epoch 80% of the training dataset was used train the regular weights and the remaining 20% was used to train the architecture weights. The second phase has been computed in many variants but always starts from a common checkpoint after ewarmup epochs. During this phase, we progressively prune the search space using pruning threshold (see Section 3.2.2) that increases linearly from 0.15 to 0.55. At the end of each epoch, we removed blocks below the threshold from the search space. The momentum used in the pruning internal structure of a Prunode was equal to 0.4 The coefficient a in the loss function varied across different runs with analyzed values {2.0, 1.2, 1.0, 0.8, 0.6, 0.4, 0.2). but all runs used the same coefficient value of 0.6 The latency term LAT of the loss function was measured in us.
A.2.2 TRAINING DETAILS
The hyperparameters' values were based on G Wang al. (2022). For fair comparison we decided to train GPUNet-O, GPUNet-1 and GPUNet-2 using exactly the same hyperparameters (including batch size) as we used for PRUNETS.
After the architecture search, the sampled network was again trained from scratch. The training lasted for 450 epochs with total batch size of 1536 and an initial learning rate of 0.06. The learning rate decays by 0.97 times for every 2.4 epochs. crop_ pct was set to 1.0. Exponential Moving Average (EMA) was used with a decay factor of 0.9999. We used the drop path Larsson et al.) (2016) with base drop path rate of 0.2. All of the PRUNET and GPUNet networks have been trained with and without distillation Hinton et al. (013)). Knowledge distillation is a technique that transfers the knowledge from large pre- mre-triiee model to smaller one which can be deployed under real-world limited constrains. For the training with distillation, we used different teachers and different crop_pct as it is presented in Table 4
13

---

Under review as a conference paper at ICLR 2023
Table For each image resolution different teacher was selected. Additionally crop_pct was changed to match the crop_pct of the teacher.
model resolution
teacher architecture
teacher resolution
crop pct.
GPUNetl & PRUNET
288 x 288
EfficientNet-BB B3
300 300
0.904
GPUNet0
320 x 320
EfficientNet-Be B4
0,922 380 x 380
GPUNet2
384 x 384
EfficientNet-B5
456 x 456
0.934
Table 5: Search space for B.1 experiment:
For stages 2-6 the search space is defined as the sum of RELU and Swish variants
Stage
Type
Stride
Layers
Activation
Kemel
Expansion
Filters
0
Conv
Swish
3
24
Conv
RELU
(3,5,7)
Fused-IRB
RELU
[3,5]
Swish
(3,5,7)
3
Fused-IRA
2
RELU
[3.5]
8
96
Swish
(3,5,7)
4
IRB
2
RELU
[3,5]
8
160
Swish
(3,5,7)
IRB
RELU
(3,5,7)
Swish
(3,5)
IRB
2
RELU
(3,5,7)
8
448
Swish
(3,5)
7
Conv+Pool+FC
RELU
1280
A.3
COCO EXPERIMENTS
The object detection experiments were conducted on the MS COCO 2017 dataset Lin et al. 20144 4). We used EfficientDer [Tan al. (2020) as baseline model and replaced the original Efficient Tan and Le (2019) backbone with PRUNET and GPUNet. The training lasted for 300 epochs with batch size 60. The learning rate was warmed-up for the first 20 epochs with the value set to le-4. Then, used with initial learning rate of 0.65. The optimizer was SGD with momentum of 0.9 and weight decay of 4e-5. Gradient clipping of value 10.0 was introduced. The training performed in automatic mixed precision (AMP). We used the Exponential Moving Average (EMA) with decay factor of (1.99.
the cosine learning rate scheduler
B ADDITIONAL EXPERIMENTS
B.1 PRUNING BLOCKS WITHIN A STOCHASTIC LAYER
We can have two different versions of setting the pruning threshold, i.c. constant or linearly increasing, The hypothesis was that they lead to the networks of comparable Pareto frontiers, but the search is significantly faster for the linearly increasing threshold. The experiments supported this hypothesis, and the result Pareto frontiers are on par; however, the processing time for the constant threshold was around 15% 45% slower.
We used the Imagewoof Howard the obtained architectures. The search space is described Table
Hoaggroof image classification dataset both for the search and the training of
The search was performed with batch size 256, cosine learning rate scheduler with initial value of 0.1. lasted 200 epochs and for the first ewarmup 70 epochs only the regular weights were trained. The image resolution was scaled to 224 224 and crop_pct was set to 0.875. Adam optimizer was used for architecture parameters and RMSprop optimizer with initial learning rate 0.002 for the regular weights. A weight decay of le-5 was used and AutoAugment with augmentation magniture of 9 and standard deviation of 0.5. The search was performed in automatic mixed precision (AMP). In each epoch 80% of the training dataset used to train the regular weights and the reamaining
14

---

Under review as a conference paper at ICLR 2023
J.0
constant
...2
86.3
0.8
0.4
0.
linearly increasing
30-
.2
86.0
0.6
Top-1, ImageWoof
Search time (hours)
,
85.5
J.2
1.2
3.0
2.6
0.6
,300
84.5
2.
2.4-
84.0
D.4
2.6
constant linearly increasing
J.0
0.8
0.2
3.0
2.0
J.2
300
420
5C0
600
700
300
400
500
600
700
TensorRT FP18 bs-1 Late ney
TensorRT bs-1 Latency
Pareto frontiers
(b) Search time
Figure 5: Linearly increasing pruning threshold leads to a significant search phase speedup while producing similar Pareto frontier to a constant threshold method. Labels indicate the value of being used.
20% was used to train the architecture weights. For a constant pruning threshold policy, we use value of O. 133; for the second setup, we linearly increase the threshold from 0.133 to 0.55 (see Section 3.2.2). The coefficient a the loss function varied across different runs with analyzed values of (0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 2.0, 3.03, but all runs used the same coefficient 3 value of 0.6.
After the search has finished, the architectures have been re-trained from scratch for 450 epochs in automatic mixed precision (AMP) using Stochastic Gradient Descent (SGD) with batch size of 32, and initial learning rate of 0.2. The learning rate decays by 0.97 for every 2.4 epochs. The drop path with bsae drop rate of 0.2 was used. The results of those experiments are presented in Figure
B.2 PRUNING UNNECESSARY STOCHASTIC LAYERS
Proper selection of parameters & and À (which multiply the output of skip connection and the output of block; for more details, see Section 3.2.3) can influence the final length the sampled network. In particular Table Table shows that for large enough, the number of layers is inversely related to À for considered cases. To further analyze the impact of our methodology. we run additional searches with (d = (1.0, 1.0) these values correspond to base approach without Table i| is clear that for a 2 0.8, the base approach samples networks with fewer layers than the searches we performed. To prove that our approach justified, we train all 28 of the architectures found during the searches described above, and we present Pareto frontiers defined by all four sets of and X. Figure 6 shows that our search technique can find networks with up to 15% better latency with the same accuracy as the base approach. Our methodology uses the final loss of the search as zero-cost filter select good candidate architectures be evaluated. Figure 1oaure7preee presents the Pareto frontier of Search Loss concerning Final Latency. Looking at both figures, it is directly visible that if search loss is significantly better for similar searches (in this case for similar a values), we can also expect better results in terms of the final accuracy; however, the correlation is not strong. Therefore, without additional compute overhead, our methodology can find almost optimal networks, possibly still leaving room for improvement.
modification. From
15

---

Under review as a conference paper at ICLR 2023
81.50
81.25
81.00
Top-1, ImageNet
80.75
80.50
80.25
best Loss (PruNet)
80.00
1.0
1.0
q 1.1
0.85
79.75
1.1
= 0.55
1.1
79.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
TensorRT FP16 bs=1 Latency (ms)
Figure Graph focuses the architectures being searched using 0.6 < 1.2
0.65
best Loss (PruNet)
1.0
À 1.0
0.60
9 1.1
0.85
1.1
À 0.55
q 1.1
À 0.4
Final CE
0.55
0.50
0.45
0.40
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
TensorRT FP16 b Latency (ms)
Figure 7: Graph focuses on the architectures being searched using 0.6 < < 1.2, which shows the relationship between the final CE of the search and the final latency of the architecture
16

---

Under review as a conference paper at ICLR 2023
Table 6: Search results for different & and X.
For each combination of (a,, Ø, A) we run a single search and for each search the total loss is reported. The total loss sum of cross entropy loss and latency loss (Loss CE + a * LATS). #IRBs indicates the number of Fused-IRB and IRB in the final architecture. The number of the other layers is the same for all architectures. PRUNET architectures are shown in bold
o, À (1.0, L.0)
Φ 0185,
o, (1.1, 0.55)
Φ, = (1.1, 0.4)
#IRBs
Loss
#IRBs
Loss
#IRBs
Loss
#IRBs
Q 2.0
6.7149
6.6316
6.5968
6.6241
11
Q 1.2
4.2938
4.1876
10
4.2139
12
4.2039
13
1.0
3,6352
11
3.6015
13
3.6074
13
3.6138
14
0.8
3.0027
13
2.9932
14
2.9762
14
2.9692
15
0.6
2.3528
15
2.3394
15
2.316
15
2.3723
15
a 0.4
1.7154
1.7201
17
1.7296
1.6946
16
a 0.2
1,0573
18
1.0573
18
1.0687
18
1.0531
18
17


Reviews:
[{"review_id": "pf8_-rPZiH", "reviewer": "Reviewer_qypu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper expands the current supernet training by introducing 3 components:\n\n1. Prunenode: although the name is a bit fancy, the concept is straightforward: it learns the expansion ratio of inverted residual blocks.\n2. Pruning blocks with stochastic layers: it kicks out the low likelihood blocks from the search space.\n3. Pruning unnecessary stochastic layers: this tries to search over the number of supernet layers that skips certain layers.\n", "summary_of_the_review": "1. novelty: This is my main concern: a. The three claimed new components: i) searching for extension layers, ii) searching for blocks in an IRB, iii) search for the number of layers in a network are all previously discussed in Once-For-All[1].\n\n2. The curation of search space borrows from GPUNet. It will help a lot if the authors can clarify the contribution of this paper to the scientific community.\n\n[1] Cai, Han, et al. \"Once-for-all: Train one network and specialize it for efficient deployment.\" arXiv preprint arXiv:1908.09791 (2019).\n\n3. What's the performance if without starting from a good network?\n", "strength_and_weaknesses": "Strength:\n\n1. the idea in this paper is pretty straightforward\n2. the results look promising but not necessarily a solid improvement.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DC7_Flz7Vjq", "reviewer": "Reviewer_Jf9e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-objective neural architecture search that optimizer for both accuracy and latency. They use GPU-Net1 as the skeleton and search for kernel size and the expansion ratio of the intermediate Fused-IRB and IRB layers. They use masked differentiable architecture search, where the output of a layer is the weighted sum of all the outputs of the blocks. The weights of these blocks are computed using gumbel-softmax of the architecture weights of all the blocks in layer i.\n   The input and output dimensions of all the blocks in a layer are the equal. To search for the expansion ratio, rather than have 1 block for each possible configuration, they use masks similar to FBNetV2. To this end, they create two copies of a block and apply small mask on one and a large mask on another. The small mask masks half of the channels to begin with and the larger mask does not mask any of the channels.  As the search progress, the small mask and the larger mask are updated such that the distance between them reduces and finally becomes close to 0. If the architecture weights of the block with small mask is higher, then both the masks shrink. Similarly, if the architecture weights of the block with the larger mask is higher, then both the masks are increased. Finally when the small mask and the larger mask are less than granularity away, the final candidate has channels between the smaller and the larger one.\n   During the search, they also prune the blocks if their weights are lesser than the threshold. As is the case with other pruning based NAS methods, the networks weights are trained for the first X epochs. After that the bilevel optimization of network weights and architecture weights begins. The threshold keeps increasing as the search progresses.", "summary_of_the_review": "The algorithm1 is not clearly written. It is not evident why this is better than the masking technique in FBNet V2.", "strength_and_weaknesses": "Strength:\n1. They are able to find networks with higher accuracy and lower latency than all their baselines for both image classification and object detection.\n2. They use a novel search space - GPUNet which is better suited for building architectures with high accuracy and low latency.\n\nWeakness/ Questions:\n 1. Are we using masking for searching for kernels too?\n 2. The algorithm is not very straightforward to understand. \n3. How is this masking algorithm better than the masking technique used in FBNet V2? If we use the masking technique of FBNet V2 with the same search space and setup as this paper, how would it fare? FBNet V2 requires only 1 block per layer while searching for channels. So it seems to be occupying lesser memory.\n\nQuestions about algorithm 1\n1. What does progress variable denote? Is it the ratio of number of epochs done to number of epochs left?\n2. Why are we setting the architecture weights to 0 if s > 0? \n3. f weights are always non-negative, then update variable will also be non-negative. When will s become negative?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7SGDpBXqu8", "reviewer": "Reviewer_Xpgv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a differentiable NAS approach that searches for efficient CNN models on GPUs. The key idea is similar to existing differentiable NAS methods: modeling NAS as a bi-level optimization problem where both weight and architectural parameters are updated during SuperNet training. The paper proposes to dynamically shrink the search space during the procedure of super network training. After super network training, operators with the largest architectural parameter is kept for each searchable block, which is also the standard procedure in differentiable NAS.", "summary_of_the_review": "Despite good results, this paper fails to make fair comparisons with existing methods and has very limited novelty. Without a major reformat, I cannot see the possibility of recommending acceptance.", "strength_and_weaknesses": "Strengths:\n- The authors carry out very extensive experiments on ImageNet and COCO, achieving impressive accuracy results.\n- The proposed method itself is easy to understand.\n\nWeaknesses:\n- The writing of this paper needs to be improved. It seems to me that this paper is a tech report rather than a research paper. For citations in the paper, please make sure that the whole sentence can read. For example,\n\n> recent research has focused on partial training Falkner et al. (2018); Li et al. (2020a); Luo et al. (2018), performing network morphism ...\n\nin p1 does not seem to be a readable sentence. The authors should also consider to replace $*$ with $\\times$ in equation in the end of page 3.\n\n- Despite good results, the method itself has very limited novelty. Modeling the NAS problem as bi-level optimization, using Gumble softmax to convert non-differentiable HPO problems to differentiable optimization problems, and adding a latency term in the loss function are all very well-recognized techniques in the NAS community. The design space used in this paper is also highly inspired by EfficientNetV2. One can also easily find the idea of gradually shrinking the NAS design space in [AtomNAS](https://arxiv.org/pdf/1912.09640.pdf) of ICLR'20. \n- The comparison in Figure 4 is not fair. The method is clearly inspired by EfficientNetV2, but it compares itself with an apparently weaker baseline, EfficientDet (backed by EfficientNetV1). \n- The authors never made a head-to-head comparison against one-shot NAS (e.g. Single Path One Shot NAS in ECCV'20) or other differentiable NAS methods in the same search space, which makes it very hard for me to judge the merit of this paper against existing approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6vCX5pdZYcM", "reviewer": "Reviewer_GWQr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a new approach with the purpose of improving the efficiency of differentiable neural architecture search (DNAS). The authors introduce Prunode, a stochastic bi-path building block that enjoys O(1) memory and computation complexity during the search. Given this advantage, Prunode allows a much larger search space than conventional NAS algorithms. To further reduce the searching computation, the authors also propose to prune the blocks within the stochastic layer that are less likely to be chosen, as well as unnecessary layers. Experimental results show that PRUNET establishes the new state-of-the-art Pareto frontier w.r.t latency and accuracy on ImageNet.", "summary_of_the_review": "The paper studies the important problem of efficient neural architecture search, and experimental results are thorough and solid. I can see the efforts from authors to improve searching efficiency by various practical designs. Yet, my major concern is that too many heuristics involved in the proposed approach that make it hard to implement in practice. The presentation can also be improved.", "strength_and_weaknesses": "Strengths:\n\n- The authors propose to prune the search space for efficient neural architecture search. The pruning is conducted on different network angularities: inner block level, inter block level and layer level. \n\n- Experimental results show that PRUNET achieves new SOTA w.r.t the inference latency and classification accuracy on ImageNet. \n\n- The authors also provide thorough ablation studies over the hyperparameters of the algorithm.\n\n\nWeakness:\n\n- A major concern with this paper is the heuristics that make the procedure overcomplicated. There are also a number of associated hyper-parameters, e.g., constant $c$, max_distance, momentum in Section 3.2.1, $e_{\\textrm{warmup}}$, $e_{\\textrm{warmup}}$, $t_{\\textrm{initial}}$ and $t_{\\textrm{final}}$ in Section 3.2.2, and $\\lambda$, $\\phi$ in Section 3.2.3. Although authors provide some ablations, these hyperparameters make the approach hard to tune in practice.\n\n- Algo 1 is hard to follow. More explanations can be provided.  It is also not clear to me in what way can the task loss affect the mask training, since their updates are based on heuristic rules.\n\n- It seems not new to have O(1) memory and computation complexity. It is common for popular NAS methods (e.g., ProxylessNAS [1]) to sample one path from the cell that achieves the same complexity.\n\nDetailed comments:\n\n- How do you construct the latency table? Are the latency values measured based on TensorRT conversion? Besides, for baselines such as EfficientNet, do you apply the same TensorRT conversion before measuring their latencies?\n\n- It seems the algorithm cannot regularize the model size explicitly. Thus picking different sizes of PRUNET architectures from Table 6 can be tricky.\n\n- \"small parameter values mean a negative impact on accuracy but a positive impact on latency, and large parameter values mean a positive impact on accuracy but negative impact on latency\": what do you refer by \"small parameter value\"? Is it the architecture weight or the network parameter?\n\n\n[1] Cai H, Zhu L, Han S. Proxylessnas: Direct neural architecture search on target task and hardware[J]. arXiv preprint arXiv:1812.00332, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject