Uniform-Precision Neural Network Quantization via Neural Channel Expansion
Authors: Seongmin Park
Beomseok Kwon
Kyuyoung Sim
Jieun Lim
Tae-Ho Kim
Jungwook Choi
Venue:  (2021)
Keywords: deep neural network
quantization
neural architecture search
image classification
reduced precision
inference

Abstract: Uniform-precision neural network quantization has gained popularity thanks to its simple arithmetic unit densely packed for high computing capability. However, it ignores heterogeneous sensitivity to the impact of quantization across the layers, resulting in sub-optimal inference accuracy. This work proposes a novel approach to adjust the network structure to alleviate the impact of uniform-precision quantization. The proposed neural architecture search selectively expands channels for the quantization sensitive layers while satisfying hardware constraints (e.g., FLOPs). We provide substantial insights and empirical evidence that the proposed search method called neural channel expansion can adapt several popular networks' channels to achieve superior 2-bit quantization accuracy on CIFAR10 and ImageNet. In particular, we demonstrate the best-to-date Top-1/Top-5 accuracy for 2-bit ResNet50 with smaller FLOPs and the parameter size.

Under review as a conference paper at ICLR 2021
UNIFORM-PRECISION NEURAL NETWORK QUANTIZA- TION VIA NEURAL CHANNEL EXPANSION
Anonymous authors Paper under double-blind review
ABSTRACT
cnifform-precision neural network quantization has gained popularity thanks to its simple arithmetic unit densely packed for high computing capability. However, it ignores heterogeneous sensitivity the impact of quantization across the layers, resulting in sub-optimal inference accuracy. This work proposes a novel approach to adjust the network structure to alleviate the impact of uniform -erfooonpreeiioonn quan- tization. The proposed neural architecture search selectively expands channels for the quantization sensitive layers while satisfying hardware constraints (e.g., FLOPss. We provide substantial insights and empirical evidence that the proposed search method called neural channel expansion can adapt several popular networks' channels to achieve superior 2-bit quantization accuracy on CIFARIO and Ima- geNet. In particular, we demonstrate the best-to-date 1op-1/Top-5 accuracy for 2-bit ResNet50 with smaller FLOPs and the parameter size.
INTRODUCTION
Deep neural networks (DNNs) have reached human-level performance in a wide range of domains including image processing (He al. (2016); Tan & Le (2019)). object detection (Ren et al. (2015);
Liu et al. (2166;; Tan et al. (020)), ), machine translation Wu et al. 2016); Devlin et al. (2018)) and speech recognition (Zhang al. 2016): Nassif al. 2019)). However, tremendous computation and memory costs of these state-of thee- art DNNs make them challenging to deploy on resource- constrained devices such as mobile phones, edge sensors, and drones. Therefore, several edge hardware accelerators specifically optimized for intensive DNN computation have emerged, including Google's edge (Poolo 2019)) and NVIDIA's NVDLA (NVIDIA[(01))) (2019)).
One of the central techniques innovating these edge DNN accelerators is the quantization of deep neural networks (QDNN). QDNN reduces the complexity of DNN computation by quantizing network weights and activations to low-bit precision. Since the area and energy consumption of the multiply- accumulate (MAC) unit can be significantly reduced with the bit-width reduction (Sze et al. (2017)). thousands of them can be packed in small area. Therefore, the popular edge DNN accelerators are equipped with densely integrated MAC arrays to boost their performance in compute-intensive operations such matrix multiplication (MatMul) and convolution (Conv).
Early studies of QDNN focused on the quantization of weights and activations of MatMul and Cony the same bit-width (Hubara et al. (2016): Rastegari et al. (2016); Zhou al. (2066). This uniform- precision QDNN gained popularity because it simplifies the dense MAC array design for edge DNN accelerators. However, uniform bit allocation did not account for the properties of individual layers in network. Sakr & Shanbhag (2018) showed that the optimal bit bit-precision varies within neural network from layer to layer. a result, uniform precisson quantization may lead sub-optimal inference accuracy for given network.
Mixed-precision networks address this limitation by optimizing bit-widths at each layer. In this approach, the sensitivity of the layer to the quantization error is either numerically estimated (Zhou al. 2017)y Dong et al. (2019)) or automatically explored under the framework of neural architecture search (NAS, Wang et al. ) Elthakeb et al. (2018)) to allocate bit-precision properly. However, mixed- preedisieiiiion representation requires specific variable precision support in hardware, restricting computation units' density and power efficiency (Camus al. (2019)). Therefore, mixed-precision support imposes a significant barrier for the low-profile edge accelerators with stringent hardware constraints.

---

Under review as a conference paper at ICLR 2021
In this work, we propose a novel NAS based serdwar--frienlly DNN quantization method that can address the layer-wise heterogeneous sensitivity under iniform-precisiinn quantization. The proposed method explores network structure in terms of the number of channels. Different from the previous work that only includes pruning of the channels in its search space (Dong & Yang (2019). ). we further incorporate the expansion of the channels, thus called neural channel expansion (NCE). During search of NCE, search parameters associated with different numbers of channels are updated based on each layer's sensitivity the uniform- peroosioroeiioonn quantization and the hardware constraints; the more sensitive to quantization errors, the larger number of channels preferred in that layer. When the preference the larger number of channels in a layer exceeds a certain threshold, we expand the channels in that layer's search space so that the more number of channels can be explored. Therefore, NCE allows both pruning and expansion of each layer's channels, finding the sweet-spot for the trade-off between the robustness against the quantization error and the hardware cost. We analytically and empirically demonstrate that NCE can adequately facilitate the search to adapt the target model's structure for better quantization accuracy. The experimental results on CIFAR10 and ImageNet show that the network structures adapted from the popular convolutional neural networks (CNNs) achieve superior accuracy when the challenging 2-bit quantization is uniformly applied to MatMul and Conv layers. In particular, we achieve the best-to-date accuracy of 14403/91.6339 (Top-1/Top-5) for NCE-ResNet50 on ImageNet with slightly lower FLOPs and 30% reduced number of parameters.
Our contributions can be summarized as follows:
We propose a new NAS-based quantization algorithm called neural channel expansion (NCE), which is equipped with simple yet innovative channel expansion mechanism to balance the number of channels across the layers under uniform-precision quantization.
We provide an in-depth analysis of NCE, shedding light on understanding the impact of channel expansion for compensation of quantization errors.
We demonstrate that the proposed method can adapt the structure of target neural networks to significantly improve the quantization accuracy.
RELATED WORK
Neural architecture search: The goal of NAS is to find network architecture that can achieve the best test accuracy. Early studies (Zoph & Le (2016); Zoph et (2018)) often employed meta-learners such as reinforcement learning (RL) agents learn the policy for accurate network architectures. However, RL-based approaches may incur prohibitive search costs (c.g., thousands of GPU hours). As a relaxation, differentiable neural architecture search (DNAS) has been proposed (Liu et al. (218)), ) which updates the search parameters and the weights via bi-level optimization. Recent DNAS approaches considered hardware constraints such as latency, the number of parameters, and FLOPs so that the search explored the trade-off between the cross-entropy loss and the hardware constraint loss. This search resulted in the discovery of light-weight models. As an example, Dong & Yang (2019) explored channel pruning that satisfies the target hardware constraints. In this work, adopt the successful NAS framework in the domain of QDNN, for which we devise a novel channel expansion search robustify networks against the quantization errors.
Low-precision quantization of deep neural network: QDNN has been actively studied in the
literature. Early work on QDNN (Hubara et al. (216)); Rastegari al. (2016): Zhou et al. (2016) introduced the concept of a straight-through estimator (STE) for the approximation of gradients of the non-differentiable rounding operation. This approximation enabled mmiform-precisiio multi-bit) quantization during the model training procedure, which fine-tunes the weight parameters towards lower training loss. QDNN techniques have evolved to adaptively find the quantization step size (Choi et al. al. (2018) Zhhag et al. (2018); Jung et al. (2019); [ssser et al. (2020). which significantly enhanced the accuracy of the uniform precision quantization. However, this line of research lacks consideration of the heterogeneous quantization sensitivity for individual layers in a network. On the other hand, mixed -peeciiireiiion quantization allows layer-specific bit-precision optimization; the higher bit-precision is assigned to the more quantization sensitive layers. Zhou et al. (2017); Dong et al. (2019) numerically estimated the sensitivity via approximating the impact of quantization errors on model prediction accuracy. Wang et al.) (2019); Elthakeb et al. (2018) employed reinforcement learning framework to learn the bit-allocation policy. Wu et al. 2018); Cai & Vasconcelos (2020) adopted DNAS with the various bit brecpiiin operators in the search space. However, mixed-precision
(I- or
2

---

Under review as a conference paper at ICLR 2021
representation requires specific variable precision support in hardware, restricting computation units' density and power efficiency (Camus et al. (2019)). Therefore, mixed-precision support imposes a significant barrier for the low-profile edge accelerators with stringent hardware constraints. In this work, we exploit NAS to address the layer-wise heterogeneous sensitivity under uniform-precision quantization.
Channel expansion for accurate DNN quantization: Researchers have actively studied channel
expansion for accurate DNN quantization. Pioneering work by Mishra et al. (2018) (WRPN) demonstrated that an increased number of channels during the training helped regain QDNN accuracy, but this work lack discussion about the detailed mechanism of channel expansion compensating the
quantization error. IZhao et al. (2019) and Park & Choi (2019) further attempted to split the channels with large magnitude weights in the pre-trained models. This channel splitting reduced the dynamic range of weights to be represented with lower bit- bit-precision (6-8-bits). Regarding the control over the dynamic range, ( et al.) (2019) also adjusted the scale factors of the weight parameters after training to balance the dynamic range across the layers. However, these approaches focused on the numerical remedy for quantization of pre-trained models (with relatively high bit-precision). Thus, it not straightforward to extend their work for quantization-awee aware training, which is necessary for ultra-low bit QDNN. We provide insights with empirical supports that the structure with the channel expanded layers itself matters, reducing the dynamic range of activation during the quantization-awar training. These exciting insights become a pivotal motivation for us to explore channel expansion in the NAS framework.
NEURAL CHANNEL EXPANSION
In this section, we explain the detail of our neural channel expansion method. Similar to TAS (Dong & Yang (.019), we construct the search space over the number of channels C = {1 Cout with the search parameters a E RICC, Then the output activation is computed as the weighted sum of sampled activations with a different number of channels aligned via channel-wise interpolation (CWI):
Ô = Σ Softmar(ay; (oulee)) x CWT(Onc,, JEI
(1)
where output activation Oj:SsSSoon input activation X and weight parameters W quantized by the quantizer Q, and is the sampled subset of C.
Een ((x(k,;;:)) Q(W(i,k,;;:))) is computed with
During the search, the search parameters are updated via channel selection based on the trade-off between the cross-entropy loss and the hardware constraint loss (c.g.. FLOPs). In TAS, the number of channels (CC)) is fixed, limiting the exploration scope to the pruning. In NCE, we enable channel expansion of individual layers when the search parameter associated with the maximum number of channels exceeds a certain threshold. The intuition is that if one layer is susceptible to the quantization errors, its search parameters are updated toward the preference for a larger number of channels to decrease the cross-entropy loss. With this simple expansion condition, we can expand channels to those layers affected most by the quantization errors and prune channels of the other layers robust to quantization; therefore, the overall hardware constraints are met.
Algorithm| [ summarizes the overall procedure. NCE consists of three phases: warm-up, search, and train. As ad advocated by Wu et al. (018)) and [ender et al.) (2020), we first perform a warm-up of the entire super-net so that all the super-net weight parameters can be reasonably initialized. The search phase consists of the iterative updates of weights (w) and the search parameters (a) via bi-level optimization. The updated search parameter associated with the maximum number of channels is compared with threshold (pre-determined as expansion in each layer. When a channel expansion happens (= Expand). the additional weight parameters are added to that layer (and the search parameter is also copied), increasing the number of channels. Once the search is done, the candidate model is derived by the "winner-tai "wimner-takes-all" strategy; i.e,, for each layer, the number of channels with the largest magnitude search parameter's is selected.
to identify if a layer needs channel
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Neural Channel Expansion
Input:
Split the training set into two dis-joint sets: D Deeight and D Darch (n D weight)) = n D Darch))
Search Parameter: {ai al,, o E A', (A A A C A, L number of layer
Expand Threshold: T
For Warm-up Epoch do
Sample batch data D., from Dweight and network from A U(0
Calculate LosSweight on Dw to update network weights
End For
For Search Epoch do
Sample batch data Dw from weiight and network from toftmar Dw to update network weights Sample batch data Da from Darch and network from Softmar (A)
Calculate LosSweiggt
Calculate Lossarch on Da to update A. For layer do
10
+- #
If Softmax( (afE)E))) 2 T do Expand search space(a))11
13
14
a'+1 c aa
# copy search parameter
15
End if
16
End for
End for
18 Derive the searched network from A
Randomly initialize the searched network and optimize it on the training set
ANALYSIS
This section explains how NCE finds the network structures that are more robust to the uniform- precision quantization eтror while maintaining hardware constraints. We first reveal insightful observations that the expanded channel's structure reduces the input activation's 's dynamic range and suppresses quantization errors. In other words, a network structure adapted by NCE can be trained standalone from scratch and exhibit robustness to the quantization. This finding motivates to seek a NAS based exploration for channel expansion; we show that NCE can facilitate compensation of the quantization errors by selective channel expansion.
4.1 IMPACT OF CHANNEL EXPANSION ON DYNAMIC RANGE OF ACTIVATION
As discussed in See it is well studied that channel-splitinne can decrease the dynamic range of the weight parameters. However, it is not clear how much its structure itself affects quantization during neural network training. To understand the impact of expanded channel structure on DNN quantization, first show that quantization applied to given network substantially increases the dynamic range of activation, hindering successful QDNN. Fig Th shows the standard-deviation (STDEV) of input activation for ResNe120 trained from scratch the CIFAR10 1 dataset, with and without quantization during tr W(X)A(Y) indicates that weights and activations are quantized into X- and Y-bits, respectively. The substantial increase of STDEV for W32A2 and W2A2 implies that large quantization eπrоrs would occur when input activation is quantized. (See Appendix for detail discussion on the quantization error.) Such an increase of dynamic range could partially explain why quantization with the fixed model structure often suffers significant accuracy degradation when ultra-low bit- bit-precision is uniformly applied. This intriguing phenomenon can be observed in all the models we investigated (e.g., VGG16 in Fig [D).
In addition, Fig. Dp and e show the impact of expanded channel structure on the dynamic range. "2X" models indicate that the number of channels in their layers is doubled. All the hyper- parameerr
Thii quantization-awanl aware training follows the hyper-parameter settings described in See 5.1
4

---

Under review as a conference paper at ICLR 2021
Ruulict20 #266
25. Ray N6.20 VI3A
VOGUS WGA2
10-
Resive20 W12A2
Reslln22 WZAA2
1.-
24 Vet20 WGAR in8
1e:20 W22A32
2X VIGLE W222
VOZ2.4 WODAS2
Medel
FLOPs
Acc(drop)
ResNee20
(W32A32)
40.31M4
0.21M
93.04%
W
06-
0.20
6.33
ResNee20
(W2A2)
40.31M
0.77M
(-2.22%)
90 82%
0.-
0.4
0.62-
(W32A32)
BexNer20
162.5784
1.06M
94.79%
8.05
2X ResNer00 (W2A2)
162.57M
1.00M
(-0.94%)
93.95%
0.0
0.00
Layer
Lager
(6)
(b)
(c)
(d)
Figure I: (a-b) Layer-wise STDEV for ResNet20 with IX 2X channels. (c) Layer-wise STDEV for VGG16 with IX or 2X channels. (d) Test accuracy on ResNet20 with 1X or 2X channels (accuracy drop from W32A32 to W2A2).
settings are the same. Notice that both 2X ResNet20 and VGG16 models with 2-bit quantization (= W2A2) could reduce the STDEV down to the full-precision 1X models.
One might regard this dynamic range reduction due weight initialization with 2X channels, based on the idea similar to channel splitting discussed in See 2 However, we examined that the initial weight parameters have little impact on the input activation's dynamic range. More specifically, since we used "He" initialization (He et al. 2015 the number of channels determines the dynamic range of the initial weights. As shown Fig a, however, 2X ResNet20 with W2A2 quantization still experiences reduced STDEV even if its weights are initialized in the same way the 1X ResNet model initialized.
Finally, we confirm that the input activation's 's reduced dynamic range results in improved test accuracy. As shown in Fig Dp. IX ResNet20 model suffers large accuracy degradation from quantization (- 2.22%) while 2X ResNet20 model experiences relatively small accuracy degradation (-0.84%) Note that this result is consistent with WRPN (Mishra et al. (0188). However, we argue that such increased robustness against quantization errors is due to the mechanism that the expanded channel structure reduces the activation's 's dynamic range.
From these experimental results, we conjecture that expanded channel structure plays crucial role in compensating quantization errors. This finding suggests that if we employ channel expansion in the NAS framework, we can adapt the channels of the layers to find new network structure that is more robust to uniform-precision quantization when trained from scratch. In the next section, we investigate proper way of incorporating channel expansion.
4.2 IMPACT OF CHANNEL EXPANSION ON DISTINCT CHANNEL SEARCH
The previous section provides critical reasoning behind the success of channel expansion of QDNN like WRPN However, it is not practical to expand the channels of all the layers, since it will quadratically increase the computational complexity. The main idea of NCE is to allow channel expansion only when the selection of more number of channels is desirable for compensating quantization errors; otherwise, the channels are pruned meet the overall hardware constraints. In this section, we show that the channel expansion mechanism of NCE propels such compensation.
We first show that the channel selection preference can be observed by the gradient w.r.t. the search parameters. As explained in Sec. 3 the critical trade-off explored during the search is between the cross-entropy loss and the hardware constraint loss. In particular, a layer that is sensitive to the cross-entropy may select a large number of channels. In other words, the search parameters associated with a large number of channels, like ar7 or agg, receive large negative gradients Fl As an example, Fig. 2 shows the gradients of the search parameters during the TAS search of CIFAR 10- ResNet20 in full-precision. Note that the search parameter associated with the maximum number of channels (ag) initially receives the negative gradients. In contrast, the search parameter with the least number of
PI.c., the weights are initialized with the normal distribution with standard deviation inversely proportional to the square root of the number of channels.
LLe., 77 * grada- Thus the negative gradicnts increase the magnitude of the search parameter.
5

---

Under review as a conference paper at ICLR 2021
1--4
0.0
0.4
-
Kew
-6-
0.0-
NCE WZAZ
TAS, W242
TAS W32A32
100
200
300
400
500
600
300
200
300
400 500
600
11
17
10
Fprch
Fpoch
Layer
Figure 2: Experiments CIFARIO-ResNede that shows gradients of cross-entropy loss w.r.. search parameters (01 ~ (g) of a layer during search: (a) in full-precision, (b) with 2-bit quantization. (c) Kendall rank-correlation score of all layers.
channels (01) receives positive gradients. From this trend, we can conjecture that this layer initially prefers a large number of channels, but the preference diminishes over the epochs of search.
Next, we show that quantization during search excels the preference for a large number of channels. From the same experimental settings, if we apply quantization during the search, the preference to large number of channels becomes more distinctive, as shown in Fig. 2p. This phenomenon because the quantization errors affect the cross-entropy loss more. To quantify this channel preference, we calculate the Kendall rank correlation score for the gradients the search parameters and average over the search epochs; the more consistent preference to a large number of channels, the higher the Kendall score. Fig. P shows the layer-wise Kendall score with and without quantization. Note that the Kendall score increased if the quantization is applied during the architecture search. This increased Kendall score implies that quantization drives the search parameters toward a strong preference larger number of channels.
As motivated by this phenomenon, we augment TAS's search space to allow channel expansion. Thanks this new search space, channel expansion can happen selectively for the layer with a strong preference for many channels. Interestingly, the new search space gains the higher Kendall score, as shown in Fig. 2 indicating that the search space allows preference to an even larger number of channels. In other words, our simple yet novel search space of selectively expanding the channels can mitigate the limit in choosing larger number of channels, opening up more significant opportunities for compensating quantization enrors.
4.3 DISCUSSION ON NEURAL CHANNEL EXPANSION
So far, we explained how NCE was devised facilitate the channel expansion for compensating the quantization error. In this section, we further investigate two aspects of NCE to understand its strengths better.
4.3.1 BENEFIT OF QUANTIZATION ARCHITECTURE SEARCH
We first show that NCE can reflect quantization during the search to find better structure for quantization. As we discussed the previous section, quantization affects the gradients w.r.t. the search parameters, resulting in the difference in the network structures after the search. In Fig we ran NCE for CIFARIA Ne-Re with or without W2A2 quantization during the search. Then we took the models after each search and trained them from scratch with or without W2A2 quantization. After the full-precision training, both networks (searched with or without quantization) achieve the same accuracy level. In the case of W2A2, however, the network searched with quantization achieves noticeable gain in average accuracy over the network searched without quantization. Fig. Bp shows the difference in the channel section between the model searched with (=W2A2) and without Quantization=W332A33 where W2A2 prefers more channels in the later layers. This demonstrates that NCE perform quantization-ano architecture search.

---

Under review as a conference paper at ICLR 2021
Search / Retrain
Aec
120
21.
ResNel20
93. 18%
(W32x32) (W332132)
91.4
80-
ResNot20
ACC15
91 28%
(W32X32)
ResNe120
98.57%
40-
91.0
(W2N3) (W322332)
ResNel20
91.63%
(W2A2) (W2A2)
33
19
54
PLOPHIMI
(a)
(b)
(C)
Figure 3: (a) Test accuracy after search with or without quantization. (b) Model structure after search
with or without quantization. (c) Accuracy vs. FLOPs for different channel expansion strategies.
4.3.2 BENEFIT OF SELECTIVE CHANNEL EXPANSION
There are two options to search for channel expansion. One option is to start with the enlarged channels as the search space then prune, while the other option is to expand the channel selectively as NCE does. To understand the selective expansion's effectiveness, when the model is searched with 1) 1X channels accompanied by eight search parameters for each layer but with NCE, and 2) 2X channels accompanied by 16 search parameters layer. Note that the search space of the IX-NCE case is strictly a subset of the 2X case. Therefore, if the IX-NCE finds a suitable network structure, 2X should also find equally good one. However, it turns out that the search results of the 2X case are inferior to NCE. As shown in Fig. Be, for the same target FLOPs, the network structures found by the 2X case achieve inferior test accuracy than NCE. In Appendix, we confirm that the observed accuracy gain is indeed originated from the proposed selective increase in the search space.
constructed experiment
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
NCE is evaluated with popular CNNs trained on CIFARIO and IamgeNet datasets. We employ PACT (Choi et al. (2018) as the main quantization scheme. NCE is implemented in PyTorch based on the TAS framework For CIFARIO experiments, we conducted 200 epochs of warm-up followed by NCE search for 600 epochs. we used threshold (T) of 0.3 and constraint coefficient of For ImageNet experiments, we randomly choose 50 classes from the original 1000 classes to reduce the training time, similar to Wu et al. (2019). We conducted 40 epochs of warm-up followed by search for 110 epochs. used threshold (I') of 0.19 and constraint coefficient of 1.5. We optimized the weight via SGD and the architecture parameters via Adam. Regularization coefficient of PACT is 0.001. After search, the candidate model is derived by the "winner-takes-alll strategy. Unless noted otherwise, all experiments on CIFARIO 10 are repeated for three times and the average test accuracy reported. For fair comparison with prior work (e.g.. Choi al. (2018); Bung al.) (2019)), the first/last and chort-cut layers are not considered for 2-bit uniform- uniform-precision precision quantization. More detailed information about the experimental settings can be found in Appendix.
(e.g..(Choi
5.2 CIFAR1O RESULTS
To evaluate NCE on CIFARIO, we employ four popular CNNs based on the ResNet structure and the other based on the VGG structure. For each network, we apply 2-bit quantization with and without NCE. As be shown in Table NCE consistently boosts the accuracy of QDNN w/o NCE by 0.35% ~ 0.81%, demonstrating its benefit. Also, we demonstrate with CIFARIO ReeRee that the loss for hardware cost be either FL.OPs the number of parameters (PARAM): Table T shows that ResNet20 could achieve higher accuracy when NCE searches with the PARAM loss. This flexibility in hardware loss can be convenient when targeting hardware platforms with specific needs.
7

---

Under review as a conference paper at ICLR 2021
w/o NCE
w/ NCE
Network
W32A32
W2A2
FLOPs
PARAM
W2A2
FLOPs
PARAM
91.63%
39.94M
0.42M
ResNet20
92.88%
90.82%
40.81M
0.27M
91.81%
45.40M
0.27M
ResNet32
93.81%
92.22%
69.12M
0.47M
92.71%
64.26M
0.57M
ResNet56
94.26%
93.08%
125.75M
0.86M
93.43%
123.04M
0.74M
VGG16
94.24%
93.48%
313.2M
14.72M
93.94%
302.96M
5.01M
Table 1: CIFAR10: Comparision on test accuracy and computational complexity for 2-bit QDNN
with without NCE (*: constrained with PARAM loss; otherwise constrained with FLOPs).
Network
Method
Top-1 Acc
Top-5 Acc
FLOPs
PARAM
Full precision
70.56%
89.88%
1.814G
11.69M
w/o NCE(Ours)
64.08%
86.47%
1.814G
11.69M
NCE(Ours)
66.17% %
86.75%
1.747G
12.57M
ResNet18
LSQ
67.6%
87.6%
QIL
65.7%
LQ-Nets 64.9%
85.9%
1,814G
11,69M
PACT
64.4%
85.6%
EdMIPS
65.9%
86.5%
Full precision
76.82%
93.33%
4.089G
25.56M
w/o NCE(Ours)
72.36%
90.81%
4.089G
25.56M
ResNet50
w/ NCE(Ours)
74.03% %
91.63%
3.932G
17.66M
LSQ
73.7%
91.5%
LQ-Nets
71.5%
90.3%
4.089G
25.56M
PACT
72.2%
90.5%
EdMIPS
72.1%
90.6%
Table ImageNet: Top-1/Top-5 accuracy and computational complexity for 2-bit (W2A2) QDNN with or without NCE compared with the other state-of- of-the-art uniform (LSQ, QIL, LQ-Nets, PACT) and mixed- sixed-precision (EdMIPS) quantization methods.
5.3 IMAGENET RESULTS
We further evaluate NCE with ResNet structures ImageNet by comparing it with slate-of-the-an QDNN methods. We take reported Top-1 and Top-5 accuracy from the state-of-the- uniform- precision quantization techniques; LSQ(Esser al. (2020). QIL(Jung al.] (2019). LQ-Nets (Zhang ct al. 20188). ), and PACTiChho et al. (018)). We also include recent work on mixed-precision quantization; EdMIPS Vasconcelos (2020)) The comparison of accuracy is summarized in T Note that NCE consistently improves the accuracy of 2-bit quantized models (=w/o NCE) by large margins (1.67% 2.09%) thanks the structure's ''s adaptation. Although the PACT quantizer's performance in NCE is inferior to the more recent techniques (e.g.. LSQ and QIL), NCE could boost the accuracy to the extremely competitive level. the case of ResNet50, NCE even surpasses the best-to-date QDNN method, LSQ, in terms of accuracy, FLOPs, and the number of parameters involved the network. This superior performance indicates that NCE is promising solution for compensating the accuracy degradation of QDNN.
CONCLUSION
In this work, we propose a novel approach that explores the neural network structure to achieve robust inference accuracy while using the simple uni-precision arithmetic units. The key idea to explore the network structure under the impact of uni-precision quantization. Our novel differentiable neural architecture search employs the search space that can shrink and expand the channels so that the more sensitive layers can be equipped with more channels. At the same time, the overall resource requirements (e.g.. FLOPs) are maintained. We provide substantial insights and empirical supports that the proposed method can achieve superior robustness in ultra-low uni uni-precision quantization for networks in CIFAR10 and ImageNet. In particular, we demonstrate a superior ResNet50 architecture that achieves higher accuracy for 2-bit inference with smaller FL.OPs and the parameter size.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le. Can weight sharing outperform random architecture search? an investigation with tunas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14323-14332, 2020.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed mixed-precision neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2349 2358, 2020.
Vincent Camus, Linyan Mei, Christian Enz, and Marian Verhelst. Review and benchmarking of precision-scalable multiply-accumuule unit architectures for embedded neural-networt processing. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(4):697-71,, 2019.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint
2018.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In Advances in Neural Information Processing Systems, pp. 760-771, 2019.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hes- sian aware quantization of neural networks with mixeddprreissoo In Proceedings of the IEFE International Conference Computer Vision, pp. 293-302, 2019.
Ahmed T Elthakeb, Prannoy Pilligundla, FatemehSadat Mireshghallah, Amir Yazdanbakhsh, Sicun Gao, and Hadi Esmaeilzadeh. Releq: automatic reinforcement learning approach for deep quantization of neural networks. arXiv preprint arX1v:1811.01704, 2018
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra Modha. Learned step size quantization. In International Conference on Learning Representations, 2020.
Google. Edge tpu run inference at the edge, 2019. URL htt.ttps://clood.ooglee..oo edue-tpu
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniy, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, pp. 4107 2016.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4350 4359, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 09055, 2018.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.

---

Under review as a conference paper at ICLR 2021
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In International Conference Machine Learning, pp. 4486 2495, 2019.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: Wide reducee--reeisson networks. In International Conference on Learning Represenlutions, 2018.
Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE Access, 7:19143-19165, 2019.
NVIDIA. Nvidia deep learning accelerator, 2019. URL https:://vaaaa.cor
Hanmin Park and Kiyoung Choi. Cell division: weight bit-width reduction technique for convolutional neural network hardware accelerators. In Proceedings of the 24th Asia and South Pacific Design Automation Conference, pp. 286-291, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pp. 522-542. Springer, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Charbel Sakr and Naresh Shanbhag. An analytical method determine minimum per-layer precision of deep neural networks. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1090-1094. IEEE, 2018.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):225-23329 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114, 64, 2019.
Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10781-10790, 2020.
Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware aware automated quantization with mixed precision. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 8612-8620, 2019.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00099. 2018.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey. Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: (609.08/44, 2016.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV). pp. 365-382, 2018
Ying Zhang, Mohammad Pezeshki, Philémon Brakel, Saizheng Zhang, César Laurent, Yoshua Bengio, and Aaron Courville. Towards end-to-end speech recognition with deep convolutional neural networks. Interspeech 2016, pp. 410-414, 2016.
10

---

Under review as a conference paper at ICLR 2021
Max Num Channels
Accuracy
FLOPs
PARAM
1.25X ResNet20
91.45%
40.19M
0.28M
I,50X ResNet20
91.82%
39.63M
0.34M
2.00X ResNet20
91.63%
39.94M
0.42M
Table 3: Impact of constraining the maximum number of channels for NCE search.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning, pp. 7543-7552, 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 160666666000 06 060,0,066606666 2016.
Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard. Adaptive quantization for deep neural network. arXiv preprint 1rXiv:1712.00000 172.00048, 2017.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 111:0:1611.01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
7 APPENDIX
7.1 EXPERIMENTAL DETAILS
In this section, we summarize the details of our training settings. The CIFAR1O dataset (Krizhevsky Hinton (2010)) is an image classification benchmark containing 32x32 pixel RGB images. It consists of 50K training and 10K test image sets. We used RandomHorizontalFFin and Normalization for preprocessing. We used stochastic gradient descent (SGD) with a momentum of 0.9 and learning rate starting from 0.1 and scheduled by cosine annealing to update network weight. We used Adam optimizer and learning rate of 0.001 to update the search parameter. L2-regularizee with the decay of 0.0004 is applied to weight, and 0.001 is applied to the search parameter. The mini-batch size of 256 is used.
The ImageNet dataset (Russakovsky et al. (2015)) consists of 1000-categories of objects with over 1.2M training and 50K validation images. We used RandomeesizeedCrop RandomHoriiznttlFFig and Normalization for preprocessing. We used stochastic gradient descent (SGD) with a momentum of 0.9 and learning rate starting from 0.1 and scaled by 0.1 at epoch 30, 60, 85, 95 to update network weight. We used Adam and a learning rate of 0.001 to update the search parameter. L2-regularizer with the decay of 0.0001 is applied weight, and 0.001 is applied the search parameter. The mini-batch size of 256 is used.
7.2
ABLATION STUDY
In this section, we present various ablation study and analysis to better understand NCE.
7.2.1 IMPACT OF CONSTRAINING MAXIMUM NUMBER OF CHANNELS FOR NCE SEARCH
In Sec. evaluated NCE with the maximum number of channels 2X of the original network size. In this setting, NCE sometimes found the networks with the hardware cost higher than the original network if it not constrained especially when the network size is relatively small; e.g., NCE for ResNee20-CIFAR 10 achieved higher accuracy (=91.63%) when it is constrained with FLOPs (=40.81M), but with increased PARAM-0.42M.
'We followed the convention of restricting only one type of hardware cost, either PARAM or FLOPs
11

---

Under review as a conference paper at ICLR 2021
ResNet20 CI W2A2
Accuracy
PARAM
HLOPs
Full-Prec Baseline
92.88%
0.27M
40.81M
W2A2 Baseline
90.82%
0.27M
40.8TM
W2A2 TAS (#search param=8)
90.99%
0.46M
43.1TM
W2A2 2X "AS (#search param=16)
91.12%
0.42M
40.36M
W2A2 NCE (#search param=8~16)
91.63%
0.42M
39.94M
Table 4: Comparison of quantization performance between NCE and 2X TAS with controlled search
space size.
91.8
91.8
WRPN
WRPN
NCE
NCE
91.6
91.4
91.2
91.2
Acc(%)
Acc!%)
91.0
91.0
90.8
90.8
90.6
90.6
90.4
90.4
90.2
0.24
0.26
0.30
PARAM(M)
FLOPs(M)
(a)
(b)
Figure 4: Comparison of quantization performance between NCE and WRPN with respect to (a) PARAM and (b) FLOPs ResNee20-CIFAR1O.
In this section, we extend our study constrain both PARAM and FLOPs by fixing the maximum number of channels, limiting channel expansion. A detailed investigation about the impact of the maximum number of channels for the NCE search can is shown in Table As maximum number of channels decreases from 2X to 1.25X, PARAM also decreases while maintaining the overall accuracy.
7.2.2 COMPARISON OF QUANTIZATION PERFORMANCE BETWEEN NCE AND TAS
In Sec. 4.3.2 discussed the benefits of the selective channel expansion over the prior channel pruning approach, TAS (Dong & Yang (2019).. We further investigate this issue to confirm that the observed accuracy gain originates from the selective channel expansion, not from the search space with the
size. Table 4 shows the experimental results of 2-bit quantization on ResNer20-CIFAR number of channels adapted by either 2X TAS NCE (with FLOPPs=0M the hardware constraint). Note that 2X TAS performs pruning from the 2X uniform channel expansion; thus, NCE's search space subset of 2X TAS's search space. The search space with both 8 and 16 search parameters per layer is considered for the controlled experiments. We can observe from the table that 2X TAS found the network with increased channels, achieving only modest accuracy gain. In contrast, NCE achieved noticeable accuracy gain with slightly lower PARAM and FLOPs than 2X TAS. This experimental result demonstrates the benefit selective channel expansion.
7.2.3 COMPARISON OF QUANTIZATION PERFORMANCE BETWEEN NCE AND WRPN
A straightforward method to explore the trade-off between the number of channels and the robustness of quantization error is to uniformly increase the number of channels for all the layers, as discussed in WRPN (Mishra et al. 2018)). We argue that NCE explore better accuracy vs. hardware cost trade-off by expanding the number channels only for the necessary layers. To demonstrate it, we compare the trade-off curves between accuracy and the two popular hardware costs, PARAM and
12

---

Under review as a conference paper at ICLR 2021
NCE W2A2
NCE W2A2 W2A2
14
ResNet20 W2A2
MMM
0.6
SQNR(dB)
STDEV
10
0.4
8-
9 10 11 12 14
2
3
10 11 14111111111
layers
layers (b)
(a)
Figure 5: Comparison of STDEV and SQNR of activation between ResNe220CIFAR10 and NCE when the models are quantized 2-bit. (W2A2 Accuracy: ResNet20 (90.82%), NCE (91.63%))
Search ResNet50 structure
ResNet50
2000
ResNet50
1500
#channel
1250
1000
750
500
250
1
7
10
13
16
19
22
25 layers
28
31
34
37
40
43
46
49
Figure 6: Comparison of structure ResNer550-lmageeet before and after adaptation by NCE,
NCE always achieves higher accuracy in terms of both PARAM and FLOPs, highlighting its superiority over WRPN.
FLOPs.
As shown in Fig. 4
7.2.4
RELATIONSHIP BETWEEN ACTIVATION DYNAMIC RANGE AND QUANTIZATION ERROR
In Sec. 4.1 we discussed the impact of quantization on the dynamic range of activation. To provide concrete connection between the dynamic range of activation and the quantization error, compare the signal-to-quanee tattttt-iiztttin---eeeee the CosNett0-CIIAR 10 network and the adapted by NCE. Note that SQNR is well-known metric for the impact of quantization error on the model performance (Cai & Vasconcelos (2020)). Fig. shows that NCE reduces the dynamic range of activation and increases the SQNR when the
(SQNR) measured when W2A2 is applied to activation of
13

---

Under review as a conference paper at ICLR 2021
Threshold 7
Accuracy
0.30
91.60%
0.25
91.44%
0.20
91.40%
0.15
90.77%
Table 5: The accuracy of the model NCE adapted from ResNer20-CIFAR10 with different channel expansion threshold, T.
ResNet20-CIFARIO
Accuracy
FLOPs
W32A32
92.88%
Original structure (w/o NCE)
W3A3
92.45%
40.81M
W4A4
92.69%
W3A3
92.66%
39.07M
NCE
W4A4
92.75%
37.07M
Table 6: 3-bit and 4-bit Quantization of ResNet20-CIFARIO with and without NCE.
model is quantized, confirming its beneficial influence for compensating the quantization error and regain the model accuracy.
7.2.5 COMPARISON OF NETWORK STRUCTURE BEFORE AND AFTER ADAPTATION NCE
NCE adapts the channels the original neural network to improve its robustness to the quantization error. As a result, the neural network searched by NCE has the number of channels expanded or reduced across the layers while maintaining overall shape. As an example, Fig. 6 shows the comparison of the number of channels of ResNet50-Imagge Note that the original ResNet50 structure contains an abundant number channels the later layers, but these numbers of channels artificially scaled inversely proportional to the feature-map width. NCE successfully balances the number of channels to improve quantization accuracy while maintaining hardware constraints.
before and after adaptation by NCE.
7.2.6 CHANNEL EXPANSION THRESHOLD
The channel expansion threshold T in Algorithm is is the dyyer-paramette that determines which layer to request the channel expansion. Given that there are 8 search parameters initialized uniformly, T needs be higher than 1/8 0.125. Although plays critical role in enabling the selective expansion of the channels, the overall performance is not very sensitive to its choice. As shown in T there plateau in the model accuracy for the selection of T 0.15. In practice, setting T between 0.20 and 0.30 is sufficient.
7.2.7 QUANTIZATION WITH HIGHER BIT-PRECISION FOR NCE
To further understand the impact of NCE for higher bit breciiicii quantization, we experimented BesNet20-CIFARI11 10. As shown in Table NCE consistently finds neural networks with higher accuracy when they are trained with the same setting as the original network. Interestingly, FLOPs decreases as the bit-precision increases; this indicates that NCE is capable of effectively exploring the trade-off's between FLOPs and accuracy.
14


Reviews:
[{"review_id": "i8LnvyHrJNz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose neural channel expansion (NCE), a neural architecture search (NAS) and quantization method. Existing NAS+Q methods typically search for the architecture of the DNN along with the precision at each layer, maximizing accuracy while respecting some kind of hardware constraint. The result is a DNN with mixed-precision, which is challenging for most existing hardware (which only support one or a few precisions). NCE keeps precision the same in each layer, and instead uses the precision sensitivity signal in the NAS to adjust the width of the layer (expand or shrink). The result is uniform-precision, hardware-friendly DNN.\n\nNCE works by first training normally (with quantization) for 40 warmup epochs, followed by 110 epochs of search. At the end of each search epoch, NCE adjusts the channels search parameter in each layer. A few experiments show convincingly that a wider layer is indeed less sensitive to quantization. Thus using the sensitivity-to-quantization signal to adjust layer width is a good idea.\n\nExperiments on CIFAR-10 show NCE can boost quantized accuracy at 2w2a by up to 0.8%, and in the case of VGG16 trim unnecessary params. On ImageNet there is also some accuracy improvement, though only a little bit over LSQ. And for ResNet-50 NCE again can reduce param size.\n\nThe paper is that the idea is fairly simple, and the results are not too impressive. LSQ seems to already do very well and it also uses uniform quantization. One major issue I have with the comparison in Tables 1 and 2 is that, on some of the smaller networks (ResNet-32 for CIFAR and ResNet-18 for ImageNet) the NCE result has more params than the other methods. This is potentially unfair as a larger network is almost always more accurate. I think you should uniformly increase channel widths in at least the \"w/o NCE\" baseline to see if the accuracy boost is from NCE learning the layer sensitivities or just from a bigger model.\n\nThe results on larger networks (VGG and ResNet-50) is much more compelling, showing that NCE can trim unnecessary params while improving accuracy. More results like this would make the paper more convincing. I would also like to see exactly which layers were reduced in size on these networks.\n\nAnother issue is that despite being a NAS work, there aren't NAS baselines in the comparison. I understand that NCE only requires one training run while the original NAS required many retrainings. But I believe HAQ (Wang et al 2019) and DARTS (Liu et al 2018) are both NAS techniques for mixed-precision quantization that require only one training run. The authors should include a comparison against such methods or discuss why it isn't needed.\n\nMinor issues:\n - Section 4.3.2 Typo: \"results of the 2X case are inferior to the 2X case\"\n - Table 2, ResNet-18, you highlighted your own result but LSQ seems to be better in accuracy and param size?\n\nEDIT: Raised score from 4 to 6 after the authors clarified some points and added additional experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CkR7LZe55K1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview\nIn this paper, the authors proposed selectively channel pruning and expansion via neural architecture search to make an off-the-shelf neural network more robust to quantization. Under 2-bit quantization, the proposed method outperforms the original model at similar or smaller FLOPs and model size.\n\n### Clarity\nOverall, the paper shows the concept clearly. The method itself is not complicated. But some definitions are missing to make the paper self-contained. For example, the implementation of channel-wise interpolation (CWI) is not provided, and not referred to existing literature. The presentation of Figure 2(a)(b) makes it hard to see the trend. It would be better to plot the temporally averaged curve (potentially with std) for better visualization.\n\n### Pros\n1. The paper is generally well written and clearly expressed.\n\n2. It focuses on an important topic: neural architecture search under (low-bit) model quantization, which has been less explored before. The paper shows that considering the quantization during the search can improve the final accuracy.\n\n3. The proposed method shows good results. On ImageNet, the optimized model outperforms the original version at similar or lower cost after quantization.\n\n### Cons\n1. My first concern is the novelty. As acknowledged by the authors, improving quantization with channel expansion has been widely studied [Mishara et al. 2018, Zhao et al. 2019, Park&Choi 2019] and proves to be effective. The authors claimed that \"none of the approaches shed light on understanding the benefit of the increased number of channels when a network is trained for quantization\", which I think is an overclaim. The reduction of dynamic range from increased channels is already well demonstrated in previous work (e.g., Zhao et al. 2019); while the authors verify the conclusion again in this work. The author did prove that the modified/expanded architecture works well when trained from scratch, but it is not a very surprising finding since networks with higher capacity are more robust to quantization.\nAs for the NCE method itself, it is like an extension of TAS by allowing expansion in search space and under quantization. The core novelty seems to the extended search space with channel expansion, but the other parts are largely the same. \n\n2. A trivial alternative would be to apply TAS using the enlarged search space (Sec 4.3.2) with theoretically the same optimal solution. The ablation study shows that the larger search space might hinder successful optimization, leading to worse results. I think using 16 search parameters makes the optimization too difficult; a fair comparison would be to still use 8 search parameters, with channel configurations using larger strides to compare the accuracy, so that we can exclude the factor of architecture search difficulty and focus on the search space design (which is the main contribution of the paper).\n\n3. Regarding the experimental results, it outperforms the original model quantization using various methods. However, the authors did not compare to existing results that modify/expand the network architecture for better quantization accuracy (e.g., Zhao et al. 2019). Without comparing to existing methods on the same topic, it would be difficult to evaluate the contribution.\n\n4. The author mentioned that the expanded architecture itself is beneficial for quantization. For results in Table 2, if we train the optimized architecture from scratch, will we still have the same advantage?\n\n5. Minor question: it is unclear how the batch normalization layers are processed. Are they fused into convolutions or kept as float-point? \n\n\nI hope to see the authors' feedback for the final evaluation. Thanks!\n\nEDIT: Raised score from 4 to 6 after the reading authors' response. The response clarifies some of the novelty issues, and it clearly shows the advantage compared to previous methods like TAS. However, I still have concerns about the novelty; the insight why the proposed method is better than TAS is still not very clear to me. I hope the author can further improve the draft for the final version.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9RCR_2WiHn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose neural channel expansion (NCE) to adjust the network structure to compensate for the performance degradation from uniform-precision quantization. Given a hardware constraint, the proposed NCE selectively expands the width for the quantization sensitive layers. Experiments on CIFAR-10 and ImageNet shows the effectiveness of the proposed NCE. However, the novelty of this paper is limited since the proposed method is just an extension of TAS. Besides, the comparisons between NCE and the existing methods are missing. My detailed comments are as follows. \n\nPositive points:\n1. Based on TAS, the authors propose a neural channel expansion (NCE) method to search for the width across the layers under uniform-precision quantization.\n\n2. The authors provide an analysis of the proposed NCE on the effect of channel expansion for compensation of quantization errors.\n\n3. Experimental results on CIFAR-10 and ImageNet demonstrate that NCE is able to improve the performance of the quantized networks.\n\nNegative points:\n1. The novelty of this paper is limited. The authors only make minor modifications to TAS [1] and apply it to network quantization. However, neither of the modification on TAS or the application of channel expansion on network quantization is significant. \n\n2. This paper is quite similar to the work of Shen et al. [2], which also explores the optimal layer width for quantization. Thus, it would be better to compare the proposed method with Shen et al. ’s. \n\n3. As mentioned in Section 2, there are several existing works [3][4][5] that reducing the quantization error by increasing the widths of the network. More comparison between the proposed method and the existing works would make this paper more convincing.\n\n4. In Section 4.1, the authors claim that “The substantial increase of STDEV for W2A2 implies that large quantization errors would occur when input activation is quantized”. However, Figure 1a only provides the results of W2A2 and W32A32. Thus, it cannot illustrate whether the large quantization error results from the quantization of activation or not. At least, the results of W32A2 and W2A32 should be provided.\n\n5. In Section 4.1, the authors argue that improved performance is resulted from the reduction of input activation’s dynamic range according to Figures 1a and 1c. However, the STDEV concerning “2X ResNet20 (W32A32)” is missing in Figure 1a. To compare the dynamic range reduction of ResNet20 with 1x and 2x width, more results regarding the STDEV of “2x ResNet W32A32” would make this paper more convincing.\n\n6. As discussed in Section 4.1, the improved performance comes from the reduction of activation’s dynamic range. However, it is not clear how the dynamic range of activation impacts model performance. It is better to give more theoretical analyses and explanations regarding the results.\n\n7. The authors only apply NCE to 2-bit quantization. It will be better if the authors would provide results on more bitwidth quantization, e.g., 4-bit or 1-bit.\n\n8. In Algorithm 1, the threshold T is important to the proposed method. According to the experimental settings on CIFAR-10 and ImageNet, the chosen of threshold T is tricky. More results on different values of T are required. \n\n9. In the first line of Algorithm 1, the n(D_val) should be n(D_arch).\n\nReferences\n\n[1] Xuanyi Dong et al. Network pruning via transformable architecture search. NeurIPS 2019.\n\n[2] Mingzhu Shen, Kai Han, et al. Searching for accurate binary neural architectures. ICCV Workshop 2019.\n\n[3] Asit Mishra et al. Wrpn: wide reduced-precision networks. ICLR 2018.\n\n[4] Ritchie Zhao et al. Improving neural network quantization without retraining using outlier channel splitting. ICML 2019.\n\n[5] Hanmin Park et al. Cell division: weight bit-width reduction technique for convolutional neural network hardware accelerators. ASPDAC 2019.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests a NAS approach for quantization which focuses on expanding the number of channels in problematic layers, given some uniform quantization level for all the layers. The reviewers were initially all negative, but the authors added more experiments and the scores changed to borderline (6/6/5). I think that though the reviewers appreciated the effort made by the authors and clarifying many of the issues. Yet, I think two concerns still remained. First, the method is demonstrated convincingly in only one large scale case (ImageNet) - on ResNet50 W2A2. The other case (ResNet18) is less impressive since it uses more parameters and has a small gap from EdMIPS. I think using more architectures (and possibly also precision levels) is important to convincingly demonstrate the utility of the method (e.g. EdMIPS used 5 architectures). Second, the novelty of the method in comparison to the previous method was not completely clear. Though the authors added more experiments on CIFAR to compare with previous methods, the significance of the results is not clear (small differences in the figures, and no error bars), and also the explanation of why the results of NCE should be better than TAS.
Decision: Reject