Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
Authors: Yichi Zhang
Ritchie Zhao
Weizhe Hua
Nayun Xu
G. Edward Suh
Zhiru Zhang
Venue:  (2020)
Keywords: deep learning
neural network
dynamic quantization
dual precision
efficient gating

Abstract: We propose precision gating (PG), an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks.  PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy.  The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss.  Our experiments indicate that PG achieves excellent results on CNNs, including statically compressed mobile-friendly networks such as ShuffleNet. Compared to the state-of-the-art prediction-based quantization schemes, PG achieves the same or higher accuracy with 2.4× less compute on ImageNet. PG furthermore applies to RNNs. Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.7× computational cost reduction on LSTM on the Penn Tree Bank dataset.

Published as a conference paper at ICLR 2020
PRECISION GATING: IMPROVING NEURAL NETWORK EFFICIENCY WITH DYNAMIC DUAL-PRECISION ACTI- VATIONS
Yichi Zhang Comell University y224998cornel1.edi
Ritchie Zhao Coreell University I22528cornel1.c edu
Weizhe Hua Coreell University wh3990cornel11 edu
Nayun Xu
G. Edward Suh
Zhiru Zhang Cornell University
Cornell Tech
Cornell University edward. suw.oornell...eeecoooooeo edu
nx388cornel1.c edu
2hiruz@cornell.ed
ABSTRACT
We propose precision gating (PG), an end-to-end trainable dymamic dual-precision quantization technique for deep neural networks. PG computes most features low precision and only small proportion of important features in a higher pre- cision to preserve accuracy, The proposed approach is applicable to a variety DNN architectures and significantly reduces the computational cost of DNN exe- cution with almost no accuracy loss. Our experiments indicate that PG achieves excellent results CNNs, including statically compressed mobile-friendly net- works such as ShuffleNet. Compared to the state-of-the-art prediction-based quan- tization schemes, PG achieves the same higher accuracy with 2.4x less pute on ImageNet. PG furthermore applies to RNNs. Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement perplexity per word with 2.7x computational cost reduction on LSTM on the Penn Tree Bank dataset.
INTRODUCTION
In recent years, deep neural networks (DNNs) have demonstrated excellent performance many computer vision and language modeling tasks such as image classification, semantic segmentation, face recognition, machine translation, and image captioning (Krizhevsky et al.] 2012. He et al.) 2016a Ronneberger et al.) 2013 Chen et al. 2016 Zhao al. 2018 Schroff et al. 2015 Luong al. 2015 Vaswani al. 2017). One evident trend in DNN design is that researchers strive for better accuracy, both the model size and the number of DNN layers have drastically increased time (Xu et al. 2018). At the same time, there is growing demand to deploy deep learning technology in edge devices such as mobile phones, VR/AR glasses, and drones (Wu et al.) 2019). The limited computational, memory, and energy budgets on these devices impose major challenges for the deployment of large DNN models at the edge.
DNN quantization is important technique for improving the hardware efficiency of DNN execu- tion (Zhao et al.) 2017). Numerous studies have shown that full-precision floating-poin computation not necessary for DNN inference quantized fixed-point models produce competitive results with small or zero loss in prediction accuracy (Lin et al. 2016; He et al. al. 2016b; Zhou et al.) 20166] 2017]. some cases, quantization may even improve model generalization by acting as a form of regularization. Existing studies mainly focus on static quantization, in which the precision of each weight and activation is fixed prior to inference (Hubara et al., 2017 | He et al. 2016b). Along this line of work, researchers have explored tuning the bitwidth per layer (W et al. 2018b Wang al.)
2019 Dong et al. 2019) as well as various types of quantization functions (Wang et al. 2018 Cour-
bariaux et al. 2016: Li et al. 2016 Zhou et al., 2016). However, static DNN quantization methods characteristics, where certain features can be computed a lower precision during inference they contribute less to the classification result for the given input. For
cannot exploit input-

---

Published as a conference paper at ICLR 2020
example, in computer vision tasks, the pixels representing the object of interest are typically more important than the background pixels.
In this paper, reduce the inefficiency of statically quantized DNN via precision gating (PG), which computes most features with low fow-piecis arithmetic operations and only updates few impor- tant features high precision. More concretely, PG first executes DNN layer in low precision and identifies the output features with large magnitude as important features. It then computes a sparse update increase the precision of those important output features. Intuitively, small val- ues make a small contribution to the DNN's output; thus approximating them in a low precision is reasonable. Precision gating enables duel-preeision DNN execution at the granularity of each indi- vidual output feature, and therefore greatly reducing the average bitwidth and computational cost of the DNN. We further introduce a differentiable gating function which makes PG applicable to a rich variety of network models.
Experimental results show that PG achieves significant compute reduction and accuracy improve- ment on both CNNs and LSTMs. Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classification accuracy with up to 4.5x and 2.4x less computational cost for CIFAR-10 and ImageNet, respectively. On LSTM, compared to 8-bit uniform quantization PG boosts perplex- ity per word (PPW) by 1.2% with 2.8x less compute on the Penn Tree Bank (PTB) dataset. Our contributions are as follows:
1. We propose precision gating (PG), which to the best of our knowledge is the first end-to- end trainable method that enables dual-precision execution of DNNs. PG is applicable to wide variety of CNN and LSTM models.
2. PG enables DNN computation with lower average bitwidth than other state-of-the-a quan- tization methods. By employing a low-cost gating scheme, PG has the potential to reduce
DNN execution costs in both commodity and dedicated hardware.
3. PG achieves the same sparsity during back proopppppagttionn as forward propagation, which dramatically reduces the computational cost for both passes. This in stark contrast to prior dynamic DNN optimization methods that focus only on reducing the inference cost.
RELATED WORK
Quantizing activations. Prior studies show that weights can be quantized to low bitwidth without compromising much accuracy (Zhu et al. 2017); however, quantizing activations with a low bitwidth (e.g.. 4 bits) typically incurs a nontrivial accuracy degradation (Mishra et al. 2018 Zhou et al., 2016). This is partially caused by large activation and weight outliers, which stretch the quantization grid too wide and too sparse under a low precision, thus increasing the error (Park et al. 2018 Zhao et al. 2019). To address this problem. Chooi et al. 2018) propose PACT to reduce the dynamic range of the activations through clipping the outliers using a learnable threshold. PACT provides a more effective quantization scheme under very low bitwidth (e.g.. 4 bits). In this work we incorporate the PACT method in the training flow of PG to handle the large outliers.
Prediction-based execution. Prior works have explored predicting Rel.U-induced zeros and max- pooling compute redundancy to lower the computational cost of CNNs. For example, [in et al. (2017); (Son et al. (2018) propose zero-prediction to utilize a few of the most-significant bits in the input activations to predict the sign of the output activation. Zero-prediction removes redundancy by exploiting the fact that negative outputs will be suppressed by the ReLU anyway. Yet, this method only applies to ReLU activations and only when a linear layer is directly followed by ReLU. Hence such method does not apply to RNNs that use sigmoid or tanh the activation function and many modern CNN networks that use batch normalization (Toffe & Szegedy] 2015) before ReI.U. Hual jet al.) 2a ()) propose channel gating to dynamically turn off subset of channels that contribute little to the model prediction result. Precision gating is orthogonal to this pruning technique as channel gating executes the whole network at the same precision.
More recently, Cao et al. (2019) propose SeerNet, which also executes CNN model in dual pre- cision. For each convolutional layer, SeerNet first executes a quantized version of the layer and uses the results to predict the output sparsity induced by the ReLU the computational sparsity induced by the max-pot maxxpoooiig layer. For those activations that are not suppressed according to the prediction, SeerNet computes the original convolution in full-precision (32-bit float). One key dif-
2

---

Published as a conference paper at ICLR 2020
Input Activations
I
Ia << Bib + Lib
1
1
1
lns
Lbb
Figure I: Splitting an input feature into las (blue), the most-significant Bad bits, and In (orange), the remaining By bits. The total bitwidth is B.
ference between PG and SeerNet is that PG reuses the result of the low- low-precision compute partial product when it performs the high-pree oigh-precision multiplication in the update phase. In contrast, the full-precision compute in SeerNet does not reuse the output from the quantized layer, which incurs a higher execution cost.
Feature-level precision tuning. There is also prior work that uses different precision to handle the outlier quantization, Park et al.] (2018) propose value-aware quantization where the majority of data computed at reduced precision while small number of outliers are handled at high precision. Our approach is significantly different because we allow dual precision for every feature, not only for the outliers.
PRECISION GATING (PG)
In this section first describe the basic mechanism PG. We then discuss how design the gating scheme to accelerate both forward and backward passes. Finally, we consider incorporating outlier clipping to reduce the quantization error.
3.1 BASIC FORMULATION
We first define a linear layer in neural network (either convolutional fully-connected) O I * W, where O, I, and W are the output, input, and weights, respectively. Suppose is represented B-bit fixed-point format, which shown Figure [ PG partitions I into (1) L the B migt-sign bits (MSBs), and (2) Ls. the remaining By, least-significant bits (LSBs). Here
B Bug BBg- More formally, we can write:
I Lsb Bbh Ils
Here << denotes the left shift operator. We then reformulate single B-bit linear layer into two lower-precision computations as:
= O + O = W * (Ias << B + [W * Lp]
(2)
Oi is the partial product obtained by using the MSBs of input feature (In6) whereas O₁ represents the remaining partial product computed with Iss
Precision gating works two phases. In prediction phase, PG performs the computation Onh W * (Inb << Bu.). Output features of One greater than a learnable gating threshold A are considered important. the update phase, PG computes O = W * In only for the important features and adds it One- The overall execution flow of PG is illustrated in Figure where unim- portant output features only take the upper path while important ones are computed as the sum of both paths. More precisely, this can be summarized follows:
O Onb ONd +
O S A O > A
In essence, PG intends to compute the majority of (unimportant) features with Bыь bits and only a small set of important features with B bits. The importance of each element in the output O is calculated by comparing the magnitude of its partial sum O A. Let Sp be the percentage of
3

---

Published as a conference paper at ICLR 2020
Prediction Phase
Partial Sum hb
An
Batch Norm ReLU
W
1tb
Update Phase
A Gate
Iib
*
W
Full recision
Figure 2: The PG building block in CNN models Input features are split into I and Ln- In the prediction phase, Ins first convolves with the full precision filters W obtain One- In the update phase, if the partial sum Os of feature exceeds the learnable threshold A, we will update that feature to high-precision by adding Oz to Ons- Otherwise, we skip the update phase, and the output feature therefore remains computed at ultra low-pue www-precssion. The prediction and update phases share the same weights. & denotes the Hadamard product.
unimportant activations over all features. PG saves Sy-Bis fraction of the compute in the original DNN model. PG thus achieves dual-precision execution using lightweight gating mechanism which only adds comparison operation. The goal is to minimize the average bitwidth of the multiply-add operations in the DNN execution.
3.2 EFFICIENT LEARNABLE GATING SCHEME
PG automatically learns gating threshold A.,r during training for each output channel in each DNN layer. A larger A., indicates that more output features computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy. Define A as the vector containing each gating threshold Ae, We formulate the problem of optimizing A as minimizing the original model loss L along with an L2 penalty term:
min L[I, D: W, A) + I 5112 W.A
Here y is the ground truth label, is the penalty factor, and 8 is the gating larget, target value for the learnable threshold. The penalty factor and gating target are hyperparameters which allow user emphasize high computation savings (large a d) or accuracy preservation (small a or d))
Training a model with precision gating can be performed on commodity GPUs using existing deep learning frameworks. We implement PG on GPU as the equation O Oj + mask C O where mask = 1o..>4 is binary decision mask and O represents element- wise multiplication. During forward propagation, most elements in mask are O. PG therefore saves hardware execution cost by only computing sparse On, in the update phase. If PG is implemented in dedicated hardware accelerator, MSBs and L.SBs will be wired separately; The prediction phase can controlled by multiplexer, and only computes LSB convolutions while O exceeds the threshold, thus achieving savings in both compute cycles and energy.
The mask is computed using binary decision function (i.e., step function), which has gradient of zero almost everywhere. To let gradients flow through mask to A, we use a sigmoid on the backward pass to approximate the step function. Specifically, A)) only on the backward pass as the previous work Hua et al. (2019b). Here a changes the slope of the sigmoid, thus controlling the magnitude of the gradients.
define mask sigmoid(a(O),, (O
4

---

Published as a conference paper at ICLR 2020
Outlier
1111
11
1100
10
Most predictions
1000
01
Uniform
0100
Taking higher 2 bits as predictions
quantization
0000
00
1111
11
1100
Quantizing narrower region separates the values apart
8-80
1000
Predictions Seperated
Saturation Value
Outlier
01
0100
0000
00
(a)
(b)
(c)
Figure 3: Effect of clipping toy example illustrating how clip threshold helps separating pre- diction values apart. The first row is quantization and prediction without clip threshold, while the second row has a clip threshold. (a) Distribution of floating- doating-point input features I. (b) Distribution of after quantizing I to bits. (c) Distribution of Ls which takes the higher 2 bits of I.
3.3 SPARSE PACK-PROPGGATION
A sparse update phase only reduces the computational cost during the inference (forward propa- gation). We further propose to save the compute during the back-propagation by modifying the forward function of the PG block. Specifically, square the mask element-wise.
Oib mask? ④ Ou
(5)
Given that mask is binary tensor, mask? in Eq. (5) preserves the same value mask. Thus the forward pass remains unchanged. During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect the gating threshold A in Eq. (6) because of the plement-wise square. Consequently, the update of A only requires the result of mask @ Ob which has already been computed during the forward pass.
d0
d (a A)) 2 mask O
(a -))
(6)
AA
Ōmask
0s
0A
The gradient of O with respect to the weights W in Eq. 7 employs the same sparse mask @ O as the update A. Therefore, precision gating can reduce the computational cost both forward- progagtion (inference) and back-propagation by the same factor.
00 aw
00 aw
80
00 hb
mmask @ O
mask
OW
aW
OW
3.4 OUTLIER CLIPPING
PG predicts important features using low low-precision computation. One difficulty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers. The top row of Figure a) shows some activations as blue dots, including single outlier. If we quantize each value to 4 bits (sccond column) and use 2 most-significant bits in the prediction phase (third column), we see that almost all values are rounded to zero. In this case, PG can only distinguish the importance between the single outlier and the rest of the values matter what A is. Thus, the presence of large outliers greatly reduces the effectiveness of PG.
To address this, combine PG with PACT (Choi al.) 2018), which clips each layer's outputs using a learnable clip threshold. The bottom row of Figure shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid. Now
5

---

Published as a conference paper at ICLR 2020
the 2 most-significant bits effectively separate out different groups of values based on mag- nitude. We apply PACT to PG in CNNs, which commonly use an unbounded activation function such ReLU. RNNs, on the other hand, typically employ bounded activation function (e.g.. tanh, sigmoid) that often makes PACT unnecessary.
4 EXPERIMENTS
We evaluate PG using ResNet-20 (He et al. 2016a and ShiftNet-20 (Wu et al. 2018a CIFAR- 10 (Krizhevsky & Hinton| 2009), and ShuffleNet V2 (Ma al. 2018) on ImageNet (Deng et al.] 2009). ResNet is very popular CNN architecture for image classification. ShiftNet and ShuffleNet are more compact architectures designed specifically for mobile and edge devices. We set ex- pansion rate of ShiftNet to be and choose the 0.5x variant of ShifleNet V2 for all experiments. On CIFAR-10, the batch size is 128, and the models are trained for 200 epochs. The initial learning rate 0.1 and decays epoch 100, 150, 200 by factor of 0.1 (i.e., multiply learning rate by 0.1). On ImageNet, the batch size is 512 and the models are trained for 120 epochs. The learning rate decays linearly from initial value of 0.5 to O.
We also test an LSTM model (Hochreiter Schmidhuber 1997) on the Penn Tree Bank (PTB) MMar- et al., 1993) corpus. The model accuracy measured by perplexity per word (PPW), where lower PPW is better. Following the configuration used by He et al. (206b) andaaa et al. (017). the number of hidden units in the LSTM cell set to 300, and the number of layers is follow the same training setting as described in He et al. (2016b)) except that the learning rate decays by factor of 0.1 epoch 50 and 90. All experiments are conducted on Tensorflow (Abadi et al,, 2016) with NVIDIA GeForce 1080Ti GPUs. We report the top-1 accuracy for all experiments.
I. We
We replace the convolutional layers in CNNs and the dense layers LSTM with the proposed PG block. Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve low average bitwidth with high accuracy.
The full bitwidth B - this represents the bitwidth for high-precision computation in PG. on ImageNet, and or on PTB.
set to 5 or less on CIFAR-1O,
The prediction bitwidth Bi this represents the bitwidth for low-precision computation.
The penalty factor a - this is the scaling factor of the L2 loss for gating thresholds A.
The gating target 8 the target gating threshold. We variety of values 5 [-1.0, 5.0) in our experiments.
The coefficient in the backward pass controls the magnitude of gradients flowing to A. We set a to be 5 across all experiments.
Table 1: Precision gating (PG) on CNN models tested are ShiftNet-20 and ResNet-20 on CIFAR-
10, and ShuffleNet V2 0.5x on ImageNet. We compare PG against uniform quantization (UQ), PACT, and Fix- Fix-Threshold. Bag is the average bitwidth. "fp" denotes floating-point accuracy. "Sp" denotes sparsity.
Ours
Baselines
PG
UQ
PACT
Fix- Threshold
BlB
Sp (%)
Bung
Acc
Bits
Acc.
Acc.
B/B
Sp (%)
B
Bavg
Acc
ShiftNet-20
5/3
55.5
3,9
89.1
8
89.1
89.0
5/3
48.8
4.0
74.3
CIFAR-10
5/3
96.3
3.1
88.6
87.3
87.5
5/3
67.8
3.6
67.0
(fp 89.4%)
3/1
71.9
1.6
84.5
2
77.8
82.9
3/1
10.1
2.8
64.3
ResNet-20
4/3
78.2
3.2
91.7
8
91.6
91.2
4/3
58.7
3.4
88.3
CIFAR-10
3/2
90.1
2.1
91.2
91.1
90.9
3/2
71.0
2.3
74.2
(fp 91.7%)
2/1
71.5
1.3
90.6
2
84.0
90.1
2/1
21.6
1.8
71.9
ShuffleNet
6/4
57.2
4.8
59.7
8
59.1
59.1
6/4
52.6
4.9
33.6
ImageNet
6/4
62.2
4.7
59.3
6
57.8
57.1
6/4
58.5
4.8
32.7
(Iip 59.0%)
5/3
41.9
4.1
58.0
5
57.0
56.6
5/3
40.4
4.2
27.7

---

Published as a conference paper at ICLR 2020
4.1 CNN RESULTS
To compare the hardware execution efficiency across different techniques, we compute the average bitwidth (Bag) of all features in a DNN model:
Bavg = Bud + (1 Sp) x (B - -Ba))
(8)
Here Sp denotes sparsity in terms of the percentage of low-precision activations (i.e., number of unimportant features divided by total features). The computational cost of DNNs is proportional to the average bitwidth. Our results on CNNs are presented in Tables a
We first compare PG against two widely adopted quantization schemes - uniform quantization (UQ) and PACT (Choi al., 2018). In Table [ columns 3 and 4 show the average bitwidth and model accuracy of PG, respectively; columns 5-7 list the bitwidth and corresponding model accu- racy of UQ and PACT, respectively. At each row, PG achieves better accuracy with a lower average bitwidth (Bag vs. Bits). Specifically, PG achieves 6.7% and 1.6% higher accuracy with 1.25x computational cost reduction, and 6.6% and 0.5% higher accuracy with 1.54x x computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-20 for CIFAR-10 (rows 3 and 6), respectively. We observe the same trend on ShuffleNet for ImageNet, where PG improves the aceu- racy of 5-bit UQ and PACT by 1.0 % and 1.4% with x.22 computational cost reduction (row 9). It is also worth noting that PG can recover the accuracy of the floating point ShiftNet-20, ResNet-20,, and ShuffleNet V2 0.5x with 3.9, 3.2, and 4.7 average bitwidth, respectively. This demonstrates that PG, using a learnable threshold, can predict unimportant features and reduce their bitwidth without compromising accuracy. The results compared to quantization baselines are visualized Figure 4 which plots accuracy vs. average bitwidth - uniform quantization (squares), PACT (triangles), and PG (circles) are shown on separate curves. Results closer to the upper-left corner are better.
We then compare PG with Fix- Threshold, which is an extension of the zero pero-prediction (Lin et al. 2017 Song et al.] 2018). Lin et al.] (2017); Song et al. (2018) explicitly predict ReLU-induced zeros during inference. To have a lair comparison, we extend their technique to predict an arbitrary fixed threshold achieve the same or lower Bawy as PG, reported as Fix- Fix-Threshold in Table [ For CIFAR-10, we observe that PG achieves 20.2% and 18.7% % higher accuracy with 1.75x and 1.38x computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-20 (rows 3 and 6), respectively. The gap in accuracy becomes even larger on ShuffleNet V2 for ImageNet dataset. With the same or a lower average bitwidth, the accuracy of PG is at least 26% higher than Fix- Threshold (rows 7-9). In conclusion, PG consistently outperforms Fix-Threee -hrehhhshold because PG uses a learnable gate function which can adjust its threshold to the clipped activation distribution.
Table 2: Comparison with SeerNet on CNN compare PG against SeerNet under similar model prediction accuracy. In SeerNet the average bitwidth Bavg = Bbs + (1 Sp) x B.
PG
ScerNet
B/Brab
Sp (%)
Bagg
Acc
Bl
Sp (%)
Bavg
Acc
ShiftNet-20
5/3
96.3
3.1
88.6
12/8
49.7
14.0
85.4
ResNet-20
3/2
90.1
2.1
91.2
6/4
51.1
6.9
91.2
ShuffleNet V2 0.5
6/4
62.2
4.7
59.3
8/6
30.8
11.5
58.9
We further compare PG with SeerNet (Cao et al. 2019) in Table 2 For each convolutional layer, SeerNet executes a quantized version of the same layer to predict the output sparsity. It then com- putes in full precision the output activations that are not suppressed according to the prediction. Since the code of SeerNet is currently unavailable, we implement the network in Tensorflow and boost its accuracy by retraining the network. We reduce the average bitwidth of SeerNet while keep- ing a comparable accuracy as PG. For CIFAR-10, PG reduces the computational cost of ResNet-20 3.3x more than SeerNet at the same prediction accuracy. Meanwhile for the mordware-friendly ShiftNet-20, PG achieves 3.2% higher accuracy with 4.5x less compute than SeerNet. For Ima- geNet, PG on ShuffleNet also achieves 0.4% higher accuracy with 2.4x computational cost reduc- tion than SeerNet. It is worth noting that SeerNet does not reuse the outputs from the quantized layer, which may incur a nontrivial overhead in execution time. In contrast, when PG invokes the update phase, it reuses the outputs of the low-precision computation from the prediction phase.
7

---

Published as a conference paper at ICLR 2020
Table 3: Sweeping manually set thresholds we sweep series of manually thresholds for ResNet-20 on CIFAR-10. Compared to manually setting thresholds, PG achieves a better model accuracy (91.2%) with a larger sparsity (90.1%).
B/B
Fix- Ti--Theshhold
Sp (%)
Bavg
Acc
3/2
86.0
2.1
65.9
3/2
2
80.2
2.2
70.7
3/2
1
71.0
2.3
74.2
3/2
0
56.7
2.4
75.8
3/2
-1
35.2
2.6
83.5
3/2
-2
24.5
2.8
86.9
3/2
-4
13.4
2.9
88.7
order evaluate the efficacy the learnable thresholds, we further sweep series of manually set thresholds on the ResNet-20 for CIFAR-10. shols the results where use dual- precision mode of Bl = 3/2. As the threshold decreases from 3 to -4, the average bitwidth in the update phase consistently increases. This is expected because compute precision. The model prediction accuracy therefore increases. Compared to these manually set thresholds, PG achieves much improved model accuracy (91.2%) with higher sparsity (90.1%) 1)).
output features in a high
Table 4: PG with and without sparse back-propagation (SpBP) on CNNs.
PG
PG w/ SpBP
Bavg
Sp (%)
Bavg
Sp (%)
B/
Acc
4.0
49.3
3.9
55.5 (16.2)
5/3
89.1
ShiftNet-20
3.3
84.0
3.1
96.3 (t12.3)
5/3
88.6
3,4
58.2
3,2
78.2 (120.0)
4/3
91.7
ResNet-20
2.2
76.5
2.1
90.1 (113.6)
3/2
91.2
5.3
36.6
4.8
57.2 (120.6)
6/4
59.7
ShuffleNet V2 0.5
5.1
43.1
4.7
62.2 (119.1)
6/4
59.3
To quantify the impact of sparse back- pack-propagation described Section we run PG with and without sparse back-propagation on CNNs. Table the sparsity in the update phase of PG with and without back back-propagation under the same model accuracy and average bitwidth. Interest- ingly, we find that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back- propaaarpan across the tested models and datasets. For both CIFAR-10 10 and ImageNet, the sparsity increases by between 6% and 21 %. We hypothesize that sparse back-pre pack-propggttion zeros out the gradients flowing to non activated LSB convolutions in the update phase, which leads to higher sparsity.
4.2
LSTM RESULTS
PG also works well on RNNs. 5 our results applying PG on an LSTM model for the PTB corpus. Here, we only compare with the uniform quantization since PACT, Fix- Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions. Although Hubara al. (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe PPW degradation when decreases from to 4 bits in our implementation. The LSTM with 8-bit activations is therefore considered the full-accuracy model. We observe the same trend as in the CNN evaluation where PG enables the 3-bit LSTM cell to improve the PPW by 1.2 and reduce the computational cost by 2.7 compared the 8-bit uniform quantization.
8

---

Published as a conference paper at ICLR 2020
90.0
92.0
Model Accuracy (%)
Model Accuracy (%)
87.5
91.5
91.0
85.0
90.5
82.5
UQ
90.0
UQ
80.0
PACT
PACT
89.5
PG
PG
77.5
89.0
2
4
6
2
6
Average Bitwidth
Average Bitwidth
(a) ShifNet-20 on CIFAR-10
(h) ResNet-18 on CIFAR-10
60
Model Accuracy (%)
Perplexity Per Word
110
115
58
UQ
120
PACT
UQ
PG
PG
125
56
5
4
Average Bitwidth
Average Bitwidth
(c) ShuffleNet on ImageNet.
(d) I.STM on
Figure 4: Precision gating (PG) results on CNNs and LSTM compare PG against uniform
quantization (UQ) and PACT.
Table 5: PG on LSTM the dataset used is Penn Tree Bank (PTB). The metric is perplexity per
word (PPW) and lower is better. Floating-point PPW is 110.1.
Base
PG
Bits
PPW
B/B
Sp (%)
Bavg
PPW
109.8
4/2
48.4
3.0
108.5
110.8
4/2
54.9
2.9
109.3
2
124.9
3/1
53.0
1.9
118.8
4.3 SPARSE KERNEL SPREDUP
The sparse update phase of PG can be implemented efficiently with a new kernel called sampled dense-dense matrix multiplication (SDDMM), A convolutional layer with PG then factorized into a regular low-precision convolution bounded with a low-precision SDDMM. To evaluate the poten- tial speedup of PG, implement the SDDMM kernel in Python leveraging high performance JIT compiler Numba (Lam et al. 2015) and test on the ResNet-20 model for CIFAR- 10. Table shoows the layer-wise sparsity and the kernel speedup compared to the dense matrix multiplication baseline on Intel Xeon Silver 41 14 CPU (2.20GHz). With the high sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3x wall clock speedup over the general dense matrix-matrin multiplication (GEMM) kernel. The significant wall clock speedup shows good potential of deploying PG on commodity hardware.
need to replace the GEMM kernel with the SDDMM kernel to celerate the update phase. Mainstream deep learning frameworks such as Tensorflow currently do not provide a built-in operator for SDDMM, which is essential for achieving high performance GPUs. Nevertheless, Nisa et al. (2018) have recently demonstrated that highly optimized SDDMM kernel with similar sparsity shown in Table 6 can achieve about 4x speedup GEMM ker-
For GPU implementation,

---

Published as a conference paper at ICLR 2020
Table SDDMM kernel sparsity and speedup We report optimized kernel execution time and wall-clock speedup of each layer in ResNet-20 for CIFAR-10.
Layer ID
5
9
11
13
15
17
Sp
85%
94%
87%
76%
98%
99%
91%
98%
97%
Execution Time (ms)
5.4
3.3
4.9
3.6
1.5
1.1
1.5
1.0
1.2
Wall Clock Speedup
3,2x
5.1x
3,3x
2.3x
6.2x
8.3x
3.2x
6.2x
5.2x
nel. This shows strong evidence that PG has potential to obtain high speedup on GPUs well. Additionally, our approach is a good fit for the specialized accelerator architectures proposed by Lin al. (2017) and [Song et al. (2018). Due the high sparsity, DNNs with PG are estimated get at least 3 speedup and 5x energy efficiency on these dedicated hardware accelerators. We leave the deployment of the SDDMM kernel on GPU and dedicated hardware future work.
5 CONCLUSIONS
We propose precision gating, dynamic dual-preciiion quantization method that effectively reduce the computational cost of DNNs. PG assigns higher precision to important features and lower precision to the remaining features at run-time. The proposed technique is end-to-end trainable, allowing individual models to learn to distinguish important and unimportant features. Experimental results show that PG outperforms state-of-the-ar quantization and prediction approaches by large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank. We will release the source code the author's website.
ACKNOWLEDGMENTS
This work was supported in part by the Semiconductor Research Corporation (SRC) and DARPA. One of the Titan Xp GPUs used for this research was donated by the NVIDIA Corporation. We thank Jordan Dotzel (Cornell) for his helpful discussions during the camera-ready revision.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A System for Large- scale Machine Learning. USENIX Conf. on Operating Systems Design and Implementation, 2016.
Shijie Cao, Lingxiao Ma, Wencong Xiao, Chen Zhang, Yunxin Liu, Lintao Zhang, Lanshu Nie, and Zhi Yang. SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity through Low-Bit Quantization. Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Liang-Chied Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2016.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. PACT: Parameterized Clipping Activation for Quantized Neural Networks, arXiv e-print, arXiv:180S, 800.0000,008, 2018.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv e-print, 16Xiv:1602.022 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. Conf. Computer Vision and Pattern Recognition (CVPR), 2009.
10

---

Published as a conference paper at ICLR 2020
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Preciieee Precision. Int'l Conf. on Computer Vision (ICCV), 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Conf. on Computer Vision and Pattern Recognition (CVPR), 201
Qinyao He, He Wen, Shuchang Zhou, Yuxin Wu, Cong Yao, Xinyu Zhou, and Yuheng Zou. Effective Quantization Methods for Recurrent Neural Networks. arXiv e-print, arXiv:1611.101 2016b.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory, Neural Computation, 1997.
Weizhe Hua, Yuan Zhou, Christopher De Sa, Zhiru Zhang, and G. Edward Suh. Boosting the Performance of CNN Accelerators with Dynamic Fine-Grained Channel Gating. Inl'l Symp. on Microarchitecture (MICRO), 2019a.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G. Edward Suh. Channel Gating Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. Jour- nal of Machine Learning Research (JMLR), 2017.
Sergey loffe and Christian Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv e-print, 177 2015.
Alex Krizhevsky and Geoffrey Hinton. Tech report, 2009.
Learning Multiple Layers of Features from Tiny Images.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet Classification with Deep Con- volutional Neural Networks. Advances Neural Information Processing Systems (NeurIPS), 2012.
Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. Numba: A LLVM--ased Python JIT Compiler. Workshop on the LLVM Compiler Infrastructure in HPC, 2015.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary Weight Networks. arXiv e-print, arXiv:1605.044 160.00411,, 2016.
Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed Point Quantization of Deep Convolutional Networks. Ini'l Conf. on Machine Learning (ICML), 2016.
Yingyan Lin, Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. PredictiveNet: An energy- efficient convolutional neural network via zero prediction. Int'l Symp. on Circuits and Systems (ISCAS), 2017.
Thang Luong, Hicu Pham, and Christopher D. Manning. Effective Approaches to Attention- Neural Machine Translation. Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. European Conf. on Computer Vision (ECCV), 2018.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a Large Annotated Corpus of English: The Penn Treebank. Compul. Linguist., 1993.
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. WRPN: Wide Networks. Int'l Conf. on Learning Representations (ICLR), 2018
Israt Nisa, Aravind Sukumaran- Rajjam Sureyya Emre Kurt, Changwan Hong, and P. Sadayappan. Sampled Dense Matrix Multiplication for High- Perf-rerform Machine Learning. Int'l Conf. High-PPrformanc Computing (HIPC), 2018.
Eunhycok Park. Sungjoo Yoo, and Peter Vajda. Value-aware Quantization for Training and Inference of Neural Networks. European Conf. on Computer Vision (ECCV), 2018.

---

Published as a conference paper at ICLR 2020
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomed- ical Image Segmentation. Medical Image Computing and Computer- Intervention (MIC- CAI), 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. Conf. Computer Vision and Patlern Recognition (CVPR), 2015.
Mingcong Song, Jiechen Zhao, Yang Hu, Jiaqi Zhang, and Tao Li. Prediction Based Execution on Deep Neural Networks. Int'l Symp. on Computer Architecture (ISCA), 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-Aware Automated Quantization With Mixed Precision. Conf. Computer Vision and Pattern Recognition (CVPR), 2019.
Peiqi Wang, Xinfeng Xie, Lei Deng, Guoqi Li, Dongsheng Wang, and Yuan Xie. HitNet: Hy- brid Terary Recurrent Neural Network. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions. Conf. on Computer Vision and Puttern Recognition (CVPR), 2018a.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search. arXiv e-print, arXiv:1812.00090, 2018b.
Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, Tommer Leyvand, Hao Lu, Yang Lu, Lin Qiao, Brandon Reagen, Joe Spisak, Fei Sun, Andrew Tulloch, Peter Vajda, Xiaodong Wang, Yanghan Wang, Bram Wasti, Yiming Wu, Ran Xian, Sungjoo Yoo, and Peizhao Zhang. Machine Learning at Facebook: Understanding Inference the Edge. Int'l Symp. on High-Performante Computer Architecture (HPCA), 2019.
Xiaowei Xu, Yukun Ding, Sharon Xiaobo Hu, Michael Niemier, Jason Cong, Yu Hu, and Yiyu Shi. Scaling for Edge Inference of Deep Neural Networks. Nature Electronics, 2018.
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. ICNet for Real- Time Semantic Segmentation on Sigh--esoouttin Images. European Conf. on Computer Vision (ECCV), 2018.
Ritchie Zhao, Weinan Song, Wentao Zhang, Tianwei Xing, Jeng-Hau Lin, Mani Srivastava, Rajesh Gupta, and Zhiru Zhang. Accelerating Binarized Convolutional Neural Networks with Software- Programmable FPGAs. Int'l Symp. on Field- Prolrarrggaa Gate Arrays (FPGA), 2017.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving Neural Network Quantization using Outlier Channel Splitting. Int'l Conf. on Machine Learning (ICML), 2019.
Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, and Yu-Heng Zou. Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks. Journal of Computer Science and Technology. 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv preprint, arXiv: 16Xiv:1606.06110 666.06660, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained Temary Quantization. Int Conf. on Learning Representations (ICLR), 2017.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 FEATURE VISUALIZATION
In precision gating, we expect that the model will learn to compute features whose prediction values exceed the threshold using high-precision while keeping others computed using low precision. In the image recognition task, we expect that high-precision features are mostly in the region where an object lies. To provide evidence that PG can effectively learn to identity those regions, in Fig- ure 5 we visualize the decision maps extracted from the final convolutional layer that is modified to support PG in the ResNet-20 model. A decision map gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer. The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be com- puted using a high precision. The first row contains the original input images in CIFAR-10 10, and the clearly that in each decision map,
second row shows their corresponding decision maps. We the locations of bright pixels roughly align with the object in the original image.
Figure 5: Visualization of gating ratio Top: feature maps from the final precision gating block in ResNet-20 CIFAR-10. 10. Bottom: ratio of computing using high-precision (brighter pixel means higher ratio). PG effectively identifies the location of the object interest and increases bitwidth when computing in this region.
A.2 ADDITIONAL RESULTS
We provide more supplementary results in this section as shown in Table 7
Table 7: Precision gating (PG) on CNN additional models tested are ResNet-32 and ResNet-56 on CIFAR-10. We compare PG against uniform quantization (UQ), PACT, and Fix-Threshold. "fp" is floating-point accuracy. "Sp" is sparsity.
Ours
Baselines
Precision Gating
UQ
PACT
Fix Threshold
BlB
Sp (%)
Bavg
Acc
Bits
Acc
Acc
B/Bbb
Sp (%)
Bavg
Acc
ResNet-32
92.3
91.9
(fp 92.4%)
3/2
96.3
2.0
92.0
92,0
91.6
3/2
94.4
2.0
45.6
ResNet-56
4/3
93.0
3.1
93.0
92.9
92.5
4/3
91.0
3.1
90.2
CIFAR-10
3/2
98.2
2.0
92.5
92.3
92.1
3/2
96.1
2.0
50.0
(fp 92.9%)
2/1
90.4
1.1
92.0
88.5
91.8
2/1
86.9
1.1
14.2
13


Reviews:
[{"review_id": "S1lyvcKksB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper outlines a new method that allows using a variety of precision in the numerical representation of the network to increase performance (both in terms of accuracy and speed). They learn a threshold value for which all activation values above the threshold are learned at full precision, while all below are learned at reduced precision. This enables substantial performance gains. \n\nThe authors summary of the contributions made in the paper is accurate.\n\nThe paper is well written and clearly articulates a contribution to the literature. As such, I think it should be accepted. However, the major question I had as I read the paper was the efficacy on GPU, which the paper discusses, but does not implement, nor show any empirical results for, which weakens the paper. Most deep learning happens on GPUs (or similar accelerators), so until this technique is implemented there, it is of limited use. It is still a contribution to the literature, but the paper would be significantly strengthened with a GPU implementation. Additionally, the experimental evidence is lacking. More experiments would also strengthen the paper.\n\nIf these changes were made I would change my score to 8 (accept). I do think that the work is slightly premature, and would benefit significantly from adding GPU results and additional experiments. The contribution is strong, however, and should be published in some form, either now, or at a future date.\n\nFor the experimental setup, I had a few questions: \n\n1) What happens with fixed thresholds? E.g. doing a sweep over fixed values. \n\n2) How do the results vary for different initialization schemes? \n\n3) How do the results vary with the 5 hyperparameters listed? How were they chosen?\n\n4) How consistent are the results? i.e. what happens if the experiments are repeated N times? Do we see the same values?\n\nIn short, I would like to see more experiments. The results are encouraging, but brief. Evaluating on more architectures would strengthen the paper.\n\nOverall questions:\n\n- How well does the 1.2% improvement in perplexity compare to SOTA? Please add context for the numbers reported. It's not at all clear how good of an improvement is seen. \n- How do the results change with top-5 accuracy vs top-1 accuracy? \n\n\nNotes which did not affect the review score:\n\n- There are some typos, e.g. “PG computes most features in a low precision and only a small proportion of important features in a higher precision.” Saying “PG computes most features using reduced precision and only a small proportion of important features using high precision” would be more correct. There are similar typos throughout that I have not listed.\n- Tables 3 & 4, and Figure 4, are very cramped and hard to read.\nTable 1 and 2 are quite crowded; can you rearrange them so they’re easier to read?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeV9s_jFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting quantization technique that is, unusually, end-to-end trainable and not just an inference technique. According to the experiments, the method achieves better performance and computational savings as compared to other quantization method baselines. The results are admirably demonstrated on a variety of models, including CNN and RNN-based neural nets, as well as on several datasets in different domains, including ImageNet, CIFAR10, and PTB. We see the method seems to generalize across all of these.\n\nNevertheless, while I found this is very interesting work, I have a number of issues with the experiments, which I'll go into below. I feel this work is being released prematurely and could use some more polish to help sell the method better. Below are a few remarks and questions for the authors that would be helpful to be answered.\n\n* Why only report on ResNet-18? It would be far more useful to show numbers against ResNet-50. It would also be useful to show the non-quantized best results on these models and datasets.\n* I wish more effort had been spent to analyze the experiments. For example, I am not sure I understand the effects of the threshold on this method. What happens when it's set manually?\n* How exactly is computation cost savings calculated so crudely? If it uses B_avg, why not calculate the bitwidth per layer and sum things up? Using B_avg strikes me as being quite crude.\n* When the authors address runtime changes except in table 6, they changed their baseline to a vanilla ResNet-18 with dense weights. What are the runtime effects relative to ShuffleNet and ShiftNet?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlTMlMpFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Precision Gating, a novel mechanism to quantize neural network activations to reduce the average bitwidth, resulting in networks with fewer bitwise operations. The idea is to have a learnable threshold Delta that determines if an output activation should be computed in high or low precision, determined by the most significant bits of the value. Assuming that high activations are more important, these are computed at higher precision.\n\nI agree that the following three key contributions listed in the paper are (slightly re-formulated):\n1. Introducing Precision Gating (PG), the first end-to-end trainable method that enables dual-precision execution of DNNs and is applicable to a wide variety of network architectures.\n2. Precision gating enables DNN computation with a better average bitwidth to accuracy tradeoff than other state-ofthe-\nart quantization methods. Combined with its lightweight gating logic, PG demonstrates the potential to reduce DNN execution costs in both commodity and dedicated hardware.\n3. Unlike prior works that focus only on inference, precision gating achieves the same sparsity during back-propagation as forward propagation, which reduces the computational cost for both passes.\n\nThese contributions are novel and experimental evidence is provided for multiple networks and datasets. The paper is well-written and provides insightful figures to showcase the strengths of the present method. Related work is adequately cited. The paper does not contain much theory, but wherever possible equations are provided to illustrate in detail how the method works.\n\nExperimental results are shown for the datasets CIFAR-10 with ResNet-18 and ShiftNet-20, and ImageNet with ShuffleNet V2 0.5x. On both datasets, PG outperforms uniform quantization, PACT, Fix-Threshold and SeerNet in terms of top-1 accuracy and average bitwidth. \nWhat I am missing is information about the variability of results, since there are no error bars. Are the results averaged over multiple trials (if yes how many?), and is there a difference in variance between the methods? I realize that adding standard deviations to all results in the tables might be infeasible, but a qualitative statement would be interesting. In particular, the random initialization of the hb bits could play a bigger role than lb bits.\n\nThe two variants of PG, with and without sparse backpropagation are also investigated, showing that sparse backpropagation leads to more sparsity. To show that the resulting lower average bitwidth gained with PG leads to increased performance, the authors implement it in Python (running on CPU) and measure the wall clock time to execute the ResNet-18 model. Speedups $> 1$ are shown for every layer when using PG. Evidence from other papers is cited to argue that similar speedups are expected on GPUs.\n\nEven though at the moment it is unclear to me how statistically significant the results are, and I strongly recommend commenting on this in the paper, I think the idea of PG and the demonstrated benefits make the paper interesting enough to be accepted at ICLR.\n\nI also have a few questions that I could not get completely from the paper:\n1. I am a bit confused by what you call features. Fig. 2 shows by example how the method works for an input $I$. Is $I$, a single number, i.e. a single entry of your input vector, or do you mean the complete input vector?\n2. Could you give a bit more insight, how you tuned your hyperparameters, especially $\\delta$ and $\\sigma$?\n3. What exactly does e.g. $\\delta=-1$ mean? The network ideally should compute at high precision, when the result when only considering the most significant bits is above -1?\n\nFrom a hardware point of view, the paper focuses on GPU implementations. I would have hoped for a discussion of suitable custom hardware that could support PG most efficiently.\n\n\nMinor comments that I would be interested in but did not influence my score\n- It seems to me that on the top-left image of Fig. 3, one blue circle (the second largest) is too much? First part shows 8 dots, middle and right only seven?\n- Can you please cite a source that DNN activations have outliers (Sec. 3.4)?\n- You could also define e.g. one $\\delta$ and $\\Delta$ per layer, couldn't you? Would be interesting to see if  e.g. thinning out the precision over depth is possible / has advantages.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes an approach to accelerate network training by modifying the precision of individual weights, allowing a substantial speed up without a decrease in model accuracy. The magnitude of the activations determines whether it will be computed at a high or low bitwidth.

The reviewers agreed that the paper should be published given the strong results, though there were some salient concerns which the authors should address in their final revision, such as how the method could be implemented on GPU and what savings could be achieved.

Recommendation is to accept.
Decision: Accept (Poster)