Title: COMPRESSING DEEP NEURAL NETWORKS WITH LEARNABLE REGULARIZATION

Abstract: We consider learning and compressing deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision DNNs, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass for computing the loss function. Thus, the gradient descent becomes suboptimal, and accuracy loss follows. In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularization. In particular, we propose using a learnable regularization coefficient with the MSQE regularizer to reinforce the convergence of high-precision weights to their quantized values. Furthermore, we investigate how partial L2 regularization can be employed for weight pruning in a similar manner. Finally, combining weight pruning, quantization, and entropy coding, we establish a low-precision DNN compression pipeline. In our experiments, the proposed method produces low-precision MobileNet and ShuffleNet models on ImageNet classification with the state-of-the-art compression ratios. Moreover, we examine our method for image super resolution DNNs to produce low-precision models at negligible performance loss.

1. INTRODUCTION:

Deep neural networks (DNNs) have achieved performance breakthroughs in many of computer vision tasks (LeCun et al., 2015) . The revolutionary progress of deep learning comes with overparametrized multi-layer network architectures, and nowadays millions or tens of millions parameters in more than one hundred layers are not exceptional anymore. Network compression for efficient inference is of great interest for deployment of large-size DNNs on resource-limited platforms such as battery-powered mobile devices (Sze et al., 2017; Cheng et al., 2018) . In such resourceconstrained hardwares, not only memory and power are limited but also basic floating-point arithmetic operations are in some cases not supported. Hence, it is preferred and sometimes necessary to deliver compressed DNNs of low-precision fixed-point weights and activations (feature maps).

In this paper, we propose a network compression scheme that produces low-precision DNNs through learning with regularization. In particular, we let the regularization coefficient learnable, instead of treating it as a fixed hyper-parameter, to make a smooth and efficient transition of a high-precision model into a sparse quantized model. The proposed compression pipeline is summarized in Figure 1 .

• For weight pruning, we utilize partial L2 regularization to make a portion of small-value weights tend to zero so we can safely prune them after training at negligible accuracy loss. • For weight quantization, we regularize (unpruned) weights with another regularization term of the mean squared quantization error (MSQE). In this stage, we also quantize the activations (feature maps) of each layer to mimic low-precision operations at inference time. The quantization bin sizes for weights and activations are optimized to minimize their MSQEs in each layer. • The pruned and quantized model is converted into a low-precision model and its low-precision weights are further compressed in size with lossless entropy coding such as Huffman coding and universal source coding algorithms (e.g., see Cover & Thomas, 2012, Section 11. 3) for memoryefficient deployment.

It is difficult to train low-precision DNNs with standard gradient descent since the learning rate is typically set to be a small floating-point value but low-precision weights cannot be adjusted in fine resolution. To enable training low-precision DNNs, a series of papers on binary neural networks suggests utilizing high-precision shadow weights to accumulate the negatives of the gradients in fine resolution, while the gradients are obtained from the network loss function calculated with binarized (or quantized) weights (Courbariaux et al., 2015; Lin et al., 2016; Hubara et al., 2016) . That is, highprecision weights are quantized in the forward pass, but the quantization function is replaced with the linear function of slope 1 in the backward pass for gradient descent. This approximate gradient descent algorithm is further refined in subsequent works (Rastegari et al., 2016; Zhou et al., 2016; Zhu et al., 2017; Cai et al., 2017; Hou et al., 2017; Hou & Kwok, 2018; Gysel et al., 2018; Zhou et al., 2018) .

The mismatch in the forward and backward passes results in sub-optimal gradient descent that causes accuracy loss. The mismatch is more problematic for the models using lower-precision weights and activations, since the quantization error is more significant. There have been some attempts to reduce this mismatch by introducing better backward pass approximation, e.g., using clipped ReLU and logtailed ReLU instead of the linear function (e.g., see Cai et al., 2017) . Our approach is different from these efforts. We use regularization to steer high-precision weights to converge to their quantized values so that the mismatch between high-precision weights and quantized weights becomes smaller instead of enhancing the backward pass approximation.

We reduce the mismatch between high-precision weights and quantized weights with MSQE regularization. In particular, we propose making the regularization coefficient learnable. Using learnable regularization, high-precision weights are reinforced to converge to their quantized values gradually in training. We empirically show that our learnable regularization yields more accurate lowprecision models than the conventional regularization with a fixed regularization coefficient. Our scheme is different from the loss-aware weight quantization in Hou et al. (2017) ; Hou & Kwok (2018) , where optimization solutions using the proximal Newton algorithm are presented to minimize the loss function under the constraints of low-precision weights, which is however impractical for large-size networks due to the prohibitive computational cost to estimate the Hessian matrix of the loss function. No regularization is considered in Hou et al. (2017) ; Hou & Kwok (2018) .

Weight pruning curtails redundant weights completely from DNNs so that one can skip computations for pruned ones. Some of successful pruning algorithms can be found in Han et al. (2015) ; Lebedev & Lempitsky (2016) ; Wen et al. (2016) ; Guo et al. (2016) ; Lin et al. (2017) . In this paper, we discuss how partial L2 regularization can be used for weight pruning. Combining weight pruning, quantization, and entropy coding, as shown in Figure 1 , we achieve the state-of-the-art compression results for low-precision MobileNet (Howard et al., 2017) and ShuffleNet (Zhang et al., 2018) We finally note that reinforcement learning has been proposed as a promising methodology to search for quantized and/or compressed models that satisfy certain latency, energy, and/or model size requirements, given hardware specifications to deploy the models (He et al., 2018; Wang et al., 2019) . 

2. LOW-PRECISION DNN MODEL:

We consider low-precision DNNs that are capable of efficient processing in the inference stage by using fixed-point arithmetic operations. In particular, we focus on the fixed-point implementation of convolutional and fully-connected layers, since they are the dominant parts of computational costs and memory requirements in DNNs (see Sze et al., 2017 , Table II ).

The major bottleneck of efficient DNN processing is known to be in memory accesses (Sze et al., 2017, Section V-B) . Horowitz provides rough energy costs of various arithmetic and memory access operations for 45 nm technology (see Horowitz, 2014, Figure 1.1.9) , where we can find that memory accesses typically consume more energy than arithmetic operations, and the memory access cost increases with the read size. Hence, for example, deploying binary models, instead of 32-bit models, it is expected to reduce energy consumption by 32× at least, due to 32 times fewer memory accesses.

Low-precision weights and activations basically stem from linear quantization (e.g., see Gersho & Gray, 2012, Section 5.4) , where quantization bin boundaries are uniformly spaced and quantization output levels are the midpoints of bin intervals. Quantized weights and activations are represented by fixed-point numbers of small bit-width. Scaling factors (i.e., quantization bin sizes) are defined in each layer for fixed-point weights and activations, respectively, to alter their dynamic ranges.

Figure 2 shows the fixed-point design of a general convolutional layer consisting of convolution, bias addition and non-linear activation. Fixed-point weights and input feature maps are given with common scaling factors δ l and ∆ l , respectively, where l is the layer index. Then, the convolution operation can be implemented by fixed-point multipliers and accumulators. Biases are added, if present, after the convolution, and then the output is scaled properly by the product of the scaling factors for weights and input feature maps, i.e., δ l ∆ l , as shown in the figure. Here, the scaling factor for the biases is specially set to be δ l ∆ l so that fixed-point bias addition can be done easily without another scaling. Then, a non-linear activation function follows. Finally, the output activations are fed into the next layer as the input.

Using rectified linear unit (ReLU) activation, two scaling operations across two layers, i.e., scaling operations by δ l ∆ l and 1/∆ l+1 , can be combined into one scaling operation by δ l ∆ l /∆ l+1 before (or after) ReLU activation. Furthermore, if the scaling factors are power-of-two numbers, then one can even implement scaling by bit-shift. Similarly, low-precision fully-connected layers can be implemented by replacing convolution with matrix multiplication in the figure.

3. LEARNABLE REGULARIZATION FOR LOW-PRECISION DNNS:

In this section, we present the regularizers that are utilized to learn pruned and quantized DNNs of low-precision weights and activations. We first define the quantization function. Given the number of bits, i.e., bit-width n, the quantization function yields

EQUATION

where x is the input and δ is the scaling factor; round(x) = sign(x) |x| + 0.5 and clip n (x) = min(max(x, -2 n-1 ), 2 n-1 -1), where x is the largest integer smaller than or equal to x. For ReLU activation, the ReLU output is always non-negative, and thus we use the unsigned quantization function given by

Q + n (x; δ) = δ clip + n (round(x/δ)),

(2) for n ≥ 1, where clip + n (x) = min(max(x, 0), 2 n -1).

3.1. REGULARIZATION FOR WEIGHT QUANTIZATION:

Consider a general non-linear DNN model consisting of L layers. Let W 1 , W 2 , . . . , W L be the sets of high-precision weights in layers 1 to L, respectively. For notational simplicity, we let A L 1 = A 1 , A 2 , . . . , A L for any symbol A. We define the MSQE regularizer for weights of all L layers as

EQUATION

where n is the bit-width for quantized weights, δ l is the scaling factor (i.e., quantization bin size) for quantized weights, and |W l | is the number of weights in layer l. We assumed that bit-width n is the same for all layers, just for notational simplicity, but it can be easily extended to more general cases such that each layer has a different bit-width.

Including the MSQE regularizer in (3), the cost function to optimize in training is given by

EQUATION

for λ > 0, where, with a slight abuse of notation, Q n (W L 1 ; δ L l ) denotes the set of quantized weights of all L layers, E(X ; Q n (W L 1 )) is the target loss function evaluated on the training dataset X using the quantized weights, and λ is the regularization coefficient. We set the scaling factors δ L 1 to be learnable parameters and optimize them along with weights W L 1 . Remark 1. We clarify that we use high-precision weights in the backward pass for gradient descent by replacing approximately the quantization function Q n with the linear function of slope 1. In the forward pass, we use quantized weights and activations, and the main target network loss function E is also calculated with the quantized weights and activations to mimic the low-precision inferencestage loss. Hence, the final trained models are low-precision models, which can be operated on low-precision fixed-point hardwares in inference with no accuracy loss. Note that our method still has the gradient mismatch problem, similar to the existing approaches (see Section 1). However, by adding the MSQE regularizer, we encourage high-precision weights to converge to their quantized values so that we reduce the mismatch.

Learnable regularization coefficient. The regularization coefficient λ in (4) is a hyper-parameter that controls the trade-off between the loss and the regularization. It is conventionally fixed ahead of training. However, searching for a good hyper-parameter value is usually time-consuming. Hence, we propose the learnable regularization coefficient, i.e., we let the regularization coefficient be another learnable parameter.

We start training with a small initial value for λ, i.e., with little regularization. However, we promote the increase of λ in training by adding a penalty term for a small regularization coefficient, which is -α log λ for λ, α > 0, in the cost function (see ( 5)). The increasing coefficient λ reinforces the convergence of high-precision weights to their quantized values for reducing the MSQE. It consequently alleviates the gradient mismatch problem that we mentioned earlier (see Remark 1). We compare the convergence curves with learnable and fixed regularization coefficients.

The cost function in ( 4) is altered into

EQUATION

where we introduce a new hyper-parameter α, while making the regularization coefficient learnable. We note that the trade-off between the loss and the regularization is now actually controlled by the new parameter α instead of λ, i.e., the larger the value of α, eventually the more the regularization. This transfer is however beneficial since the new parameter α does not directly impact either the loss or the regularization, which leads to a smooth transition of high-precision weights to their quantized values. For gradient descent, we need the gradients of (5) with respect to weights, scaling factors and the regularization coefficient, respectively, which are provided in Appendix A.1.

Learnable versus fixed regularization coefficients. In Table 1 , we compare the performance of quantized ResNet-18 (He et al., 2016 ) models for ImageNet ILSVRC 2012 dataset (Russakovsky et al., 2015) when we use learnable and fixed regularization coefficients, respectively. Observe that the proposed learnable regularization method outperforms the conventional regularization method with a fixed coefficient in various low-precision settings. In Figure 4 , we compare the convergence curves when learnable and fixed regularization coefficients are used, respectively. Using a learnable regularization coefficient, the MSQE regularization term decreases (although there is a bump in the middle) while λ increases in training. However, using a fixed regularization coefficient, the MSQE regularization term saturates and even increases after some point as training goes on, which implies that the mismatch of the forward and backward passes is not resolved. The unresolved mismatch eventually turns into accuracy loss, as shown in the figure.

Evolution of weight histogram. Figure 3 presents an example of how high-precision weights are gradually quantized by our regularization scheme. We plotted weight histogram snapshots captured at the second convolutional layer of the MNIST LeNet-5 model1 while a pre-trained model is quantized to a 4-bit fixed-point model. The histograms in the figure from the left to the right correspond to 10k, 21k, 23k, and 30k batch iterations in training, respectively. Observe that the weight distribution gradually converges to the sum of uniformly spaced delta functions and all high-precision weights converge to quantized values completely in the end.

Comparison to soft weight sharing. In soft weight sharing (Nowlan & Hinton, 1992; Ullrich et al., 2017), a Gaussian mixture prior is assumed, and the model is regularized to form groups of weights that have similar values around the Gaussian component centers (e.g., see Bishop, 2006, Section 5.5.7). The learnable regularization coefficient can be related to the learnable variance in the Gaussian mixture prior. However, our weight regularization method is different from soft weight sharing since we consider linear quantization and optimize quantization bin sizes, instead of optimizing individual Gaussian component centers for non-linear quantization. We employ the simple MSQE regularization term for quantization, so that it is applicable to large-size DNNs. Note that soft weight sharing yields the regularization term of the logarithm of the summation of exponential functions, which is sometimes too complex to compute for large-size DNNs. In our method, the additional computational complexity for MSQE regularization is not expensive. It only scales in the order of O(N ), where N is the number of weights. Hence, the proposed scheme is easily applicable to the state-of-the-art DNNs with millions or tens of millions weights.

3.2. QUANTIZATION OF ACTIVATIONS:

We quantize the output activation (feature map) x of layer l for 1 ≤ l ≤ L and yield Q + m (x; ∆ l ), where Q + m is the quantization function in (2) for bit-width m and ∆ l is the learnable scaling factor for quantized activations of layer l.2 Similar to (3), we assumed that activation bit-width m is the same for all layers, but this constraint can be easily relaxed to cover the cases where each layer has a different bit-width. We assumed ReLU activation and used the unsigned quantization function Q + m while we can replace Q + m with Q m in case of general non-linear activation. We optimize ∆ l by minimizing the MSQE for activations of layer l, i.e., we minimize where A l is the set of activations of layer l for 1 ≤ l ≤ L. In the backward pass, we first perform gradient descent for weights and their scaling factors using the loss function in ( 5), and then we update ∆ l with gradient descent using (6). We do not utilize (6) in gradient descent for weights.

EQUATION

Backpropagation through quantized activations: Backpropagation is not feasible through quantized activations analytically since the gradient is zero almost everywhere. For backpropagation through the quantization function, we adopt the straight-through estimator (Bengio et al., 2013) .

In particular, we pass the gradient through the quantization function when the input is within the clipping boundary. If the input is outside the clipping boundary, we pass zero.

3.3. REGULARIZATION FOR WEIGHT PRUNING:

For weight pruning, we propose using partial L2 regularization. In particular, given a target pruning ratio r, we find the r-th percentile of weight magnitude values. Assuming that we prune the weights below this r-th percentile value in magnitude, we define a L2 regularizer only for them as follows:

P r (W L 1 ) = 1 N L l=1 w∈W l |w| 2 1 |w|<θ(r) ,

where θ(r) is the r-th percentile of weight magnitude values, which is the threshold for pruning.

Adopting the learnable regularization coefficient as in (5), we have

EQUATION

The partial L2 regularizer encourages the weights below the threshold to move towards zero, while the other unregularized weights are updated to minimize the loss due to pruning. The threshold θ(r) is also updated at every iteration of training based on the instant weight distribution. We note that the threshold θ(r) decreases as training goes on since the regularized weights gradually converge to zero (see Figure 5 ). After finishing the regularized training, we finally have a set of weights clustered very near zero. The loss due to pruning these small-value weights is negligible.

4. EXPERIMENTS:

We evaluate the proposed low-precision DNN compression for ImageNet classification and image super resolution. Image super resolution is included in our experiments as a regression problem since its accuracy is more sensitive to quantization than classification. Note that Tensorflow Lite3 already supports a very efficient 8-bit weight and activation quantization tool for network development on mobile platforms. Thus, our experimental results focus on more extreme cases of quantization using less than 8 bits, where a more sophisticated algorithm is needed for smaller loss. The detailed experimental settings and some of the experimental results are provided in Appendix B due to the space constraints. We use FLP and FXP to denote the floating-point and fixed-point formats, respectively.

4.1. EXPERIMENTAL RESULTS:

ResNet-18 quantization on ImageNet classification. Figure 6 presents the accuracy of the lowprecision ResNet-18 (He et al., 2016 ) models obtained from our quantization method. For ablation For ablation study, we compare pruning-only results and pruning+quantization results with various low-precision setting. We also show the compression results with and without entropy coding, where we used bzip2 as a specific entropy coding scheme. study, we compare weight and activation quantization for various low-precision settings. The loss due to weight quantization is relatively less than the loss due to activation quantization, which is consistent with the results from DoReFa-Net (Zhou et al., 2016) . We compare our method to DoReFa-Net for AlexNet (Krizhevsky et al., 2012) quantization, which can be found in Appendix B.2. We also compare the low-precision models obtained with and without the power-of-two scaling constraint. In fixed-point computations (see Figure 2 ), it is more appealing for the scaling factors (i.e., quantization bin sizes) to be powers of two so they can be implemented by simple bit-shift, rather than with scalar multiplication. For power-of-two scaling factors, we perform rounding of scaling factors into their closest power-of-two integers in the forward pass, while the rounding function is replaced with the linear function of slope 1 in the backward pass. We observe small performance degradation due to the constraint of power-of-two scaling factors in our experiments.

MobileNet/ShuffleNet compression on ImageNet classification. We mainly evaluate our method to obtain compressed low-precision MobileNet (Howard et al., 2017) and ShuffleNet (Zhang et al., 2018) models for ImageNet classification. For MobileNet and ShuffleNet compression, we prune 50% weights from their pre-trained models as described in Section 3.3 so that the accuracy loss due to pruning is marginal. Then, we employ our weight and activation quantization method. After converting into sparse low-precision models, universal source coding with bzip2 (Seward, 1998) follows to compress the fixed-point low-precision weights. In Figure 7 , we compare the compression ratios of our scheme and the existing network compression methods in Han et al. (2016) ; Park et al. (2017) ; Tung & Mori (2018) ; Wang et al. (2019) . Our lowprecision network compression scheme shows comparable compression ratios to the state-of-the-art weight compression schemes. We emphasize that our compression scheme produces low-precision models of fixed-point weights and activations that support efficient inference of fixed-point operations, while the previous compression schemes, except the one from Wang et al. (2019) , produces quantized weights that are still floating-point numbers and thus floating-point operations are necessary to achieve the presented accuracy of them. The hardware-aware automated quantization in Wang et al. (2019) achieved impressive compression results by searching for a quantized model with "mixed" precisions for different layers with reinforcement learning, but not all hardwares support mixed precision arithmetic operations. Our method used a fixed precision for all layers assuming more limited hardware capability.

Image super resolution. The image super resolution problem is to synthesize a high-resolution image from a low-resolution one. The DNN output is the high-resolution image corresponding to the input low-resolution image, and thus the loss due to quantization is more prominent. We evaluate the proposed method on SRCNN (Dong et al., 2016) and cascade-trained SRCNN (CT-SRCNN) (Ren et al., 2018) for image super resolution. The objective image quality metric measured by the peak signal-to-noise ratio (PSNR) and the perceptual score measured by the structural similarity index (SSIM) (Wang et al., 2004 ) are compared for Set-14 image dataset (Zeyde et al., 2010) in Table 3 for 9-layer CT-SRCNN. Additional experimental results for 3-layer SRCNN and 5-layer CT-SRCNN can be found in Appendix B.2. Observe that our method successfully yields low-precision models of 8-bit weights and activations at negligible loss, and they are better than the results that we obtain with one of the previous works, Ristretto (Gysel et al., 2018) . It is interesting to see that the PSNR loss of using binary weights and 8-bit activations is 0.5 dB only.

5. CONCLUSION:

In this paper, we proposed a method to quantize deep neural networks (DNNs) by regularization to produce low-precision DNNs for efficient fixed-point inference. The proposed scheme alleviates the mismatch problem in the forward and backward passes of low-precision network training by using MSQE regularization. Moreover, we proposed a novel learnable regularization coefficient to reinforce the convergence of high-precision weights to their quantized values when using MSQE regularization. We also discussed how a similar regularization technique can be employed for weight pruning with partial L2 regularization.

We showed by experiments that the proposed quantization algorithm successfully produces lowprecision DNNs of binary weights for classification problems, such as ImageNet classification, as well as for regression and image synthesis problems, such as image super resolution. For MobileNet and ShuffleNet compression, we obtained sparse (50% weights are pruned) low-precision models of 5-bit weights and 8-bit activations with compression ratios of 7.13 and 6.79, respectively, at marginal accuracy loss. For image super resolution, we only lost 0.04 dB PSNR when using 8-bit weights and activations, instead of 32-bit floating-point numbers.

:

https://github.com/BVLC/caffe/tree/master/examples/mnist

We note that ∆ l is the scaling factor for activations of layer l whereas it denotes the scaling factor for input feature maps of layer l in Section 2 (see Figure2). This is just one index shift in the notation, since the output of layer l is the input to layer l + 1. We adopt this change just for notational simplicity.

https://www.tensorflow.org/lite

https://github.com/openimages/dataset

https://github.com/BVLC/caffe

https://github.com/ppwwyyxx/tensorpack/tree/master/examples/DoReFa-Net

Figures and Tables:

Figure 1: Our low-precision DNN compression pipeline. We utilize partial L2 regularization and MSQE regularization to transform a pre-trained high-precision model into a sparse low-precision model with fixed-point weights and activations. The low-precision weights are further compressed in size with lossless entropy source coding.

Figure 3: Weight histogram snapshots of the MNIST LeNet-5 second convolutional layer captured at different training batch iteration numbers while a pre-trained model is quantized to have 4-bit weights and activations with the proposed regularization method.

Figure 4: ResNet-18 model training convergence curves for binary weights and 2-bit activations. We compare the convergence curves with learnable and fixed regularization coefficients.

Figure 5: Weight histogram snapshots of the MNIST LeNet-5 at different training batch iteration numbers when trained from scratch with the partial L2 regularizer for 90% sparsity (r = 90).

Figure6: Ablation study of ResNet-18 quantization on ImageNet classification. We use "W: Weight precision" and "A: Activation precision" to denote weight and activation precisions, respectively. FLP and FXP stands for floating-point and fixed-point formats, respectively.

Figure 7: Comparison of our low-precision MobileNet and ShuffleNet compression results to the ones of the state-of-the-art network compression methods on ImageNet classification. We use "W: Weight precision" and "A: Activation precision" to denote weight and activation precisions used in the compressed models, respectively.

Comparison of the weight regularization methods with learnable and fixed regularization coefficients for ResNet-18 on ImageNet classification.

Table content: <table/>

)

Table content: <table><tr><td>68.1</td><td>88.4</td><td>68.1</td><td>88.3</td><td>68.1</td><td>68.1</td><td>88.3</td><td>88.4</td><td>67.4</td><td>67.5</td><td>87.9</td><td>88.1</td><td>60.6</td><td>59.8</td><td>83.7</td><td>83.0</td><td>61.3</td><td>59.2</td><td>83.7</td><td>82.5</td><td>60.2</td><td>58.9</td><td>83.2</td><td>82.3</td><td>55.6</td><td>50.6</td><td>79.6</td><td>76.1</td><td>38.9</td><td>33.1</td><td>65.4</td><td>59.7</td></tr><tr><td colspan="2">W:32-bit FLP,</td><td colspan="2">W:8-bit FXP,</td><td colspan="4">W:8-bit FXP,</td><td colspan="4">W:4-bit FXP,</td><td colspan="4">W:2-bit FXP,</td><td colspan="4">W:1-bit FXP,</td><td colspan="4">W:1-bit FXP,</td><td colspan="4">W:1-bit FXP,</td><td colspan="4">W:1-bit FXP,</td></tr><tr><td colspan="2">A:32-bit FLP</td><td colspan="2">A:32-bit FLP</td><td colspan="4">A:8-bit FXP</td><td colspan="4">A:4-bit FXP</td><td colspan="4">A:2-bit FXP</td><td colspan="4">A:8-bit FXP</td><td colspan="4">A:4-bit FXP</td><td colspan="4">A:2-bit FXP</td><td colspan="4">A:1-bit FXP</td></tr><tr><td/><td/><td/><td colspan="11">32-bit FLP scaling factors, Top-1 accuracy (%)</td><td/><td colspan="12">Power-of-2 scaling factors, Top-1 accuracy (%)</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="11">32-bit FLP scaling factors, Top-5 accuracy (%)</td><td/><td colspan="12">Power-of-2 scaling factors, Top-5 accuracy (%)</td><td/><td/><td/><td/><td/></tr></table>

Low-precision MobileNet and ShuffleNet compression results for ImageNet classification.

Table content: <table/>

quantization results for upscaling factor 3.

Table content: <table><tr><td>Model</td><td>Method</td><td>Weights</td><td colspan="5">Activations Set-14 PSNR (dB) Set-14 SSIM PSNR (dB) loss SSIM loss</td></tr><tr><td/><td>Pretrained model</td><td colspan="2">32-bit FLP 32-bit FLP</td><td>29.71</td><td>0.8300</td><td>-</td><td>-</td></tr><tr><td/><td/><td>8-bit FXP</td><td/><td>29.67</td><td>0.8288</td><td>0.04</td><td>0.0012</td></tr><tr><td>CT-SRCNN 9-layer</td><td>Ours</td><td>4-bit FXP 2-bit FXP</td><td>8-bit FXP</td><td>29.63 29.37</td><td>0.8285 0.8236</td><td>0.08 0.34</td><td>0.0015 0.0064</td></tr><tr><td/><td/><td>1-bit FXP</td><td/><td>29.20</td><td>0.8193</td><td>0.51</td><td>0.0107</td></tr><tr><td/><td colspan="2">Gysel et al. (2018)* 8-bit FXP</td><td>8-bit FXP</td><td>29.05</td><td>0.8065</td><td>0.74</td><td>0.0234</td></tr><tr><td>Bicubic</td><td>-</td><td>-</td><td>-</td><td>27.54</td><td>0.7742</td><td>-</td><td>-</td></tr><tr><td/><td/><td colspan="6">* from our experiments using their code at https://github.com/pmgysel/caffe.</td></tr></table>

Peer review (iclr2020):
This paper presents an approach for quantizing neural networks, in order to make them more efficient. Quantization is difficult because training a quantized net can lead to gradient mismatches that impair accuracy. In this paper, the authors circumvent this difficulty using a three step compression approach that relies on retraining with regularization. First, they prune the synaptic connections of a pre-trained un-compressed network after retraining with an additional L_2 regularizer. Second, they retrain the network with a regularizer that uses the mean-squared error between the weights and their nearest quantized value. Third, they re-train the network with a regularizer that uses the mean squared error between the activations and their nearest quantized value. In some cases they also then use a final compression algorithm to render the quantized network as compressed as possible. They show that the accuracy losses on ImageNet associated with this technique are not too bad, and result in comparable or even better accuracy when compared to the TensorFlow 8-bit quantization, even when using less than 8-bits. Ultimately, this is a good paper. It was easy to read and the ideas were sound. The major thing it is missing, in my opinion, is some analysis, particularly: 1) How bad is the impact on the gradient calculations of this technique compared to other approaches? This is ultimately a key claim in their paper, that they do a better job of following the gradient. But it is not directly demonstrated, only indirectly shown via good accuracy. An explicit comparison in the bias on gradient calculations between their technique and other techniques for quantization would be ideal. One possibility is to do something like show the angle between the weight updates given and the true gradient, for the different approaches. A direct demonstration of this central claim is an important missing factor in this paper. 2) Can any analytical guarantees, or even estimates, be given on the accuracy drop in the event of the gradient being well-followed? It would be nice to know how badly the quantization would affect performance even in the best case scenario, for comparison sake. Minor items: - Figure 2 doesn’t add much. It could be removed to make space. - It would be easier to see what is going on in Figure 5d if the y-axis range were reduced and the peak clipped. As it stands, it looks like all the weights are zeroed out. - Why is Figure 6 presented as a bar graph rather than a table?