Title: COMPUTATION-EFFICIENT QUANTIZATION METHOD FOR DEEP NEURAL NETWORKS

Abstract: Deep Neural Networks, being memory and computation intensive, are a challenge to deploy in smaller devices. Numerous quantization techniques have been proposed to reduce the inference latency/memory consumption. However, these techniques impose a large overhead on the training procedure or need to change the training process. We present a non-intrusive quantization technique based on re-training the full precision model, followed by directly optimizing the corresponding binary model. The quantization training process takes no longer than the original training process. We also propose a new loss function to regularize the weights, resulting in reduced quantization error. Combining both help us achieve full precision accuracy on CIFAR dataset using binary quantization. We also achieve full precision accuracy on WikiText-2 using 2 bit quantization. Comparable results are also shown for ImageNet. We also present a 1.5 bits hybrid model exceeding the performance of TWN LSTM model for WikiText-2.
3 ITERATIVE QUANTIZATION Choromanska et al. (2015) shows that minima of high quality (measured by test accuracy) for largesize networks occur in a well-defined band. Choromanska et al. (2015) also conjectured that training

1. INTRODUCTION:

Different variants of Deep Neural Networks have achieved state-of-the-art results in various domains from computer vision to language processing (Krizhevsky et al., 2012; Ren et al., 2015; Vaswani et al., 2017; Cho et al., 2014) . However, newer models are becoming more memory and computation intensive to achieve performance improvements. For example, winner of ILSVRC 2015 ResNet (He et al., 2016) increased the number of layers by over 4x to gain less than 2% Top-1 accuracy improvement on ImageNet (Russakovsky et al., 2015) . Compression techniques, such as knowledge distillation, pruning, low rank approximation and quantization, have been proposed to reduce the model size (Vapnik & Izmailov, 2015; Han et al., 2015; Sainath et al., 2013; Courbariaux et al., 2015) . These compression techniques are evolving the field of model compression towards the goal of deploying DNN models on mobile-phone and other embedded devices. Courbariaux et al. (2015) proposed the widely used technique for training quantized neural networks, where binary weights are used during forward and backward propagation, while full precision weights are preserved for accumulating gradients. Binary weights are approximated from full precision weights every iteration. (Zhou et al., 2018; 2017a) have proposed incremental quantization training procedure where the range for the weights is incrementally reduced. Choi et al. (2017) use Hessian weighted k-means clustering for quantization. Lee & Kim (2018) used iterative procedure of quantizing, de-quantizing and complete retraining of the full precision model, performed multiple times. All the techniques aimed to reduce the quantization error (error between full precision model and corresponding quantized model). However, most of these quantization techniques either adds extra set of hyper-parameters or modifies/lengthens the original training procedure.

Designing a neural network model consists of two main steps -choose a proper architecture/model given the characteristics of the task, and optimize the hyper-parameters for convergence and accuracy. Hyper-parameter search varies from number of layers in a model (Zoph et al., 2018) to learning rate (lr), batch-size (bs) combo (compare ResNet (He et al., 2016) and Inception (Szegedy et al., 2016) networks with altogether different hyper-parameter set). Courbariaux et al. (2015) requires updating the back-propagation procedure to train quantized networks. (Zhou et al., 2018; Lee & Kim, 2018) only every few steps (specifically once every 500 iterations for the experiments). • Maintained the total number of iterations and time required to train the quantized network compared to the full precision network. • Achieve full precision accuracy for WikiText-2 and CIFAR dataset with 2-bit and 1-bit quantization respectively. Present a hybrid 1.5 bits LSTM models for WikiText-2 outperforming TWN LSTM model. Achieve performance comparable to existing works for ImageNet.

2. RELATED WORK:

Quantization. Courbariaux et al. (2015) proposed the idea of training binary neural networks with quantized weights. Rastegari et al. (2016) introduced shared scaling factors to allow more range for binary values. (Hubara et al., 2016; Zhou et al., 2016; Lin et al., 2017; McDonnell, 2018; Hubara et al., 2018) built upon the training methodology along with introduction of binary activation units. Lee & Kim (2018) performs full precision retraining multiple times to train a quantized network. Ternary quantization was proposed (Zhu et al., 2017; Li et al., 2016; Wang et al., 2018) to mitigate the gap between full precision and quantized weight networks. Let W ∈ R k×c×f ×f represent a weight of a convolution layer l with total n elements where k, c, f represents output channels, input channels and size of the filter respectively. Rastegari et al. (2016) splits W into binary weight B ∈ {-1, +1} k×c×f ×f and scaling factor α ∈ R +k shared per output, where

B = sign(W) α = B,W /n (1)

obtained by minimizing W -αB 2 . Binary quantization is extended to ternary where B ∈ {-1, 0, +1} n×c×k×k . Ternary quantization introduces a threshold factor l to assign the ternary value to a weight. (Li et al., 2016; Zhu et al., 2017; Wang et al., 2018) have proposed various methodologies to evaluate the threshold. Lee & Kim (2018) performed ternary quantization by combining pruning and binary quantization.

Binary quantization was extended to multi-bit quantization using a greedy methodology by Guo et al. (2017) . For k-bit quantization, minimizing Ŵiα i B i for i th bit quantization resulted in,

B i = sign( Ŵi ) α i = B i , Ŵi /n where Ŵi = W - i-1 j=1 α j B j (2)

referred as the greedy approach. Greedy approach was improved by refined method, where α i is computed by using (( Xu et al. (2018) improved refined method by performing a binary search on the given refined α set and alternately evaluating α and B. Low precision networks have also been proposed to reduce the gap with quantized activation units (Zhuang et al. (2018) ). Quantization has also been applied to RNNs and Long Short Term Memory (LSTM) models as well (Hou et al. (2017) ; Guo et al. (2017) ; Zhou et al. (2017b) ; Xu et al. (2018) ; Lee & Kim (2018) ). We use greedy quantization in this work due to its simple operations (although alternating yields the better results at the cost of higher computation overhead). Next section describes our quantization training procedure in detail. using methods like stochastic gradient descent, simulated annealing converges to a minimum in the band. Minima in the band can have varying flatness, where flat minima have smaller error introduced to the accuracy upon adding distortion to the weight (Hochreiter & Schmidhuber (1995) ). However, exploring through multiple minima has been a challenging task. Simulated annealing1 (Kirkpatrick et al., 1983) explores through various minima, but does not aim to find wider minima. Motivated from simulated annealing, we propose a training technique to enable exploration of wider minimum among multiple minima. The technique allows escaping from relatively sharper minima and aims to find wider minima in the band. Our training procedure consists of two phases. Phase1 trains the full precision network with quantization. Phase2 fine-tunes the binary network obtained from phase1.

B T i B i ) -1 B T i W) T .

3.1. PHASE1: STEP TRAINING:

The goal of phase1 is to produce a full precision model with optimized B and reduced quantization error (error between full precision and quantized model). Phase1 does not modify the original training procedure. Instead, in addition to the original training, phase1 just adds an extra distortion step using quantization (referred as quantized-distortion step), performed once every few iterations. Applying quantized-distortion to the weights consists of 3 parts -quantize the weights of each layer (quantization), convert the quantized weights back to full precision format (de-quantization), and update the full precision model with the de-quantized weights. Quantized-distortion is performed once every Quantized Step Size (QSS) iterations. Original training procedure combined with quantization-distortion is referred as Step Training (Figure 1a ).

Step training is performed in phase1 until the convergence of training (in principle).

Full precision training of the network for QSS iterations explores the curvature of the local convex surface of the current local minimum. Applying quantized-distortion post-training moves the model to the quantized lattice point on the training contour. Suppose that the quantized lattice point exist outside the curvature around a sharp minimum. Then, the network escapes such sharper minima in phase1. In contrast to existing quantization training methods, step training does not store quantized weights. Instead, step training updates and replaces full precision weights with their quantized version every few iterations.

Quantization:

Step Size. QSS determines the amount of retraining to be done between two quantized-distortion steps. QSS needs to be big enough to compensate for the error added by the distortion and let the network explore the current local curvature. However, QSS should not be too large to diverge the weights far away from a nearby quantized lattice point. Comparing big vs small QSS -big QSS allows the weights to explore farther allowing the binary representation of the weights to change (weights need large amount of updates to change their sign). On the other hand, small QSS allows the training to exploit the current local curvature and fine-tune α.

We observe that for a training procedure with fixed learning rate, starting with a big QSS and reducing QSS over the training period results in better convergence. However, the same behavior can be approximated with a fixed QSS and a varying learning rate. Figure 1b shows the movement of accuracy using step training with step-wise reducing learning rate and fixed QSS for ResNet32 on CIFAR-10. Larger learning rate enables larger amount of updates (and hence more curvature exploration) given the same gradient from the back propagation (fluctuations in the accuracy). On the other hand, smaller learning rate helps exploit (fine-tune the parameters inside the current minimum) as shown by a smoother rise in accuracy. Hence, we use fixed QSS (500 iterations) in the rest of the manuscript, although one can use varying QSS for further fine-tuning.

Convergence of B.:

We observe that B converges earlier compared to α during step training. Such an observation is demonstrated in our experiment with step training for CIFAR-10 with ResNet32.

Figure 2 shows the movement of weight with step training2 . Initially, the sign bits of the weight flip frequently (with higher learning rate). However, with smaller learning rate (after 80K iterations for CIFAR-10), B do not change and only α is optimized.

Convergence of α. Let W be a tensor of full precision weights with n elements. W is quantized into B (binary tensor) and α (shared scaling factors).

Step training updates W every iteration.

On the other hand, (B, α) are calculated every QSS iterations. Let W denote the total update accumulated for W since the last distortion step. Let α denote the change between the new α and the α calculated at previous distortion step. For binary quantization, the updated α is given by-

α + α = 1 n |W + W| (3)

where |.| is the absolute function. Starting from a common absolute quantized value (α), the weights sharing the same α update independently (∀i, j w i , w j ∈ W, ∂w i /∂w j = 0). With large QSS, as the weights diverge from α, the update for α becomes inefficient and noisy. Although phase1 results in optimized B, phase1 does not completely optimize α (within the limited number of iterations). Need for improved convergence of α forms the motivation for phase2. Rastegari et al., 2016) (k is the number of output features, c is the number of inputs features and f is the filter size). There are total k α, each shared by c × f × f number of weights. Each output feature in the convolution layer has c × f × f weights. Thus, there is only 1 α shared by all the weights for an output feature. As the quantized weights can only take the value of {+α, -α}, all the inputs for an output feature can only be weighted by the same absolute factor α. Hence, the representative power of the quantized network is limited.

W ∈ R k×c×f ×f in a convolution layer, W is reshaped to 2D matrix W ∈ R k×cf f (

To alleviate this problem, we convert the 4D tensor W to 2D matrix differently. W is transposed to f × k × c × f and then reshaped to 2D matrix R f ×kcf (referred as skewed matrix). Next, each row of size k × c × f is split into k/f sub-rows. Each sub-row has a different α. Total number of α remains the same as above (k). Furthermore, the inputs for an output feature can now be weighted by f number of unique α. Note that some α will be shared among different output features as well.

In our experiments, skewed matrix shows better results. Section 5.1 shows the benefit in accuracy using the skewed matrix for quantization.

4. K-MEANS LOSS AND SHUFFLING:

This section aims to limit the divergence of weights from each other during training to get more accurate estimation of α. L 2 regularization (in the form of L 2 loss) is frequently used in training of neural networks. L 2 loss prevents the weights from exploding and suppresses the magnitude of the weights. However, L 2 loss does apply any restriction on the variance of the weights. As α is obtained using Equation 1, higher variance in weights results in higher quantization error. We aim to reduce quantization error by introducing L KM loss function to reduce the variance of the weights.

Let W i represent all the weights in a layer i. Let w ∈ W i be a subset of weights sharing common α (w are referred as clusters from now). The new loss is represented as:

L KM = c W 0 ∀w∈W w -avg(|w|) 2 (4)

where c is a constant, avg(.) computes the average of the inputs. L KM divides the weights W into different clusters w with common α and limits the divergence of weights from the cluster average of its absolute values α. L KM restricts the independent movement of weights. Similar with L 2 loss, a diverged weight increases the L KM . In addition, a diverged weight also shifts the cluster average, increasing the L KM loss further. Thus L KM encourages a lower variance in the weights with common α and improve quantization as a result. The constant factor c is set to be the same as weight decay rate (the constant for L 2 loss is reduced to mitigate the effect of L 2 ).

Shuffling. L KM can be extended for multiple tables, where a row of a quantized matrix has multiple shared α. With multiple tables, let w correspond to a subset of row, where the subset shares the common α. L KM helps in better approximation for α by forcing a predetermined group of weights to exhibit low variance. We could also achieve a better approximation for α by re-arranging the weights so that similar values are grouped together. Note that rearranging is applicable for DNNs with multiple tables only.

Let the weight matrix between two fully connected layers l i and l i-1 be represented by W i,i-1 . Two nodes in a layer l i given as l i j and l i k can be swapped by switching the rows j, k of weight matrix W i,i-1 and the columns j, k of W i+1,i . The layout of the weight matrix can be set to cluster the desired weights without impacting the output of the network. Swapping the nodes in layer l i , l i-1 swaps the rows and columns of W i,i-1 respectively. Thus, nodes in each layer can be swapped independently. K-means clustering is used to find an optimized configuration of weights in terms of grouping similar values together first. And then the nodes in the layer are swapped to enforce such configuration, reducing the quantization error. The methodology of finding and applying the optimal swapping configuration is termed as shuffling of nodes.

Because applying shuffling of nodes to the current layer requires a layer before and after the current layer, we introduce a shuffle layer in the start and end of the network to allow shuffling in first and last layer of the network. The shuffle layer stores the shuffle configuration and behaves as a mapping layer. The overhead of the shuffle layer is less than 1% of the model size (same as the size of bias in a layer). Shuffling is applied in the weight distortion step in phase1, when quantizing the weights.

5. EXPERIMENTS:

Experiments are performed with CNNs and RNNs to show the effectiveness of the proposed method. All the experiments are performed using Tensorflow (Abadi et al., 2016) using 2 Titan X GPUs. Full precision models for CNNs are obtained from tensorflow models repository3 , while RNN models are obtained from Verwimp et al. (2017) 4 . We quantize all the layers of the network unless specified otherwise. Iterative quantization training is performed on pre-trained full precision models. Greedy quantization using Equation 1 is used for all the experiments. Quantization Step Size is set to 500 constant throughout all the experiments.

5.1. CIFAR:

ResNet32 is trained on CIFAR-10 (Krizhevsky, 2009) for 90k iterations, where the first 60k iterations are performed using step training and remaining 30k iterations are performed with α training. 60% pruning rate is set for ternary quantization. Training ResNet32 using our proposed quantization training method does not incur any increase in training time.

Table 1 shows the improvement by combining the techniques discussed in previous sections in an incremental manner for ResNet32. We use the k α for quantization following Rastegari et al. (2016) as default quantization mode (k is the number of output features for a convolution layer). Different QSS schedules were tried where QSS starts with a high value and is reduced over the training procedure. QSS schedule produces better accuracy compared to fixed QSS for step training. Note that, however, α training eliminates the need to fine-tune over the QSS and achieves the same accuracy. Number of tables for skewed mode have been set to have the same model size as default mode (resulting in the same number of α).

Table 2 provides our final accuracy for CIFAR dataset with all the techniques combined using ResNet32 and WideResNet 28-10 models (70x bigger model size compared to ResNet32). Our model provides similar performance compared to TTQ (Zhu et al., 2017) using ResNet32. Our method achieves full precision accuracy for WideResNet 28-10 on both CIFAR-10 and CIFAR-100, compared to the results by McDonnell (2018) without changing the training procedure. We believe that WideResNet demonstrates smaller quantization error than ResNet32 because Li et al. (2018) reported that wider networks facilitates flatter minima.

5.2. WIKITEXT-2:

LSTM model with 1 layer consisting of 512 nodes is used for WikiText-2 (Merity et al., 2017) dataset, with same hyper-parameter settings as followed by Xu et al. (2018) . Performance is mea- sured with Perplexity Per Word metric (PPW). Full precision PPW is 100.2. Activation quantization requires quantization to be performed every iteration for inference, wiping out the speed up obtained with quantized weights for inference. Activation quantization slows down training as well. Thus, we use 32bit activations while 3bit activations used by Xu et al. (2018) . Our 2-bit alternating quantization (greedy quantization replaced with alternating quantization in our proposed method) reaches full precision PPW (Table 3 ). Hybrid LSTM. In the WikiText-2 LSTM model, Embedding and Softmax layer each forms over 45% of the full precision model size. Therefore, we selectively optimize the number of quantization bits for each layer to achieve higher compression rate. 1 bit quantization was found to be sufficient for Embedding layer. However, other layers required more number of bits. We fix 1bit quantization for Embedding layer (1 table per row), TWN for Softmax layer and vary the number of quantization bits for LSTM layer. Using 2bit for LSTM layer (1.53 bits per weight in total) provides PPW better than our TWN greedy model, with 25% smaller model size.

5.3. ABLATION STUDY:

Random Initialization. The distribution of bit-flips and convergence of accuracy for step training with randomly initialized model and with pre-trained model is observed to be similar (Figure 2 ).

The accuracy gap between Binary:

Step Training model with pre-trained model and BST without pre-trained model less than 0.1% (Table 1 ).

One-Step Quantization. We experiment the degradation in accuracy with just one-step quantization (no retraining). ResNet32 with full precision accuracy of 92.47% on CIFAR-10 produces 44.33% accuracy. To examine the potential of L KM , ResNet32 is again trained with random initialization in full precision mode with L KM (without any form of quantization). Although, the full Robustness. Table 4 shows the robustness of iterative quantization with varying QSS over 2x in the order of magnitudes. As explained in section 3.1, varying learning rate can provide the same functionality as varying QSS. As most of the modern neural networks use special learning rate policy (such as exponential decay, step-wise decay), the training procedure is overall robust to the choice of QSS. The simplicity of the algorithm and robustness to the added hyper-parameter facilitate quick adoption of our proposed technique.

5.4. IMAGENET:

Full precision ResNet18 is trained on ImageNet (Russakovsky et al., 2015) following the base repository 3 with batch size of 256. Our binary model reaches 60.6% Top1 accuracy compared to full precision accuracy of 69.6% (Table 5 ). Our model shows comparable accuracy compared to existing quantization methods. We believe our model can reach higher accuracy by using layer-by-layer quantization as done by Zhou et al. (2018) .

5.5. QUANTIZATION OVERHEAD:

Let training time per iteration be defined as the combined time to perform forward-propagation and back-propagation on a batch of data. We evaluate the overhead of performing a quantization step once relative to the defined training time per iteration. All the timings are averaged over 1000 iterations, averaged over ResNet32 and WideResNet. We observed that overhead of using greedy quantization is the lowest (8% and 12% of the training time for 1bit and 2bit quantization). More sophisticated quantization methods using regression or iterative procedures, namely refined and alternating quantization, have overhead of 5x and 40x respectively over the training time. compares the benefit of using these quantization methods, where alternating quantization shows the best performance despite the biggest overhead. As our training method, unlike existing methods, performs quantization once every 500 iterations, the overhead of the quantization is reduced by 500x. As a result, the overhead of the most expensive quantization remains to be 10% of the training time.

6. CONCLUSION:

In this work, we have presented an iterative quantization technique performing quantization once every few steps, combined with binary model α training.

Step training explores flatter minima while escaping sharp minima and α training performs exploitation of the chosen minima. We also presented a loss function L KM which allows weights to be adjusted for improved quantization.

We demonstrated full precision accuracy recovery with CIFAR and WikiText-2 dataset with our quantized models. We also presented a hybrid model with 1.5 bits performing better than the our TWN model. cropping and random horizontal flipping. However, validation data was pre-processed with 1 single central crop.

Step-wise decay learning rate schedule was followed with initial learning rate of 0.1 and decayed at epochs 30, 60, 80 and 90 by a factor of 0.1. The complete training procedure was performed for 100 epochs with a batch size of 256. Momentum training optimizer was used for training with momentum set 0.9. 0.0005 was set as weight decay rate. Quantization Step Size was set as 500 during step training.

B.4 BATCH NORMALIZATION:

Batch normalization (Ioffe & Szegedy, 2015) parameters (µ and σ) are updated using moving average. Consequently, the effect of quantized-distortion performed even 100 iterations earlier would have less than 1% effect on the BN parameters (with momentum=0.99). As a result, BN parameters are not suited well for quantized model and results in drop in evaluation accuracy for the quantized model. To avoid the drop in evaluation accuracy by BN, the BN parameters are re-evaluated over 1 train epoch (keeping the other parameters fixed) before performing evaluation for the phase1. Phase2 does not require any special care for batch normalization as there is no distortion step. Error Gap (Goodfellow et al., 2015; Keskar et al., 2017) . Q i denote the weight set after performing quantizeddistortion for the i th time while performing step training. F P i correspond to the full precision weight set just before performing the i th quantized-distortion. The plot is for cross entropy along a line segment containing the two points. Specifically for a ∈ [0, 1], we plot f (aQ i + (1 -a)F P i ).

EQUATION

The same is plotted for F P i and Q i+1 . Step training is performed with 160k iterations, with learning rate decayed by 0.1 every 40k steps to investigate the effects of 4 different learning rates. All the quantized points are sampled randomly at different iterations at different learning rate. Q 1 is obtained at learning rate 0.1, Q 2 at 0.01, Q 3 at 0.001 and Q 4 at 0.0001. (b) Starting from these quantized weight sets, the model is retrained using full precision training method for the remainder of the iterations. This retraining gives us F P 1 , F P 2 , F P 3 , F P 4 . The rise in the loss function along the path between two full precision points shows the existence of full precision weights in different local minimum. 

:

Simulated Annealing explores neighbors of the current solution in a randomized order. Worse neighboring solutions are also explored with relatively high probability (temperature) in the start (exploration). Temperature is reduced over time and later, only nearby better solutions are explored (exploitation).

All the figures are presented with 160k iterations, with learning rate decayed by 0.1 every 40k steps to investigate the effects of 4 different learning rates.

https://github.com/tensorflow/models/

https://github.com/lverwimp/tf-lm

Figures and Tables:

require very long training time because of multiple iterations of training and extra introduced hyper-parameters. Over time, focus on reducing the model size and the corresponding inference latency has led to either lengthening or major modifications to training procedure. Our Proposed quantization technique addresses these issues resulting in easy adoption of our technique. Our contributions include but are not limited to • A simple quantization training method based on re-training without requiring major modifications to the original training procedure. Training consists of two phases: phase1 trains the full precision model (with quantization) and phase2 trains the binary model constructed by phase1. • Reduce the overhead of expensive quantization techniques as quantization is performed

Figure 1: (a) Quantization Training algorithm (b) Convergence of accuracy using step training (Phase1) for ResNet32 on CIFAR-10 dataset.

Figure 2: (left) Histogram of bit flips for all the weights of ResNet32 for CIFAR-10. (right) weights flipping their signs over the course of step training. Weight is randomly chosen from layer 30 of ResNet32 for CIFAR-10. Larger learning rate allows for more exploration and flipping of weights while small learning rate allows for fine-tuning and final convergence.

Figure 3: Comparing different skewed and default quantization mode(Rastegari et al., 2016) with multiple tables. Skewed quantization mode converts a 4d tensor W ∈ R k×c×f ×f into R f ×kcf , and default quantization mode convert into R k×cf f (where k is number of output features, c is number of input features, fxf is the filter size). For a kernel of shape 128x128x3x3, skewed mode with 42 tables has a memory footprint equivalent to default quantization mode.

Figure 5: Convergence of weight when training using Step Training. Compares two scenarios where (a) bit flips once and (b) bit does not flips during the course of step training. Step training was performed for ResNet32 using CIFAR-10 dataset.

Figure 6: (a) Shows the convergence of quantization error with decrease in learning rate. The distance between consecutive quantization weights (QSS iterations apart) and distance between corresponding full precision weights is also shown. Distance between consecutive quantization weights is directly correlated with the number of weight flips (Figure 2). (b) Shows the movement of loss with step training for ResNet32 using CIFAR-10 dataset. The loss rises for the last learning rate as distortion causes more damage compared what training with the small learning rate can repair.

Figure 7: (a) Shows the same behavior of histogram of total bit flips for step training with or without pre-trained model for ResNet32 with CIFAR-10 dataset. (b) Parametric 1-D plots as described in(Goodfellow et al., 2015;Keskar et al., 2017). Q i denote the weight set after performing quantizeddistortion for the i th time while performing step training. F P i correspond to the full precision weight set just before performing the i th quantized-distortion. The plot is for cross entropy along a line segment containing the two points. Specifically for a ∈ [0, 1], we plot f (aQ i + (1 -a)F P i ). The same is plotted for F P i and Q i+1 .

Parametric 1D plot for full precision weights.

Figure 8: (a) Parametric 1-D plots quantized weight sets at different learning rates while performing step training.Step training is performed with 160k iterations, with learning rate decayed by 0.1 every 40k steps to investigate the effects of 4 different learning rates. All the quantized points are sampled randomly at different iterations at different learning rate. Q 1 is obtained at learning rate 0.1, Q 2 at 0.01, Q 3 at 0.001 and Q 4 at 0.0001. (b) Starting from these quantized weight sets, the model is retrained using full precision training method for the remainder of the iterations. This retraining gives us F P 1 , F P 2 , F P 3 , F P 4 . The rise in the loss function along the path between two full precision points shows the existence of full precision weights in different local minimum.

Figure 9: Distribution of weights of the WikiText-2 LSTM model for full precision and quantized trained model.

construct the full precision model. Phase2 is faster compared to phase1 due to fewer training parameters, use of binary weights, and no quantized-distortion step. Phase2 is performed with a smaller learning rate after the bit-flips do not occur anymore in phase1. Similar to phase1, phase2 also uses the original training procedure but with fewer number of trainable parameters.In the complete training procedure, we first perform phase1 followed by phase2. The trained binary model at the end of phase2 represents the output of the complete training procedure. This complete training procedure uses the same number of iterations as that in the original training procedure. As a result, the total training time combining phase1 and phase2 is equivalent to the original full precision training time.3.3 SPECIAL CARE FOR CNNSThis section compares different ways to apply quantization on a 4D tensor kernel. Quantization for a 2D weight matrix W with n elements outputs a binary 2D matrix B ∈ {-1, +1} n and shared scaling factor α per row of W . All the weights in a row share the same α, where α ∈ α. Further, a row in the matrix can be split into t sub-rows (referred as tables), where each table has a different α. For quantizing a 4D tensor kernel

Table content: <table/>

Accuracy improvement by each method incrementally for ResNet32 on CIFAR-10.

Table content: <table><tr><td/><td colspan="2">Config</td><td/><td>Accuracy %</td><td/></tr><tr><td colspan="2">Full Precision</td><td/><td/><td>92.47</td><td/></tr><tr><td colspan="3">Binary Step Training (BST)</td><td/><td>88.18</td><td/></tr><tr><td colspan="3">BST without pre-trained</td><td/><td>88.09</td><td/></tr><tr><td colspan="3">Ternary Step Training (TST)</td><td/><td>89.27</td><td/></tr><tr><td colspan="2">TST + QSS schedule</td><td/><td/><td>90.45</td><td/></tr><tr><td colspan="2">TST + α training</td><td/><td/><td>90.4</td><td/></tr><tr><td colspan="2">TST + skewed matrix</td><td/><td/><td>91.3</td><td/></tr><tr><td colspan="3">TST + skewed matrix + L KM</td><td/><td>91.8</td><td/></tr><tr><td colspan="4">TST + skewed matrix + L KM + α training</td><td>92.36</td><td/></tr><tr><td colspan="6">Table 2: Accuracy Comparison for CIFAR using ResNet32 and WideResNet for TWN and binary</td></tr><tr><td colspan="6">models. TTQ: Zhu et al. (2017) (TWN model), Wide-1b: McDonnell (2018) (binary model)</td></tr><tr><td>Config</td><td/><td/><td>Accuracy %</td><td/><td/></tr><tr><td colspan="6">CIFAR-10 ResNet32 CIFAR-10 WideResNet CIFAR-100 WideResNet</td></tr><tr><td>Ours</td><td>TTQ</td><td>Ours</td><td>Wide-1b</td><td>Ours</td><td>Wide-1b</td></tr><tr><td>Full Precision 92.47</td><td>92.33</td><td>95</td><td>95.77</td><td>78.3</td><td>81.37</td></tr><tr><td>Binary Model 92.36</td><td>92.37</td><td>95.02</td><td>95.54</td><td>78.3</td><td>81.06</td></tr></table>



Table content: <table/>

Comparison of Perplexity Per Word (PPW) for LSTM models on WikiText-2 dataset. Multitable models use 1 quantization bit with multiple tables (8 tables (8 α per row) for Embedding and Softmax layer, 16 tables for LSTM layer). Hybrid models use 1 bit quantization for Embedding layer and TWN quantization for Softmax layer. 2 to 4 quantization bits for LSTM layer provides 1.53 to 1.65 bits per weight models. Results forGuo et al. (2017) are taken fromXu et al. (2018).

Table content: <table><tr><td>2-bit models</td><td>PPW</td><td>1.5-bits models</td><td>PPW</td><td>Hybrid models</td><td>PPW</td></tr><tr><td colspan="2">Guo et al. (2017) 105.8</td><td>Multi-table</td><td colspan="2">117.13 1.53 bit</td><td>108.1</td></tr><tr><td>Xu et al. (2018)</td><td>102.7</td><td>Multi-table + L KM</td><td colspan="2">115.26 1.6 bit</td><td>105.46</td></tr><tr><td>Our Greedy</td><td colspan="4">104.15 Multi-table + Shuffle 116.08 1.65 bit</td><td>103.58</td></tr><tr><td>Our Alternating</td><td>100.3</td><td>TWN</td><td>115.05</td><td/><td/></tr></table>

Robustness of Step Training to QuantizationStep Size for CIFAR-10 with ResNet32. Accuracy varies within a range of 3% with QSS ranging from 10 to 2500. 8%, one-step quantization accuracy goes up to 76.32%. Increasing the regularization constant for L KM yields the one-step quantization accuracy as 84.51%.

Table content: <table><tr><td>Quantization Step Size</td><td>10</td><td>50</td><td>100</td><td>500</td><td>1000 2500 5000 10000</td></tr><tr><td>Accuracy %</td><td colspan="5">86.77 88.03 88.97 89.15 88.88 86.52 83.01 78.54</td></tr><tr><td colspan="2">precision accuracy drops to 91.</td><td/><td/><td/><td/></tr></table>

Accuracy Comparison for Imagenet using ResNet18 for 1 bit quantization.

Table content: <table><tr><td>Config</td><td>Top1 Accuracy</td></tr><tr><td>Full Precision</td><td>69.6</td></tr><tr><td>Li et al. (2016)</td><td>57.5</td></tr><tr><td>Dong et al. (2017)</td><td>58.36</td></tr><tr><td>Our binary model</td><td>60.6</td></tr><tr><td>Rastegari et al. (2016)</td><td>60.8</td></tr><tr><td>Zhou et al. (2018)</td><td>64.72</td></tr></table>

Peer review (iclr2019):
 This paper proposes a method based on re-training the full-precision model and then optimizing the corresponding binary model. It consists of two phases: (1) the full-precision model training where the quantization step is introduced through QSS to train the network, and (2) fine tuning of quantized networks, where the trained network was converted into a binary model. In addition, using the skewed matrix for quantization improves the accuracy. Then a loss function based on the k means form is used to normalize the weight for reducing the quantization error. Quantization experiments for CNNs or LSTMs have been conducted on CIFAR10, CIFAR100, IMAGENET, and WikiText-2 dataset. This paper has been presented clearly. However, it can be improved by introducing the motivation of the tricks(e.g. skewed matrix and loss related to k-means ) used for quantization. In the experiments, the precision improvement on the CIFAR and ImageNet dataset performs worse than some competitors. For example, the precision of the proposed method was significantly worse than Zhou et al, 2018 on ImageNet. It is better to analyze the reason. In addition, as claimed from the introduction, the contribution of this paper was to reduce the overhead of expensive quantization. However, no experimental results on computation time and parameter size have been shown.