Title: TRAINED TERNARY QUANTIZATION

Abstract: Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it's as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16× smaller than fullprecision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.

1. INTRODUCTION:

Deep neural networks are becoming the preferred approach for many machine learning applications. However, as networks get deeper, deploying a network with a large number of parameters on a small device becomes increasingly difficult. Much work has been done to reduce the size of networks. Halfprecision networks (Amodei et al., 2015) cut sizes of neural networks in half. XNOR-Net (Rastegari et al., 2016) , DoReFa-Net (Zhou et al., 2016) and network binarization (Courbariaux et al.; 2015; Lin et al., 2015) use aggressively quantized weights, activations and gradients to further reduce computation during training. While weight binarization benefits from 32× smaller model size, the extreme compression rate comes with a loss of accuracy. Hubara et al. (2016) and Li & Liu (2016) propose ternary weight networks to trade off between model size and accuracy.

In this paper, we propose Trained Ternary Quantization which uses two full-precision scaling coefficients W p l , W n l for each layer l, and quantize the weights to {-W n l , 0, +W p l } instead of traditional {-1, 0, +1} or {-E, 0, +E} where E is the mean of the absolute weight value, which is not learned. Our positive and negative weights have different absolute values W p l and W n l that are trainable parameters. We also maintain latent full-precision weights at training time, and discard them at test time. We back propagate the gradient to both W p l , W n l and to the latent full-precision weights. This makes it possible to adjust the ternary assignment (i.e. which of the three values a weight is assigned).

Our quantization method, achieves higher accuracy on the CIFAR-10 and ImageNet datasets. For AlexNet on ImageNet dataset, our method outperforms previously state-of-art ternary network (Li & Liu, 2016 ) by 3.0% of Top-1 accuracy and the full-precision model by 1.6%. By converting most of the parameters to 2-bit values, we also compress the network by about 16x. Moreover, the advantage of few multiplications still remains, because W p l and W n l are fixed for each layer during inference. On custom hardware, multiplications can be pre-computed on activations, so only two multiplications per activation are required.

2. MOTIVATIONS:

The potential of deep neural networks, once deployed to mobile devices, has the advantage of lower latency, no reliance on the network, and better user privacy. However, energy efficiency becomes the bottleneck for deploying deep neural networks on mobile devices because mobile devices are battery constrained. Current deep neural network models consist of hundreds of millions of parameters. Reducing the size of a DNN model makes the deployment on edge devices easier.

First, a smaller model means less overhead when exporting models to clients. Take autonomous driving for example; Tesla periodically copies new models from their servers to customers' cars. Smaller models require less communication in such over-the-air updates, making frequent updates more feasible. Another example is on Apple Store; apps above 100 MB will not download until you connect to Wi-Fi. It's infeasible to put a large DNN model in an app. The second issue is energy consumption. Deep learning is energy consuming, which is problematic for battery-constrained mobile devices. As a result, iOS 10 requires iPhone to be plugged with charger while performing photo analysis. Fetching DNN models from memory takes more than two orders of magnitude more energy than arithmetic operations. Smaller neural networks require less memory bandwidth to fetch the model, saving the energy and extending battery life. The third issue is area cost. When deploying DNNs on Application-Specific Integrated Circuits (ASICs), a sufficiently small model can be stored directly on-chip, and smaller models enable a smaller ASIC die.

Several previous works aimed to improve energy and spatial efficiency of deep networks. One common strategy proven useful is to quantize 32-bit weights to one or two bits, which greatly reduces model size and saves memory reference. However, experimental results show that compressed weights usually come with degraded performance, which is a great loss for some performancesensitive applications. The contradiction between compression and performance motivates us to work on trained ternary quantization, minimizing performance degradation of deep neural networks while saving as much energy and space as possible.

3. RELATED WORK:

3.1 BINARY NEURAL NETWORK (BNN) Lin et al. (2015) proposed binary and ternary connections to compress neural networks and speed up computation during inference. They used similar probabilistic methods to convert 32-bit weights into binary values or ternary values, defined as:

EQUATION

Here w b and w t denote binary and ternary weights after quantization. w denotes the latent full precision weight.

During back-propagation, as the above quantization equations are not differentiable, derivatives of expectations of the Bernoulli distribution are computed instead, yielding the identity function:

∂L ∂ w = ∂L ∂w b = ∂L ∂w t (2)

Here L is the loss to optimize.

For BNN with binary connections, only quantized binary values are needed for inference. Therefore a 32× smaller model can be deployed into applications.

3.2 DOREFA-NET Zhou et al. (2016) proposed DoReFa-Net which quantizes weights, activations and gradients of neural networks using different widths of bits. Therefore with specifically designed low-bit multiplication algorithm or hardware, both training and inference stages can be accelerated.

They also introduced a much simpler method to quantize 32-bit weights to binary values, defined as:

EQUATION

Here E(| w|) calculates the mean of absolute values of full precision weights w as layer-wise scaling factors. During back-propagation, Equation 2 still applies.

3.3. TERNARY WEIGHT NETWORKS:

Li & Liu (2016) proposed TWN (Ternary weight networks), which reduce accuracy loss of binary networks by introducing zero as a third quantized value. They use two symmetric thresholds ±∆ l and a scaling factor W l for each layer l to quantize weighs into {-W l , 0, +W l }:

w t l =    W l : wl > ∆ l 0 : | wl | ≤ ∆ l -W l : wl < -∆ l (4)

They then solve an optimization problem of minimizing L2 distance between full precision and ternary weights to obtain layer-wise values of W l and ∆ l :

EQUATION

And again Equation 2 is used to calculate gradients. While an additional bit is required for ternary weights, TWN achieves a validation accuracy that is very close to full precision networks according to their paper. Han et al. (2015) proposed deep compression to prune away trivial connections and reduce precision of weights. Unlike above models using zero or symmetric thresholds to quantize high precision weights, Deep Compression used clusters to categorize weights into groups. In Deep Compression, low precision weights are fine-tuned from a pre-trained full precision network, and the assignment of each weight is established at the beginning and stay unchanged, while representative value of each cluster is updated throughout fine-tuning.

3.4. DEEP COMPRESSION:



4. METHOD:

Our method is illustrated in Figure 1 . First, we normalize the full-precision weights to the range [-1, +1] by dividing each weight by the maximum weight. Next, we quantize the intermediate full-resolution weights to {-1, 0, +1} by thresholding. The threshold factor t is a hyper-parameter that is the same across all the layers in order to reduce the search space. Finally, we perform trained quantization by back propagating two gradients, as shown in the dashed lines in Figure 1 . We back-propagate gradient 1 to the full-resolution weights and gradient 2 to the scaling coefficients.

The former enables learning the ternary assignments, and the latter enables learning the ternary values.

At inference time, we throw away the full-resolution weights and only use ternary weights.

4.1. LEARNING BOTH TERNARY VALUES AND TERNARY ASSIGNMENTS:

During gradient descent we learn both the quantized ternary weights (the codebook), and choose which of these values is assigned to each weight (choosing the codebook index). To learn the ternary value (codebook), we introduce two quantization factors W p l and W n l for positive and negative weights in each layer l. During feed-forward, quantized ternary weights w t l are calculated as:

w t l =    W p l : wl > ∆ l 0 : | wl | ≤ ∆ l -W n l : wl < -∆ l (6)

Unlike previous work where quantized weights are calculated from 32-bit weights, the scaling coefficients W p l and W n l are two independent parameters and are trained together with other parameters. Following the rule of gradient descent, derivatives of W p l and W n l are calculated as:

EQUATION

Here

I p l = {i| wl (i) > ∆ l } and I n l = {i|(i) wl < -∆ l }.

Furthermore, because of the existence of two scaling factors, gradients of latent full precision weights can no longer be calculated by Equation 2. We use scaled gradients for 32-bit weights:

EQUATION

Note we use scalar number 1 as factor of gradients of zero weights. The overall quantization process is illustrated as Figure 1 . The evolution of the ternary weights from different layers during training is shown in Figure 2 . We observe that as training proceeds, different layers behave differently: for the first quantized conv layer, the absolute values of W p l and W n l get smaller and sparsity gets lower, while for the last conv layer and fully connected layer, the absolute values of W p l and W n l get larger and sparsity gets higher.

We learn the ternary assignments (index to the codebook) by updating the latent full-resolution weights during training. This may cause the assignments to change between iterations. Note that the thresholds are not constants as the maximal absolute values change over time. Once an updated weight crosses the threshold, the ternary assignment is changed.

The benefits of using trained quantization factors are: i) The asymmetry of W p l = W n l enables neural networks to have more model capacity. ii) Quantized weights play the role of "learning rate multipliers" during back propagation.

4.2. QUANTIZATION HEURISTIC:

In previous work on ternary weight networks, Li & Liu (2016) proposed Ternary Weight Networks (TWN) using ±∆ l as thresholds to reduce 32-bit weights to ternary values, where ±∆ l is defined as Equation 5. They optimized value of ±∆ l by minimizing expectation of L2 distance between full precision weights and ternary weights. Instead of using a strictly optimized threshold, we adopt different heuristics: 1) use the maximum absolute value of the weights as a reference to the layer's threshold and maintain a constant factor t for all layers:

EQUATION

and 2) maintain a constant sparsity r for all layers throughout training. By adjusting the hyperparameter r we are able to obtain ternary weight networks with various sparsities. We use the first method and set t to 0.05 in experiments on CIFAR-10 and ImageNet dataset and use the second one to explore a wider range of sparsities in section 5.1.1.

We perform our experiments on CIFAR-10 ( Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015) . Our network is implemented on both TensorFlow (Abadi & et. al o, 2015) and Caffe (Jia et al., 2014) of 0.0002 is used as regularizer. Most of our models converge after 160 epochs. We take a moving average on errors of all epochs to filter off fluctuations when reporting error rate.

We compare our model with the full-precision model and a binary-weight model. We train a a full precision ResNet (He et al., 2016) on CIFAR-10 as the baseline (blue line in Figure 3 ). We fine-tune the trained baseline network as a 1-32-32 DoReFa-Net where weights are 1 bit and both activations and gradients are 32 bits giving a significant loss of accuracy (green line) . Finally, we fine-tuning the baseline with trained ternary weights (red line). Our model has substantial accuracy improvement over the binary weight model, and our loss of accuracy over the full precision model is small. We also compare our model to Tenary Weight Network (TWN) on ResNet-20. Result shows our model improves the accuracy by ∼ 0.25% on CIFAR-10.

We We further train and evaluate our model on ILSVRC12 (Russakovsky et al. (2015) ). ILSVRC12 is a 1000-category dataset with over 1.2 million images in training set and 50 thousand images in validation set. Images from ILSVRC12 also have various resolutions. We used a variant of AlexNet (Krizhevsky et al. (2012) ) structure by removing dropout layers and add batch normalization (Ioffe & Szegedy, 2015) for all models in our experiments. The same variant is also used in experiments described in the paper of DoReFa-Net.

Our ternary model of AlexNet uses full precision weights for the first convolution layer and the last fully-connected layer. Other layer parameters are all quantized to ternary values. We train our model on ImageNet from scratch using an Adam optimizer (Kingma & Ba (2014) ). Minibatch size is set to 128. Learning rate starts at 10 -4 and is scaled by 0.2 at epoch 56 and 64. A L2-normalized weight decay of 5 × 10 -6 is used as a regularizer. Images are first resized to 256 × 256 then randomly cropped to 224 × 224 before input. We report both top 1 and top 5 error rate on validation set.

We compare our model to a full precision baseline, 1-32-32 DoReFa-Net and TWN. After around 64 epochs, validation error of our model dropped significantly compared to other low-bit networks as well as the full precision baseline. We draw the process of training in Figure 4 , the baseline results of AlexNet are marked with dashed lines. Our ternary model effectively reduces the gap between training and validation performance, which appears to be quite great for DoReFa-Net and TWN. This indicates that adopting trainable W p l and W n l helps prevent models from overfitting to the training set. We also report the results of our methods on ResNet-18B in Table 3 . The full-precision error rates are obtained from Facebook's implementation. Here we cite Binarized Weight Network(BWN) Rastegari et al. (2016) results with all layers quantized and TWN finetuned based on a full precision network, while we train our TTQ model from scratch. Compared with BWN and TWN, our method obtains a substantial improvement. 

6. DISCUSSION:

In this section we analyze performance of our model with regard to weight compression and inference speeding up. These two goals are achieved through reducing bit precision and introducing sparsity. We also visualize convolution kernels in quantized convolution layers to find that basic patterns of edge/corner detectors are also well learned from scratch even precision is low.

6.1. SPATIAL AND ENERGY EFFICIENCY:

We save storage for models by 16× by using ternary weights. Although switching from a binaryweight network to a ternary-weight network increases bits per weight, it brings sparsity to the weights, which gives potential to skip the computation on zero weights and achieve higher energy efficiency.

6.1.1. TRADE-OFF BETWEEN SPARSITY AND ACCURACY:

Figure 5 shows the relationship between sparsity and accuracy. As the sparsity of weights grows from 0 (a pure binary-weight network) to 0.5 (a ternary network with 50% zeros), both the training and validation error decrease. Increasing sparsity beyond 50% reduces the model capacity too far, increasing error. Minimum error occurs with sparsity between 30% and 50%.

Published as a conference paper at ICLR 2017 We further analyze parameters from our AlexNet model. We calculate layer-wise density (complement of sparsity) as shown in Table 4 . Despite we use different W p l and W n l for each layer, ternary weights can be pre-computed when fetched from memory, thus multiplications during convolution and inner product process are still saved. Compared to Deep Compression, we accelerate inference speed using ternary values and more importantly, we reduce energy consumption of inference by saving memory references and multiplications, while achieving higher accuracy.

Validation Error Train Error 1

We notice that without all quantized layers sharing the same t for Equation 9, our model achieves considerable sparsity in convolution layers where the majority of computations takes place. Therefore we are able to squeeze forward time to less than 30% of full precision networks.

As for spatial compression, by substituting 32-bit weights with 2-bit ternary weights, our model is approximately 16× smaller than original 32-bit AlexNet.

6.2. KERNEL VISUALIZATION:

We visualize quantized convolution kernels in Figure 6 . The left matrix is kernels from the second convolution layer (5 × 5) and the right one is from the third (3 × 3). We pick first 10 input channels and first 10 output channels to display for each layer. Grey, black and white color represent zero, negative and positive weights respectively.

We observe similar filter patterns as full precision AlexNet. Edge and corner detectors of various directions can be found among listed kernels. While these patterns are important for convolution neural networks, the precision of each weight is not. Ternary value filters are capable enough extracting key features after a full precision first convolution layer while saving unnecessary storage. Furthermore, we find that there are a number of empty filters (all zeros) or filters with single non-zero value in convolution layers. More aggressive pruning can be applied to prune away these redundant kernels to further compress and speed up our model.

Layer:

Full 

:

https://github.com/BVLC/caffe/wiki/Models-accuracy-on-ImageNet-2012-val

Figures and Tables:

Figure 1: Overview of the trained ternary quantization procedure.

Figure 3: ResNet-20 on CIFAR-10 with different weight precision.

Figure 4: Training and validation accuracy of AlexNet on ImageNet

Figure 5: Accuracy v.s. Sparsity on ResNet-20 6.1.2 SPARSITY AND EFFICIENCY OF ALEXNET

expand our experiments to ternarize ResNet with 32, 44 and 56 layers. All ternary models are fine-tuned from full precision models. Our results show that we improve the accuracy of ResNet-32, ResNet-44 and ResNet-56 by 0.04%, 0.16% and 0.36% . The deeper the model, the larger the improvement. We conjecture that this is due to ternary weights providing the right model capacity and preventing overfitting for deeper networks. Error rates of full-precision and ternary ResNets on Cifar-10

Table content: <table><tr><td>Model</td><td colspan="3">Full resolution Ternary (Ours) Improvement</td></tr><tr><td>ResNet-20</td><td>8.23</td><td>8.87</td><td>-0.64</td></tr><tr><td>ResNet-32</td><td>7.67</td><td>7.63</td><td>0.04</td></tr><tr><td>ResNet-44</td><td>7.18</td><td>7.02</td><td>0.16</td></tr><tr><td>ResNet-56</td><td>6.80</td><td>6.44</td><td>0.36</td></tr><tr><td>5.1 IMAGENET</td><td/><td/><td/></tr></table>

Finally our model reaches top 1 error rate of 42.5%, while DoReFa-Net gets 46.1% and TWN gets 45.5%. Furthermore, our model still outperforms full precision AlexNet (the batch normalization version, 44.1% according to paper of DoReFa-Net) by 1.6%, and is even better than the best AlexNet results reported (42.8% 1 ). The complete results are listed in Table2.

Table content: <table><tr><td colspan="2">Error Full precision</td><td>1-bit (DoReFa)</td><td>2-bit (TWN)</td><td>2-bit (Ours)</td></tr><tr><td>Top1</td><td>42.8%</td><td>46.1%</td><td colspan="2">45.5% 42.5%</td></tr><tr><td>Top5</td><td>19.7%</td><td>23.7%</td><td colspan="2">23.2% 20.3%</td></tr></table>

Top1 and Top5 error rate of AlexNet on ImageNet

Table content: <table/>

Top1 and Top5 error rate of ResNet-18 on ImageNet

Table content: <table/>

Alexnet layer-wise sparsity

Table content: <table><tr><td/><td colspan="2">precision</td><td colspan="2">Pruning (NIPS'15)</td><td>Ours</td><td/></tr><tr><td/><td colspan="3">Density Width Density</td><td>Width</td><td colspan="2">Density Width</td></tr><tr><td>conv1</td><td>100%</td><td>32 bit</td><td>84%</td><td>8 bit</td><td>100%</td><td>32 bit</td></tr><tr><td>conv2</td><td>100%</td><td>32 bit</td><td>38%</td><td>8 bit</td><td>23%</td><td>2 bit</td></tr><tr><td>conv3</td><td>100%</td><td>32 bit</td><td>35%</td><td>8 bit</td><td>24%</td><td>2 bit</td></tr><tr><td>conv4</td><td>100%</td><td>32 bit</td><td>37%</td><td>8 bit</td><td>40%</td><td>2 bit</td></tr><tr><td>conv5</td><td>100%</td><td>32 bit</td><td>37%</td><td>8 bit</td><td>43%</td><td>2 bit</td></tr><tr><td>conv total</td><td>100%</td><td>-</td><td>37%</td><td>-</td><td>33%</td><td>-</td></tr><tr><td>fc1</td><td>100%</td><td>32 bit</td><td>9%</td><td>5 bit</td><td>30%</td><td>2 bit</td></tr><tr><td>fc2</td><td>100%</td><td>32 bit</td><td>9%</td><td>5 bit</td><td>36%</td><td>2 bit</td></tr><tr><td>fc3</td><td>100%</td><td>32 bit</td><td>25%</td><td>5 bit</td><td>100%</td><td>32 bit</td></tr><tr><td>fc total</td><td>100%</td><td>-</td><td>10%</td><td>-</td><td>37%</td><td>-</td></tr><tr><td>All total</td><td>100%</td><td>-</td><td>11%</td><td>-</td><td>37%</td><td>-</td></tr></table>

Peer review (iclr2017):
The paper shows a different approach to a ternary quantization of weights. Strengths: 1. The paper shows performance improvements over existing solutions 2. The idea of learning the quantization instead of using pre-defined human-made algorithm is nice and very much in the spirit of modern machine learning. Weaknesses: 1. The paper is very incremental. 2. The paper is addressed to a very narrow audience. The paper very clearly assumes that the reader is familiar with the previous work on the ternary quantization. It is *what is new in the topic* update, not really a standalone paper. The description of the main algorithm is very concise, to say the least, and is probably clear to those who read some of the previous work on this narrow subject, but is unsuitable for a broader deep learning audience. 3. There is no convincing motivation for the work. What is presented is an engineering gimmick, that would be cool and valuable if it really is used in production, but is that really needed for anything? Are there any practical applications that require this refinement? I do not find the motivation *it is related to mobile, therefore it is cool* sufficient. This paper is a small step further in a niche research, as long as the authors do not provide a sufficient practical motivation for pursuing this particular topic with the next step on a long list of small refinements, I do not think it belongs in ICLR with a broad and diversified audience. Also - the code was not released is my understanding.