Title: INSTANT QUANTIZATION OF NEURAL NETWORKS US-ING MONTE CARLO METHODS

Abstract: Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose to apply Monte Carlo methods and importance sampling to sparsify and quantize pre-trained neural networks without any retraining. We obtain sparse, low bit-width integer representations that approximate the full precision weights and activations. The precision, sparsity, and complexity are easily configurable by the amount of sampling performed. Our approach, called Monte Carlo Quantization (MCQ), is linear in both time and space, while the resulting quantized sparse networks show minimal accuracy loss compared to the original full-precision networks. Our method either outperforms or achieves results competitive with methods that do require additional training on a variety of challenging tasks.

1. INTRODUCTION:

Developing novel ways of increasing the efficiency of neural networks is of great importance due to their widespread usage in today's variety of applications. Reducing the network's footprint enables local processing on personal devices without the need for cloud services. In addition, such methods allow for reducing power consumption -also in data centers. Very compact models can be fully stored and executed on-chip in specialized hardware like for example ASICs or FPGAs. This reduces latency, increases inference speed, improves privacy concerns, and limits bandwidth cost.

Quantization methods usually require re-training of the quantized model to achieve competitive results. This leads to an additional cost and complexity. The proposed method, Monte Carlo Quantization (MCQ), aims to avoid retraining by approximating the full-precision weight and activation distributions using importance sampling. The resulting quantized networks achieve close to the full-precision accuracy without any kind of additional training. Importantly, the complexity of the resulting networks is proportional to the number of samples taken.

First, our algorithm normalizes the weights and activations of a given layer to treat them as probability distributions. Then, we randomly sample from the corresponding cumulative distributions and count the number of hits for every weight and activation. Finally, we quantize the weights and activations by their integer count values, which form a discrete approximation of the original continuous values. Since the quality of this approximation relies entirely on (quasi)random sampling, the accuracy of the quantized model is directly dependent on the amount of sampling performed. Thus, accuracy may be traded for higher sparsity and speed by adjusting the number of samples. On the challenging tasks of image classification, language modeling, speech recognition, and machine translation, our method outperforms or is competitive with existing quantization methods that do require additional training.

2. RELATED WORK:

The computational cost of neural networks can be reduced by pruning redundant weights or neurons, which has been shown to work well (Han et al., 2015; Mocanu et al., 2018; LeCun et al., 1990) . Alternatively, the precision of the network weights and activations may be lowered, potentially introducing sparsity. Using low precision computations to reduce the cost and sparsity to skip computations allows for efficient hardware implementations (Lin et al., 2015; Venkatesh et al., 2017) . This is the approach used in this paper.

BinaryConnect (Courbariaux et al., 2015) proposed training with binary weights, while XNOR-Net (Rastegari et al., 2016) and BNN (Hubara et al., 2016) extended this binarization to activations as well. TWN (Li et al., 2016) proposed ternary quantization instead, increasing model expressiveness. Similarly, TTQ (Zhu et al., 2016) used ternary weights with a positive and negative scaling learned during training. LR-Net (Shayer et al., 2017) made use of both binary and ternary weights by using stochastic parameterization while INQ (Zhou et al., 2017) constrained weights to powers of two and zero. FGQ (Mellempudi et al., 2017) categorized weights in different groups and used different scaling factors to minimize the element-wise distance between full and low-precision weights. Wang et al. (2019) used the hardware accelerator's feedback to perform hardware-aware quantization using reinforcement learning. Zhang et al. (2018) jointly trained quantized networks and respective quantizers. Reagen et al. (2017) used Bloomier filters to compactly encode network weights.

Similarly, quantization techniques can also be applied in the backward pass. Therefore, some previous work quantized not only weights and activations but also the gradients to augment training performance (Zhou et al., 2016; Gupta et al., 2015; Courbariaux et al., 2014) . In particular, RQ (Louizos et al., 2018) propose a differentiable quantization procedure to allow for gradient-based optimization using discrete values and Wu et al. (2018) recently proposed to discretize weights, activations, gradients, and errors both at training and inference time.

These quantization techniques have great benefits and have shown to successfully reduce the computation requirements compared to full-precision models. However, all the aforementioned methods require re-training of the quantized network to achieve close to full-precision accuracy, which can introduce significant financial and environmental cost (Strubell et al., 2019) . On the other hand, our method instantly quantizes pre-trained neural networks with minimal accuracy loss as compared to their full-precision counterparts without any kind of additional training.

3. NEURAL NETWORKS AND MONTE CARLO METHODS:

Neural networks make extensive use of randomization and random sampling techniques. Examples are random initialization of network weights, stochastic gradient descent (Robbins & Monro, 1951) , regularization techniques such as Dropout (Srivastava et al., 2014) and DropConnect (Wan et al., 2013) , data augmentation and data shuffling, recurrent neural networks' regularization (Merity et al., 2017a) , or the generator's noise input on generative adversarial networks (Goodfellow et al., 2014) .

Many state-of-the-art networks use ReLU (Nair & Hinton, 2010) , which has interesting properties such as scale-invariance. This enables a scaling factor to be propagated through all network layers without affecting the network's original output. This principle can be used to normalize network values, such as weights and activations, as further described in Section 3.1. After normalization, these values can be treated as probabilities, which enables the simulation of discrete probability densities to approximate the corresponding full-precision, continuous distributions (Section 3.2).

3.1. NETWORK NORMALIZATION:

Assuming the exclusive use of the ReLU activation function in the hidden layers, the scale-invariance property of the ReLU activation function allows for arbitrary scaling of the weights or activations without affecting the network's output. Given weights w l-1,i,j connecting the i-th neuron in layer l -1 to the j-th neuron in layer l, where i ∈ [0, N l-1 -1] and j ∈ [0, N l -1], with N l-1 and N l the number of neurons of layer l -1 and l, respectively. Let a l,j be the j-th activation in the l-th layer and f ∈ R + :

a l,j = max 0, N l-1 -1 i=0 w l-1,i,j a l-1,i + b l,j = f • max 0, N l-1 -1 i=0 w l-1,i,j a l-1,i +b l,j f .

Biases and incoming weights for neuron j in layer l may then be normalized by f = w l-1,j 1 = N l-1 -1 i=0 |w l-1,i,j |, enabling weights to be seen as a probability distribution over all connections to a neuron. A similar procedure could be used to normalize all activations a l,j of layer l.

Propagating these scaling factors forward layer by layer results in a single scalar (per output), which converts the outputs of the normalized network to the same range as the original network. This technique allows for the usage of integer weights and activations throughout the entire network without requiring rescaling or conversion to floating point at every layer.

3.2. NETWORK QUANTIZATION:

Taking advantage of the normalized network, we can simulate discrete probability densities by constructing a probability density function (PDF) and then sampling from the corresponding cumulative density function (CDF). The number of references of a weight is then the quantized integer approximation of the continuous value. For simplicity, the following discussion shows the quantization procedure for weights; activations can be quantized in the same way at inference time.

Without loss of generality, given n weights, assuming 

0 = P 0 P n-1 = 1 P 1 P 2 P n-2 |w 1 | |w 2 | |w n-1 | (1)

Then, given N uniformly distributed samples x i ∈ [0, 1), we can approximate the weight distribution as follows:

EQUATION

where j i ∈ {0, . . . , n -1} is uniquely determined by P ji-1 ≤ x i < P ji .

One can further improve this sampling process by using jittered equidistant sampling. Thus, given a random variable ξ ∈ [0, 1), we generate N uniformly distributed samples x i ∈ [0, 1) such that

x i = i + ξ N

, where i ∈ {0, . . . , N -1}. The combination of equidistant samples and a random offset improves the weight approximation, as the samples are more uniformly distributed. The variance of different sampling seeds is discussed in the Appendix.

4. MONTE CARLO QUANTIZATION (MCQ):

Our approach builds on the aforementioned ideas of network normalization and quantization using random sampling to quantize an entire pre-trained full-precision neural network. As before, we focus on weight quantization; online activation quantization is discussed in Section 4.4. Our method, called Monte Carlo Quantization (MCQ), consists of the following steps, which are executed layer by layer:

(1) Create a probability density function (PDF) for all N l,w weights of layer l such that N l,w -1 i=0 |w l,i | = 1 (Section 4.1).

(2) Perform importance sampling on the weights based on their magnitude by sampling from the corresponding cumulative density function (CDF) and counting the number of hits per weight (Section 4.2). (3) Replace each weight with its quantized integer value, i.e. its hit count, to obtain a low bit-width, integer weight representation (Section 4.3).

The pseudo-code for our method is shown in Algorithm 1 of the Appendix. Figure 1 illustrates both the normalization and importance sampling processes for a layer with 10 weights and 1 sample per weight, i.e. K = 1.0.

4.1. LAYER NORMALIZATION:

Performing normalization neuron-wise, as introduced in Section 3.1 may result in an inferior approximation, especially when the number of weights to sample from is small, as for example in convolutional layers with a small number of filters or input channels. To mitigate this, we propose to normalize all neurons simultaneously in a layer-wise manner. This has the additional advantage that samples can be redistributed from low-importance neurons to high-importance neurons (according to 0 1 2 3 4 5 6 7 8 and uniformly sample from the corresponding CDF (c). The sampling process produces quantized integer network weights based on the number of hits per weight (d). Note that since weights 7, 8, and 9 were not hit, sparsity is introduced which can be exploited by hardware accelerators. some metric), resulting in an increased level of sparsity. Additionally, there is more opportunity for global optimization, so the overall weight distribution approximation improves as well.

We use the 1-norm of all weights of a given layer l as the scaling factor f used to perform weight normalization. Thus, each normalized weight can be seen as a probability with respect to all connections between layer l -1 and layer l, instead of a single neuron. This layer-wise normalization technique is similar to Weight Normalization (Salimans & Kingma, 2016) , which decouples the neuron weight vector magnitude from its direction.

4.2. IMPORTANCE SAMPLING:

As introduced in Section 3.2, we generate ternary samples (hit positive weight, hit negative weight, or no hit), and count such hits during the sampling process. Note that even though the individual samples are ternary, the final quantized values may not be, because a single weight can be sampled multiple times. For jittered sampling, we use one random offset per layer, with a number of samples N = K • N values , where K ∈ R + is a user-specified parameter to control the number of samples and N values represents the number of weights of a given layer. By varying K, the computational cost of sampling can be traded off better approximation (more bits per weight) of the original weight distribution, leading to higher accuracy. In our experiments, K is set the same for all network layers.

One simple modification to enhance the quality of the discrete approximation is to sort the continuous values prior to creating the PDF. Applying sorting mechanisms to Monte Carlo schemes has been shown to be beneficial in the past (L'Ecuyer et al., 2008; 2018) . Sorting groups smaller values together in the overall distribution. Since we are using a uniform sampling strategy, smaller weights are then sampled less often, which results in both higher sparsity and a better quantized approximation of the larger weights in practice. This effect is particularly significant on smaller layers with fewer weights.

Since the quantized integer weights span a different range of values than the original weights, and biases remain unchanged, care must be taken to ensure the activations of each neuron are calculated correctly. After the integer multiply-accumulate (MAC) operation, the result must then be scaled by f N before adding the bias. This requires the storage of one floating point scaling value per layer. However, weights are stored as low bit-width integers and the computational cost is greatly reduced since the MAC operations use low-precision integers only instead of floating point numbers.

4.3. LAYER QUANTIZATION:

The number of bits required for the weights B W l ∈ N, for layer l and its quantized weights Q(w l,i ), corresponds to the bit amount needed to represent the highest hit count during sampling, including its sign:

B W l = 1 + log 2 (max 0≤i≤Nw-1 |Q(w l,i )|) + 1.

Alternatively, positive and negative weights could be separated into two sets.

4.4. ONLINE QUANTIZATION:

While weights are quantized offline, i.e. after training and before inference, activations are quantized online during inference time using the same procedure as weight quantization previously described. Thus, in the normalization step (Section 4.1), all N l,a activations of a given layer l are treated as a probability distribution over the output features, such that N l,a -1 j=0 |a l,j | = 1. Then, in the importance sampling step (Section 4.2), activations are sub-sampled using possibly different relative sampling amounts, i.e. K, than the ones used for the weights (we use the same K for both weights and activations in all of our experiments). The required number of bits B A l for the quantized activations Q(a l,j ) can also be calculated similarly as described in Section 4.3, although no additional bit sign is required when using ReLU since all activations are non-negative.

5. EXPERIMENTS:

The proposed method is extensively evaluated on a variety of tasks: for image classification we use CIFAR-10 ( Krizhevsky & Hinton, 2009) , SVHN (Netzer et al., 2011) , and ImageNet (Deng et al., 2009) , on multiple models each. We further evaluate MCQ on language modeling, speech recognition, and machine translation, to assess the preformance of MCQ across different task domains.

Due to the automatic quantization done by MCQ, some layers may be quantized to lower or higher levels than others. We indicate the quantization level for the whole network by the average number of bits, e.g. '8w-32a' means that on average 8 bits were used for weights and 32 bits for activations on each layer.

Many works note that quantizing the first or last network layer reduces accuracy significantly (Han et al., 2015; Zhou et al., 2016; Li et al., 2016) . We use footnotes1 ,2 , and3 to denote the special treatment of first or last layers respectively. For MCQ we report the results with both quantized and full-precision first layer. We do not quantize Batch Normalization layers as the parameters are fixed after training and can be incorporated into the weights and biases (Wu et al., 2018) .

Tables 1 to 4 show the accuracy difference ∆ between the quantized and full-precision models. For other compared works this difference is calculated using the baseline models reported in each of the respective works. We didn't perform any search over random sampling seeds for MCQ's results.

5.1. CIFAR-10:

The best accuracies on VGG-7, VGG-14, and ResNet-20 produced by our method using K = 1.0 on CIFAR-10 are shown in Table 1 . We refer to the Appendix for model and training details. MCQ outperforms or shows competitive results showing minimal accuracy loss on all tested models against the compared methods that require network re-training. The full-precision baselines for BNN (Hubara et al., 2016) and XNOR-Net (Rastegari et al., 2016) are from BC (Courbariaux et al., 2015) as these works use the same model. Similarly, BWN (Rastegari et al., 2016) 's results on VGG-7 are the ones reported in TWN (Li et al., 2016) since they did not report the baseline in the original paper.

Figure 2 shows the effects of varying the amount of sampling, i.e. using K ∈ [0.1...2.0].The average percentage of used weights/activations per layer and corresponding bit-widths of the final quantized model is also presented on each graph. We observe a rapid increase of the accuracy even when sparsity levels are high on all tested models.

5.2. SVHN:

For SVHN, the tested models are identical to the compared methods. Models B, C, and D have the same architecture as Model A but with a 50%, 75%, and 87.5% reduction in the number of filters in each convolutional layer, respectively (Zhou et al., 2016) . We refer to the Appendix for further model and training details.

Table 2 shows MCQ's results for several models on SVHN using K = 1.0. On bigger models, i.e. VGG-7* and Model A, we see minimal accuracy loss when compared to the full-precision baselines. For the smaller models, we observe a slight accuracy degradation as model size decreases due to the reduction in the sample size, resulting in a poorer approximation. However, we used only about 4 bits per weight/activation for such models. Thus, increasing the number of samples would improve -0.48 (6.1W-32A) / +0.04 1 (6.1W-32A) -1.04 (6.7W-32A) / -0.50 1 (6.8W-32A) -0.84 (5.1W-32A) / -0.54 1 (5.1W-32A) ∆ MCQ (QUANTIZED A) -0.12 1 (32W-5.68A) -0.06 1 (32W-5.51A) -0.28 1 (32W-6.3A) ∆ MCQ (QUANTIZED W + A) -0.58 (6.1W-5.6A) / -0.13 1 (6.1W-5.6A) -1.08 (6.6W-5.3A) / -0.54 1 (6.8W-5.5A) -1.77 (5.1W-5.3A) / -1.21 1 (5.1W-5.3A) accuracy while still maintaining a low bit-width. Figure 3 illustrates the consequences of varying the number of samples. Less samples are required than on CIFAR-10 for bigger models to achieve close to full-precision accuracy. Potentially this is because layers have a larger number of weights and activations, so a larger sample size reduces quantization noise since the important values being more likely to be better approximated.

∆ TTQ (2W-32A) - - -0.64 1 ∆ DLAC (2W-32A) - -3.0 / -1.4 1 - ∆ TWNS (2W-32A) -0.06 - - ∆ BC (1W-32A) +0.74 - - ∆ BNN (1W-1A) +0.49 1 - - ∆ BWN (1W-32A) -0.36 / +0.76 1 - - ∆ XNOR-NET (1W-1A) +0.47 1 - - ∆ RQ (8W-8A)) +0.25 - - ∆ LR-NET (2W-32A) -0.

5.3. IMAGENET:

For ImageNet, we evaluate MCQ on AlexNet, ResNet-18, and ResNet-50 using the pre-trained models provided by Pytorch's model zoo (Paszke et al., 2017) ). Table 3 shows the results on ImageNet with K = 5.0 for the different models. The results shown for DoReFa, BWN, TWN (Zhou et al., 2016; Rastegari et al., 2016; Li et al., 2016) are the ones reported in TTQ (Zhu et al., 2016) .

Figure 4 shows the accuracy of the quantized model when using different sample sizes, i.e., K ∈ [0.25, ..., 5.0]. We observe that more sampling is required to achieve a close to full-precision model accuracy on ImageNet. On this dataset, sorting the CDF before sampling didn't result in any improvements, so reported results are without sorting. All the quantized models achieve close to full-precision accuracy, though more samples are required than for the previous datasets resulting in a higher required bit-width.

5.4. EXPERIMENTS ON ADDITIONAL TASKS:

To assess the robustness of MCQ, we further evaluate MCQ on several models in natural language and speech processing. We evaluate language modeling on Wikitext-103 using a Transformer-based model (Baevski & Auli, 2018) and Wikitext-2 using a 2-layer LSTM (Zhao et al., 2019) , speech recognition on VCTK using Deepspeech2 (Amodei et al., 2015) , and machine translation on WMT-14 English-to-French using a Transformer (Ott et al., 2018) . Additional details are provided in the Appendix. Table 4 shows the comparison to full-precision models for these various tasks.

Table 2 : Accuracy results on SVHN when quantizing weights, activations, or both. On VGG-7*, MCQ shows minimal accuracy loss when quantizing both weights and activations and close to no accuracy loss when not quantizing the first layer. For models A, B, C, and D the accuracy lowers as the model size decreases. Quantizing only the activations barely lowers the baseline accuracy. -0.30 (7.3W-32A) / -0.02 1 (7.0W-32A) -0.20 1 (5.1W-32A) -0.30 1 (4.8W-32A) -1.48 1 (4.1W-32A) -2.17 1 (4.1W-32A) ∆ MCQ (QUANTIZED A) -0.04 (32W-7.15A) +0.01 1 (32W-5.28A) -0.03 1 (32W-5.11A) -0.12 1 (32W-4.88A) -0.11 1 (32W-4.58A) ∆ MCQ (QUANTIZED W + A) -0.32 (7.2W-6.0A) / -0.06 1 (7.0W-5.5A) -0.40 1 (5.1W-4.2A) -0.56 1 (4.8W-4.1A) -2.13 1 (4.1W-3.9A) -3.72 1 (4.1W-3.7A) 

EQUATION

6. DISCUSSION AND FUTURE WORK:

The experimental results show the performance of MCQ on multiple models, datasets, and tasks, demonstrated by the minimal loss of accuracy compared to the full-precision counterparts. MCQ either outperforms or is competitive to other methods that require additional training of the quantized network. Moreover, the trade-off between accuracy, sparsity, and bit-width can be easily controlled by adjusting the number of samples. Note that the complexity of the resulting quantized network is proportional to the number of samples in both space and time.

One limitation of MCQ, however, is that it often requires a higher number of bits to represent the quantized values. On the other hand, this sampling-based approach directly translates to a good approximation of the real full-precision values without any additional training. Recently Zhao et al. (2019) proposed to outlier channel splitting, which is orthogonal work to MCQ and could be used to reduce the bit-width required for the highest hit counts.

There are several paths that could be worth following for future investigations. In the importance sampling stage, using more sophisticated metrics for importance ranking, e.g. approximation of the Hessian by Taylor expansion could be beneficial (Molchanov et al., 2016) . Automatically selecting optimal sampling levels on each layer could lead to a lower cost since later layers seem to tolerate more sparsity and noise. For efficient hardware implementation, it's important that the quantized Table 3 : Accuracy results on ImageNet when quantizing weights, activations, or both. When quantizing weights only, accuracy drops less than 1% in all tested models. Quantizing only the activations generally leads to a lower accuracy loss compared to quantizing weights. Quantizing both weights and activations leads to an additional accuracy loss of 0.6% in the worst case, i.e. ResNet-50. -0.99 (8.00W-32A) / -0.68 1 (8.00W-32A) -0.72 (8.00W-32A)/ -0.63 1 (8.00W-32A) -0.73 (8.28W-32A)/ -0.20 1 (8.28W-32A) ∆ MCQ (QUANTIZED A) +0.02 1 (32W-8.36A) -0.58 1 (32W-7.36A) -0.76 1 (32W-7.45A) ∆ MCQ (QUANTIZED W + A) -1.05 (7.88W-8.46A) / -0.75 1 (8.00W-7.2A) -1.13 (8.00W-7.35A) /-1.03 1 (8.00W-7.36A) -1.64 (8.26W-7.43A) / -1.21 1 (8.28W-7.45A) Table 4 : Evaluation of MCQ on language modeling, speech recognition, and machine translation. All quantized models reach close to full precision performance. Note that, as opposed to the image classification task, we did not study different sampling amounts nor the effect of quantization on specific network layers. A more in-depth analysis could then help to achieve close to full-precision accuracy at a lower bit-width on these additional models. network can be executed using integer operations only. Bias quantization and rescaling, activation rescaling to prevent overflow or underflow, and quantization of errors and gradients for efficient training leave room for future work.

∆ FGQ (2W-8A) -7.79 3 - -4.29 ∆ TTQ (2W-32A) +0.3 1 , 2 -3.0 1 , 2 - ∆ TWNS (2W-32A) -2.7 1 , 2 -4.3 1 , 2 - ∆ BWN (1W-32A) +0.2 -8.5 1 , 2 - ∆ XNOR-NET (1W-1A) -12.4 -18.1 1 , 2 - ∆ DOREFA (1W-32A) -3.3 1 , 2 - - ∆ INQ (5W-32A) -0.15 -0.71 -1.59 ∆ RQ (8W-8A) - +0.43 - ∆ LR-NET (2W-32A) - -6.

7. CONCLUSION:

In this work, we showed that Monte Carlo sampling is an effective technique to quickly and efficiently convert floating-point, full-precision models to integer, low bit-width models. Computational cost and sparsity can be traded for accuracy by adjusting the number of sampling accordingly.

Our method is linear in both time and space in the number of weights and activations, and is shown to achieve similar results as the full-precision counterparts, for a variety of network architectures, datasets, and tasks. In addition, MCQ is very easy to use for quantizing and sparsifying any pre-trained model. It requires only a few additional lines of code and runs in a matter of seconds depending on the model size, and requires no additional training. The use of sparse, low-bitwidth integer weights and activations in the resulting quantized networks lends itself to efficient hardware implementations.

A ALGORITHM An overview of the proposed method is given in Algorithm 1.

Input: Pre-trained full-precision network Output: Quantized network with integer weights for K=0 to L-1 do unsorted idxs ← argsort(W K ); W sorted ← sort(W K ); W abs ← abs(W sorted ); // Create PDF

W P DF ← W abs W K 1 ; // Create CDF W CDF ← |W P DF | i=1 W P DFi ; N ← ceil(|W K | * K); start idx ← 0; ξ ← random(0, 1) ; // Initialize discrete weights with zeros W K ← [0] × |W K |; // Start subsampling for i=0 to N -1 do x i ← i + ξ N ; hit idx ← argmax(W CDF [start idx :] ≥ x i ) + start idx ; start idx ← hit idx ; unsorted idx ← unsorted idxs [hit idx ] // Update counter if W K [unsorted idx ] > 0 then W K [unsorted idx ]++; else W K [unsorted idx ]

--; end end // Update to integer weights W K ← W K ; // Update layer's precision B W K ← 1 + f loor(log 2 (max(abs(W K )))) + 1 ; end Algorithm 1: Monte Carlo Quantization (MCQ) on network weights. L represents the number of trainable layers, K indicates the percentage of samples to be sampled per weight. The process is performed equivalently for quantizing activations at inference time. Our algorithm is linear in both time and space in the number of weights and activations.

B AVOIDING EXPLODING ACTIVATIONS:

When using integer weights, care has to be taken to avoid overflows in the activations. For that, activations can be scaled using a dynamically computed shifting factor as in Wu et al. (2018) . With Monte Carlo sampling, since we know the expected value of the next-layer activations, we can scale accordingly.

EQUATION

With the activation equation presented in Section 3.1 and N I connections from the input layer to every neuron in the second layer: tion images. The full-precision pre-trained models are taken from Pytorch's model zoo https://pytorch.org/docs/stable/torchvision/models.html (Paszke et al., 2017) .

C.4 VCTK CSTR's VCTK Corpus (Centre for Speech Technology Voice Cloning Toolkit) includes speech data uttered by 109 native speakers of English with various accents, where each speaker reads out about 400 sentences, most of which were selected from a newspaper. The evaluated model uses 2 convolutional layers and 5 GRU layers of 768 hidden units, using code from https://github.com/SeanNaren/deepspeech.pytorch (Veaux et al., 2017) .

C.5 WIKITEXT:

The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers -all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. The WikiText-2 model was a 2-layer LSTM with 650 hidden neurons, and an embedding size of 400. It was trained using the setup and code at https://github.com/salesforce/awdlstm-lm (Merity et al., 2017b) . The WikiText-102 model was a pretrained model available at https://github.com/pytorch/fairseq/tree/master/examples/language model, along with evaluation code (Baevski & Auli, 2018) .

C.6 NMT:

The dataset is WMT14 English-French, cmobining data from several other corpuses, amongst others the Europarl corpus, the News Commentary corpus, and the Common Crawl corpus (Machacek & Bojar, 2014) .

The model was a pretrained model available at https://github.com/pytorch/fairseq/tree/master/examples/scaling nmt, along with evaluation code (Ott et al., 2018) . 9 , and 10 show the effects of varying the amounts of sampling when quantizing only the activations. We observe less sampling is required to achieve full-precision accuracy when quantizing only the activations when compared to quantizing the weights only. 

D QUANTIZING WEIGHTS ONLY:



:

Not quantizing weights in the first layer.

Not quantizing weights in the last layer.

Using higher precision (8w-8a) for the first layer.

Figures and Tables:

|w k | = w 1 = 1 and defining a partition of the unit interval by P m := m k=1 |w k | we have the following partitions:

Figure 1: Starting from full-precision weights (a), we create a PDF of the sorted absolute values (b)and uniformly sample from the corresponding CDF (c). The sampling process produces quantized integer network weights based on the number of hits per weight (d). Note that since weights 7, 8, and 9 were not hit, sparsity is introduced which can be exploited by hardware accelerators.

Figure2: Results of quantizing both weights and activations on CIFAR-10 using different sampling amounts. The quantized models reach close to full-precision accuracy at around half the sample size while using only around half the weights and one-third of the activations of the full-precision models.

Figure3: Results of quantizing both weights and activations on SVHN using different sampling amounts. The quantized VGG-7* model reaches close to full-precision accuracy using around 0.5 samples per weight/activation, requiring around 8 bits and using 22% of the weights of the original model, with 22% nonzero activations. Model A, B, C, and D are less redundant models that require more sampling to achieve close to full-precision accuracy.

Figure 4: Results of quantizing both weights and activations on ImageNet using different sampling amounts. All quantized models reach close to full-precision accuracy at K = 3.

Figures 5, 6, and 7  show the effects of varying the amounts of sampling when quantizing only the weights.

Figure 5: Quantized weights on CIFAR-10.

Figure 8: Quantized activations on CIFAR-10.

Figure 9: Quantized activations on SVHN.

Figure 10: Quantized activations on ImageNet.

Accuracy results on CIFAR-10 when quantizing either weights or activations or both. Quantizing only the weights leads to an accuracy loss of ≈ 1.0% in the worst case. Quantizing both weights and activations does not reduce accuracy on VGG-7 while ResNet-20's accuracy decreases by ≈ 1.0%. Quantizing the first layer results in an additional ≈ 0.5% accuracy loss on all models.

Table content: <table><tr><td>METHOD</td><td>VGG-7</td><td>VGG-14</td><td>RESNET-20</td></tr><tr><td>FULL PRECISION (32W-32A)</td><td>91.23</td><td>92.49</td><td>95.02</td></tr><tr><td>∆ MCQ (QUANTIZED W)</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary The paper introduces a novel method for quantizing neural network weights and activations that does not require re-training or fine-tuning of the network. To this end the paper notes that weights and activations in a ReLU network can be scaled by a multiplicative factor, in particular such that weight-magnitudes of a layer sum to one, thus defining a probability mass function. The main idea of the paper is to cleverly sample from this PMF such that a histogram of the sampled values produces an approximate integer-representation of the original weights. The quality of this approximation can be refined arbitrarily by taking more samples, however, the value of the bin with the highest count defines the maximally required bit-precision, such that higher approximation accuracy comes at the cost of requiring larger bit-precision. The method is evaluated on several networks on CIFAR-10, SVHN and ImageNet, as well as some language models. On the image classification tasks, the paper also reports previously published relevant results from competitor methods for comparison. Quality, Clarity, Novelty, Impact The paper is well written, the main ideas are presented clearly and concisely and necessary experimental details are given in the appendix. The main idea of the paper is neat and I have not seen its application to neural network quantization before. Unfortunately I am not sure how the paper exactly fits within existing methods: while the method could in principle be used to train low bit-width networks (<4bit), the results suggest that in this domain the method is outperformed by other methods. Particularly <=2-bit weights and activations are interesting since they can be used to construct networks that replace multipliers with much faster and more energy-efficient bit-wise operations (XNOR bit-counting). But this regime seems not to be the strong suit of the proposed method. On the other hand models in the >=8-bit regime can be readily obtained with most major deep learning frameworks (e.g. Tensorflow lite and Pytorch), such that having to train or fine-tune a model is rarely a problem, unless one does not have access to training data. But even for the latter, many papers have proposed fine-tuning on dummy-data, or not fine-tuning at all (using linear or non-linear quantization schemes). If this is the targeted regime, then the literature review is missing quite a few important references that should also be compared against in the experiments section. So if the method is specifically targeted at the regime between 4- and 8-bit when re-training or fine-tuning is not possible (for some reason), then the motivation, literature comparison and experimental comparison should be much more targeted at methods that perform well in this regime. While I agree that the main idea is neat and it is nice to see it performing reasonably well, I currently do not see how the method provides strong and convincing advantages over the existing body of methods. I am very happy to be convinced of the opposite during rebuttal by the authors or other reviewers of course. Currently however, I would suggest to take a bit more time and think about particular use-cases that the method is targeted at and flesh this out much stronger - I am not sure whether the rebuttal phase is sufficient for this and therefore (currently) vote for rejection. Improvements a) The paper and the method needs a clear focus. I personally would roughly group methods into >=8-bit or <2-bit. The first one is becoming the new out-of-the-box standard and the latter allows for very efficient multiplier-free implementations (even on standard hardware to some degree, particularly for ternary/binary networks). Anything in-between might be interesting but would also probably require non-standard hardware, or variable precision hardware to be practically useful. If the target regime is between 2- and 8-bit, please expand the literature review accordingly and compare against relevant methods previously reported that operate well in this regime. b) If the emphasis is on training-free methods, also adjust the literature and comparisons accordingly, but also have a strong point when and why training-free methods are important (especially when they come with an overhead for each forward-pass during test-time). This does not mean to shorten the literature review or remove comparisons, but add the most relevant ones given that there is such a large body of network compression methods published in the last three years. c) I currently have a hard time judging to which degree the benefits reported could actually be exploited by (somewhat realistic future) hardware. Is it possible to sketch an efficient way of implementing the activation-quantization (together with quantized weights of course) on some low-bit-width hardware? While the results show that the method can work in theory with <8-bit activations, I am not sure how the quantization scheme could be efficiently implemented on actual hardware. Is it possible to do some back-of-the-envelope calculations on how much computational overhead the quantizations would add to each forward-pass (e.g. for one of the CIFAR-10/ImageNet networks). For instance, how many 32-/16-bit floating point operations would the sampling of activations roughly require, and how many would it then correspondingly save by performing 4- to 8-bit operations? Or could the whole scheme be implemented in low bit-width (<16bits)? Comments ii) Related to c) above: at which point would the computational overhead for sampling quantized activations in each forward-pass exceed the cost of retraining a network that can natively deal with low-bit activations. I don*t expect a quantitatively precise answer here, but just something that gives a rough idea of the orders of magnitude. Currently I find it very hard to judge how expensive the sampling would be after deployment. iv) In the results-tables the fractional numbers for the bit-widths (e.g. 5.6bit) are a bit misleading. While it is ok to report them, the comparison against other methods should probably use the bit-widths rounded up to the next integer. This should be mentioned at least in the text.