Year: 2021

Abstract:
Low bit quantization of weights in increasingly large deep convolutional neural networks (DCNNs) can be critical for their implementation in memory constrained hardware systems. Post-training quantization consists in quantizing a model without retraining, which is user-friendly, fast and data frugal. In this paper, we propose LatticeQ, a new post-training weight quantization method designed for DCNNs. Instead of the standard scalar rounding widely used in state-of-theart quantization methods, LatticeQ uses a quantizer based on lattices discrete algebraic structures which we show are able to exploit the inner correlations between the model parameters. LatticeQ allows us to achieve state-of-the-art results in post-training quantization, enabling us to approach full precision accuracies for bitwidths previously not accessible to post-training quantization methods in similar experimental settings. In particular, we achieve ImageNet classification results close to full precision on the popular Resnet-18/50, with only 1% and 3% accuracy drop for the 4-bit weights and 3-bit weights model architectures respectively.

heading: 1 INTRODUCTION
text: Complex tasks such as image recognition on large datasets (Krizhevsky et al., 2017; He et al., 2015) and speech recognition (Hinton et al., 2012) are very efficiently solved by deep convolutional neural networks. To improve precision, deep learning models have become continually larger, more memory demanding and computationally heavier. Popular models like ResNet (He et al., 2015) use millions of parameters (Resnet-50 uses 25Mio of them, which represents a storage of almost 100MB of data). However, for embedded systems applications, storing such quantities of information is often impossible in practice. Some architectures are specifically designed to reduce this memory footprint like MobileNet (Howard et al., 2017; Sandler et al., 2018) (Mobilenet-v2 requires just 14MB of memory). But compressing existing architectures is easier than inventing new ones for specific needs. Therefore, various methods were designed to improve memory efficiency and reduce computational complexity of these models, in order to be able to use DCNNs for low power and/or low memory applications.
There are two major ways of reducing the size of a deep neural network :
• Reducing the total number of parameters, through pruning (Blalock et al., 2020), weight sharing, low-rank factorization.
• Reducing the memory footprint of its parameters and operation complexity, mainly through quantization (Krishnamoorthi, 2018).
Recently, quantization of popular models down to 4 and even 3-bit has been made possible, with little to no degradation of task performance (Esser et al., 2020; Jin et al., 2019). These breakthroughs often rely on retraining the model from scratch with stochastic gradient descent, using the full training dataset. This class of methods is known as quantization-aware training, and leads to the best performance in DCNNs quantization. However, using these methods can sometimes be impractical. The first drawback is the need for the full dataset. In some cases we only have access to a trained full precision network, and for confidentiality reasons, it may be impossible to access the full training dataset of this network. A second drawback is their practicality: training a quantized network from scratch can be extremely demanding in computation resources, last generation GPUs, which might not be available for the developer.
To overcome these constraints, several methods have been suggested that do not rely on retraining a network from scratch. This framework is called post-training quantization. It is commonly acknowledged that post-training methods do not lead to task performance as good as quantization aware training (Krishnamoorthi, 2018), especially when dealing with bitwidths 4 or lower, but they do solve the aforementioned problems, and can be critical for rapid deployment.
Inspired by Nagel et al. (2019), we propose a classification of quantization methods given their level of complexity and data requirements, which extends their own classification. We will use this framework to help the reader compare our method with those of other authors.
• Level 1a No data and no finetuning of the network’s parameters. “As simple as an API call”. The required inputs are only the model definition and its weights. (Nagel et al., 2019).
• Level 1b No data and no finetuning of the network’s parameters. Limited data can be used to calibrate activation quantization thresholds. Per-channel quantization is widely used. (Banner et al., 2019; Zhao et al., 2019).
• Level 2a Some samples of application data are needed, no finetuning of the network’s parameters. Data determines the quantization function (i.e. step, threshold, channel splitting, mixed precision), or updates batch normalization statistics. Per-channel and per-layer quantization coexist in the state of the art. (Choukroun et al., 2019; Liu et al., 2021).
• Level 2b Some samples of application data are needed, restricted finetuning pipeline for the network’s parameters. Data is used for layerwise optimization (Nagel et al., 2020; Wang et al., 2020) or blockwise optimization (Li et al., 2021) to improve quantization locally. From this level on, per-layer quantization is standard.
• Level 3 Quantization aware training. Requires the full dataset and parameter optimization, needs finetuning and hyperparameter tuning to achieve good performance.
In this paper, we introduce a new level 1b post-training quantization technique for deep convolutional neural networks, which achieves level 1b state-of-the-art classification performance on ImageNet down to 4-bit weights on Resnet architectures, with less than 1% accuracy drop compared to full precision models. We also show good performance for 3-bit quantization with 3% accuracy drop compared to full precision. Our method relies on a new quantizer that uses linear correlations between the parameters of convolution layers to minimize its error. The goal of this paper is not to show absolute best performance among all post-training quantization methods, but rather to show some improvement over the scalar quantizer is a standard and simple experimental setting.

heading: Our contributions
text: 1. We introduce a new quantizer, based on the structure of lattices, to allow for more flexible and adaptive quantization of weights, contrary to most other techniques that rely on uniform quantizers.
2. We explore the impact of combining parameter and activation quantization for our quantizer, to allow for faster inference and low complexity computation.
3. We provide insights and analysis of how our quantizer works, and its memory overhead.
This paper is organized as follows: section 2 presents previous work on post-training quantization as well as quantization-aware training. Section 3 analyzes the parameter correlations inside DCNNs and introduces the intuition behind our approach. In section 4, we detail our approach and compare it to existing state-of-the art methods. In section 5, we give additional results of our quantizer without enhancements as an ablation study. Finally, in section 6, we provide analysis of the characteristics of our quantizer, including its quantization error, its distribution, and its memory overhead.

heading: 2 RELATED WORK
text: The performance of DCNN quantization schemes depends heavily on hypotheses: availability of training data, computation time and hardware capacities. The best results are currently achieved by quantization-aware training, which is the least restrictive: unlimited access to training data, unlimited computation time and resources, heavy hyperparameter finetuning and human expertise. Recently, promising results in this area have been achieved by Esser et al. (2020), Jin et al. (2019), which both used uniform quantizers for weights and activations. Non uniform quantizers, including vector quantizers relying on clustering techniques, have also been successfully employed (Han et al., 2016; Stock et al., 2020).
Post-training quantization meets stricter specifications than quantization-aware training. Nagel et al. (2019) for example assumed that no data is available at all for quantization. Banner et al. (2019) introduced bias correction and per-channel bit allocation. Choukroun et al. (2019) specifically designed a quantizer to minimize the MSE loss of the quantization operation. Other approaches that use a few samples of data have been proposed (Nagel et al., 2020), which used data in order to learn whether to round the weights up or down. Wang et al. (2020) proceeded by bit optimization, and Liu et al. (2021) exploited multipoint quantization to approximate full precision weight vectors with a linear combination of low bit vectors.

heading: 3 PRELIMINARY OBSERVATIONS
text: Most post-training schemes try to optimize task performance by adapting their quantizer to better fit the typically bell-shaped density function of the model weights. Moreover, these scalar methods do not pay attention to potential correlations between these parameters. As can be seen on Figure 1, correlations may exist between the weight values of trained DCNNs.
Figure 2 is the correlation matrix of the 9 distributions of filter parameters in a 3 × 3 convolution layer. The first distribution is the one of weights located in the upper left corner of a filter, the second distribution is the one of weights located in the upper center of a filter, etc. Formally, we plot in line i and column j the points (wi, wj) for each filter f = (w1, ..., w9) in a chosen layer. On the long diagonal, we plot the histogram of wi. We notice that filters tend to have correlated coordinates, with quite high correlation coefficients. The same observation can be made in other 3 × 3 layers. From this observation, we justify the main assumption of our method: a quantizer “shaped as a parallelogram” is more data efficient than a uniform quantizer “shaped as a square”.

heading: 4 METHODS
text: Many state-of-the-art quantization methods rely on scalar uniform quantizers, which require only a few additional parameters: step size, bitwidth, and/or threshold. Other approaches use vector quantization, which may either be scalar or multidimensional, leading to more flexible quantization sets (Han et al., 2016; Stock et al., 2020) but are often limited by the need to use codebooks. LatticeQ takes the best of both approaches. It adds a limited number of additional parameters to encode quantization bases, and offers a broader variety of possible quantization sets than scalar uniform quantizers.

heading: 4.1 LATTICE BASED WEIGHT QUANTIZER
text: In order to quantize the weights, LatticeQ uses lattices, which are algebraic structures that discretize the notion of vector space (see APPENDIX A for more details). Each lattice has a basis, meaning that each point of the lattice can be written as an integer linear combination of the vectors of this basis. This integer linear combination is the encoding of our quantization. A lattice has infinite cardinality. In order to use this structure as our quantizer we need a finite number of quantization points. We truncate our lattice in the following fashion : let Λ be a lattice, and B = (bi)1≤i≤n ∈ Rn a basis of Λ. Given b the bitwidth, our quantization set is Q = {q ∈ Rn, q = ∑n i=1 µibi, ∀i ∈ {1, ..., n}, −2b−1 ≤ µi ≤ 2b−1 − 1}. In memory, each quantized block is represented by its coordinates in the quantization basis. Let us suppose B = (b1,b2,b3) = ((b1,1, b1,2, b1,3), (b2,1, b2,2, b2,3), (b3,1, b3,2, b3,3)) is our quantization basis (with each bi,j a scalar, possibly quantized with a uniform min/max quantizer), and:
F q = ( f1 f2 f3 f4 f5 f6 f7 f8 f9 )
is a 3×3 quantized filter. It consists in 3 quantized blocks: (f1, f2, f3), (f4, f5, f6) and (f7, f8, f9). The convolution represented by F q is the concatenation of three 1× 3 vectors:
F = Dequant(F q) = ( (f1b1 + f2b2 + f3b3) (f4b1 + f5b2 + f6b3) (f7b1 + f8b2 + f9b3) ) Note that if we choose B as a uniform scaling of an orthonormal basis (B = λ×In), the quantization set is exactly the one of classic scalar quantization (we call this simplified version “Cubic LatticeQ”).
The quantization process for a 3 × 3 layer is simple: we flatten the weights and group them by blocks of 3. Then, using the quantization basis, we search for the quantization point nearest to this block. The vector found is the quantization point for this block. In practice, we can use Dequant to calculate the convolution in the quantized network. However, for per-layer quantization, we propose an optimized way of dealing with the inference in a LatticeQ-quantized network in APPENDIX E.
Quantizing literally means defining a function from an infinite set to a finite set. In the scalar model, the quantization operation relies on a simple round function. In order to quantize on a lattice, we need to use a more complex algorithm. Actually, the problem of finding the closest lattice point to a real vector is known to be NP-hard (Micciancio, 1998), and is called Closest Vector Problem (CVP):
“Given x ∈ Rn and Λ a lattice of Rn, find λ ∈ Λ such that d(x, λ) = min{d(x, l), l ∈ Λ}.”
Algorithm 1 Nearest plane algorithm 1: function BABAI(Basis B, Vector t) 2: B∗ = GramSchmidt(B) 3: b← t 4: for j ∈ {n, ..., 1} do 5: uj ← <b,b∗j>
<b∗j ,b ∗ j>
6: b← b− bujebj 7: end for 8: return x = ∑n j=1bujebj = t− b 9: end function
Algorithm 2 Gram-Schmidt algorithm 1: function GRAMSCHMIDT(Basis B) 2: b∗1 ← b1 3: for j ∈ {2, ..., n} do 4: b∗j ← bj − ∑j−1 i=1 < bj , b ∗ i > b ∗ i
5: end for 6: return B∗ = (b∗j )1≤j≤n 7: end function
In order to solve the closest vector problem, we use the nearest plane algorithm (Babai, 1986). It is fairly easy to understand and implement, computationally light, and still provides good approximations in low dimension. We detail the implementation in algorithms 1 and 2.
Now that we have an algorithm to quantize on a lattice, we want to find a good lattice. For the data free approach, we opt for a simple random search with restart as described in algorithm 3. Restarts simply consist in running the algorithm several times in a row and keeping the best result of all the runs. We look for a lattice that reduces the mean cube error loss (MCE) between the full precision weights of the layer (or channel), and their quantized version. We chose this loss rather than MSE because it performed slightly better (Table 1). We assume this is because the MCE loss gives more importance to larger weights that are far away from any quantization point compared to the MSE.
Algorithm 3 FindBasis 1: function FINDBASIS(Weight tensor W , Basis dimension dim, Bitwidth bits, Temperatures T ) 2: B ← Idim 3: for 0 ≤ s ≤ |T | do 4: B′ ← B′ + Sample(G(0, σ = T (s), |B|)) 5: if MCELoss(W,W qΛB′ ) < MCELoss(W,W q ΛB
) then 6: B ← B′ 7: end if 8: end for 9: return B
10: end function
The random transformation we use is a multivariate gaussian noise addition with standard deviation T (s), where T is a sequence of temperature steps, that can be adjusted as a hyperparameter, see APPENDIX B. In order to perform low bit computations, we may want to quantize the basis itself to integer values, which we did in all our experiments, by adding a simple min/max quantizer for B′ between line 4 and line 5 in algorithm 3.
Since the approximation factor of the nearest plane algorithm grows in square root in the basis dimension n (Babai, 1986), there is no interest in choosing high dimensional bases. Moreover, it increases the storage requirements, since an n-dimensional basis requires n2 entries in memory. Experimentally, we notice that the most accurate dimension for the quantization basis is 3 for 3×3 layers (Table 2), and 2 for 1×1 layers.

heading: 4.2 ACTIVATION QUANTIZATION
text: As typically done in post-training scenario, we chose to perform activation quantization on one batch of data right before testing the quantized network. In real life, this can be done right before deployment on one batch of application data. We can either quantize activations per layer or per channel. In case we quantize the activations in a layer-wise fashion, the performance drops significantly below the 8-bit setting. In case we quantize the activations in a channel-wise fashion, it is possible to reach an activation precision down to 4-bit without much loss of accuracy by using a quantile trick similar to McKinstry et al. (2019). Let C be an activation channel to quantize. Let αC be the quantile 0.9997 of C’s distribution, which means pc∈C(c ≤ αC) = 0.9997. We estimate αC by running a forward pass with one batch of data. Then we use αC as our activation threshold for the channel C. We chose different quantiles depending on the network and activation bitwidths, see APPENDIX C.

heading: 4.3 EXPERIMENTAL SETUP
text: We evaluate our method on the ImageNet (Russakovsky et al., 2015) classification task. All experiments are made using PyTorch (Paszke et al., 2019), and the pretrained models used all come from the torchvision.models library. The basis is always quantized with the same bitwitdh as the activations. Based on experiments, we set bases dimension to 3 for 3 × 3 convolution layers, 2 for 1 × 1 layers and linear layers, and 1 for the first layer (since we did not notice any advantage in increasing the dimension for this particular layer). To quantize activations, we use a calibration batch of 512 images. As it is common practice in such applications, we quantize the pooling layers, first layer and last layer to 8-bit, and we do not quantize layer biases (only VGG (Simonyan & Zisserman, 2015) has biases). We report bitwidths settings and top-1 accuracy for each model tested, and we also provide the results from Banner et al. (2019) and Choukroun et al. (2019) for comparison. In part 4.4, we report the results of per-channel LatticeQ with data free enhancements in Table 3. In part 5, we report the results of per-channel LatticeQ without bias correction (refered to as our baseline) in Table 4 and the results of per-layer LatticeQ without bias correction in Table 5. Finally, in APPENDIX F, we report the results of weights-only quantization.

heading: 4.4 DATA FREE PER-CHANNEL ENHANCEMENT
text: For this part, we rely on the work of Banner et al. (2019). First, we use per-channel quantization, to allow for more accurate quantization in each channel. We then add bias correction. Quantization operations tend to alter the moments (mean and variance) of the weight distribution in each channel, which is taken into account by bias correction. In our case, bias correction is also computed during the execution of FindBasis for bitwidths lower than 4, so that the quantization basis found takes bias correction into account. Line 5 of FindBasis is, in this case, replaced by :
if MCELoss(W, biascorrection(W qΛB′ )) < MCELoss(W, biascorrection(W q ΛB )) then B ← B′ end if
The results in Table 3 show the substantial improvement of LatticeQ over state-of-the-art approaches. Our method outperforms Banner et al. (2019) with similar hypotheses, and even the level 2a OMSE+opt method, in almost all settings for all presented models.
As we read the Table 3 from left to right, quantization is more and more aggressive. We find that LatticeQ manages to stay within 1% of full precision model accuracy on Resnets and VGG in 4- bit weights and 8-bit activation, and within 3% in 3-bit weights and 8-bit activation. As expected, compact models like Densenet (Huang et al., 2018) and Mobilenet-V2 (Sandler et al., 2018) are less resilient to quantization than Resnets (He et al., 2015) and VGG (Simonyan & Zisserman, 2015). It is noticeable that our method reaches a top-1 accuracy of 54.2% for Densenet in 3-bit weights and 3-bit activation setting while the method proposed by Banner et al. (2019) only achieved 18.7%.

heading: 5 ABLATION STUDY
text: In this section, we present an ablation study which objective is twofold. First, we provide the results of our per-channel quantizer without bias correction (Table 4). It is noticeable that our results remain close to full precision accuracy on Resnets and VGG (within 2% in per-channel W4A8, and within 10% in W3A8 for instance). Bias correction increases our method’s accuracy, but our baseline significantly outperforms baselines of other scalar quantization methods (Banner et al., 2019; Choukroun et al., 2019). Based on those results, we want to emphasize that lattice quantization is indeed the cornerstone of the quantization process.
Next, we provide the results of our per-layer quantizer (Table 5). Quantizing per layer in W8A8 does not impact performance on Resnets. Although it remains challenging to quantize weights per layer under our assumptions (no calibration data for weights), we show promising accuracy that lays the foundations of potential level 2 applications.
1OMSE+opt (Choukroun et al., 2019), uses calibration data to setup its quantizer (level 2a).
Table 5: LatticeQ per-layer baseline quantization of weights and activations.
Top-1 accuracy Network FP32 W8A8 W4A8 W3A8
Resnet-18 69.6 69.5 59.5 23.8 Resnet-50 76.0 75.9 70.2 41.9 VGG-16bn 73.4 73.3 68.5 42.5 Densenet 74.4 71.3 59.4 11.4
Mobilenet-v2 71.9 70.6 12.9 0.2

heading: 6 ANALYSIS

heading: 6.1 LATTICE QUANTIZATION VERSUS CUBIC QUANTIZATION
text: Table 6: Comparison between baseline perchannel LatticeQ and baseline per-channel Cubic LatticeQ (scalar quantization).
Network Method FP32 W4A8
Resnet-18 LatticeQ 69.6 67.2Cubic LatticeQ 69.6 57.6
Figure 4: Left : Cubic LatticeQ quantization points (red) and 1×3 filter blocks (blue), Right : LatticeQ quantization points (red) and 1×3 filter blocks (blue). These are the 2-bit quantization points for visualization.
We have demonstrated experimentally the advantages, both in quantization error - as can be seen in Figure 3 - and task loss (Table 6), in using a deformable lattice for quantization rather than a cubic lattice (which is equivalent to uniform scalar quantization). This confirms our hypothesis that the inner correlations of the parameters of a neural network can be exploited for the purpose of quantization. We also show how the distribution of the quantization points indeed fits the multidimensional distribution of the network’s parameters thanks to our method (Figure 4). On this figure, each full precision filter block is represented by a blue dot in the 3D space, and each quantization point of our method is represented by a red dot in the 3D space. LatticeQ increases the concentration of quantization points in the most critical areas of the filters’ multidimensional distribution.

heading: 6.2 MEMORY OVERHEAD
text: In this section, we compute the memory overhead due to using our method, both in per-channel and per-layer settings. Each time a basis is used to quantize either a channel or a layer, we need to store 32 + n2.b bits where n is the dimension of the basis and b is the number of quantization bits used for basis elements. 32 comes from the scaling factor. See Table 7 for a few examples among the networks we experimented with in this paper. We report the compression ratios of the full models. The compression rate penalty due to quantization bases is negligible in the per-layer setting, and always less than 1% in the per-channel setting. The compression that can be achieved by descending to lower bitwidths therefore largely offsets the memory overhead.

heading: 7 DISCUSSION AND FUTURE WORK
text: In this paper, we introduced LatticeQ, a new post-training method which exploits the flexibility of lattice quantizers for the purpose of DCNN quantization. LatticeQ is particularly useful in cases where we want to deploy deep learning models trained in floating point precision on lightweight architectures without requiring a single training sample (which could happen for confidentiality, safety reasons, or for the sake of simplicity). We showed that our quantizer significantly outperforms the scalar quantizer for 3-bit quantization on several well-known architectures, and by up to 20% on Resnet-18, with less than 1% additional memory costs. LatticeQ does not require any finetuning or hyperparameter optimization, which makes it simple to use in practice.
Since lattice quantizers are a generalization of uniform quantizers, every uniform quantization method has a lattice extension. Therefore, we believe that lattice quantizers could have a high potential under other experimental hypotheses, like quantization-aware training. Using limited calibration data in order to perform parameter optimization after quantization could also improve performance, which we leave for future work.

heading: 8 REPRODUCIBILITY STATEMENT
text: We want to make sure our results are reproducible. As suggested in the author guide, we will make a comment directed to the reviewers and area chairs and put a link to an anonymous repository to submit our code. Due to the randomness of our optimization strategy and the choice of calibration data, results may slightly vary. We mitigated this issue by adding restarts to our random search.

heading: APPENDIX A

heading: Lattices
text: The following part is a very synthetic introduction to the (very rich) theory of lattices. It states only what the reader needs to know to understand how our quantization method works. If the reader wants to know more about lattices, they are welcome to start with Lenstra (2008).
Definition 1
A lattice Λ of Rn is a discrete additive subgroup of Rn, such that span(Λ) = Rn, where span(Λ) is the set of linear combinations of the vectors of Λ.

heading: Example
text: Zn is a lattice. It is called the cubic lattice.
Property 1
Let Λ be a lattice of Rn, there is a sequence B of cardinality n of lattice vectors such that any vector that belongs to Λ can be uniquely expressed as an integer linear combination of the elements of B. Such a sequence is called basis of Λ.

heading: Examples
text: • ((1, 0), (0, 1)) is a basis of Z2
• ((1, 0), (1, 1)) is a basis of Z2
• (δi,i)1≤i≤n is a basis of Zn

heading: Observation
text: The knowledge of a basis of a lattice Λ is sufficient to know every point of the lattice. Given that we want to use this type of structure to quantize a weight distribution, this observation allows us to avoid keeping the whole lattice stored as a codebook in memory.

heading: Important observation
text: We shall emphasize that the uniform scalar quantization scheme widely used for neural networks quantization is nothing else but a cubic lattice quantization scheme. Therefore, lattice quantization is nothing else but a generalization of scalar techniques.

heading: APPENDIX B

heading: Heuristic details
text: This paragraph is dedicated to providing the details of our heuristic FindBasis (algorithm 3). Let sc = 1
2b−1 where b is the bitwidth of the channel, or layer, to be quantized. Basis B is ini-
tialized as sc104 ∗ In. T consists in a sequence of 800 steps at each of these deviation values : [ sc104 ,sc, sc 2 , sc 3 , sc 5 , sc 7 , sc 9 , sc 15 , sc 30 ]. We apply this algorithm with 5 restarts and keep the best basis at each restart. Note that it is possible to reduce the number of steps and hence shrink computation time.

heading: APPENDIX C

heading: Quantile tables

heading: APPENDIX D

heading: Intuition on vector quantization
text: We know for a fact that correlated distributions share mutual information, in the sense of Shannon information theory. Shannon’s mutual information between continuous random variables X and Y is calculated as a double integral :
I(X;Y ) = ∫ Y ∫ X p(X = x, Y = y)log2( p(X = x, Y = y) p(X = x)p(Y = y) ) dx dy (1)
If X and Y are independant, I(X;Y ) = 0, they share no mutual information. When X = Y , I(X;Y ) = H(X) = H(Y ). Now, we model the pairs of correlated weights of the layer we want to quantize with (w1, w2) which follows a bivariate normal distribution with location µ = (0, 0) and covariance matrix :
Σ =
( σ21 ρσ1σ2
ρσ1σ2 σ 2 2
) (2)
where σ1, σ2 are the standard deviations of w1 and w2 and ρ their correlation coefficient. This yields:
I(w1;w2) = − 1
2 log2(1− ρ2) (3)
Therefore, the more w1 and w2 are correlated, the more information they share, which means that an independent coding of w1 and w2 scalars is suboptimal, because of information redundancy. This is the idea behind our approach and behind vector quantization in general.

heading: APPENDIX E
text: Optimized inference for LatticeQ networks In this section, we suppose that we have a 3 × 3 convolution layer Λ quantized using our per-layer method, with in input channels and out output channels. We want to compute a forward pass through this layer. Let (Cini )1≤i≤in the set of input channels and (C out j )1≤j≤out the set of output channels. We want to compute Coutj for each j. Let W q the tensor of quantized weights. Let the quantization basis B = (b1,b2,b3,b4,b5,b6,b7,b8,b9). Note that in this notation, our basis has dimension 9, which is not the choice we made in the paper. This is not a big deal, since we can choose to fill coordinates with zeros: b1 = (b1,1, b1,2, b1,3, 0, 0, 0, 0, 0, 0), b4 = (0, 0, 0, b1,1, b1,2, b1,3, 0, 0, 0), b7 = (0, 0, 0, 0, 0, 0, b1,1, b1,2, b1,3). In this manner, our 3D bases can be expanded in 9D bases.
Usually, in a full precision network, Coutj is computed as :
Coutj = in∑ i=1 Conv(Wi,j ;C in i ) (4)
In our case, we change the order of operations :
Coutj = 9∑ k=1 in∑ i=1 W qi,j,kConv(bk;C in i ) (5)
Since Conv(bk;Cini ) does not depend on the output channel, we can start by computing Conv(b1;C in i ) for each i (in convolutions). b1 is uniformly quantized using a simple min/max quantizer, therefore we can use low-bit operators to compute these convolutions. Then, we only need to multiply the result by W qi,j,1 (which is an integer) for each j, and store the result in the corresponding output channel. Then, we reiterate the process with b2, b3, etc.
With this method, the complexity of the forward pass through Λ is 9× in low-bit convolutions and 9× in× out scalar multiplications and additions.

heading: Weight-only quantization

heading: APPENDIX F

Review summaries:
This paper proposes a new post-training quantization method for DCNNs using vector quantization format w/o additional gradient updates. The proposed method calibrates the quantized model using sampled dataset only for activations. For the weights, lattice quantization, which seems to be a sort of vector quantization methods, is used without any data for the calibration/re-training step. For the activations, a small dataset is used for conventional activation quantization. This paper also includes the outperformed results compared to previous methods, AdaQuant and OMSE. 

LATTICE QUANTIZATION provides a novel way to quantize weights.  This paper relies on weight correlation inside convolution filters so we can select better basis to represent weights (so called "Lattice quantization").  Firstly authors quantize numbers to lattice and then find an good lattice.  Due to weight correlation, a good lattice quantization can be more efficient than conventional uniform number representation (Cubic LatticeQ).    In the same bit precision, Lattice quantization achieved less quantization error in convolution layers.  Less quantization in Lattice quantization also translates to better Neural Network accuracy (compared to conventional scalar uniform quantization).    

The paper explores the post-training inference quantization. Observing that the weight distributions between filters are often highly correlated, the authors propose a new quantizer based on algorithmic theory of lattices for the quantization of weight tensors. The method is verified on popular vision models with precision down to 4/3 bits.