WAVEQ: GRADIENT-BASED DEEP QUANTIZATION OF NEURAL NETWORKS THROUGH SINUSOIDAL REGULARIZATION
Authors: Ahmed T. Elthakeb
Prannoy Pilligundla
Tarek Elgindi
Fatemehsadat Mireshghallah
Charles-Alban Deledalle
Hadi Esmaeilzadeh
Venue:  (2021)

Abstract: Deep quantization of neural networks below eight bits can lead to superlinear benefits in storage and compute efficiency. However, homogeneously quantizing all the layers to the same level does not account for the distinction of the layers and their individual properties. Heterogenous assignment of bitwidths to individual layers is attractive but opens an exponentially large non-contiguous hyperparameter space (${Available Bitwidths}^{\# Layers}$). As such finding the bitwidth while also quantizing the network to those levels becomes a major challenge. This paper addresses this challenge through a sinusoidal regularization mechanism, dubbed WaveQ. Adding our parametrized sinusoidal regularizer enables us to not only find the quantized weights but also learn the bitwidth of the layers by making the period of the sinusoidal regularizer a trainable parameter. In addition, the sinusoidal regularizer itself is designed to align its minima on the quantization levels. With these two innovations, during training, stochastic gradient descent uses the form of the sinusoidal regularizer and its minima to push the weights to the quantization levels while it is also learning the period which will determine the bitwidth of each layer separately. As such WaveQ is a gradient-based mechanism that jointly learns the quantized weights as well as the heterogeneous bitwidths. We show how WaveQ balance compute efficiency and accuracy, and provide a heterogeneous bitwidth assignment for quantization of a large variety of deep networks (AlexNet, CIFAR-10, MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11) that virtually preserves the accuracy. WaveQ is versatile and can also be used with predetermined bitwidths by fixing the period of the sinusoidal regularizer. In this case. WaveQ enhances quantized training algorithms (DoReFa and WRPN) with about 4.8% accuracy improvements on average, and outperforms multiple state-of-the-art techniques. Finally, WaveQ applied to quantizing transformers


Under review as a conference paper at ICLR 2021
WAVEQ: GRADIENT-BASED DEEP QUANTIZATION OF NEURAL NETWORKS THROUGH SINUSOIDAL REGU- LARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep quantization of neural networks below eight bits can lead to superlinear ben- efits in storage and compute efficiency. However, homogeneously quantizing all the layers to the same level does not account for the distinction of the layers and their individual properties. Heterogeneous assignment of bitwidths the layers is attractive but opens an exponentially large non non-contiguous hyperparameter space (Available Bitwidths Laayen, Thus, finding the bitwidth while also quantizing the network those levels becomes major challenge. This paper addresses this chal- lenge through a sinusoidal regularization mechanism, dubbed WaveQ. Adding our parametrized sinusoidal regularizer enables WaveQ to not only find the quantized weights, but also learn the bitwidth of the layers by making the period of the si- nusoidal regularizer trainable parameter. addition, the sinusoidal regularizer itself is designed to align its minima the quantization levels. With these two innovations, during training, stochastic gradient descent uses the form of the si- nusoidal regularizer and its minima to push the weights to the quantization levels while it is also learning the period which will determine the bitwidth of each layer separately. As such WaveQ is gradient- based mechanism that jointly learns the quantized weights as well as the heterogeneous bitwidths. We show that WaveQ balances compute efficiency and accuracy, and provides heterogeneous bitwidth assignment for quantization of large variety of deep networks (AlexNet, CIFAR- 10. MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11) that virtually pre- serves the accuracy. WaveQ is versatile and can also be used with predetermined bitwidths by fixing the period of the sinusoidal regularizer. In this case, WaveQ, average, improves the accuracy of quantized training algorithms (DoReFa and WRPN) by ~ 4.8%, and outperforms multiple state-of-the-arr techniques. Finally, WaveQ is applicable quantizing transformers and yields significant benefits.
INTRODUCTION
Quantization, in general, and deep quantization (below eight bits) (Krihhaamoortii 2018). 8), in par- ticular, aims to not only reduce the compute requirements of DNNs but also reduce their memory footprint (Zhou et al.] 2016) Judd et al. 2016b, Hubara et al. 2017: Mishra et al.) 2018 Sharma al. 2018). Nevertheless, without specialized training algorithms, quantization can diminish the accuracy. As such, the practical utility of quantization hinges upon addressing two fundamental challenges: (1) discovering the appropriate bitwidth of quantization for each layer while consider- ing the accuracy; and (2) learning weights in the quantized domain for given set of bitwidths.
This paper formulates both of these challenges as gradient-based joint optimization problem by in- troducing an additional novel sinusoidal regularization term in the training loss, called WaveQ. The following two main insights drive this work. (1) Sinusoidal functions (sin?) have inherent periodic minima and by adjusting the period, the minima can be positioned on quantization levels corre- sponding to a bitwidth at per-layer granularity. (2) As such, sinusoidal period becomes a direct and continuous representation of the bitwidth. Therefore, WaveQ incorporates this continuous variable (i.e.. period) as a differentiable part of the training loss the form of a regularizer. Hence, WaveQ is differentiable regularization mechanism, it piggy backs the stochastic gradient descent that trains the neural network to also learn the bitwidth (the period). Simultaneously this parametric sinusoidal regularizer pushes the weights to the quantization levels (sin? minima).
By adding our parametric sinusoidal regularizer to the original training objective function, our method automatically yields the bitwidths for each layer along with nearly quantized weights for

---

Under review as a conference paper at ICLR 2021
those bitwidths. In fact, the original optimization procedure itself is harnessed for this purpose, which is enabled by the differentiability of the sinusoidal regularization term. As such, quantized training algorithms (Zhou et al. 2016: Mishra et al. 2018) that still use some form of backpropa- gation (Rumelhart et al. 1986) can effectively utilize the proposed mechanism by modifying their loss. Moreover, the proposed technique is flexible as it enables heterogeneous quantization across the layers. The WaveQ regularization can also be applied for training model from scratch, or for fine-tuning a pretrained model.
In contrast to the prior inspiring works (Uhlich et al. 2019 Esser et al. 2019), WaveQ is the only technique that casts finding the bitwidths and the corresponding quantized weights as a simultane- ous gradient-based optimization through sinusoidal regularization during the training process. We also prove a theoretical result to provide an insight on why the proposed approach leads to so- lutions preserving the original accuracy during quantization. We evaluate WaveQ using different bitwidth assignments across different DNNs (AlexNet, CIFAR-10, MobileNet, ResNet- 18, ResNet- 20, SVHN, and VGG-11). To show the versatility of WaveQ, it is used with two different quan- tized training algorithms, DoReFa (Zhou et al., 2016) and WRPN (Mishra et al., 2018). Over all the bitwidth assignments, the proposed regularization,
average, improves the top-1 accuracy of DoReFa by 4.8%. The reduction in the bitwidth, on average, leads to 77.5% reduction in the en- ergy consumed during the execution of these networks. Finally, we apply WaveQ to Transformer DNNs citeppby augmenting their loss with WaveQ parametric sinusiodal regularization. In this case, the conventional stochastic gradient descent plus WaveQ regularization is used to quantize the big Transformer model from (Ott et al. 2018) for machine translation on the IWSLT14 Berman-Englisg dataset (TWS). For 5, 6, and 7-bit quantization, training with WaveQ yields 0.46, 0. 14, 0.04 im- proved Bil Bingul Evaluation Understudy (BLEU) score, respectively. As a point of reference, the original big Transformer model from (Ou et al.] 2018) improves the BLEU by only 0.1 over the sfate-of-the-art. Code available at /tttttss//giii //ggt com/ htnn/wave--reee /waveq
JOINT LEARNING OF LAYER BITWIDTHS AND QUANTIZED PARAMETERS
Our proposed method WaveQ exploits weight regularization in order to automatically quantize a neural network while training. To that end, Section [. describes the role of regularization in neural networks and then Section 2.2 explains WaveQ in more details.
2.1 PRELIMINARIES
Quantizer. We discuss how quantization of weight works. Consider a floating-point variable Wy to be mapped into a quantized domain using (b + 1) bits. Let @ be a set of (2k + 1) quantized values, where k = 26 1. 1. Considering linear quantization, Q can be represented as & ..... , 0, E .... ا-ع 11. where 1 is the size of the quantization bin. Now, wy can be mapped to the b-bit quantization Zhou et al. (2016) space as follows.
{-1, k-1
tanh(wy) maxx(Ianh( (I) +
1 -/2
Uyo quantize,
(2.1)
In Equation 2.1 quantize, polrroun((?) (2 1)z). w is a scalar, Wy is a vector, and Ugo is a scalar in the range [-1, 1]. Then, a scaling factor e is determined per layer to map the final quantized weight wy into the range [-c, +c]. As such, Wo takes the form eWg where e > 0, and Wgo Q.
Soft constraints through regularization and the loss landscape of neural networks. Neu- ral networks' loss landscapes are known to be highly non-convex and it has been empirically verified that loss surfaces for large networks have many local minima that essentially pro- vide equivalent test errors Choromanska et al. ( Li et al. (2018). This opens up the pos- sibility of adding soft constrains as extra cus- tom objectives during the training process, in addition to the original objective (i.e., to mini- mize the accuracy loss). The added constraint could be with the purpose of increasing gener-
Loss surface (Joss conatent)
Pertodic pattern of minima
Wopt
Wepp
(4)
Inseritted rofieal
Figure 1: Sketch for a hypothetical loss surface (original task loss to be minimized) and an ex- tra regularization term in 2-D weight space: for (a) weight decay, and (h) WaveQ. alization or imposing some preference on the weights values.
2

---

Under review as a conference paper at ICLR 2021
Sirusoidal Requiarization
140
R 2
2
(b)
-1.00
-0.75
Weight W
130
0.55
(c)
Weight:
Rog 1010
Phate
Phase
Phase
Briwitt Molmization
Broidh
Weggle
0.5 08
(a)
(d)
Bhwldh 8
Earations
Figure 2: (a) 3-D visualization of the proposed generalized objective WaveQ. (b) WaveQ 2-D profile, w.r.t weights, adapting for arbitrary bitwidths, (c) example of adapting ternary quantization. (d) WaveQ 2-D) profile w.r.t bitwidth. (e) Regularization strengths profiles, Aw and Ag, across training iterations.
Regularization in action. Regularization effectively constrains weight parameters by adding corresponding term (regularizer) to the objective loss function. A classical example is Weight De- cay Krogh & Hertz (I99I) that aims to reduce the network complexity by limiting the growth of the weights. This soft constraint is realized by adding term, proportional to the L2 norm of the weights the objective function as the regularizer that penalizes large weight values. WaveQ, on the other hand, uses regularization to push the weights to the quantization levels. For the sake of simplicity and clarity, Figure a and (b) illustrate geometrical sketch for a hypothetical loss sur- face (original objective function to be minimized) and extra regularization term in 2-D weight space, respectively. For weight decay regularization (Figure [ (a)), the faded circle shows that as we get closer to the origin, the regularization loss is minimized. The point Wopt is the optimum just for the loss function alone and the overall optimum solution is achieved by striking a balance between the original loss term and the regularization loss term. Similarly, Figure shows representation of the proposed periodic regularization for fixed bitwidth B. A periodic pattern of minima pockets are seen surrounding the original optimum point. The objective of the optimization problem is to find the best solution that is the closest to one of those minima pockets where weight values are nearly matching the desired quantization levels, hence the name quantization -itiiiiaatioo-ieellly...
2.2 WAVEQ REGULARIZATION
The proposed regularizer is formulated in Equation (2.2) where the first term pushes the weights to the quantization levels and the second correlated term aims to reduce the bitwidth of each individual layer heterogeneously.
sin² (Tw:,(2'. 1))
R(w; B) Xu ΣΣ
B
3
(2.2)
26.
Weights quantization regularization
Bitwidth regularization
In Equation (2.2). Am is the weights quantization regularization strength which governs how strongly weight quantization errors are penalized, and A is the bitwidth regularization strength. The param- eter 83 is proportional to the quantization bitwidth which is claborated later in this section. Figure (a) shows a 3-D visualization of our regularizer, R. Figure[(b) (c) show a 2-D profile w.r.t weights (w), while (d) shows 2-D profile w.r.t the bitwidth (3)).
Periodic sinusoidal regularization. As shown in Equation equation 2.2 the first regularization term is based periodic function (sinusoidal) that adds smooth and differentiable term to the original objective, Figure (b... The periodic regularizer induces a periodic pattern of minima that correspond to the desired quantization levels. Such correspondence is achieved by matching the period to the quantization step (1/(28: - 1)) based on a particular number of bits (3,) for given layer
Learning the sinusoidal period. The parameter 3; in (Equation 2.2 controls the period of the sinusoidal regularizer. Thereby 3, is directly proportional to the actual quantization bitwidth (b:) of layer i as follows:
(2.3) In Equation B.2) where a; E R is a scaling factor. Note that by E Z is the only discrete param- eter, while 3, R is continuous real valued variable, and [+]. is the ceiling operator. While the first term in Equation equation 2.2 is only responsible for promoting quantized weights, the second term (As Σ, BB) aims to reduce the number of bits for each layer i individually while the
b, = [[,], and a, = 2//////
3

---

Under review as a conference paper at ICLR 2021
overall loss is aiming maximize accuracy. As such, this term is a soft constraint that yields het- erogeneous bitwidths for different layers. The main insight here is that 8,, which also controls the period of the sinusoidal term, is continuous valued parameter by definition. As such, 3 acts an ideal optimization objective constraint and a proxy to minimize the actual quantization bitwidth 4;- Therefore, WaveQ avoids the issues of gradient-based optimization for discrete valued parameters. Furthermore, the benefit of learning the sinusoidal period is two-fold. First, it provides a smooth differentiable objective for finding minimal bitwidths. Second, simultaneously learning the scaling factor (a;) associated with the found bitwidth.
Leveraging the sinusoidal properties, WaveQ learns the following two quantization parame- ters simultaneously: (I) a per-layer quantization bitwidth (6;) along with (2) a scaling fac- tor (a;)) through learning the period of the sinusoidal function. Additionally, by exploit- ing the periodicity, differentiability, and the local convexity profile in sinusoidal functions, WaveQ automatically propels network weights towards values that are inherently closer to quantization levels according to the jointly learned quantizer's parameters by ag as follows.
Rune
Mapping this to the quantizer's ss design, for (b+ 1) bits quantization, & = 2 1. and c = a. These learned parameters (b, a), as explained above, can be mapped to the quantizer param- eters explained in Equation equation 2.1 For (b+1) bits quantization (the extra bit is the sign bit):
auPude, Unsuunaded
Dounded
k == 2 1 and c= = /28 (2.4)
Bounding the gradients. The denomina- tor in the first term of Equation equation 2.2
Baube Uakaersee
sin sin" (2ww)(2" 22:
Desnded
is used to control the range derivatives of the proposed regular- ization term with respect to 3. This denomina- tor is chosen limit vanishing and exploding gradients during training. To this end, we com- pared three variants of equation equation 2.2 with different normalization defined, for k = 0. 1, and 2, as:
E,
(a)
(b)
(c)
Figure 3: Visualization for three variants of
the proposed regularization objective using dif- ferent normalizations and their respective first and second derivatives with respect to B. (a) Ro B), (b) Rí (w; 19), and (c) R2(w; B).
sin (Imiy (2 13) 243;
(w: 8) ΣΣ
(2.5)
Figure 3 (a), (b), (c) provide visualization on how each of the proposed scaled variants affect the first and second derivatives. For Rı=o and Rı=2, there are regions of vanishing or exploding gradients. Only the regularization R&=1 (the proposed one) is free of such issues.
Setting the regularization strengths. The convergence behavior depends the setting of the regularization strengths A and Ag. Our proposed objective seeks to learn multiple quantization parameterization conjunction. As such, the learning process can be portrayed as three phases (Figure 2(e)). In Phase ①), the learning process optimizes for the original task loss Eg. Initially, the small A and Àø values allow the gradient descent explore the optimization surface freely. As the training process moves forward, we transition phase gradually engage both the weights quantization regularization and the bitwidth regularization, re- spectively. Note that, for this work, the strength of the weights quantization regularization A should be higher than the strength of the bitwidth regularization Å such that bitwidth per layer could be properly evaluated and eventually learned during this phase. After the bitwidth regular- izer converges
where the larger A and Xg
bitwidth for each layer, we transition to phase bitwidths and gradually decay À while we keep An high. The criterion for choosing X and Xg is to balance the magnitude of regularization loss to be smaller than the magnitude of accuracy loss. The mathematical formula used to generate Xn and A profiles can be found the supplementary material. (Figure 8).
where we fix the learned
4

---

Under review as a conference paper at ICLR 2021
3
THEORETICAL ANALYSIS
The results of this section are motivated as follows. Intuitively, we would like to show that the global minima of E' = Eg + R are very close to the minima Eg that minimizes R. In other words, we expect to extract among the original solutions, the ones that are most prone to be quantized. To establish such result, we will not consider the minima of = Eg + R. but the sequence S, of minima of E,, Eg 5, R defined for any sequence 6, of real positive numbers. The next theorem shows that our intuition holds true, at least asymptotically with v. provided 8n + 0.
Theorem 1. Let Eg. R R + 00) be continuous and assume that the set SE of the global minima of Eo is non-empty and compact. As S is compact, we can also define SEo, C SE, as the set of minima of Eg which minimizes R. Let on be a sequence of real positive numbers, define En Eo + 5n R and the sequence S,, Fr,, of the globul minima of En. Then, the following holds tre:
8f +0 and Sn + S., then S. S RR
2. 8 + then there is a subsequence on - and a non-empty set S. S S so that Sm S..
where the convergence of sets, denoted by Sn - Sa, is defined the convergence 0 of their Haussdorff distance, i.e., lim d S.) = 0.
n-000 Proof. For the first statement, assume that Sn S.. We wish show that S. S S R. Assume that In is a sequence of global minima of F + 6, converging to I.. It suffices to show that T, S Ea.R. First let observe that x. BB- Indeed, let
== inf Fo(x)
EER
and assume that SE. Then,
À - Eo(xn) S (Eo + 8n R)()) & (Eo +,, = X R,.(x).
Thus, since En is continuous and vn + H, we have that Eo(x,) = which implies x, E SE Next, define
# inf R(x).
zESEo
Let E SF n so that R(t) μr. Now observe that, by the minimality of a. we have that
8,11 14 (Eg + 6, R)) 2 (Fo 5, RIx,)) À 5, R(rn))
Thus, R(x,) S for all n. Since R is continuous and xn + I. we have that R(x,) S j which implies that R(x.)= = μ since x. SF. Thus, x., SF. R. The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the first statement.
O
Theorem M implies that by decreasing the strength of R. one recovers the subset of the original
solutions that achives the smallest quantization loss. In practice, minima, and we should not decrease much the strength of R. In our context, Theorem [ should then be understood Experiments carried out in the next section will support this claim. Additionally, note that while the theorem is stated in terms of limit the regularization parameter vanishes, the proof in fact gives a corresponding stability result. Namely, if the regularization parameter is sufficiently small relative to the main loss then the minimizers will be "almost" quantized. For the interested reader, we provide more detailed version of the above analysis in the supplementary material.
not interested in global
proof of concept why the proposed approach leads the expected result.
EXPERIMENTAL RESULTS
To demonstrate the effectiveness of WaveQ, we evaluated it on several deep neural networks with different Image Classification datasets (CIFARIO, SVHN, and ImageNet), and one Transformer- based network, which is the big Transformer model from(Ott et al. (2018) on the IWSLT14 German- English dataset (IWS). We provide results for two different types of quantization. First, we show quantization results for learned heterogeneous bitwidths using WaveQ and we provide different anal- yses to asses the quality of these learned bitwidth assignments. Second, we further provide results assuming a preset homogeneous bitwidth assignment as special setting of WaveQ. This is, in some cases, a practical assumption that might stem from particular hardware requirements or constraints. Table I provides summary of the evaluated networks and datasets for both learned heterogeneous bitwidths, and the special case of training preset homogeneous bitwidth assignments. We com- pare our proposed WaveQ method with PACT (Choi al. 2018a), LQ-Nets (Zhang et al. 2018).
5

---

Under review as a conference paper at ICLR 2021
Table 1:
Comparison with state-of-the-art quantization methods on ImageNet. The W/A values are the bitwidths of weights/activations.
Benchmack
AlexNet
ReaNes-18
Vo
Quamization
Method
Assignment
Top-1
Top-5
Top-1
Top
Top-1
Top-5
W32/322
Precision
Homogenous
N0.2
70.1
71x8
90.3
PACT
Homogenous
55.6
68.1
88.2
LQ-Nets
Homnogenous
87.9
W3/A3
DSQ
Homogenous
Homogenous
75.1
87.5
58.3
W3/A3
DoReFa + WaveQ
Preset
55.8
77.2
68.9
89.9
60.4
83.1
Homogenous
Imprvement
0.2%
2.1%
0.2%
1.7%
5,0%
PACT
Homogenous
55.7
69.2
89.0
61.4
83.7
LQ-Nets
Homogenous
69.3
88.8
W4:44
DSQ
Homogenous
69.6
64.8
WRPN
Homogenous
75.4
68
84.5
DoReFa
Homogenous
76.3
69.1
64.6
Preset
WIM
DoReFa + WareQ
56.2
79.2
69.8
89.1
65.4
85.5
Homugenous
Improvement
0.5%
2.9%
0.6%%
0,4%8
WL
DoReFa WiveQ
Learned
56.5
79.8
70.0
89.3
65.8
85.8
Heterogenous
W3.85
W3.57
W3.95
Improvement
0.6%
0.4%
Energy Saving
Tiae Aecumey
Pricsion SO nor FOT wayy Bit
(9)
(6)
Figure 4: Quantization bitwidth assignments across layers. (a) AlexNet (average bitwidth 3.85 bits). (b) R1 (average bitwidth 3.57 bits)
DSQ (Gong et al., 2019), and DoReFa, which are current state-of-the-an methods that show re- sults with homogeneous 3-, and 4-bit weight/activation quantization for various networks (AlexNet, ResNet-18, and MobileNet).
Experimental setup. We implemented WaveQ on top of DoReFa inside Dis- tiller (Zmora et al., 2018), open source framework for quantization by Intel that implements various quantization techniques. The reported accuracies for DoReFa and WRPN are with the built-in implementations in Distiller, which may not exactly match the accuracies reported in their respective pa- pers. However, an independent implementa- tion from major company provides an un- biased foundation for comparison. We quan- tize all convolution and fully connected lay- ers, except for the first and last layers which use 8-bits. This assumption is commensurate with the previous techniques.
Table 2: Accuracies of different networks ing plain WRPN, plain DoReFa and DoReFa + WaveQ on homogeneous weight quantization.
WIA Quantication
Beathmark
SanpleSer on CIPARIO
on CIAAIIO
Ro-Na-30
On CIPARIO 10
VGG-II
SVHNS
SVHN
Accurscy
W12A
Full Pratsion
74.11
91.3
96.11
96:47
Doleel
65.11
61.57
78.74
81:45
W3AA22
Wavee)
73.68
Improvement
8.529
118
14
13.92
65.23
80.24
DulkeFa
20.75
08.34
36.94
89 56
W40032
WareO
74.14
96.12
Improvenvnnl
3.99
5.9%
690
6.59r 1
71.17
90.84
73.41
90.34
91.63
92.5 02.56
DoKaPa Waree)
74.45
96.11
96.42
Improvemnnt
0899
242
4.1 LEARNED HETEROGENEOUS BITWIDTHS
As for quantizing both weights and activations, Table I shows that incorporating WaveQ into the quantized training process yields best accuracy results outperforming PACT, LQ-Net, DSQ, and DoReFa with significant margins. Furthermore, it can be seen that the learned heterogeneous bitwidths yield better accuracy as compared to the preset 4-bit homogeneous assignments, with

---

Under review as a conference paper at ICLR 2021
lower, on average, bitwidth (3.85, 3.57, and 3.95 bits for AlexNet, ResNet-18, and MobileNet, spectively). Figure (b) (bottom bar graphs) show the learned heterogeneous weight bitwidths over layers for AlexNet and ResNet-18, respectively. As seen, WaveQ parametric regularization yields a spectrum of varying bitwidth assignments to the layers which vary from 2 bits to 8 bits with irregular pattern. These results demonstrate that the proposed regularization, WaveQ, auto- matically distinguishes different layers and their varying importance with respect to accuracy while learning their respective bitwidths. To assess the quality of these bitwidths assignments, we conduct a sensitivity analysis on the relatively big networks (see next subsection).
Benefits of heterogeneous quantization. Figure 4 a))(b) (top graphs) show various comparisons and sensitivity results for learned heterogeneous bitwidth assignments for bigger networks (AlexNet and ResNet-18) 18). It is infeasible to enumerate these networks' respective quantization spaces. Com- pared to 4-bit homogeneous quantization, learned heterogeneous assignments achieve better accu- racy with lower, on average, bitwidth 3.85 bits for AlexNet and 3.57 bits for ResNet-18. This demonstrates that a homogeneous (uniform) assignment of the bits is not always the desired choice to minimize accuracy loss. Furthermore, Figure 4 also shows that decrementing the learned bitwidth for any single layer at time results in 0.44% and 0.24% average reduction in accuracy for AlexNet and ResNet- 18, respectively. The dotted blue line with markers shows how differently decrement- ing the bitwidth of various individual layers affect the accuracy. This trend further demonstrates the effectiveness of learning with WaveQ to find the lowest bitwidth that maximizes the accuracy.
Energy savings. To demonstrate the energy savings of the solutions found by WaveQ, we evaluate on Stripes (Judd et al., 6a). a custom accelerator designed for DNNs, which exploits bit-serial computation to support flexible bitwidths for DNN operations. As shown in Table I the reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.
4.2 PRESET HOMOGENOUS BITWIDTH QUANTIZATION
Gercutt
We also consider a preset homogeneous bitwidth quantization which can also be sup- ported by WaveQ under special settings where we fix 8 (to a preset bitwidth). Hence, only the first regularization term is engaged for pro- pelling the weights to the quantization levels.
Table 2 shows accuracies of different net- works (SimpleNet-5, ResNet-220, VGG-11, and SVHN-8) using plain WRPN, plain DoReFa and DoReFa+ + WaveQ for 3, 4, and 5 bitwdiths. As depicted, the results concretely show the ef- fect of incorporating WaveQ into existing quan- tized training techniques and how it outper- forms the previously reported accuracies.
Weight distributions during training. Fig- ure shows the evolution of weights distribu- tions over fine-tuning epochs for different lay- of CIFAR1O, SVHN, AlexNet, and ResNet- 18 networks. The high-precision weights form clusters and gradually converge around the quantization centroids as regularization loss is minimized along with the main accuracy loss.
Figure 5: Evolution of weight distributions over training epochs at different layers and bitwidths for different networks. (a) CIFAR10, (b) SVHN (c) AlexNet, (d) ResNet18.
4.3 WAVEQ FOR TRANSFORMER QUANTIZATION
Table 3: Performance of WaveQ for quantizing Transformers.
Transformers (encoder-decoder architectures) have been shown to achieve best results for NLP tasks including machine translation (Vaswani et al.) (017)) and automatic speech recog- nition (Mohamed et al. 2019). A Transformer layer relies key-value self eeff-attention mechanism for learning relationships between distant concepts, rather than relying on recurrent con- nections and memory cells. Herein, we extend the application of WaveQ regularization improve the accuracy of deeply
BLEU score
Bitwidth
Unregularized Training (without Wave()
Regularized Training (with WaveQ)
Full precision
34.93
7-bit
34.79
34.83 (0.04
6-bit
34.39
34.53 (0.14
5-bit
32.74
33.20 (0.46 f)
7

---

Under review as a conference paper at ICLR 2021
quantized (below 8 bits) Transformer models. We run our experiments on IWSLT 2014 German- English (DE-EN) dataset. We use the Transformer model implemented in the pairseq-py toolkit (Ott et al. 2019). All experiments are based on the big transformer model with 6 blocks in the encoder and decoder networks. Table S shows the effect of applying WaveQ regularization into the training process for 5, 6, and 7-bit quantization
the final accuracy (BLEU score). WaveQ consistently improves the BLEU score of quantized models at various quantization bitwidths (7-5
bits). Moreover, higher improvements are obtained at lower bitwidths.
DISCUSSION
We conduct an experiment that uses WaveQ for training from scratch. For the sake of clarity, we are considering in this experiment the case of preset bitwidth assignments (i.e.. A3- = 0). Figure (ow(())))Co(() shows weight trajecto- ries without WaveQ as a point of reference. Row(II-Col()) shows the weight trajectories when WaveQ is used with a constant Au-
Colamin 01
Column (4)
Column [
GINGE W/ 330
of Rorts
CINIDO of Wowg 4b10
200L Mleed Lil
kmity
Nowe
Jumpping
100e 200
400
As the Figure illustrates. using constant Au results in the weights being stuck in a re- gion close to their initialization, (i.e,, quanti- zation objective dominates the accuracy objec- tive). However, if we dynamically change the Au following the exponential curve in Figurelo) (ow(II))Col(I) during the from-scratch train- ing, the weights no longer get stuck. Instead, the weights traverse the space (i.e. jump from wave to wave) as illustrated in Figure 6 Cols(II) and (III) for CIFAR and SVHN respectively. In these two columns, Rows (I, (II), (III), correspond to quantization with 3, 4, bits, respectively. citepInitially, the smaller Xu values allow the gra- dient descent to explore the optimization surface freely, as the training process moves forward, the larger Ar gradually engages the sinusoidal regularizer, and eventually pushes the weights close to the quantization levels. Further convergence analysis is provided in the supplementary material.
Sepil intortiinn Day
CIRaA of Wheel 5km
JAN A/WINO, 1kte
How (0
Disted
Figure 6: Weight trajectories. The 10 colored lines in each plot denote the trajectory of 10 different weights.
Next, we provide results comparing two profiles of the regularization strength (Aw). Profile 1: A grad- ually increases as training proceeds then gradually decays towards the end of training. (Figure a))).
Profile 2: Am gradually increases as training pro- ceeds and remains high (Figure ()). Figure ((a.)
dhrigg
depicts different loss components and Figure (b.d) visualizes weights trajectories. Both profiles show that accuracy loss is unimpededly minimized along with WaveQ loss. Our theoretical results align with Profile 1 (Figure w reviewer insightfully pointed out. Although X decays back towards end of training, the weights mostly remain tied to their quantization levels except for a few deflections that cause slight increase of the regularization loss towards end of training. In terms of test accuracy, both profiles yield similar results (Profile 1, 74.95% vs (Profile 2, 74.45%) Note that while the theorem is stated in terms of a limit as the regularization parameter vanishes, the proof in fact gives a corresponding stability result. Namely, if the regularization parameter is sufficiently small relative to the main loss then the minimizers will be "almost" quantized.
00
510
Tra mig
Traning Sendion
Figure 7: Weight trajectories and training losses for different A, profiles.
RELATED WORK
This research lies at the intersection of (1) quantized training algorithms and (2) techniques that dis- cover bitwidth for quantization. The following discusses the most related works in both directions. Most related methods define a new optimization problem and use a special method for solving it. For example, (Bai et al.) 2019) uses a proximal gradient method (adds prox step after each stochas- tic gradient step), (Yang et al., 2020) uses ADMM, and (Tung & Mori 2018) takes a Bayesian approach. This only makes the training more difficult, slower and increases the computational com-
8

---

Under review as a conference paper at ICLR 2021
plexity. In contrast, WaveQ exploits the conventional stochastic gradient descent method while jointly optimizing for the original training loss while softly constraining it to simultaneously learn the quantized parameters, and more importantly bitwidths. The differentiability of the adaptive sinu- soidal regularizer enables simultaneously learning both the bitwidths and pushing the weight values to the quantization levels. As such, WaveQ can be used complementary method some of these efforts, which is demonstrated by experiments with both DoReFa-Net (Zhou et al. 2016) and WRPN (Mishra et al. 2018). Our preliminary efforts (Anonymous) and another concurrent work Naumov et al. (0188) use a sinusoidal regularization to push the weights closer to the quantization levels. However, neither of these two works make the period a differentiable parameter nor find
bitwidths during training.
Quantized training algorithms There have been several techniques (Zhou et al. 2016 Zhu et al., 2017 Mishra et al., 2018) that train neural network in a quantized domain after the bitwidth of the layers is determined manually. DoReFa-Net (Zhou et al. (0016) uses straight through estimator (Ben-) gio et al. 2013) for quantization and extends it for any arbitrary & bit quantization in weights, acti- vations, and gradients. WRPN (Mishra et al. 2018) is training algorithm that compensates for the reduced precision by increasing the number of filter maps in a layer (doubling or tripling). TTQ (Zhu (017) (017) quantizes the weights to ternary values by using per layer scaling coefficients learnt dur- ing training. These scaling coefficients are used to scale the weights during inference. PACT (Choi eet al., 2018a) proposes technique for quantizing activations by introducing an activation clipping parameter at. This parameter (a:) is used to represent the clipping level in the activation function and learned via proppagapan during training. More recently, VNQ (Achterhold et al. 2018) uses variational Bayesian approach for quantizing neural network weights during training. VNQ requires a careful choice prior distribution of the weights, which is not straightforward, and the model is of- ten intractable. In contrast, WaveQ is directly applicable without introducing extra hyperparameters to optimize. Additionally, VNQ takes on probabilistic approach, while WaveQ is a deterministic approach towards soft quantization.
ett
Loss-aware weight quantization. Recent works pursued loss-aware minimization approaches for quantization. (Hou et al. 2017) and (Hou & Kwok 2018) developed approximate solutions us- ing proximal Newton algorithm to minimize the loss function directly under the constraints of low bitwidth weights. One effort (Choi et al. [21 8b) proposed to learn the quantization of DNNs through a regularization term of the caae---uaard-qunntzattlee to jointly train the network and its quantizers. DSQ (Gong et al. (219)) employs a series of tanh functions to gradually approximate the staircase function for low-bit quantization (c.g., sign for 1- bit case), and meanwhile keeps the smoothness for easy gradient calculation. Although some of these techniques use regularization to guide the process of quantized training, none explores the use of adaptive sinusoidal regularizers for quantization. Most recently, (Nguyen et al.) 2020) suggests using |co|| function as a regularizer. Moreover, unlike WaveQ, these techniques do not find the bitwidth for quantizing the layers.
error. I.Q-Net (Zhang et al., 2018) proposes
Techniques for discovering quantization bitwidths. A recent line of research focused on methods which can also find the optimal quantization parameters, e.g., the bitwidth, the stepsize, in parallel to the network weights. Recent work (Ye et al. 2018)) based on ADMM (adm) runs a binary search to minimize the total square quantization error in order to decide the quantization levels for the layers. Most recently, (Uhlich et al., 2019) proposed to indirectly learn quantizer's 's parameters via Straight Through Estimator (STE) (Bengio et al. 2013) based approach. In similar vein, (Esser al. 2019) has proposed to learn the quantization mapping for each layer in a deep network by approximating the gradient to the quantizer step size that is sensitive to quantized state transitions. On another side, recent works (Elthakeb et al. 2018; WWang et al.) November 21, 2018) proposed reinforcement learning based approach to find an optimal bitwidth assignment policy.
Quantizing Transformers. FullyQT (Prato al. 2019) uses a bucketing based uniform quanti- zation proposed by QSGD (Alistarh et al. 2016) and extends it to Tranformers. Q8BERT (Zafrir al. 2019) quantizes all the GEMM (General Matrix Multiply) operations to 8 bit by adding an additional term for quantization loss during training, which is calculated based on the rounding ef- fect of floating point values (Shaw et al.) 2018). WaveQ, however, uses a sinusoidal regularizer to automatically push the weights towards the quantization levels.
7 CONCLUSION
This paper devised WaveQ that casts the two problems of finding layer bitwidth and quantized weights as a gradient- based optimization through parametric sinusoidal regularization. WaveQ pro- vides significant improvements over the state-of-the-art and is even applicable to the Transformers.

---

Under review as a conference paper at ICLR 2021
REFERENCES
IWSLT 14 german to english dataset. http:///onrsspp0000..lLs..ccoooo proceeding.
Admm. http:://stanforddsed//ooooy
Jan Achterhold, Jan M. Köhler, Anke Schmeink, and Tim Genewein. Variational network quantiza- tion. In 6th ICLR, 2018.
Dan Alistarh, Demjan Grubic, Jungshian Li, Ryota Tomioka, and Milan Vojnovie. Qsgd: Communication-opiime stochastic gradient descent, with applications to training neural net- works. 2016.
Anonymous.
arXiv preprint arXiv:
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal op- erators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432 2013. URL
abs htttp:/arxiv..r wwwpaawwwww..rra///a/aaaa////// 3432
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. CoRR, abs/1805.06085 2018a.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Learning low precision deep neural networks through regularization. CoRR, abs/1809.00099. 2018b.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. In Artificial Intelligence and Statistics, 2015.
Ahmed T. Elthakeb, Prannoy Pilligundla, Fatemehsadat Mireshghallah, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. ReLeQ: A reinforcement learning approach for deep quantization of neural networks. Advances in Neural Information Processing Systems (NeurIPS) Workshop Machine Learning for Systems, 2018. URL httph:://///mm http:////aaall.ooo///////// 01704
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S. Modha. Learned step size quantization. 8th ICLR, 2020, abs/1902.08153. 2019. URL
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. CoRR, abs/1908.0500 055033, 2019. StL/htpp::/// http:////aavvv..a wwwoorggaaaa. abs / 1908. 05033
Lu Hou and James T. Kwok. Loss-aware weight quantization of deep networks. In 6th ICLR, 2018
Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. In 5th ICLR, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Elnia, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. J. Mach. Learn. Res., 2017.
Patrick Judd, Jorge Albericio, Tayler H. Hetherington, Tor M. Aamodt, and Andreas Moshovos. Stripes: Bit-serial deep neural network computing. In 49th Annual IEEE/ACM International Symposium on Micraachitecture, MICRO 2016, Taipei, Taiwan, October 15-19, 2016, pp. 19:1- 19:12. IEFE Computer Society, 2016a. doi: 10. 119/MICRO
URL https:
7/doi. oww.wo.rg11.... 1109/MI100/MCRR0 2016. 7783722
Patrick Judd, Jorge Albericio, Tayler H. Hetherington, Tor M. Aamodt, and Andreas Moshovos. Stripes: Bit-serial deep neural network computing. 49th MICRO, pp. 1-12, 2016b.
10

---

Under review as a conference paper at ICLR 2021
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018.
Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In Proceed- ings of the 4th International Conference on Neural Information Processing Systems, NIPS 91, pp- 950-957, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558602224.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Land- scape of Neural Nets. In NIPS, 2018.
Asit K. Mishra, Eriko Nurvitadhi. Jeffrey Cook, and Debbie Marr. WRPN: Wide Reduced- Precision Networks. In ICLR, 2018.
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolutional context for ast, 2019.
Maxim Naumov, Utku Diril, Jongsoo Park, Benjamin Ray, Jedrzej Jablonski, and Andrew Tulloch. On periodic functions as regularizers for quantization of neural networks. CoRR, abs/181 htss/8811.9986. 2018. /ttp::/:://rxx wttp://wwww.w////..../.... 11810/986111119
Hieu Duy Nguyen, Anastasios Alcxandridis, and Athanasios Mouchtaris. Quantization aware training with absolute-cosine regularization for automatic speech recognition. In Helen Meng,
Bo Xu, and Thomas Fang Zheng (eds.), Interspeech 2020, 21st Annual Conference of the In- ternational Speech Communication Association, Virtual Event, Shanghai, China, 25-29 Octo- ber 2020, pp. 3376- ISCA, 2020. doi: 222444/7/nterspecchh2222222 7/doi
URL https:
21 httpg/g//0....113eeeeeeeeeeeeeee
2020-1997
Myle Out, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. Proceedings of the Third Conference on Machine Translation: Research Papers, 2018, doi: 10.
18653/v1/w18 11--6330 URL htttp://xx.coi..oo// dwi.
6301
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grang- ier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceed- ings of NAACL-HLT 2019: Demonstrations, 2019. URL/attps://gittuus com/pytorch/
fairseq
Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for improved translation. ArXiv, abs/1910. 10485, 2019.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning internal representations by eиror propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume I: Foundations, pp. 318-362. MIT Press, Cambridge, MA, 1986.
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. ISCA, pp. 764-775, 2018
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NATCL-HLT, 2018
Frederick Tung and Greg Mori. CLIP-Q: deep network compression learning by in- in-parallel pruning- quantization. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7873-7882, IEEE Computer Society, 2018. doi: 10.1109/CVP..2018..oM 1.0021...
Stefan Uhlich, Lukas Mauch, Kazuki Yoshiyama, Fabien Cardinaux, Javier Alonso García, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is a good parametrization. abs/1905.1 1452, 2019. http:ttpp///aii// org/ halgg//b///aa 1905. 11452
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-Aware Automated Quantization. arXiv preprint 181 1.088110888 1.08886, November 21, 2018.
11

---

Under review as a conference paper at ICLR 2021
Haichuan Yang. Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by
joint learning: A constrained optimization-baxe approach. In 2020 IEEE/CVF Conference on Computer Vision and Pattem Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 2175-2185, 2020. doi: 10. 1.11099922244420002020000 000225.
Shaokai Ye, Tianyun Zhang, Kaiqi Zhang, Jiayu Li, Jiaming Xie, Yun Liang, Sijia Liu, Xue Lin, and Yuankai Wang. A unified framework of dnn weight pruning and weight clustering/qunnizzatinn using admm. CoRR, (bs//1111.0190) 2018.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. ArXiv, abs/1910.06188, 2019.
Dongqing Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, pp. 373-390, 2018. doi: 10.1007797833333000123333333
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoRcFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. CoRR, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained Ternry Quantization. In ICLR, 2017.
Neta Zmora, Guy Jacob, and Gal Novik. Neural network distiller, June 2018.
12

---

Supplementary Material for "WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Regularization"
A DETAILED THEORETICAL ANALYSIS
A.1 MOTIVATION
The results of this section are motivated by the following question.
Question A.1. Suppose that a function Pn - [0, oo) has many global minima and that Q C lk closed. How do isolate the global minima of F that are closest without actually computing the full set of global minima of F?
Intuitively, we would like show that if is very small, then the global minima of the function
F(x) ed(2, Q)
are very close to the global minima of F closest to Q. To achieve this we will have to introduce first the concept of convergence of sets and then will show that our intuition correct by proving that the set of global minima to the above relaxed function converges to subset of global minima of F closest to Q.
A.2 RELEVANT DEFINITIONS
Definition A.2. FR. [0, oo) satisfies /im F(x) +00, we will say that F is coercive.
Definition A.3. For a coercive function F R oo) we let SF for
ER F(x)
minyeR" F(y)} be coercive.
Lemma A.4. Assume that F :R" - [0, oo) continuous and coercive. Then F has at least one global minimum. That is, Sy is non-empty. Furthermore, Sr is compact set.
Definition A.S. Let G.G: R [0, oo) are continuous and assume that F is coercive. Define
S = {a E SF G(x) = inf G(y),, yESp
the minima of F which minimize G among the minima of F. Definition A.6. Let Q C IR" be closed set and assume that r E IR. Define the distance from x to the set Q to be
d(z. Q) = inf ||: yl|-
vEQ
Observe that since Q is closed set we have that x E Q if and only if d(x,, Q) and otherwise d(x, Q) 0.
Definition A.7. Let A, B C R be compact sets. We define the Hausdorff distance between A and B by
du (A, B) max d(x; A), sup d(y. B))- zeB
yeA
Observe that (A, B) = and only if A B.
Definition A.8. Let {S5)6>0 family of compact subsets of I... We say that limg- S3 S,
lim (S5, S.) 0.
8-+0
Lemma A.9. Let Ss be a family of compact subsets of TR.", then lim Sa S. if and only if the following two conditions hold.
1. If ra E Sg converges to x, then x S.
2. For every x E S., there exists a family as Sa with 28 - x.
The lemma is just exercise in the definition.

---

Under review as a conference paper at ICLR 2021
A.3 STATEMENT OF THE THEOREM
Theorem 2. Let F, G R + [0, oo) are continuous and assume that F is coercive. Consider the sets Sp+8G, the set of points which F 8G is globally minimum. The following are true:
1. òn 0 and Sp-5aG Sw, then
S. C SF.
2. If 8n - then there a subsequence on + and a non-empty set S. C Sp. so that S S bn, S..
Proof. The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the first statement. For the first statement, assume that SF+5, + S., We wish to show that S_ C SF.G. Assume that Tn is a sequence of global minima of F ò G converging to It suffices to show that 2, Sp.G. First let observe that I, E Sp. Indeed, let
inf F())
ZER"
and assume that E Sp. Then,
x F(x,) S (F+ 8, S (F 5,,)))((=) (,G(() +
Thus, since continuous and In + Ta we have that F(x.) À which implies v, E Sp. Next, define
inf G(x).
ZESP
Let SF.G so that G(t)= μ.. Now observe that, by the minimality of xn we have that
A 8m11 = (F + 8,G)( 2 (F + (x.)) 2 8, G(xa)
Thus,
G(Ix) Sp
for all Since G is continuous and Fa V. have that G(x.) S μ which implies that G(æ.) since v, E Sp. Thus, I, E Sp.G.
B QUANTIZER
Here, we give an overview about the used quantization method. Consider a floating-poit vari- able wy to be mapped into quantized domain using (b + 1) bits. Let Q be set of (2k + 1) quantized values, where k = 2 - 1. Considering linear quantization, Q can be represented as 0, E 8-1 17. where the size the quantization bin. Now, wy can be mapped to the b-bit quantization (Zhou et al. 2016) space as follows:
{-1, A
&
tanh(wy) 2 maxx(tanh(W,)) + -
Wgo = 2 x quantize,
(B.1)
where quantize, -ー round 1)x), wy is scalar, Wy is a vector, and Wgo is a scalar and tanh is used to limit its range to [-1, 1]. Then, scaling factor is determined per layer to map the final quantized weight wiy into the range [-c, +cJ. As such, wi takes the form cWggs where O. and Wyo E Q.
These learned parameters (b, a). as explained Section 2.2 can be mapped to the quantizer pa- rameters explained in Equation equation B.I For (b + 1) bits quantization (the extra bit is the sign
bit):
= 2b and c 2²/28
(B.2)
2

---

Under review as a conference paper at ICLR 2021
Table 4: Hyperparameters settings.
Epocho
Phase
Phose
Phuse
Simlleeeet
on OFAR10
0.001
CIFAR1
0.0001
Reshect200
126
Acclet
0.01
0.0001
Bestit-18
256
0.0001
Mobiette:
Imspelied
256
0.01
0.0001
0.9
C CONVERGENCE ANALYSIS
Figure (a), (b) show the convergence behavior of WaveQ by visualizing both accuracy and regu- larization loss over finetuning epochs for two networks: CIFARIO and SVHN. As can seen, the regularization loss (WaveQ Loss) is minimized across the finetuning epochs while the accuracy is maximized. This demonstrates a validity for the proposed regularization being able to optimize the two objectives simultancously. Figure WaveQ for the case of training from scratch for VGG-11. As can be seen, at the onset of training, the accuracy in the presence of WaveQ is behind that without WaveQ. This can be explained as result of optimizing for an extra objective in case of with WaveQ compared to without. Shortly thereafter, the regularization effect kicks in and eventually achieves ~ 6% accuracy improvement.
(d) contrasts the convergence behavior with and without
The convergence behavior, however, is primarily controlled by the regularization strengths (Awr Ag). As briefly mentioned in Section 2.2 (Xany Aş) E [0, oc) is hyperparameter that weights the relative contribution of the proposed regularization objective to the standard accuracy objective.
We reckon that careful setting of A Ag across the layers and during the training epochs is essential for optimum results (Choi et al. 2018b).
Accuraty
Accuracy
weo SinAReC SHARUG
SiNARte Loss
SivAReO
(a)
Epochs
(b)
Epochs
(c)
Coochs
12
Spochs
Figure 8: Convergence behavior: accuracy and WaveQ regularization loss over fine-tuning epochs for (a) CIFARIO, (b) SVHN. Comparing convergence behavior with and without WaveQ during training from scratch (c) accuracy, (d) training loss. Network: VGG-11, 2-bit DoReFa quantization
Lieralions):
Phose
Phose
Phase
200 rise edge starting iteration
583 fall edge starting iteration
Value
determines smoothness the transitions (rise/Fall)
Ave
f1 8.5 (l=tanh((1-n)33) ()))))
(L-tonn((s-d)/3):
Ap
Lorbda.w.voo torbdn_bets_vaauee 0.85*(P1-F2)
rise
fall
Iterations
(a)
(b)
Figure Math formula for setting An and ag during training iterations.
3

---

Under review as a conference paper at ICLR 2021
Table 5: Performance of WaveQ BERT.
MODEL: CamemBERT
SPOKEN
PARTUT
BITWIDTH: W4,5/88
UPOS
L.88
UPOS
LA$
Baseline
96.99
81.37
93.33
Quantized w/
89.91
72.32
90.89
84.25
Unregularized Finetuning
Quantized w/ WaveQ
93.41
79.34
95.76
91.55
Regularized Finetuning
Table 6: Validation top-I accuracy for training from scratch w/ WaveQ vs w/o WaveQ.
CIFAR10 (FP Accuracy = 74.53 %)
SVHN (FP Accuracy 96.4 %)
4 bits
bits
bits
bits
bits
Training w/o WaveQ
31.8
61.7
79.1
Training WaveQ
Improvement (%)
(+35.2)
(+34.8)
(+2.9)
(+17.6)
(+6.0)
(+4.2) f
D WAVEQ PERFORMANCE ON BERT
Additionally, Table 5 provides layer-wise quantization with heterogeneous mix of and bits for the BERT model. In all cases, WaveQ improves UPOS and LAS metries for two French treebanks (SPOKEN, PARTUT).
E TRAINING FROM SCRATCH
Table 6 shows comparison between training from scratch with WaveQ vs without. can be seen that incorporating WaveQ into the training process achieves strictly better accuracy than the baseline training without WaveQ across all cases. Moreover, higher improvements are obtained at lower bitwidths reaching 35%
F REGULARIZATION STRENGTHS
Having regularization strength is normal setting associated with any regularization method. The criterion for choosing A and Äg is to balance the magnitude of regularization loss to be smaller than the magniude of accuracy loss. We then perform grid search over few points and chose the ones with the best convergence.
From the theoretical perspective, while the theorem stated in of a limit the regularization parameter vanishes, the proof in fact gives corresponding stability result. Namely, if the regular- ization parameter sufficiently small relative the main loss then the minimizers will be "almost" quantized.
G HETEROGENEOUS COMPARISON
For Mobilenet-V V2, WaveQ quantizes the network to average bitwidth of 3.95(20.66 GBOPS) compared to 5.90(29.16 16 GBOPs) reported by Uhlich et al. (2019). Similarly, for Resnet-18. 18, WaveQ achieves average bitwidth of 3.57(62.56 GBOPs) compared to 5.47(65.90 GBOPs) by Uhlich ctt al. (2019). Table shows this comparison. However this is not a fair comparison since WaveQ is regularizza larizatiin method and not full-blown quantization technique.
4

---

Under review as a conference paper at ICLR 2021
Table 7: Comparison to heuristic- based bitwidth selection method.
MobileNet-V2
ResNet-18
Accuracy (Learned/FP Baseline)
Accuracy
GBOPs
Avg. BW
GBOPs
Avg.
(Learned/FP Baseline)
Ref [31] (Learned)
70.5%
2002.30
5.90
70.6%//70.39
414.42
5.47
Ours (Learned)
65.8%
1418.26
3.95
70.0%/70.11%
392.76
3.57


Reviews:
[{"review_id": "gnKffpHu_rY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a regularization term to control the bit-width and encourage the DNN weights moving to the quantization intervals. The key in such regularization is the Sinusoidal function, where the penalty is maximized in the middle of quantization levels and minimized at the quantization points. The sinusoidal period is regarded as the continuous representation of the bit-width.\n\nPros:\n1. The paper is well written and easy to understand. Illustration figures help a lot to present the proposed method. E.g., Figure 2 clearly shows the key idea of this paper.\n\n2. The idea of using the sinusoidal period as a continuous representation is novel and it makes it natural to use gradient descent methods to optimize the bit-width.\n\n3. Experiments on different datasets and tasks show that the proposed method can bring improvement on accuracy by using better regularization and / or better bit-width allocation.\n\nCons:\n1. The quantization regularization can only be used on weight quantization, while activation quantization is very important and sometimes more sensitive to the accuracy drop. On this point, the proposed method cannot be used to allocate the bit-width of activation, while activations can actually have different bit-width to the weights.\n\n2. Some related works are missing.\nFor quantization regularization, it is used in ProxQuant[1] as well, where the weights are regularized with a norm-based quantization regularization. It also discussed the similar observation that the regularization can help quantization-aware training.\nFor bit-width allocation, [2] formulated the bit-width allocation as a knapsack problem and used ADMM to iteratively optimize the NN compression. [3] used Bayesian optimization to allocate the bit-width for different layers.\n\n3. The procedure to tune the regularization strengths is a little complicated. Although the authors proposed a detailed procedure for tuning these hyper-parameters, it's questionable that the same procedure can be widely applied in different networks/optimizers/datasets. For $\\lambda_\\beta$, it actually determines the number of total bits for the entire network, so it should depends on the expected compression rate of the quantized DNN.\n\n4. Using the sinusoidal period to decide the bit-width has an assumption that the range of the weights is fixed: otherwise one can reduce both the period and value range to keep the bit-width unchanged. However, the range of weights is usually not fixed in the training.\n\n5. The experiments on bit-width allocation may be not enough to show the effectiveness of the proposed method as a bit-width allocation method. It's better to compare with some other bit-width allocation methods.\n\nIn general, my rating is borderline. I hope the authors can give some response to the cons listed above.\n\nReference:\n\n[1] Bai, Y., Wang, Y.X. and Liberty, E., 2018. Proxquant: Quantized neural networks via proximal operators. arXiv preprint arXiv:1810.00861.\n\n[2] Yang, H., Gui, S., Zhu, Y. and Liu, J., 2020. Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach. In CVPR 2020.\n\n[3] Tung, F. and Mori, G., 2018. Clip-q: Deep network compression learning by in-parallel pruning-quantization. In CVPR 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "P5FD1Wdg5e", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using a sinusoidal regularizer for neural network quantization. The regularizer “WaveQ” (sin^2) pushes floating-point parameters towards quantized values. Because the period of the function is highly related to the required bit-width, it can be used to determine the bit-width while keeping good characteristics - continuous and trainable. The authors provide experiments on both CNN and Transformers. The proposed method is widely adaptable and easy-to-use with quite promising results.\n\nMajor questions/suggestions:\n\n1.\tIn equation 2.1, the period (=quantization step) is s= 1/ (2^beta – 1), so the regularizer tries to move ‘w’ towards the multiple of ‘s’. As I understand, the desired range of ‘w’ is (-1, 1), to properly match the actual quantized value. But there is no clipping nor normalization applied to ‘w’. Is there something I missed?\n\n2.\tThere is a missing work [Nyuyen, 2020] which suggests a similar regularizer (|cos|). However, this paper is still valuable and does not hurt novelty.\n[Ngugen, 2020] Quantization Aware Training with Absolute-Cosine Regularization for Automatic Speech Recognition\n\n3.\tIt seems that WaveQ is also used for activation quantization (Section 4.1.), but there are not enough details about applying WaveQ for activations.\n\nMinor issues:\n\n1.\tEquation B.2 is frequently used in Section 2. Also, the ‘alpha’ term needs more explanation when it first appears. (‘scaling factor’ is not enough) Consider moving the equation B.1, B.2. to the front.\n\n2.\tMany recent quantization papers use w / max(|w|) to normalize weight. Is there any advantage to use tanh(w) / max(tanh(w)) in Equation B.1?\n\n3.\tTypo: ‘bitwidh’ in Sec.4.1, ‘citep’ in Sec.5, ‘ofcitepp’ in Sec.6, \n\n4.\tI believe the theoretic part is OK but am not sure about the exactness and its importance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DnXIqb4p1S2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes WaveQ which proposes a sinusoidal regularization approach for quantizing Neural Networks. The motivation is to enable mixed-precision quantization, since homogeneously quantizing the model to low precision can lead to accuracy degradation.\n\nThe main problem with heterogeneous/mixed-precision bit-width is that its search space is exponentially large. To address this the authors propose to automatically learn this bit-precision by adding a sinusoidal regularization where the period learn able.\n\nWhile the approach is interesting but the theoretical and empirical results are not satisfactory and as such it is not clear how the proposed method is superior to other methods proposed in the literature. In particular:\n\n\n- The theoretical analysis provided does not apply to the proposed method. This is because the theory requires that the regularization function vanish after many iterations, whereas in the experiments the opposite approach is used.\n\n- The accuracy results provided in the empirical section incur significant degradation as compared to the baseline. Furthermore, the comparison is performed with old quantization methods. Newer mixed-precision quantization results using for example HAQ leads to much better quantization. As such, it is not clear what is the advantage of the proposed method?\n\n- How is this approach different than Achterhold et al., 2018?", "extra_scores": {}, "extra_text": {}}, {"review_id": "5j07Sf3fUeB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "When training quantized neural networks, one typically first fixes the desired bitwidth $b$ (of weights and activations). While training, one maintains and updates full-bitwidth weights during backprop and \"cheats\" by using $b$-quantized versions of these full-precision weights during forward propagation. The quantization scheme used may vary.\n\nWhat if we wished instead to *learn* $b$? This is especially attractive, for instance, if we wish a distinct $b_i$ for each layer $i$. We could introduce a variable $\\beta_i$ to represent $b_i$ during training. We would probably add a regularizer to minimize $\\Sigma \\beta_i$ to encourage training toward small bitwidths. However, this still leaves the problem that the $\\beta_i$ would be real, not integer, as required for quantization. We could cheat here by using $\\lceil \\beta_i \\rceil$, but note that quantization error $\\beta_i - \\lceil \\beta_i \\rceil$ may be quite large relative to the small integer values $\\beta$ is expected to take. Is there a way to ensure that training automatically produces $\\beta$s that are close to integer values?\n\nThis is where the current paper introduces a neat trick. By embedding $\\beta$ inside a (suitably scaled) sinusoidal loss function, they ensure that the $\\beta$s tend to have integer values (that correspond to the minima of the sinusoid).\n\nI haven't seen this trick before, and it seems like an elegant approach to the problem of favoring integer values within the global optimization process. The goal of producing distinct bitwidths for each layer is definitely a valid one, so this technique does also address a practical problem.\n\nThe technique's practical impact seems limited. In particular, it seems that the main effect is to reduce the average bitwidth of SOTA quantized models from 4 to >= 3.6 at roughly the same accuracy. This should correspond roughly to a 10% gain in performance (runtime), which is quite good but not great. I have no idea how to to interpret the power numbers, so I will ignore those for now.\n\nOne question I have though is how the technique compares to a decent heuristic variable-bitwidth baseline. The authors do compare to a  \"decrement the bitiwidth of a single layer\" baseline, which is a good start. However, what if you do something more informed than that as baseline? Can you tell us what bitwidths your techniques chose for Resnet, for instance? Is there a simple pattern there (e.g. high bitwidth for early layers, and lower later on)? Based on this, what is the \"best\" non-learned bitwidth selection you can come up with?\n\nIn any case, the sinusoid trick for learning integer-like values is a good one for people to know, and it seems that the authors have shown that it can be implemented to have practical impact. Even if bitwidth reduction is currently modest, I can imagine follow-up work to increase it. So I support acceptance of the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a regularization term to control the bit-width and encourage the DNN weights moving to the quantization intervals. The paper is well-written and the idea of using the sinusoidal period as a continuous representation is novel. However, the theoretical analysis provided are not consistent with the proposed method.

As for the experimental results, the proposed method incurs significant degradation as compared to the baseline, and comparison with recent quantization methods is lacking.
Decision: Reject