Mirror Descent View For Neural Network Quantization
Authors: Thalaiyasingam Ajanthan
Kartik Gupta
Philip H. S. Torr
Richard Hartley
Puneet K. Dokania
Venue:  (2020)
Keywords: mirror descent
network quantization
numerical stability

Abstract: Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. NN quantization is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the continuous parameters (unconstrained) as the dual of the quantized ones, we introduce a Mirror Descent (MD) framework (Bubeck (2015)) for NN quantization. Specifically, we provide conditions on the projections (i.e., mapping from continuous to quantized ones) which would enable us to derive valid mirror maps and in turn the respective MD updates. Furthermore, we discuss a numerically stable implementation of MD by storing an additional set of auxiliary dual variables (continuous). This update is strikingly analogous to the popular Straight Through Estimator (STE) based method which is typically viewed as a “trick” to avoid vanishing gradients issue but here we show that it is an implementation method for MD for certain projections. Our experiments on standard classification datasets (CIFAR-10/100, TinyImageNet) with convolutional and residual architectures show that our MD variants obtain fully-quantized networks with accuracies very close to the floating-point networks.

Under review as a conference paper at ICLR 2020
MIRROR DESCENT VIEW FOR NEURAL NETWORK QUANTIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due reduced memory and time complexity. NN quantization is usually formulated as a constrained optimization problem and optimized via modified version of gradient descent. In this work, by interpreting the continuous parameters (unconstrained) as the dual of the quantized ones, we introduce a Mirror Descent (MD) framework (Bubeck) (2015)) for NN quantization. Specifically, we provide conditions on the projections (i.e.. mapping from continuous to quantized ones) which would enable us derive valid mirror maps and in turn the respective MD updates. Furthermore, we discuss numerically stable implementation of MD by storing an additional set of auxiliary dual variables (unconstrained), This update is strikingly analogous to the popular Straight Through Estimator (STE) based method which is typically viewed as a "trick" to avoid vanishing gradients issue but here show that it is an implementation method for MD for certain projections. Our experiments on standard classification datasets (CIFAR- 1ORRR-00100 10/100, TinylmageNet) with convolutional and residual architectures show that our MD variants obtain fully-quantized networks with accuracies very close to the floating- point networks.
INTRODUCTION
Despite the success of deep neural networks in various domains, their excessive computational and memory requirements limit their practical usability for real-time applications or in resource-limited devices. Quantization is prominent technique for network compression, where the objective is to learn a network while restricting the parameters to take values from a small discrete set (usually binary). This leads to dramatic reduction in memory (a factor of 32 for binary quantization) and inference time as enables specialized implementation using bit operations.
Neural Network (NN) quantization is usually formulated constrained optimization problem minge f(x), where f(.) denotes the loss function by abstracting out the dependency on the dataset and X C IR' denotes the set of all possible quantized solutions. Majority of the works in the literature (Hubara et al. (2017); Yin et al. )): Ajanthan et al. (2019)) convert this into an unconstrained problem by introducing auxiliary variables (x) and optimize (stochastic) gradient descent. Specifically, the objective and the update step take the following form:
min f(P(x))
g x (x)((
where P: IR" - X' is a mapping from the unconstrained space to the quantized space (sometimes called projection) and 17 >> 0 is the learning rate. In cases where the mapping P is not differentiable, a suitable approximation is employed (Hubara et al. (2077).
In this work, by noting that the well-known Mirror Descent (MD) algorithm, widely used for online convex optimization (Bubeck (2015)), provides a theoretical framework to perform gradient descent in the unconstrained space (dual space, IR') with gradients computed in the quantized space (primal space, X'), we introduce an MD framework for NN quantization. In essence, MD extends gradient de- scent to non- -un-Euclid spaces where Euclidean projection is replaced with more general projection defined based on the associated distance metric. Briefly, the key ingredient of MD is a concept called mirror map which defines both the mapping between primal and dual spaces and the exact form of

---

Under review as a conference paper at ICLR 2020
the projection. Specifically, in this work, by observing P in Eq. ( mapping from dual space to the primal space, we analytically derive corresponding mirror maps under certain conditions on P. This enables us to derive different variants of the MD algorithm useful for NN quantization.
Furthermore, as MD is often found to be numerically unstable (Hsich et al. (0188), we discuss a
numerically stable implementation of MD by storing additional set of auxiliary variables similar to the existing methods. As will be shown later, this update is strikingly analogous to the popular Straight Through Estimator (STE) based gradient method (Hubara al. (2017); [ai al. (2019)) which is typically viewed as a "trick" to avoid vanishing gradients issue but here we show that it an implementation method for MD under certain conditions on the mapping P. We believe this connection sheds some light on the practical effectiveness of STE.
We evaluate the merits of our MD variants on CIFAR- 10/100 and TinyImageNet classification datasets with convolutional and residual architectures. Our experiments show that the quantized networks obtained by the MD variants yield accuracies very close the noating-point counterparts while outperforming directly comparable baselines. Finally, we would like to emphasize that even though our formulation does not necessarily extend the theory of MD, we believe showing MD as a suitable framework for NN quantization with superior empirical performance opens up new ways of designing MD-inspired update rules for NNs.
PRELIMINARIES
We first provide some background on the MD algorithm and NN quantization. Then we discuss the link between them and provide our MD framework for NN quantization.
2.1 MIRROR DESCENT
The Mirror Descent (MD) algorithm is first introduced in Nemirovsky & Yudin (1983) and it has been extensively studied the convex optimization literature ever since. In this section we provide a brief overview and we refer the interested reader to Chapter 4 of (Bubeck (2015)). In the context of MD, we consider a problem of the form:
min f(x), xEX
where f : + IR is a convex function and C. IR' is a compact convex set. The main concept of MD is to extend gradient descent to a more general non-Euclidean space (Banach space"]. } thus overcoming the dependency of gradient descent on the Euclidean geometry. The motivation for this generalization that one might be able to exploit the geometry of the space to optimize much more efficiently. One such example is the simplex constrained optimization where MD converges at a much faster rate than the standard Projected Gradient Descent (PGD).
To this end, since the gradients lie in the dual space, optimization is performed by first mapping (quantized space, X') the dual space B (unconstrained space, IR"., then
the primal point x performing gradient descent in the dual space, and finally mapping back the resulting point to the primal space B. If the new point x lie outside of the constraint set X' C B, it is projected the set X. Both the primal/dual mapping and the projection are determined by the mirror map. Specifically, the gradient of the mirror map defines the mapping from primal to dual and the projection is done via the Bregman divergence of the mirror map. We first provide the definitions for mirror map and Bregman divergence and then turn to the MD updates.
Definition 2.1 (Mirror map). Let C C IR' be a convex open set such that X' c Ć (C denotes the closure of set C) and X' CC + 0. Then, & C - IR a mirror map if satisfies:
1. + is strictly convex and differentiable.
2. (Cb(C) = IR", Le., 4 takes all possible values in IR'.
3. limx +ac [x)( = oo (&C denotes the boundary of C), iee., V diverges on the boundary of Definition 2.2 (Bregman divergence). Let & - IR be a continuously differentiable, strictly convex function defined on a convex set C. The Bregman divergence associated with & for points P. q E C is the difference between the value of & at point and the value of the first-order Taylor expansion of t around point q evaluated at point p. i.e.,
(p q) = ((p) ( (VD(q). q)
A Banach space is a complete normed vector space where the norm is not necessarily derived from inner product.
2

---

Under review as a conference paper at ICLR 2020
Notice, (p) 0 with (p. p) 0, and Dp(p.qu q) is convex on p.
Now we are ready to provide the mirror descent strategy based on the mirror map Φ. x" angminxxxxcc (xx) be the initial point. Then, for iteration k 2 0 and step size y the update of the MD algorithm can be written as:
Let
V V ng",
where g" 0f((x) and ykt EC,
(4)
x = argmin Do (x, y
xEXne
Note that, in Eq. (4). the gradient g" is computed at x* E X'nc (solution space) but the gradient descent is performed in IR" (unconstrained dual space). Moreover, by simple algebraic manipulation, it is easy to show that the above MD update 4 can be compactly written in a proximal form where the Bregman divergence of the mirror map becomes the proximal term (Beck & Teboulle (2003))
++1
argmin (ng". x) + (x(xxx))
xEX'nc
Note, if (x) |1 then (x, problem and optimized for x, leads to exactly the same update rule as that PGD. However, MD allows us to choose various forms of & depending on the problem at hand.
| x -/2 which when plugged back to the above
2.2
NEURAL NETWORK QUANTIZATION
Neural Network (NN) quantization amounts to training networks with parameters restricted to a small discrete representing the quantization levels. Here review two constrained optimization formulations for NN quantization: 1) directly constrain each parameter to be in the discrete set; and 2) optimize the probability of each parameter taking label from the set of quantization levels.
2.2.1
PARAMETER SPACE FORMULATION
Given a dataset D yx,, y NN quantization can be written as:
min L(w; D) w
((w; (x);
Here, ((-) denotes the input-output mapping composed with a standard loss function (e.g., cross- entropy loss), w is the dimensional parameter vector, and with |Q| is predefined discrete set representing quantization levels (e.g.. Q 11 or @ = (,,,0,1})
The approaches that directly optimize in the parameter space include BinaryConnect (BC) (Cour bariaux et al. (2015)) and its variants (Hubara et al. (2017); Rastegari et al. (2016)), where the constraint discrete. In contrast, recent approaches (Bai (2019) Yin al. (2018) relax this constraint set to be its convex hull: conv(Q")) = qquin [max]
(7)
where Zonin and fmax represent the minimum and maximum quantization levels, respectively. In this case, a quantized solution is obtained by gradually increasing an annealing hyperparameter.
2.2.2 LIFTED PROBABILITY SPACE FORMULATION
Another formulation is based on the Markov Random Field (MRF) perspective to NN quantization recently studied in (Ajanthan al.) (2019)). It treats Eq. (6) as a discrete labelling problem and introduces indicator variables Uj E [0, 1 for each parameter w, where E (1...., m) such that
Uy:A = 1 if and only wi = X E Q. For convenience, by denoting the vector of quantization levels
as q. parameter vector E Q" can be written in matrix vector product as:
Σx u 1,
V }
Vj
w = uq where u E 1
(8)
Ug:A (0,1),
Here, is d matrix (i..,, each row u, {uJA | A 2)). and q is column vector of dimension Note that, u E ym is an overparametrized (i.e., lifted) representation w E Q"" Now, similar the relaxation in the parameter space, one relax the binary constraint in nm form its convex hull:
= conv conv(y" (
Σx U 11 Vj uyA
"
AJ.A
3

---

Under review as a conference paper at ICLR 2020
The set A is in fact the Cartesian product of the standard (d - 1)- probability simplexes embedded in I Therefore. for a feasible point u E A", the vector u, for each j (ý-th row matrix u) belongs to the probability simplex A. Hence, we can interpret the value Uj:A as the probability of assigning the discrete label À to the weight wj. This relaxed optimization can then be written as:
min Luuq D) = Σ ((uq: (x, U TD
(10)
In fact, this can be interpreted as finding a probability distribution u E A such that the cost L(u) is minimized. Note that, the relaxation of u from V to A translates into relaxing from QTH to the convex region conv (2").. Even in this case, a discrete solution u E ymm can be enforced via an annealing hyperparameter or using rounding schemes.
MIRROR DESCENT FRAMEWORK FOR NETWORK QUANTIZATION
Before introducing the MD formulation, we first write NN quantization as a single objective unify- ing (6) and TT) as:
min f(x)
(11)
XEX
where f(-) denotes the loss function by abstracting out the dependency on the dataset D. and X denotes the constraint set (Qm or y depending on the formulation). Note that, as discussed in Sec. 2.2 many recent NN quantization methods optimize over the convex hull of the constraint set. Following this, we consider the solution space X' in Eq. (TT) to convex and compact. To employ MD, need to choose mirror map (refer Definition 2.1 suitable for the problem at hand. In fact, as discussed in Sec. 2.1 mirror map is the core component of an MD algorithm which determines the effectiveness of the resulting MD updates. However, there is straightforward approach to obtain mirror map for a given constrained optimization problem, except in certain special cases.
To this end, observe that the usual approach to optimize the above constrained problem is via version of projected gradient descent, where the projection is the mapping from the unconstrained auxiliary variables (high-precision) to the quantized space X. Now, noting the analogy between the purpose of the projection operator and the mirror maps in the MD formulation, we intend to derive the mirror map analogous to a given projection. Precisely, we prove that if the projection invertible and strictly monotone, a valid mirror map can be derived from the projection itself. This does not necessarily extend the theory of MD as finding a strictly monotone map hard as finding the mirror map itself. However, this derivation is interesting as it connects existing PGD type algorithms to their corresponding MD variants. For completeness, we now state it as a theorem for the case X' C and the multidimensional case can be easily proved with an additional assumption that the vector field (x) is conservative.
Theorem 3.1. Let X be compact convex sel and P IR - C be an invertible function where C C IRt is a convex open set such that X' = (C denotes the closure of C). Now, if
1. P is strictly monotonically increasing.
2. lima-sac| ||P (x)]l = Do (ŌC denotes the boundary of C).
Then, (xx) 1 zz
P 1 (y)dd dy is valid mirror map.
Proof. This can be proved by noting that V&((x) P (x)). Please refer to Appendix A.
The MD update based on the mirror map derived from given projection illustrated in Fig.
P
|
IRT
Note that, to employ MD to the problem [1].. theory, any mirror map satisfying Definition 2.1 whose domain (the closure of the domain) is superset of the constraint set X' can be chosen. However, the above theorem provides method to derive only a subset of all applicable mirror maps, where the closure of the domain of mirror maps is exactly equal to the constraint set X.
x
k+1
P
x X
Figure 1: MD formulation where mirror map is de- rived from the projection P. Note, g² is computed in the primal space (X) but it is directly used to update the auxiliary variables in the dual space.
We now give some example projections useful for nn quantization (tanh for w-space and softmax for u-space) and derive their corresponding mirror maps. Given mirror maps, the MD updates are straightforward based on Eq. (5). Even though consider differentiable projections, Theorem 3.1
4

---

Under review as a conference paper at ICLR 2020
does not require the projection to be differentiable. For the rest of the section, assumee 1, i.e., consider projections that are independent for each j E {l....,,) m).
Example 3.1 (w-space, binary, tanh). Consider the tanh function, which projects a real value to the interval [-1, 1|:
P ( = (anh(3@) exp(23m) - exp(2,30) +
(12)
where 3 > 0 the annealing hyperparameter and when + oc, tanh approaches the step function. The inverse of the tanh is:
1
1 w log
P- (w) tanh (w) 2
(13)
1 1-w
Note that, P-1 monotonically increasing for fixed B. Correspondingly, the mirror map from The- orem 3.1 3.J can be written
((w)= p-
1 [1 + w) log(1 w) + (1 - ww) log(1 w)]
(14)
2,3
Here, the constant from the integration is ignored. can be easily verified that ((w) is in fact valid mirror map. The projection, its inverse and the corresponding mirror map are illustrated in Fig. 2a Consequently, the resulting MD update (5) takes the following form:
1tw" (xp(--281g'))
wh argmin (ng w) + Du( w.(--
I-w
(15)
Itwk I- - exp(- -2,0gg))))))
The update formula is derived using the KKT conditions (Boyd & Vandenberghe (2009). For the detailed derivation please refer to Appendix B. A similar derivation can also be performed for the sigmoid function, where C X 0. 1]. Note that the sign function has been used for binary quantization in (Courbariaux et al. (2015)) and tanh can be used as a soft version of sign function as pointed by (Zhang et al. (2015)). Mirror map corresponding tanh is used for online linear optimization in (Bubeck et al. 20122) but here we use it for NN quantization.
Example 3.2 (u-space, multi-label, softmaxx. Now we consider the softmax projection used in Proximal Mean-Fiee (PMF) (Ajanthan et al. (2019) to optimize in the lifted probability space. In this case, the projection is defined as P(ū) := softmax (Bū) where P IR + C with = X' A. Here dimmennsional probability simplex and Q = Note that the soft projection
is the (d - 1) not invertible as it is many-to-one mapping. In particular, it is invariant to translation, i.e.
exp(ūx)
11 = ma + el) softmax
where UA = EnEo exp(ūp))
(16)
for any scalar e e IR (1 denotes a vector of all ones). Therefore, the softmax projection does not satisfy Theorem 3.1 However, one could define the inverse of softmax as follows: given E A, find a unique point + cl, for particular scalar c, such that u softmax(i). Now, by choosing lo exp (ū,,) softmax can be written as:
softmax(()
where uu = exp(ūx), AA
(17)
Now, the inverse of the projection can be written as:
$. = P (u) = softmax-1 P(u). where
ÚA log(ux):
AY.
(18)
Indeed, log is a monotonically increasing function and from Theorem 3.1 by summing the integrals, the mirror map can be written as:
(u))
Σ ax log(ux) ux
H(u) 1/3
(19)
Here, ΣA == = as as u E A, and (u) is the entropy. Interestingly, as the mirror map in this case the negative entropy (up to constant), the MD update leads to the well-known Exponentiated
5

---

Under review as a conference paper at ICLR 2020
1.0
1.0
0.5
0.2
0.5
0.6
0.
-0.5
-05
-1.0
8(8).8
-08
-0.4
1-
(w),
-1.0
0.5
(a) tanh, its inverse, and mirror map
(b) shifted tanh and its inverse
Figure 2: Plots of tanh and shifted tanh projections, and their inverses corresponding to the tanh projection. Note that, the inverses are monotonically increasing and the mirror map is strictly convex. Moreover, when 3 + oo, the projections approaches their respective hard versions.
Gradient Descent (EGD) (or Entropic Descent Algorithm (EDA)) (Beck & Teboulle (2003):
Bubeck
(2015)). Consequently, the update takes the following form:
ux (xp(=)
A
(20)
EnEO U cxp(-Bngf)
The derivation follows the same approach as in the tanh case above. It is interesting to note that the MD variant of softmax is equivalent to the well-known EGD. Notice, the authors of PMF hinted that PMF is related to EGD but here we have clearly showed that the MD variant of PMF under the above reparametrization
is exactly EGD.
Example 3.3 (w-space, multi-label, shifted lanh). Note that, similar to softmax, we wish to extend the tanh projection beyond binary. The idea is to a function that is addition of multiple shifted tanh functions. To this end, as an example we consider ternary quantization, with Q = (-1,0, and define our shifted tanh projection P IR- C as:
w P(w)= = 2 [anh (3(w + 0.5)) + tanh (8(m - 0.5))]
(21)
where 3 > 0 and w = C X 1-1, 1]. When 3 + co. P approaches a stepwise function with inflection points at 0.5 and 0.5 (here, =0.5 is chosen heuristically), meaning w move towards one of the quantization levels in the set Q. This behaviour together with its inverse is illustrated in Fig. 2b Now, one could potentially find the functional form of p-1 and analytically derive the mirror map corresponding to this projection. Note that, while Theorem 3.1 provides an analytical method to derive mirror maps, in some cases such as the above, the exact form of mirror map and the MD update might be nontrivial. In such cases, as will be shown subsequently, the MD update can be easily implemented by storing an additional set of auxiliary variables w.
Effect of Annealing. Note that, ensure a discrete solution, projection is parametrized by a scalar 3 and it is annealed throughout the optimization. This annealing hyperparameter translates into a time varying mirror map (refer Eqs. (14p and (19) in our case. Such an adaptive mirror map gradually constrains the solution space X' to its boundary and in the limit enforces a quantized solution. Since this adaptive behaviour can affect the convergence of the algorithm, in our implementation capped at arbitrarily chosen maximum value, and empirically, the algorithm converges to fully quantized solutions in all tested cases. We leave the theoretical analysis annealing for future work.
Numerically Stable form of Mirror Descent. We showed a few examples of valid projections, their corresponding mirror maps, and the final MD updates. Even though. in theory, these updates can be used directly, they are sometimes numerically unstable due to the operations involving multiple logarithms, exponentials and divisions (Hsieh et al. (2018). To this end, we provide numerically stable way of performing MD by storing an additional set of auxiliary parameters during training.
careful look at the Fig. suggests that the MD update with the mirror map derived from Theorem 3. can be performed by storing auxiliary variables X = px-(x). In fact, once the auxiliary variable x* is updated using gradient g it is directly mapped back to the constraint set X' via the projection. This is mainly because of the fact that the domain of the mirror maps derived based on the Theorem 3.] are exactly the same as the constraint set. Formally, with this additional set of variables, one can write the MD update
4 corresponding to the projection P as:
++1 = =x* "g"
update in the dual space
(22)
x x+1 p(x ) X, projection to the primal space

---

Under review as a conference paper at ICLR 2020
where 7
(x*). Experimentally observed these updates to show 0, (x(xx) and x
stable behaviour and performed remarkably well for both the tanh and softmax.
Note, above updates can be seen as optimizing the function f(P())) using gradient descent where the gradient through the projection (i.e., Jacobian) Jp = a /0x is replaced with the identity matrix. This is exactly the same the Straight Through Estimator (STE) for NN quantization (following the nomenclature of (Bai et al.] (2019): Yin et al. )). Despite being a crude approximation, STE has shown be highly effective for NN quantization with various network architectures and datasets (Yin) et al. (2018); Zhou et al. ). However, a solid understanding of the effectiveness of STE is lacking in the literature except for its convergence analysis in certain special cases (Li et al. (2017):
Yin et al. (2019). In this work, by showing STE based gradient descent as an implementation method of MD under certain conditions on the projection, we provide justification on the effectiveness of STE. Besides, as shown in Example in cases where the MD formulation is nontrivial, the STE based implementation can be used. The pseudocodes of original and numerically stable versions of our MD algorithm for tanh are presented in Appendix B.
Comparison against ProxQuant. The connection between the dual averaging version of MD and STE was recently hinted in ProxQuant (PQ) (Bai et al. (2019). However, no analysis of whether an analogous mirror map exists to the given projection is provided and their final algorithm is not based on MD. In particular, following our notation, the final update equation of PQ can be written as:
x++1 =x ng"
assumes x" and g" are in the same space
(23)
x k+1 = -rrox(x*+1)
prox IR" - IR' is the proximal mapping defined in (Bai et al. 2019
where 7) > O, and g* E fx*x)) Note that, as opposed to MD (refer to Eq. 22 PQ assumes the
point x and gradient g" are in the same space. Then only the formula x* 78" is valid. This would only be true for the Euclidean space. However, as discussed in Sec. 2.1 MD allows gradient
descent to be performed on more general non-Euclidean space by first mapping the primal point x* to a point * in the dual space via the mirror map. Such an ability enabled theoretical and practical research MD for the past three decades.
Convergence of MD in the Nonconvex Setting. We would like to point out that MD is originally developed for convex optimization, however, in this paper we directly apply MD to NN quantization where the loss highly nonconvex and gradient estimates are stochastic, and empirically evaluate its convergence behaviour and performance. Theoretical analysis of MD for nonconvex, stochastic setting an active research area (Zhou et (2017a)b)) and MD has been recently shown to converge the nonconvex stochastic setting under certain conditions (Zhang & He (2018)) ). We believe convergence analysis of MD for NNS could constitute to a completely new theoretical paper.
4 RELATED WORK
In this work we consider parameter quantization, which usually formulated as constrained problem and optimized using a modified projected gradient descent algorithm, where the methods (Courbariaux jet (2015); 5): Carreira-Perpinán & Idelbayev (2017); Yin et al. (2018); Bai et al. (2019); Ajaanthan et al. (2019) mainly differ in the constraint set, the projection used, and how backpropagation through the projection performed. Among them, STE based gradient descent is the most popular method as enables backpropagation through nondifferentiable projections and it has shown to be highly effective in practice (Courbariaux et al. 2015)). In fact, the success of this approach lead to various extensions by including additional layerwise scalars (Rastegari et al. (2016), ) relaxing the solution space ( Yin let al.] (2018)) and even to quantizing activations (Hubara et al. (2017), and/or gradients (Zhou al. (0166). Moreover, there are methods focusing on loss aware quantization (Hou al. (2017)) quantization for specialized hardware (Esser et (2015), and quantization based on the variational approach (Achterhold al. (2018); Louizos et al. (2017) 2019).. We have only provided brief summary of relevant methods and for a comprehensive survey we refer the reader to (Guo (2018)
5
EXPERIMENTS
Due to the popularity of binary neural networks (Courbariaux et al. 2015); Rastegari et al. (0116). mainly consider binary quantization and set the quantization levels as [-1,1). We would
7

---

Under review as a conference paper at ICLR 2020
like to point out that we quantize all learnable parameters, meaning approach results in 32 times less memory compared to the floating-pon point counterparts.
We evaluate two MD variants corresponding to tanh and softmax projections, CIFAR- 10, CIFAR- 100 and TinyImageNef datasets with VGG-16 and ResNet-18 18 architectures. We also evaluate the numerically stable versions of our MD variants (denoted with suffix "-s") performed by storing auxiliary parameters during training as explained in Eq. (22). The results are compared against parameter quantization methods, namely BinaryConnect (BC) (Courbariaux et al. (2015)) ProxQuant (PQ) (Bai et al. 22019) and Proximal Mean-Field (PMF) (Ajanthan et al. (2019)) In addition, for completeness, we also compare against a standard PGD variant corresponding the tanh projection (denoted as GD-tanh), f.e., minimizing ftaanh (x)) using gradient descent. The only difference of this to our MD-tanh-S is that, Eq. (2). the Jacobian of tanh is directly used the updates. Note that, numerous techniques have emerged with BC as the workhorse algorithm by relaxing constraints such as the layer-wise scalars (Rastegari al. (2016), and similar extensions are straightforward even in our case. Briefly, our results indicate that the binary networks obtained by the MD variants yield accuracies very close to the Noating-point counterparts while outperforming the baselines.
For all the experiments, standard multi-class cross-entropy loss is used. We crossvalidate the hyper- parameters such as learning rate, learning rate scale, rate of increase of annealing hyperparameter B, and their respective schedules for all tested methods including the baselines. This extensive crossvali- dation improved the accuracies of previous methods by large margin, e.g., up to 3% improvement for PMF. We provide the hyperparameter tuning search space and the final hyperparameters in Appendix C. Our algorithm is implemented in PyTorch (Paszke et al. (2017)) and the experiments are performed on NVIDIA Tesla-P100 GPUs. Our code will be released upon publication.
CIFAR-10 10
CIFAR 100
TinylmageNet ResNet- 18
Algorithm
Space
VGG-16
ResNet-18
VGG-16
18
REF (float)
w
93.33
94.84
71.50
76.31
58.35
BC
W
89.04
91.64
59.13
72.14
49.65
PQ
85.41
90.76
39.61
65.13
44.32
PQ*
W
90.11
92.32
55.10
68.35
49.97
PMF
11
90.51
92.73
61.52
71.85
51.00
PMF*
UI
91.40
93.24
64.71
71.56
51.52
GD-tanh
91.47
93.27
60.67
71.46
51.43
MD-softmax
U1
90.47
91.28
56.25
68.49
46.52
Ours
MD-tanh
91.64
92.97
61.31
72.13
54.62
MD-softmax-S
91.30
93.28
63.97
72.18
51.81
MD-tanh-S
W
91.53
93.18
61.69
72.18
52.32
Table 1:
Classification accuracies on the set for different methods for binary quantization. PQ* denotes performance with biases, fully-connected layers, and shortcul layers in floating-point (original PQ setup) whereas PQ represents full quantization. PMF* denotes the performance of PMF after crossvalidation similar to our AD-variants and the original results from the paper are denoted as PMF. Note our MD variants obtained accuracies virtually the same as the best performing method and it outperformed the best method by large margin in much harder TinylmageNer dataset.
5.1 RESULTS
The classification accuracies of binary networks obtained by both variants of our algorithm, namely, MD-tanh and MD-softmax, their numerically stable versions (suffix "-s"" and the baselines BC, PQ, PMF, GD-tanh and the floating point Reference Network (REF) are reported in Table [ Both the numerically stable variants consistently produce better or on par results compared other binarization methods while narrowing the performance gap between binary networks and floating point counterparts to large extent, on multiple datasets.
Our stable MD-variant perform slightly better than MD- MD-softmax, whereas for tanh, MD updates either perform on par or sometimes even better than numerically stable version of MD-Lanh. We
wwgenet. http:///inynmmmmeeeeeeeeeerrrraa
8

---

Under review as a conference paper at ICLR 2020
Tranne LeKe
035
Eporas
Figure 3: Training curves for binarization for CIFAR-IO (first two) and CIFAR-I00 (last two) with R8.Ner-18. Compared to original MD variants, stable MD variants are less noisy and after the initial exploration phase (up to 60 in CIFAR-10 and 25 epochs CIFAR-100) the validation accuracies rise sharply and show gradual improvement afterwards.
believe, the main reason for this empirical variation results for our MD-variants due to numerical instability caused by the foating-poin arithmetic of logarithm and exponential functions in Eq. (15) and Eq. (20). Furthermore, even though our two MD-variants, namely MD-softmax and MD-tanh optimize in different spaces, their performance is similar in most cases. This may be explained by the fact that both algorithms belong to the same family where a "soft" projection to the constraint set (in fact the constraints sets are equivalent in this case, refer Sec. 2.2.2) is used and annealing hyperparameter is used to gradually enforce quantized solution.
Note, PQ (Bai et al. 2019h) does not quantizee the fully connected layers, biases and shorteut layers. For fair comparison, we crossvalidate PQ with all layers binarized and original PQ settings, and report the results denoted PQ and PQ* respectively in Table [ Our MD-variants outperform PQ consistently on multiple datasets in equivalent experimental settings. This clearly shows that entropic or tanh-based regularization with our annealing scheme is superior simple "W" shaped regularizer and emphasizes that MD is a suitable framework for quantization.
Furthermore, the superior performance of MD-tanh against GD-tanh and on par or better performance of MD-softmax against PMF for binary quantization empirically validates that MD is useful even in a nonconvex stochastic setting. This hypothesis along with our numerically stable form of MD can be particularly useful to explore other projections which are useful for quantization and/or network compression in general.
The training curves for our MD variants for CIFAR-10 and CIFAR- 100 datasets with ResNet- 18 are shown Fig. 3 The original MD variants show unstable behaviour during training which attributed to the fact that it involves logarithms and exponentials in the update rules. In addition, we believe, the additional annealing hyperparameter also contributes to this instability. Regardless, by storing auxiliary variables, the MD updates are demonstrated to be quite stable. This clear distinction between MD variants emphasizes the significance of practical considerations while implementing MD especially in NN optimization. For more experiments such training curves comparison other methods and ternary quantization results please refer to the Appendix C.
6 DISCUSSION
In this work, we have introduced an MD framework for NN quantization by deriving mirror maps corresponding to various projections useful for quantization. In addition, have discussed numerically stable implementation of MD by storing an additional set of auxiliary variables and showed that this update is strikingly analogous to the popular STE based gradient method. The superior performance of our MD formulation even with simple projections such as tanh and softmax is encouraging and we believe, MD would be suitable framework for not just NN quantization but for network compression in general. Finally, some theoretical aspects such as the use of time-varying mirror maps and the combination of MD and stochastic optimizer such as Adam are left unattended in this paper, which we intend to analyze in a future work.
REFERENCES
Gradient theorem. http:///ww/////..o/./aaaa/////// theorem. Accessed:
31-07-2019.

---

Under review as a conference paper at ICLR 2020
J. Achterhold, J. M. Kohler, A. Schmeink, and T. Genewein. Variational network quantization. ICLR, 2018.
Thalaiyasingam Ajanthan, Puneet K Dokania, Richard Hartley, and Philip HS Torr. mean meen-fiell for neural network quantization. ICCV, 2019.
Proximal
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. ICLR, 2019.
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 2003.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2009.
Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 2015.
Sébastien Bubeck, Nicolo Cesa-Bianchi, and Sham M Kakade. Towards minimax policies for online linear optimization with bandit feedback. Conference on Learning Theory, 2012.
2012.
Miguel A Carreira-Perpinán and Yerlan Idelbayev. Model compression as constrained optimization, with application to neural nets. part ii: Quantization. NeurIPS Workshop on Optimization for Machine Learning, 2017.
Matthieu Courbariaux, Yoshua Bengio, and -ean-Pierre David, Binaryconnect: Training deep neural networks with binary weights during propagations. NeurIPS, 2015.
S. K. Esser, Appuswamy, A. Merolla, V. Arthur, and D. S. Modha. Backpropagation for energy-efficient neuromorphic computing. NewrlPS, 2015.
Yunhui Guo. A survey on methods and theories of quantized neural networks. arXiv preprint 180088088,47 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Lu Hou, Quanming Yao, and James T Kwok. Loss-aware binarization of deep networks. ICLR, 2017.
Ya-Ping Ping Hsieh, Ali Kavis, Paul Rolland, and Volkan Cevher. Mirrored langevin dynamics. NeurIPS. 2018.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. ICLR, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. JMLR. 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. ICML, 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H S Torr. SNIP: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. NeurIPS, 2017.
C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning. NeurIPS, 2017.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling, Relaxed quantization for discretized neural networks. ICLR, 2019.
Arkadii Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. ECCV, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. SIMSS, 2018
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understand- ing straight-through estimator in training activation quantized neural nets. ICLR, 2019.
Ruimao Zhang, Liang Lin, Rui Zhang, Wangmeng Zuo, and Lei Zhang. Bit-scalable deep hashing with regularized similarity learning for image retrieval and person re-identification. TIP, 2015.
Siqi Zhang and Niao He. On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization. CoRR, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, 2016.
Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Stephen Boyd, and Peter Glynn. Mirror descent in non-convex stochastic programming. CoRR, 2017a.
Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Stephen Boyd, and Peter W Glynn. Stochastic mirror descent in variationally coherent optimization problems. NeurlPS, 2017b.
Appendices
Here, we first provide the proof of the theorem and the technical derivations. Later we give additional experiments and the details of our experimental setting.
A DERIVING MIRROR MAPS FROM PROJECTIONS
Theorem A.I. Let X be compact convex and P IR - C be invertible function where C IR is a convex open set such that X = (C denotes the closure of C). Now; if
1. P is strictly monotonically increasing.
2. lim | 2o (8C denotes the boundary of C).
Then, (xx) - zo
p-1 p-1(y) (yydy is valid mirror map.
Proof. From the fundamental theorem of calculus, the gradient of 4(x) satisfies, (=xx)= Since P is strictly monotonically increasing and invertible, P-1 is strictly monotonically increasing. Therefore, ((x) is strictly convex and differentiable. Now, from the definition of projection and since it is invertible (i.e., p-1 is one-to-one and onto). (
P-I(C) = IR. Therefore, together with condition (2), we can conclude that ((x) _ p p (v)dy is valid mirror map (refer Definition 2.2 in the main paper). For the multi--imensional case, we need an additional condition that the vector field p-1 (xx is conservative. Then by the gradient theorem (gra), there exists a mirror map (xx Sx p- (yydy for some arbitrary base point Xo.
11

---

Under review as a conference paper at ICLR 2020
B MD UPDATE DERIVATION FOR THE TANH PROJECTION
We now derive the MD update corresponding to the tanh projection below. From Theorem A. the mirror map for the tanh projection be written as:
1 (1 + w) log(1 + w) + (1 w) log(1 - -w) 23
(w)
P (w)dw
(24)
Correspondingly, the Bregman divergence can be written as:
Dø(w, (w, r) = (w=) { Φ ('()) - u): where ('(v) 23
log 1+c
(25)
1-v
(1 w1 v)
w log
+ log(1 w)(1 + +w) log(1 r)(1 - v)
2,9
(1 w) + v)
Now, consider the proximal form of MD update
w k+1
argmin (n ww) + Do(w, xE(-1.1)
(26)
The idea is to find w such that the KKT conditions are satisfied. To this end, let us first write the Lagrangian of Eq. (26) by introducing dual variables y and corresponding to the constraints -1 and w < 1, respectively:
1
(1 w) w*
F(w, x. y) n w +
log (1 w1 (1 wh)
+ log(1 w)(( w) log(1 w² )1 w²)
w*
23
(27)
+ y( ( 1) + z(w 1).
Now, setting the derivatives with respect to w to zero:
OF = ng* 23
1
(1 + w)( - w²)
log
y +==
(28)
dw From complementary slackness conditions,
(1 w)(1 + w²)
y(-w - 1) 2(w 1) 0,
since
w > -1 ==
(29)
since w < 1 = = 0.
Therefore, Eq. (28 now simplifies to:
OF
1
(1 w)( w²)
ng k
log
(30)
dw
23
(1 w) + w*)
(1 (1 (1- w*)
log
exp( (2,8ngn))))))))
(1 w)(1 (1 + w*)
1 1 w
1 + =
1 wi exp( (28ngg
1
1 w-
exp( 22,,ng**))))
W
Lowk T-
exp(-2.8np)
The pseudocodes of original (MD-tanh) and numerically stable versions (MD-tanh-S) for tanh are presented in Algorithms T and respectively.
C ADDITIONAL EXPERIMENTS
We first give training curves of all compared methods, and provide ternary quantization results as proof of concept. Later, we provide experimental details.
Convergence Analysis. The training curves for CIFAR- 10 and CIFAR- 100 datasets with ResNet- 18 are shown in Fig. Notice, after the initial exploration phase (due to low 3) the validation accuracies of our MD-tanh-S increase sharply while this steep rise is not observed in regularization methods such as PQ. The training behaviour for both our stable MD-variants (softmax and tanh) is quite similar.
12

---

Under review as a conference paper at ICLR 2020
Algorithm 1 MD-tanh
Require: K,b, hpp I, Ensure: w E Q
1: w
E RR". 3 t-1
Initialization Projection
2: w tanh(Bw")
3: for f- O...., K do
4: D6 (x);
D
Sample mini-bater D Gradient w.r.t at w (Adam based gradients)
5:
g < V (w; Do) w=
6:
for j t 1, ,,..,,, m do
exp( 05)
7:
w
b MD update
expl 28n +1
8: end for
9: B t p
> Increase B
end for
11: w f (
> Quantization
Algorithm 2 MD-tanh-S
Require: K. b, (1/*)
[np 1, L,D.L
Ensure: w E Q
1: w E "R,, B x-1
Initialization
2: for k <- 0,..., K do
3:
w - k
Projection
4:
D6 = (x);
D
Sample a mini-batch
5:
g - "wl((w D
Gradient w.r.t. at w (Adam based gradients)
w-wk
6:
w+tl t- we 1/" g
Gradient descent on w
7:
p8p
> Increase B
8:
end for
9: w t- sign (w
>
Quantization
Ternary Quantization. As proof of concept for our shifted tanh projection (refer Example 3.3 we also show results for ternary quantization with quantization levels Q {-1, 0, 1} in Table Note that the performance improvement of our ternary networks compared to their respective binary networks is marginal as only 0 is included as the 3rd quantization level. In contrast to us, the baseline method PQ (Bai et al. (2019) optimizes for the quantization levels (differently for each layer) as well in an alternating optimization regime rather than fixing it Q = {-1, O, 1). Also, PQ does ternarize the first convolution layer, fully-connected layers and the shortcut layers. We cross-validate hyperparameters for both the original PQ setup and the equivalent setting of our MD-variants where we optimize all the weights and denote them as PQ and PQ respectively.
Our MD-tanh variant performs on par sometimes even better in comparison tanh projection results where gradient is calculated through the projection instead of performing MD. This again empirically validates the hypothesis that MD yields in good approximation for the task of network quantization. The better performance of PQ in their original quantization setup, compared to our approach in CIFAR-10 can be accounted to their non-quantized layers and different quantization levels. We believe, similar explorations are possible with our MD framework as well.
Experimental Details. As mentioned in the main paper the experimental protocol is similar to (Ajanthan et al. (2019)). To this end, the details of the datasets and their corresponding ex- periment setups are given in Table 3 For 1IFAR-10/100 and TinyImageNet, VGG-16 (Simonyan & Zisserman ))15)) and ResNet-18 (He et al. (2016) architectures adapted for CIFAR dataset are used. In particular, for CIFAR experiments, similar to (Lee et al. (2019)) the size of the fully-connected (FC) layers of VGG-16 is set to 512 and no dropout layers employed. For Tiny of the first convolutional layer of ResNet- 18 is set to 2 to handle the image size (Huang et al. (2017). In all the models, batch normalization (loffe & Szegedy (015)) (with no learnable parameters) and
the stride
13

---

Under review as a conference paper at ICLR 2020
Vplidetion Focuacy
Trawing Loge
Trann Pg Love
Figure 4: Training curves for binarization for CIFAR-10 10 (first two columns) and CIFAR-100 (last two columns) with ResNet-18. Compared to BC, our MD-Lanh-S and PMF are less noisy and after the initial exploration phase (up to 60 in CIFAR-I0 and 25 epochs CIFAR- (00), the validation accuracies rise sharply and closely resembles the floating point network afterwards. This steep increase is not observed in regularization methods such PQ.
CIFAR-10 10
CIFAR-IO 100
TinylmageNet ResNet- 18
Algorithm
Space
VGG-16
ResNet-18
VGG-16
ResNet-18
REF (float)
W
93.33
94.84
71.50
76.31
58.35
PQ
W
83.32
90.50
32.16
59.18
41.46
PQ*
92.20
93.85
57.64
70.98
45.72
GD-tanh
W
91.21
93.20
53.88
69.48
50.65
Ouss
MD-ssfltmax-S
91.69
93.30
65.11
72.01
52.21
MD-lanh-S
W
91.70
93.42
66.15
71.29
52.69
Table 2: Classification accuracies the test for ternary quantization. PQ* denotes performance with fully-connected layers, first convolution layer and shortcut layers in floating point whereas PQ represent results with all layers quantized. Also, PQ optimize for the quantization levels well (different for each layer), in contrast we fix it to Q = {-,,0,11 GD-tanh denotes results without using STE and actually calculating the gradient through the projection.
ReLU nonlinearity are used. Only for the floating point networks (i.e., REF), we keep the learnable parameters for batch norm. Standard data augmentation (i.e., random crop and horizontal flip) is used.
For both our MD variants, hyperparameters such as the learning rate, learning rate scale, annealing hyperparameter 3 and its schedule are crossvalidated from the range reported in Table 4 and the chosen parameters are given in the Tables [5] and 6 To generate the plots, we used the publicly and [
available codes of BCD pqJ
All methods are trained from a random initialization and the model with the best validation accuracy is chosen for each method. Note that, in MD, though enforce a discrete solution, the chosen network may not be fully-quantized (as the best model could be obtained an early stage of training). Therefore, simple argmax rounding is applied to ensure that the network is fully-quantized.
increasing schedule for B to
14

---

Under review as a conference paper at ICLR 2020
Dataset
Image
class
Train / Val.
b
K
Hyperparameters
Fine-tuning grid
MNIST
28 28
10
50k 10k
100
20k
learning rate
0..1..011, 0.001, 0.0001] 0.2,,0. 0.5] [1.01, 1.02, 1.05, 1.1, 100, 200, 500, 1000, 2000]
CIFAR-I0
32 32
10
45k 5k
128
100k
Ir Ir_seale
CIFAR-100
x 32
100
45k
100k
beta scale
TinylmageNet
64 x 64
200
100k / 10k
128
100k
beta beta_scale sat__salleeteevvalll interval
Table 3: Experiment setup. Here, is the batch size and K is the total number of iterations for all the methods.
The hyperparameter search space for all the experiments. Chosen
Table 4:
parameters are given in Tables S Sand 6
CIFAR-I0 ResNet-18
MD MD-softmax
MD tanh
MD-softmax.S
MD MD-tanb.s
PMF*
GD-tanb
BC
PQ
learning_rait
0.001
0,001
0.001
0.001
0.001
0,001
0.001
0.01
Ir Ir_seale
0.2
0.3
0.3
0.3
0.3
0.3
0,3
0.5
1.02
1.02
0.0011
bela_xcale _caleeinteevl
200
100
200
200
1000
1000
CIFAR- 100 with ResNet-18
MD suftmax
MD. MD-tanh
MD softmaax-
MD- tanb-S
PMF*
GD-tanh
BC
PQ
learning_rate rate
0.001
0.001
0.001
0.001
0.001
0,001
0.001
0.1
scale
0.2
0.2
betn, betn_scale scalle
1.05
1.05
1.1
1.2
1.01
1.01
0.001
ain__scale_intrvv
500
500
100
CIFAR 10 with VGG- 16
MD-softmax
MD-softmax--
MD-tauh-S
GD-tanb
PQ
learning_rate
0.01
0.001
0.001
0.001
0.001
0.001
0.0001
0.01
Ir_scale
0.2
0.3
0.3
0.2
0.5
0.3
0.3
0.5
beta, sccle
1.05
1.2
1.05
0.0001
keta_scale_jne iiterval
500
1000
2000
2000
500
1000
CIFAR VGG-
MD-softmmx
MD-tanh
MD-softmax-S
MD-tanh-S
PMF*
GD-tanh
BC
PQ
learning
0.001
0.001
0.0001
0.001
0.0001
0,001
0.0001
0.01
Ir. scale
0.3
0.3
0.2
0.5
0.5
0.5
0.2
0.5
scale
1.01
1.05
0.0001
betn_scale interval
100
500
500
500
200
1000
ImyIImggeNet with ResNet. 18
MD-softmax
MD-tanh
MD-softmax-S
MD-tanh-S
PMR*
GD-tanh
BC
PQ
learning rate
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.01
scale
beta_scale (ours)
1.02
1.2
1.02
1.2
1.01
1.01
0.0001
Aeta_scale_intere
200
2000
500
100
Table 5: Hyperparameter settings used for the binary quantization experiments. Here, the learning rate is multiplied by Ir scale after every 30k tterations and annealing hyperparameter (3) is multiplied by beta_ scale after every beta scallsscllleaaaa iintrralt iterations. We use Adam optimizer with zero weight decay. For PQ, beta_scal scale denotes regularization rate.
15

---

Under review as a conference paper at ICLR 2020
CIFAR-10 with ResNet-18
REF (float)
MD-softmax-8
MD-tanh-5
GD-tanh
PQ
PQ
learning rate
0.1
0.001
0.01
0.01
0.01
0.01
Ir scale
0.3
0.2
0.5
0.3
beta_scale (ours)
1.05
1.2
1.02
0.0001
0.0001
cottassalc_interval
500
1000
500
weight_decay
0.0001
0
0
0
0.0001
CIFAR-100 100 with ResNet-18
REF (float)
MD-softmax-S
MD-tanh-s
GD-tanh
PQ
PQ
learning_rate
0.1
0.001
0,001
0.01
0.01
0.001
Ir_scale
0.1
0.1
0.5
0.5
0.2
beta_scale (ours)
1.1
1.02
0.0001
0.0001
heta_ satta_scll__ua satltsscaee__eeraalll interval
100
500
1000
weight_decay
0.0001
0
0
0.0001
CIFAR- 10 with VGG-16
REF (float)
MD-softmax-S
MD-tanh-S
GD-tanh
PQ
PQ
learning rate Ir scale
0.1
0.001
0.01
0.01
0.01
0.1
0.2
0.3
0.3
0.3
beta_scale (ours)
1.05
1.1
1.01
1c-07
0.0001
iite_scale_intere
500
1000
500
weight_docay
0.0001
0
0
0
0.0001
CIFAR 100 with VGG-16 16
REF (float)
MD-softmax-S
MD-tanh-S
GD-tanh
PQ
PQ
learning rate
0.0001
0.001
0.01
0.01
0.0001
Ir scale beta_scale (ours)
0.2
0.3
0.5
0,2
1.05
1.1
1.05
0.0001
0.0001
peta_scale scale_interval interval
100
500
2000
0.0001
0
0
0.0001
TinylmageNet with ResNet-18
REF (float)
MD-softmax-S
MD-tanh-S
GD-tanh
PQ
PQ
learning
0.1
0.001
0.01
0.01
0.01
0.01
Ir scale
0.1
0.5
beta_scale (ours) Deta_scalc_inter jnterval
1.2
1.2
1.05
0.01
0.0001
500
2000
2000
weight_decay
0.0001
0
0
0
0
0.0001
Table 6: Hyperparamettr settings used for the ternary quantization experiments. Here, the learning rate is multiplied by It scale after every 30k tterations and annealing hyperparameter (3) is multiplied by beta scale after every beta scale interval iterations. We use Adam optimizer except for REF for which SGD with momentum 0.9 used. For PQ, beta scale denotes regularization rate.
16


Reviews:
[{"review_id": "ryg5FaBptB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Mirror Descent (MD) framework for the quantization of neural networks, which, different with previous quantization methods, enables us to derive valid mirror maps and the respective MD updates. Moreover, the authors also provide a stable implementation of MD by storing an additional set of auxiliary dual variables. Experiments on CIFAR-10/100 and TinyImageNet with convolutional and residual architectures show the effective of the proposed model. \n\nOverall, this paper is well-written and provide sufficient material, both theoretical and experimental evidence to support the proposed method. Although the novelty of this work is somehow limited, i.e. appling MD from convex optimization to NN quantization, the authors provides sufficient effort to explore how to success to adopted it the literature. Hence, I lean to make an accept suggestion at this point. \n\nConcern: it would better to provide the code to validate the soundness of the model.\n\n##post comments\nThe rebuttal addresses my concerns and I will not change my score. Thanks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gG6JwAYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A good paper that uses the Mirror Descent paradigm for learning quantized networks.  \nThough Mirror Descent is not their original idea, but using it in the context of learning quantized network is novel and interesting.  \nEmpirically, they showed better results than existing method, with comparisons with reasonable baselines including using relaxed projected gradient descent.  \n\nOverall, I don’t have much concerns, but here are some more specific comment/questions (most relates to writing)\n\nIn the intro, it would be great to mention some past success on using MD, as opposed to just saying it’s well-known. Also you mention MD can be used for more than quantization, but compression in general, it’d be better to add that discussion, or remove this sentence. \n\nIn the beginning of Section 2.1, it'd be easier for the readers to make clear that the primal space corresponds to the quantized weights and the dual space corresponds to the unconstrained space in the rest of the paper.\n\nAt the top of page 3 you describe MD for the first time, but it’s unclear to me how y^0 is handled.\n\nThe end of section 3 and section 4 talk quite a bit about STE, maybe it'd be clear if the authors can provide a concise description.\n\nAs someone not super familiar with NN quantization, this work seems like a good contribution.  My only possible concerns would be somehow comparisons to existing methods are not comprehensive enough (if this will be pointed out by the other reviewers)\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlXGL9j2B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use the mirror descent algorithm for the binary network. The key point is Theorem 3.1, which enables the mirror map. The paper is easy to read and follow, and the main contributions are clearly stated.\n\nHowever, I suggest a weak rejection of this paper. The reasons are\n\nQ1. As Review #3, it is better for authors to provide more theoretical analysis, which better includes the nonconvex objective function and the effect of annealing. \n\nQ2. It is not clear to me, why mirror descent is better than proximal gradient descent, i.e., proxQuant, in this application. The authors repeatedly claim \"MD allows gradient descent to be performed on a more general non-Euclidean space\". This cannot be told by Table 1, which is just overall performance. So, it is better to empirically show this point by an ablation study.\n\nQ3. Since the technical contributions are not enough, I expect more experimental comparisons.\n- Could the authors perform experiments on ImageNet?\n- While VGG and ResNet are taken as a protocol for experimental comparison, it is better to do an extra comparison with STOA networks. VGG and ResNet are too old and easy to be compressed, compression these networks are of little practical values. EfficientNet [1], Mobilenets [2], and Shufflenet [3] can be good ones. The paper will be more convincing with these methods.\n\n[1]. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\n[2]. Mobilenets: Efficient convolutional neural networks for mobile vision applications\n[3]. Shufflenet: An extremely efficient convolutional neural network for mobile devices", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1li8fEhYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural network (NN) quantization based on Mirror Descent (MD) framework. The core of the proposal is the construction of the mirror map from the unconstrained auxiliary variables to the quantized space. Building on that core, the authors derive some mapping functions from the corresponding projection, i.e. tanh, softmax and shifted tanh. The experimental result on benchmark datasets (CIFAR & TinyImageNet) and basic architectures (VGG & ResNet-18) showed that the proposed method is suitable for quantization. The proposed method is a natural extension of ProxQuant, which adopted the proximal gradient descent to quantize NN (a.k.a $\\ell_2$ norm in MD). Different projections in NN quantization lead to different Bregman divergences in MD. \n\nHowever, the authors do not analyze the convergence of the MD with nonconvex objective function in NN quantization neither how to choose the projection for mirror mapping construction. Moreover, it is better to discuss with [Bai et al, 2019] to clarify the novelty of the proposed method. So I concern about the novelty and the theoretical contributions \n\nYu Bai, Yu-Xiang Wang, Edo Liberty. \nProxQuant: Quantized Neural Networks via Proximal Operators. ICLR 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use the mirror descent algorithm for the binary network. It is easy to read. However, novelty over ProxQuant is somehow limited. The theoretical analysis is weak, in that there is no analysis on the convergence and neither how to choose the projection for mirror mapping construction. Experimental results can also be made more convincing, by adding comparisons with bigger datasets, STOA networks, and ablation study to demonstrate why mirror descent is better than proximal gradient descent in this application.
Decision: Reject