CBP: backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
Authors: Guhyun Kim
Doo Seok Jeong
Venue: NeurIPS 2021 Poster (2021)
Keywords: backpropagation with constraints
weight quantization
pseudo-Lagrange multiplier method
learning

Abstract: Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints — binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post-training method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For most cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6\%, 74.4\%, and 64.0\% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP is publicly available at \url{https://github.com/dooseokjeong/CBP}.

CBP: Backpropagation with constraint on weight precision using pseudo-Lagrange multiplier method
Guhyun Kim, Doo Seok Jeong"
Division of Materials Science and Engineering Hanyang University, Republic of Korea
gubyunkim01@gmail. com, dooseok Shanyang ac kr
Abstract
Backward propagation of errors (backpropagation) is method minimize objec- tive functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) its objective function. We considered various types of constraints binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post- training method, CBP applied to AlexNet, R ResNet-50, and GoogLeNet ImageNet, which were pre-trained using the conventional backpropagation, For most cases, the proposed algorithm outperforms the state-of-the-arr methods on ImageNet, e.g., 66.6%, 74.4%, and 64.0% top-1 accuracy for ResNet-18, 18, ResNet- 50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP publicly :tttps://gitan h/tp:/gg.tttt.tt/// com wwwoosseoeeoogoooooooooooo
available htttt:::////
Introduction
Currently, deep learning-based methods applied in variety of tasks, including the classification of static data, e.g., image recognition [1] 2]; classification of dynamic data, e.g., speech recogni- tion (3:6: function approximations, which require the output of precise predictions, e.g., electronic structure predictions [7] and nonlinear circuit predictions (8). All of the aforementioned tasks require discriminative models. Additionally, generative models, including generative adversarial networks and variants [30 comprise another type of deep neural network. Despite the diversity in applica- tion domain and model type used, almost all deep learning- bassed methods use backpropagation as common learning algorithm.
Recent developments in deep learning have primarily focused increasing the size and depth of deep neural networks (DNNs) to improve their learning capabilities as in the case of of the-art DNNs like VGGNet (14 and ResNet (15). Given that the memory capacity required by DNN is proportional to the number of parameters (weights and biases), memory usage for DNN becomes severe. Additionally, significant number of multipy--acummulat operations during the
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training and inference stages impose prohibitive workload on hardware. Thus. efficient hardware- resource consumption is critical the optimal performance of deep learning. One way address this requirement is the use of weights of limited precision, such as binary [16 17 and ternary weights [8 19]. To this end. particular constraints are applied to weights during training, and additional algorithms for weight quantization are used in conjunction with backpropagation. This is because such constraints are not considered during the minimization of the objective function (loss function) when backpropagation is executed.
We adopt the Lagrange multiplier method (LMM) to combine basic backpropagation with additional constraint algorithms and produce a single constrained backpropagation (CBP) algorithm. We refer the adopted method as pseudo-LMM because the constraint functions cs (x) are nondifferentiable at Fm (= arg m cs (æ)), rendering LMM inapplicable. Nevertheless, pseudo-LMM successfully attains the optimal point under particular conditions as for LMM. In the CBP algorithm, the optimal weights satisfying a given set of constraints are evaluated via basic backpropagation algorithm. is implemented by simply replacing the conventional objective function (loss function) with a Lagrangian function L that comprises the loss and constraint functions as sub-functions that are subjected to simultaneous minimization. Therefore, this method perfectly compatible with conventional deep learning frameworks. The primary contributions of this study are follows.
We introduce a novel and simple method to incorporate given constraints into backpropa- gation by using a Lagrangian function as the objective function. The proposed method is able to address any set of constraints the weights insomuch the constraint functions are mathematically well-defined.
We introduce pseudo-LMi LMM with constraint functions cs (w) that are nondifferentiable at Wm (= arg min (w)) and analyze the kinetics of pseudo-LMM in the continuous time domain.
We introduce optimal (sawtooth-like) constraint functions with gradually vanishing unconstrained-weigle windows and provide guiding principle for the stable co-optimization of weights and Lagrange multipliers in quasi-static fashion.
We evaluate the performance of CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet (pre-trained using backpropagation with full-precision weights) with four different constraints (binary, ternary, one-bit shift, and two-bit shift weight constraints) on ImageNet as proof-of- -roo--ef-concept examples. The results highlight the classification accuracy outperforming the previous state-of sfate-of-the- the--arttt art results.
2 Related work
The simplest approach to weight quantization the quantization of pre-trained weights. Gong et al. (20] proposed several methods for weight quantization and demonstrated that binarizing weights using a sign function degraded the top-1 accuracy on ImageNet by less than 10%. Mellempudi et al. [21] proposed tine-graine quantization algorithm that calculates the optimal thresholds for the ternarization of pre-trained weights. The expectation backpropagation algorithm [22] implements a variational Bayesian approach to weight quantization. It uses binary weights and activations during the inference stage. Zhou et al. (23) proposed the incremental network guantization method (INQ) that iteratively re- re-trains group of weights to compensate for the performance loss caused by the rest of weights which are quantized using pre-set quantization thresholds.
Several methods of weight quantization utilize auxiliary real-valued weights in conjunction with quantized weights during training. The srreight-through-stie
(STE) comprises the conduction of forward and backpropagation using quantized weights but relies the auxiliary real-valued weights for the update of weights 3.. BinaryConnect [66] utilizes weights binarized by a sign function for forward and backpropagation, and the real-valee real-valued weights updated via backpropagation with binary weights. The hinary-weight-ne the real-valued weights using scaling factor, and it exhibits higher classification accuracy than BinaryConnect on ImageNet. The binarized neural nets 24 and XNOR-Nets [7 are extensions of BinaryConnect and BWN, respectively, which utilize binary activations alongside binary weights.
(BWN) [17] identifies the binary weights closest to
Lin et al. 18 proposed TernaryConnect and Ternary- weinrry-wwight-etwoor (TWN), which are similar to BinaryConnect and BWN but use weiggt-ternarizztion methods instead. Trained ternary quantization

---

(TTQ) 125 also uses ternary weights that are quantized using trainable thresholds for quantization. LQ-Nets proposed by Zhang et al. [26] utilize activation- and weight-quantizers considering the
actual distributions of full-precision activation and weight, respectively. DeepShift Network [27] includes the LinearShift and ConvShift operators that replace the multiplications of fully-connected layers and convolution layers, respectively. Qin et al. (28) introduced IR-Nets that feature the use of error decay estimators to approximate sign functions for weight and activation binarization to differentiable forms. Gong et al. [29] and Yang et al. [30] employed functions similar to step functions but differentiable. Pouransari et al. 3IT proposed the least squares quantization method that searches proper scale factors for binary quantization. Elthakeb et al. [32) attempted to quantize weights and activations by applying sinusoidal regularization. The regularization involves two hyperparameters that determine weeight-qanttiznn
and bitwidth-reuularizatie strengths.
To take into account the constraint on weight precision, Leng et al. 3 used an augmented Lagrangian function as an objective function, which includes a constraint-agreement sub-function. The method was successfully applied to various DNNs on ImageNet; yet, the method failed to reach the accuracy level for full- fill-precision models even when 3-bit precision was used.
The CBP algorithm proposed in this study also utilizes LMM; but CBP essentially differs from 33 given that (i) the basic differential multiplier method (BDMM), rather than ADMM, is used to apply various constraints on weight precision, (ii) particularly designed constraint functions with gradually vanishing unconstrained- -weight windows are used, and (iii) substantial improvement on the classification accuracy on ImageNet achieved.
3 Optimization method
3.1 Pseudo-Lagrange multiplier method
LMM calculates the maximum or minimum value of differentiable function f under a differentiable constraint cs = 0. (34) Let us assume that a function f (z, y) attains its minimum (or maximum) value m satisfying a given constraint ( i.e., f (I 3mm = m. Further, cs (Im, 3mm) as the constraint is satisfied at this point. In this case, the point of intersection between the graphs of the two functions, ff (x, y) m and cs (I, y) = 0, is (mm: Vm). Because the two functions have a common tangent at the point of intersection, the following equation holds:
V Vayl =-VV As,,ss
at (z Vm), where X is Lagrange multiplier (35).
The Lagrangian function is defined as L (x. v, = f (I.1) + des (I, (I.y). Consider a lo- cal point (x,z) at which the gradient of L (z. )) is zero. Therefore, V V.y.all yr, A) V.. [f (z. y) + Aes (I. y)1 + VA [des v)l. Thus, Vanal following equations.
(I. (a, = equivalent to the
VE. [f (a, v) + Acs (z. y) 0, Va Dess (z. v)] = 0
V Vayaa (I, =0 +
Vanf (x,y) cs (I, v)
-V esycs (x. v)
= =0
This satisfies the condition in Eq. ( as well as the constraint cs (x, y) = 0. Therefore, the local point (x. y) corresponds to the minimum (or maximum) point of the function f under the constraint.
We define pseudo-LMM to address similar minimization tasks but with continuous constraint func- tions that are nondifferentiable at (Im, 3/m) arg ming,y y cs (I, v). Thus, Eq. ( cannot be satisfied at the optimal point. Nevertheless, pseudo-LMM enables us to minimize the function f (zr) subject to the constraint condition es (ar) = 0 using the Lagrangian function L (æx, A).
Definition 3.1 (Pseudo-LMM). Pseudo-LMM is a method to attain the optimal variables æ,, that minimize function / (a)) subject to the constraint condition es (ax) = 0, where the function es (x) is nondifferentiable at a. but reaches the minimum at Fm, i.e., es (xmm) = 0.
Theorem 3.1. Minimizing the Lagrangian function L (æx, A), which is given by L (æx, A) = f (ar) + Acs (z), is equivalent to minimizing the function ff (x) subject to cs (x) 0
3

---

Proof. The Lagrangian function L (a. A) is always differentiable with respect to the Lagrangian multiplier X, so that the equation cs (x) holds at the optimal point mm Thus, we have
minimize
L( X)
minimize L (a. A) + Æ,
(3)
subject to cs (x) (x)= = 0.
When the constraint satisfied, i.e. cs (x) O, the Lagrangian function C (x; A) equals the function (x), so that the task to minimize C (x, A) with respect to x and À corresponds to the task to
minimize (x) subject cs (x) =
Given Theorem 3.1 Dseudo-LMM can attain the optimal point by minimizing the Lagrangian function L. in spite of the nondifferentiabiiity of the constraint function es (æx) at the optimal point mm- Note that not all functions have zero gradients at their minimum points; for instance, the function [a]] attains its minimum at z = 0 but the gradient at the minimum point is not defined. However, all convex functions have zero gradients their minimum points. Thus, LMM to minimize the function f (x) subject to the constraint convex function cs (x) with minimum point 2m is subset of pseudo-LMM. this case, the constraint function has zero gradient at the minimum point, so that Eq. becomes
Vzl (x) = (x, = + { subject to cs (x) 0.
We will consider continuous constraint functions with a few nondifferentiable points their variable domains, including their minimum points. Other than such nondifferentiable points, we will use the gradient descent method to search for the minimum points within the framework of Pseudo-LMM. Hercafter, when the gradient of the Lagrangian L function is remarked, its variable domain excludes such nondifferentiable points.
The optimal solution to Eq. can be found using the basic differential multiplier method (BDMM) (66] that calculates the point at which (æx, X) attains its minimum value by driving toward the constraint subspace E (cs (T) 0). The BDMM updates x and X according to the following relations.
x サ x - 12 Vx,
(5)
and
A x V
(6)
BDMM is cheaper than Newton's 's method in terms of computational cost. Additionally, Eq. (5) is
identical to the solution used in the gradient descent method during backpropagation, except for the use of Lagrangian function instead of loss function. This indicates the compatibility of pseudo-LMM with the optimization framework based on backpropagation.
3.2 Constrained backpropagation using the pseudo-Lagrange multiplier method
We utilize pseudo-LMM to train DNNs with particular sets of nright-constraints We define Lagrangian function L in the context of feedforward DNN using the following relation.
C y
W. >)
C. (1))) (g)) w) Ccss (W).
(k),
c8 = [0s] (WI) ...,) c8n (Wn.))
A Diy.... An
where (*) and ý (8) denote the actual output vector for the kth input data and correct label, respec- tively. The set W denotes a set weight matrices, including n., weights in aggregate, and the function C' denotes a loss function. Each weight w; (1 < < n.w)) is given one constraint function cs; (w:) and one multiplier X;
We chose sawtooth-shapel constraint functions. We quantize the real-valued weights into ng values in {q ye where q, qi+1 for all i. We also employ set of the medians of neighboring
the set Q

---

Constraint function
Constraint function
Binary weight constraint, g=2
Ternary weight constraint, g=2
UCS window
UCS
UCS
1
0
0
-2
-2
Weight
Weight
Constraint function
Constraint function
Binary weight constraint, g=10
Ternary weight constraint, g=10
-2
-1
0
-2
-1
1
Weight
Weight
Figure 1:
Binary- and termary-weight constraint functions for g 2 and 10. Blue-filled regions indicate unconstrained-wweihh windows.
values in the set Q: M [m; where mn (gi (:+1) /2. Using Q and M, we define 0, 1 S ng. and ng.
partial constraint function yı for i
W < q -2 (w 41) otherwise,
yo (w)
0
2 ww - my + gi+1 q
if 4: S w < 9++1
Yi (w)
otherwise,
2 2(w (mg) if w 2 On otherwise,
Ung (w)
(8)
The constraint function es is the summation of the partial constraint functions Y (w) E (w), gated by the unconstrained-weit weight window ues (w) parameterized by variable g.
respectively.
Cs (w: Q, 2,M,9) ucs (w) Y (w),
where
Bg -1
Wcs (w) = Σ İ=U
H
(g++1 9:) - |/ - m. + +el) el) 2g
(10)
where c + 0+, and denotes the Heaviside step function. The function ucs (w) realizes the imconstrained-weigee
window function of g (2 1). When g == 1, the function outputs zero for Q1 S w < In merely confining w to the range @1 I w < 9n, without weight quantization, whereas, when g oo, the window vanishes, allowing the constraint function quantize the weight in the entire weight range. Examples of function ucs (w) are shown in Fig.
The inccostrained-wwiieee window variable is initially set to one and updated such that it keeps increasing during training, i.c., the window gradually vanishes. The window gradually vanishing allows sequential weight quantization such that the further the initial weights from their nearest q. the later their weights are subject to quantization, which is otherwise subject to simultancous (abrupt) quantization. It is likely that the further the initial weights from their nearest gr, the larger the increase in loss function C' when they are quantized. Thus, the sequential guantization from the weights close to their 4; likely avoids abrupt increase in the loss. Further, while the closer weights are being quantized, the further weights (not subject to quantization yet) are being updated to reduce the loss given the partially quantized weights. This effect will be discussed in Sections 3.3
For every training batch, the weights are updated following a method similar to conventional back- propagation. Nevertheless, the use of the Lagrangian function in Eq. 8 rather than loss function

---

only, as an objective function constitutes a critical difference. The Lagrange multipliers À are sub- sequently updated using the gradient ascent method in Eq. 6 Updating cross-coupled variables. such as W and X, often experiences difficulties in convergence toward the optimal values hecause of oscillation around the optimal values. feasible solution involves quasi-static update. To this end, we significantly reduce the update frequency of the Lagrange multipliers À compared with weights W.
Weight update: Weights W are updated once every iteration as for the conventional backpropagation but using the Lagrangian function L.
Lagrange multipliers À are conditionally updated once every training epoch. The update is allowed if the summation of all L in given epoch (C is not smaller than f.uu for the previous epoch (Cemm) or the multipliers À have not been updated in the past Pmaz epochs. This achieves the convergence of W for given À in quasi-static manner.
Lagrange multiplier update:
Unconstrained weight window update: Unconstrained-weighh window variable g is updated on the same condition as for the Lagrange multipliers X. Unlike weights W and multipliers X, the variable g (initialized to one) constantly increases when updated such that Ag = 1 when g < 10, Ag 10 for 10 S < 100, and Ag 100 otherwise.
The detailed learning algorithm is shown in the pseudocode in Appendix A.2
3.3 Learning kinetics
Learning with BDMM using the Lagrangian function is better understood in the continuous time domain. We first address the kinetics of learning without the unconstrained weight window ues (w). The change in the Lagrangian function L at given learning step in the discrete time domain is equivalent to the derivative of with time at time in the continuous time domain, which is given by
dC
New
DC
Ocs: Dwi
2
TW
Ow;
+
T
CS
(11)
The Lagrange multiplier X; time is given by
X; (1) - No (0) S. cs.dt.
(12)
The constraint functions cs; approach zero as the weights approach their corresponding quantized values gi, and thus the Lagrange multipliers in Eq. (12) asymptotically converge to their limits.
Eqs. 11 and 12 are derived Appendix A.I
At equilibrium, the Lagrange function is no longer time-dependent, i.e., dC/dt 0. This requires the Lagrange multipliers reaching their limits, which turn requires the weights reaching their corresponding quantized values W leading to 0. For convenience, we define the integration of cs; in Eq. (12) as Acs; (2 0).
Acs, L' cs;dt, if cs; 0 as co.
(13)
Thus, the equilibrium Lagrange multiplier Aie0 can be expressed as X,ce X, (0) + T Aes.
(14)
Therefore, is evident from Eq.
TTT that the equilibrium leads to
ac (A: (0) + - c88
Ocs: Ow:
Vi,
(15)
Ow;
We consider sawtooth constraint functions with slopes Ls, i.e., Ocs;/Ow; =s. where > 0.
Eq. (8) is the case 2. Generally, the Lagrange multiplier is initialized to zero, i.e., X; (0) = 0. Therefore, the gradient of loss function C at the equilibrium point a is given by
ac
Vi, Ow:
= t sAcs; =\
(16)

---

Consider that the loss function C' has the equilibrium point W (= arg min w C). Eq. (16) eluci- dates the increase of loss by attaining the equilibrium point W In this regard, A corresponds to the cost of weight quantization. Assuming the convexity of the loss function C' in domain D including W and W* C (W *c*) keeps increasing as A increases. If the initial pre-trained weights equal their corresponding quantized weights, i.e., W w then Acs; O, and thus X = 0 according to Eq. (14) Eq. (T)) consequently yields ac = O, indicating zero cost of quantization.
Considering the gradually vanishing mncontrraine-wweiht window ues (w) yields the derivative of L with time time L. in the continuous time domain follows.
dC
OC
0Y; Ajucsi
(ucs:Y.)
(17)
TW
Ows
Ows
The derivation of Eq. (77) weight window Dues from the others at a given time i, Eq. (17) can be written by
is given in Appendix A.33 Distinguishing the weights in the unconstrained-
dC = TW W Σ dt
aC
2
ac Ow.
+ Ai OY, Ows
)
Σ
TW
xx Y?
(18)
Owi
we E Draa
The latter term the right-hand side of Eq. (18) indicates that the weights outside the window Dues are being quantized at the cost of increase of loss. However, as indicated by the former term, the weights in the window Ducs are being optimized only to decrease the loss function with partially quantized weights. Compare this gradual quantization with abrupt quantization without the gradually vanishing unconstrained-we weight window, where all weights quantization. The gradual quantization allows the weights in the window to further reduce the loss function regarding the weights that have already been quantized eventual cost of quantization is likely smaller than the simultaneous quantization case.
subject to simultaneous (abrupt)
being quantized, and thus the
4 Experiments
To evaluate the performance of our algorithm, we trained three models (AlexNet, and ResNet- 18 and 50) on the ImageNet dataset [37 with four different weight constraints (binary. ternary, and one-bit, and two-bit shift weight constraints). ImageNet consists of approximately 1.2 million training images and 50 thousands validation images, All training images were pre-processed such that they were randomly cropped and resized to 224 x 224 with mean subtraction and variance division. Additionally, random horizontal flipping and color jittering were applied. For validation, the images were resized to 256 x 256 and their centers in 224 x 224 were cropped. We evaluated the top-1 and top-5 classification accuracies the validation set.
We considered binary, ternary, one-bit shift and two-bits shift weight constraints validate the CBP algorithm as a general weight-quantizatioon framework. For all cases, we introduced layer-wise scaling factors such that (1) (for the (th layer) is given by a w II1 (n) where w and (!()) denote the weight matrix of the /th layer and the number of elements of (!)), respectively. As for [17 and [19], the weight matrices of the first and last layers were not quantized. The quantized weights employed for each constraint case is elaborated as follows.
weight constraint: A set of quantized weights Q is {-a,a}, a).
The other weight constraints:
A set of quantized weights Q is (0. +2 +2-44) D and 2 for the ternary, one-bit shift, and two-bit shift weight constraints. Each ternary weight needs 2- memory while each of one-bit and two-bit shift weight needs 3-bit memory.
d=o- where D 0, 1
We adopted the STE 38 to train the models such that the forward pass is based on quantized weights Wg.
Tly
Σ (q++1 q.) (sign (w m.) /2,
Wy = q1 +
whereas the backward pass uses the real-valued weights w that are subject to quantization, OC/Ow OC aC/Ow,
7

---

For all cases, the DNN was pre-trained using conventional backpropagation with full-precision weights and activations, which was followed by post-learning using CBP. We used the stochastic gradient descent with momentum to minimize the Lagrangian function L with respect to W and Adam [39 maximize L with respect to A. The initial multiplier-learning rate 7) and Peaar were set to 10-a and 20, respectively. The weight-learning rate Vjw decreased to 10-1 times the initial rate when g reached 20 for all cases except GoogLeNet with the binary-weight constraint (the weight- learning rate decayed when y 200). The hyperparameters used are shown in Appendix A.5 which were found using manual searches.
By asymptotically minimizing the Lagrangian function L, the constraint function cs (W) approaches O. The degree of constraint- per weight was evaluated based on the constraint-failum score (CFS), which is defined as
1 Σ CFS =
Y (wo Q,
(19)
Thw
where new denotes the total number of weights. The CBP algorithm was implemented in Python on workstation (CPU: Intel Xeon Silver 41 10 2.10GHz, GPU: Titan RTX).
It should be noted that used CBP post-training method, so that the random seed effect is involved only when organizing the mini-batches. The accuracy deviation is consequently marginal.
Table I: Top- 1/Top5 accuracy of AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet
Algorithm
Binary
Ternary
One-bit shift
Two-bit shift
Full-precision
AlexNet
BWN 17
56.8%/79 4.%
ADMM 33
57.0%
58.2%//0.669
60.0%
LQ-Nets 266
60.5%
60.0%
TTQ 22 CBP
57.5% 7%
58.0% 60%
58.8% /
60.8%/82.co 62.6%
50.9%//2.8%%
ResNet-18
BWN
17
60.8%/83.0%
TWN
19
61.8%/84.2%
INQ
23
66.0%/87.11
68.1%/88.44
ADMM
33
(4.8%//8..0 .%%
67. 77.00%/87.5%
67.5% 97%
68. 98%8
69.6%/89.22%
QN 30
66.5%//87.8 733%
69.1 11/88.889
69.9%
70.4%//89.
IR-Nets
28
66.5%
LQ-Nets
26
68.0% 08.0% 00%
69.3 69.3%
TTQ 25
66.6%//87.2%
DSQ 29
63.71%
LS 31
66. 16/6.5
CBP
66.6% 0176//87.11 1
69.1
1011%%889.00
88.00%
69.6% 89.3%
69.6% /893%
ResNet-50
BWN
17
63.9%.
TWN
19
65.6%
76.0%/
ADMM
33
68.7%//8..
72.5%/990.7%
73.9% 55.5%
74.0% 166
QN 30
72.8%//91.39
75.2%
75.5%
76.2%6/93.29
CBP
74.4% 992.1%
75.1% /92.5%%
76.0% 76.0% 99..9%
76.0% 76.0% /92.9%
GoogLeNet
BWN
17
59.0%//2.44%
TWN
19
61.2%
71.0%/90.8%
ADMM B3
60.3%//8.29%
63.1%/
65.9%//7.39%
66.3%/87.55%
CBP
64.0% 64.0% 86.0%
66.0% /87.3%
69.8%/89.7% 10.1%
4.1 AlexNet
Aet is a simple convolutional networks which consists of five convolutional layers and three fully-connected layers (2. We used AlexNet with batch normalization (40) as in [7 19 (3). The
initial weight-learning rate 7Jw was set to 10 for the binary- and dernary-weighe constraints and

---

10-4 for the other constraints. The batch size was set to 256. We used a weight decay rate (L2- regularization) of 5 x 10-4. The CBP algorithm exhibited state-of- the-art results as listed in Table. The detailed behaviors of networks with binary- and ternary-weight constraints are addressed in Appendix [B] The behaviors highlight asymptotic increases in the top-1 and top-5 recognition accuracy with asymptotic decrease in CFS.. Consequently, the weight distribution bifurcates asymptotically, fulfilling the constraints imposed on the weights.
4.2 ResNet-18 and ResNet-50
We also evaluated our algorithm on ResNet-18 and ResNer-50 L5 which were pre-trained using conventional backpropagation. For 18, the initial weight-learning rate Iiw was set to 10-3 for all constraint cases. The batch size was 256. For ReNet-50, the initial weight-learning rate 'w was set to 10-3 for binary- and ternary-wiight constraints and 10 for the other cases. The batch size was set to 128. The weight decay rate (L2-regularization) was set to 10 for both ResNet-18 and ResNet-50. The results are summarized in Table IT] highlighting state-of-the-ar performance compared with previous results. Notably, CBP with the one- and two-bit shift weight constraints almost reaches the performance of the full-precision networks. Particularly, for ResNet-50, CBP with the binght-weight constraint significantly outperforms other methods. The detailed behaviors of weight quantizations for ResNet- 18 and ResNet-50 are addressed in Appendix
4.3 GoogL.eNet
GoogLeNet consists of 22 layers organized with the inception modules 411. We evaluated our algorithm on GoogLeNet which was pre-trained using conventional backpropagation. The weight- learning rate 7h was initially set to 10 for all constraint cases. The batch size, Pmnrr and weight decay rate were set to 256, 10 and 10-4, respectively. For the binary-weigh constraint case, the weight-learnngg rate V/ decreased to 10-1 times the initial rate when g reached 200. The results summarized in [able Notably, CBP significantly outperforms the previous results for all constraint cases. The detailed behaviors of weight quantizations for GoogLeNet are addressed in Appendix B
Discussion
To evaluate the effect of the constraint function on training performance, we considered three different cases of post-training DNN using CBP (i) with and (ii) without the unconstrained-weeig window, and (iii) without the constraint function at all, i.c. conventional backpropagation with STE only. Because all DNNs in this work include STE, the comparison between these three cases highlights the effect the constraint function in addition to STE. For all cases, pre-training using conventional backpropagation preceded the three different post-training schemes. Table 2 addresses the comparison, highlighting accuracy and CFS improvement in Case (i) over Case (iii). This indicates that CBP allows the DNN to learn features while the weights are being quantized by the constraint function with the gradually vanishing unconstrained-weiiee window. On the contrary, CBP without the unconstrained- weight window (Case (ii)) rather degraded the accuracy compared with Case (iii), whereas the improvement on CFS was significant. This may be because the constraint function without unconstrained-weitt weight window strongly forced the weights to be quantized without learning the features.
Table 2: Top-1 accuracy of ResNet- 18 trained in various conditions
Post-training algorithm
Accuracy
CFS
CBP with update of g
66.6% 1% 60.2%
1.19 x10-3
CBP without encconstraine-wwiieee window Backpropagation--TT
1.05 10
64.6%
3.58x10 10-2
We used manual searches for the hyperparameters, weight-learning rate vw, multiplier-learning rate 7), multiplier update scheduling variable Pmaz, and unconstrained-weit
window variable
Ag. We used identical parameters f/,, Pmars and Ag for all four models, each with the four distinct constraints, i.c., 12 cases in total. CBP is unlikely susceptible to the hyperparameters for different

---

models. Therefore, the hyperparameters used in this work may serve as the decent initial values for other models.
CBP needs floating-point operations (FLOPs) for the Lagrange multiplier update addition to FLOPs for the weight update, which causes additional computational complexity. Given that Lagrange multiplier is assigned to each weight, the additional complexity scales with the number of weights. The total computational complexity of CBP exceeds the conventional backpropagation by approximately 2% for ResNet-18 and ResNet-50 whereas by approximately 25% for AlexNet. The complexity estimation is elaborated in Appendix
We used CBP as a post-training method. That is, the networks considered were pre-trained using conventional backpropagation. Applying CBP to untrained networks hardly reached the accuracies of classification listed in [ When efficiency in training is the most important concern, CBP may not be the best choice. However, when efficiency in memory usage is of the most important concern CBP may be the optimal choice with regard to its excellent learning capability with maximum 3-bit weight precision, which almost reaches the classification accuracy of the full-precision networks. The use of one-bit or two-bit shift weights can avoid multiplication operations that consume a considerable amount of power, so that can significantly improve computational efficiency. Additionally, CBP is not a method tailored to particular models. Therefore, our work may have a broader impact on various application domains where memory capacity is limited and/or computational efficiency is of significant concern.
6 Conclusion
In this study, we proposed the CBP algorithm that trains DNNs by simultaneously considering both loss and constraint functions. It enables the implementation of any well-defined set of constraints on weights in a common training framework, unlike previous algorithms for weight quantization, which were tailored to particular constraints. The evaluation of CBP on ImageNet with with different constraint functions (binary, ternary, one-bit shift and two-bit shift weight constraints) demonstrated its high capability, highlighting its state-of-the-a art accuracy of classification.
Acknowledgments and Disclosure of Funding
This work was supported by the Ministry of Trade, Industry & Energy (grant no. 20012002) and Korea Semiconductor Research Consortium program for the development of future semiconductor devices and by National R&D Program through the National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (202IMFF3244010755)
References
[1] Y. Taigman, M. Yang. M. Ranzato, and L. Wolf, "Deepface: Closing the gap to human-level performance in face verification," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1701-1708.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25, 2012, pp. 1097- 1105.
[3] G. Hinton, I.. Deng, D. Yu, G. E. Dahl, A.r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
[4] T. N. Sainath, A. Mohamed, B. Kingsbury, and B. Ramabhadran, "Deep convolutional neural networks for LVCSR," in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013, pp. 8614-8618.
[5] G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
10

---

[6] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. pp. 1735-1780, 1997.
[7] K. Lee, D. Yoo, W. Jeong, and S. Han, "SIMPLE-NN: an efficient package for training and executing neural-network interatomic potentials," Computer Physics Communications, vol. 242, pp. 903, 2019.
[8] G. Kim, V. Kornijeuk, D. Kim, I. Kim, C. Hwang, and S. Jeong, "Artificial neural network for response inference of a nonvolatile resistance-switch array," Micromachines, vol. 10, no. 4, p. 219, 2019.
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in Neural Information Processing Systems 27, 2014, pp. 2672-2680.
[10] A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep con- volutional generative adversarial networks," in 4th International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2016.
[11] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, "Unrolled generative adversarial networks," in 5th International Conference on Learning Representations, ICLR, Conference Track Proceed- ings, 2017.
[12] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, "InfoGAN: inter- pretable representation learning by information maximizing generative adversarial nets," in Advances in Neural Information Processing Systems 29, 2016, pp. 2172-2180.
[13] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70. PMLR, 2017, pp. 214-223.
[14] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," " in 3rd International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2015.
[15] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[16] M. Courbariaux, Y. Bengio, and J.-P. David, "BinaryConnect: training deep neural networks with binary weights during propagations," in Advances in Neural Information Processing Systems 28, 2015, pp. 3123-3131.
[17] M. Rastegari, V. Ordonez, Redmon, and A. Farhadi, "XNOR-Net. imagenet classification using binary convolutional neural networks," in European Conference on Computer Vision, 2016, pp. 525-542.
[18] Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, "Neural networks with few multiplica- tions," in 4th International Conference on Learning Representations, ICLR, Conference Track Proceedings. 2016.
[19] Li, B. Zhang, and Liu, "Ternary weight networks," 2016,
[20] Y. Gong, L. Liu, M. Yang, and L. Bourdev, "Compressing deep convolutional networks using vector quantization," 2014, arXiv:1412.6111
[21] N. Mellempudi, A. Kundu, D. Mudigere, D. Das, B. Kaul, and P. Dubey, "Ternary neural networks with fine-grained quantization," " 2017, arXin:1705.01462
[22] D. Soudry, I. Hubara, and R. Meir, "Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights," " in Advances in Neural Information Processing Systems 27, 2014, pp. 963-971.
[23] A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, "Incremental network quantization: Towards lossless cnns with low-precision weights," 5th International Conference on Learning Repre- sentations, ICLR, Conference Track Proceedings, 2017.

---

[24] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, "Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -,," 2016, arXiv11602002830.
[25] C. Zhu, S. Han, H. Mao, and W. J. Dally, "Trained ternary quantization," " arXiv preprint 1rXiv:6612.01064, 2016.
[26] D. Zhang. J. Yang. D. Ye, and G. Hua, "Lq-nets: Learned quantization for highly accurate and compact deep neural networks," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 365-382.
[27] M. Elhoushi, Z. Chen, F. F.aaffq, Y. H. Tian, and J. Y. "Deepshift: Towards multiplication-les neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, June 2021, pp. 2359-2368.
[28] H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song, "Forward and backward information retention for accurate binary neural networks," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition; 2020, pp. 2250-2259.
[29] R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan, "Differentiable soft quantiza- tion: Bridging full cull-precision and low-bit neural networks," in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[30] J. Yang, X. Shen, J. Xing, X. Tian, H. Li, B. Deng. J. Huang, and X.-s. Hua, "Quantization networks," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[31] H. Pouransari, Z. Tu, and O. Tuzel, "Least squares binary quantization of neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[32] A. T. Elthakeb, P. Pilligundla, F. Mireshghallah, T. Elgindi, C.-A. Deledalle, and H. Es- maeilzadeh, "Waveq: Gradient-based deep quantization of neural networks through sinusoidal adaptive regularization," arXiv preprint arXiv:2003.00146, 2020.
[33] C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, "Extremely low bit neural network: Squeeze the last bit out with admm," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
[34] D. Bertsekas and W. Rheinboldt, Constrained Optimization and Lagrange Multiplier Methods, ser. Computer science and applied mathematics. Elsevier Science, 2014.
[35] D. Luenberger and Y. Ye, Linear and Nonlinear Programmiing. Springer, 2015.
[36] C. Platt and A. H. Barr, "Constrained differential optimization." " in Proceedings of the 1987 International Conference on Neural Information Processing Systems, 1987, 612-621.
[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision (IJCV), vol. 1 115, no. 3, pp. 211-252, 2015.
[38] Y. Bengio, N. Léonard, and A. Courville, "Estimating propagating gradients through stochas- tic neurons for conditional computation," arXiv preprint 1308.3330 2013.
[39] D. Kingma and Ba, "Adam: A method for stochastic optimization." in 3rd International Conference Learning Representations, ICLR, Conference Track Proceedings, 2015.
[40] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," " in International conference on machine learning. PMLR, 2015, pp. 448-456.
[41] C. Szegedy, W. Liu, Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
12


Reviews:
[{"review_id": "lLxQmFFuf1p", "reviewer": "Reviewer_J3PT", "summary": "This paper proposes a technique for quantization aware training. The idea is to train in full precision using the back-propagation algorithm but add a constraint to the weights that is supposed to mimic the effects of quantization. Essentially, the constraint is a sawtooth function whose minima coincide with the quantization levels of the low bit-width number representation that is desired. Experimentally, the method is shown to achieve state-of-the-art accuracy on ImageNet using AlexNet, ResNet-18, and ResNet-50.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed idea is interesting, but I have some questions/concerns. \n\n- In section 3.1, the Lagrange multiplier \\lambda is made into a learnable parameter whose gradient is required and over which the optimization is carried over. This is odd as in the usual case the Lagrange multiplier acts as a regularizer and its value is a hyperparameter tuned as part of the training process. There is no explanation as to why differentiating the loss function with respect to the Lagrange multiplier is required. Can the authors provide some explanations and/or insights?\n\n- The idea of using a regularization that makes weights fall into the quantization levels has been explored before, using a sinusoidal regularization function. Such a function has the added benefit of being continuous and differentiable at its minima, which coincide with the quantization levels of the number representation. Can the authors compare their work with this one and provide some comments: https://arxiv.org/pdf/2003.00146.pdf\n\n- What is the cost of constrained back-prop in terms of training difficulty? How does the convergence look? Since such a complicated optimization mechanism is being employed, it is to be suspected that training becomes more difficult. Perhaps the authors can provide some data on training difficulty pooled out from the experimental results.\n\nMinor comment: there are several typos and LaTeX errors such as broken cross-references.\n\n\nPost-Rebuttal Comments:\nI wish to thank the reviewers for their thorough response. After reading both the author feedback, as well as the other reviews, I am keeping my original score and review as is. The method proposed in the paper is interesting, but in my opinion, the paper in its current form has significant room for improvement, particularly in its exposition and in the assessment of the overhead of the proposed method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not Applicable.", "review_text": "", "extra_scores": {"ethical_concerns": "Not Applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1-2 hrs"}, "extra_text": {}}, {"review_id": "U3OErFehH5L", "reviewer": "Reviewer_KcPD", "summary": "The paper describes the use of the Lagrange multiplier method (LMM) to train deep nets with constraints on weight precision incorporated as an intrinsic part of back prop (BP). The proposed method referred to as constrained BP (CBP) allows one to obtain low weight precision networks whose accuracy is close to that of full-precision networks. The paper describes the LMM method and its application to training DNNs. Results on AlexNet, ResNet-18 and ResNet-50 on ImageNet is shown.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very well-written paper. It was enjoyable to read due to its clarity of exposition and simplicity of the concept. The proposed idea viz. to augment the loss function by including weight constraints, is novel and interesting. I was hoping that the proposed method CBP would provide some advantage over existing methods. Unfortunately, this was not the case. The main issues are:\n\n1. It was only at line number 225 and 235 that it became clear that CBP was applied post full-precision standard BP training! This means that CBP is a post-training method but with a high training complexity (compared to both post and in-training quantization methods) since training proceeds in two phases - standard BP followed by CBP. The paper does not mention how many epochs of CBP training was required. Please provide this information. \n\n2. Related 1., why wasn't CBP applied with a cold-start? Does it not converge or does it converge to a poor solution?\n\n3. The accuracy numbers reported match but do not significantly exceed state-of-the-art reported numbers. This is probably too much to expect since weight quantization is a mature topic already. Nevertheless, it would have been nice to see CBP come out ahead in some other respect, e.g., computational complexity. \n\nOverall, though I was rooting for this paper due to the elegance of the method, I am hard-pressed to find a tangible benefit ensuing from it. Perhaps, I am missing something. I am willing to adjust my ratings if the authors or the other reviewers are able to shed some light.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not discussed the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no obvious ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "KZmPT0-bJmA", "reviewer": "Reviewer_kc15", "summary": "This paper presents a method to quantize weights of neural networks by imposing constraints using a pseudo-Lagrange multiplier. The proposed method was used to quantize weights of networks to 1 bit, 2 bits and 3 bits. It's shown that the proposed method can quantize AlexNet, ResNet-18 and ResNet-50 successfully on ImageNet dataset.\n  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are the main pros and cons of the paper.\nPros: \n+ Mathematical formulation for the proposed approach\n+ SOTA results on ImageNet dataset on different networks\n\nCons:\n- Since the paper addressed shift computations, the proposed method could have been compared with shift networks such as DeepShift Network\n- Quantization of activations was not considered in this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "I have no concern.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses a Lagrange multiplier method to train neural networks with constraints on the values of the weights that allows them to be quantized. While some reviewers appreciated the simplicity and effectiveness of this method, they found that the paper didn't properly explain the additional costs of the method, or justify its practical benefits. There also seems to be some questions about whether or not the baseline methods compared to represent the SOTA for weight quantization. The quality of the writing also seems pretty rough in places, based on my own quick scan. While I think this paper may contain a solid contribution to the field, the draft could use some work.
Decision: Accept (Poster)