CBP: backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
Authors: Guhyun Kim
Doo Seok Jeong
Venue: NeurIPS 2021 Poster (2021)
Keywords: backpropagation with constraints
weight quantization
pseudo-Lagrange multiplier method
learning

Abstract: Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints — binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post-training method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For most cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6\%, 74.4\%, and 64.0\% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP is publicly available at \url{https://github.com/dooseokjeong/CBP}.

CBP: Backpropagation with constraint on weight precision using pseudo-Lagrange multiplier method
Guhyun Kim, Doo Seok Jeong"
Division of Materials Science and Engineering Hanyang University, Republic of Korea
gubyunkim01@gmail. com, dooseok Shanyang ac kr
Abstract
Backward propagation of errors (backpropagation) is method minimize objec- tive functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) its objective function. We considered various types of constraints binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post- training method, CBP applied to AlexNet, R ResNet-50, and GoogLeNet ImageNet, which were pre-trained using the conventional backpropagation, For most cases, the proposed algorithm outperforms the state-of-the-arr methods on ImageNet, e.g., 66.6%, 74.4%, and 64.0% top-1 accuracy for ResNet-18, 18, ResNet- 50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP publicly :tttps://gitan h/tp:/gg.tttt.tt/// com wwwoosseoeeoogoooooooooooo
available htttt:::////
Introduction
Currently, deep learning-based methods applied in variety of tasks, including the classification of static data, e.g., image recognition [1] 2]; classification of dynamic data, e.g., speech recogni- tion (3:6: function approximations, which require the output of precise predictions, e.g., electronic structure predictions [7] and nonlinear circuit predictions (8). All of the aforementioned tasks require discriminative models. Additionally, generative models, including generative adversarial networks and variants [30 comprise another type of deep neural network. Despite the diversity in applica- tion domain and model type used, almost all deep learning- bassed methods use backpropagation as common learning algorithm.
Recent developments in deep learning have primarily focused increasing the size and depth of deep neural networks (DNNs) to improve their learning capabilities as in the case of of the-art DNNs like VGGNet (14 and ResNet (15). Given that the memory capacity required by DNN is proportional to the number of parameters (weights and biases), memory usage for DNN becomes severe. Additionally, significant number of multipy--acummulat operations during the
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training and inference stages impose prohibitive workload on hardware. Thus. efficient hardware- resource consumption is critical the optimal performance of deep learning. One way address this requirement is the use of weights of limited precision, such as binary [16 17 and ternary weights [8 19]. To this end. particular constraints are applied to weights during training, and additional algorithms for weight quantization are used in conjunction with backpropagation. This is because such constraints are not considered during the minimization of the objective function (loss function) when backpropagation is executed.
We adopt the Lagrange multiplier method (LMM) to combine basic backpropagation with additional constraint algorithms and produce a single constrained backpropagation (CBP) algorithm. We refer the adopted method as pseudo-LMM because the constraint functions cs (x) are nondifferentiable at Fm (= arg m cs (æ)), rendering LMM inapplicable. Nevertheless, pseudo-LMM successfully attains the optimal point under particular conditions as for LMM. In the CBP algorithm, the optimal weights satisfying a given set of constraints are evaluated via basic backpropagation algorithm. is implemented by simply replacing the conventional objective function (loss function) with a Lagrangian function L that comprises the loss and constraint functions as sub-functions that are subjected to simultaneous minimization. Therefore, this method perfectly compatible with conventional deep learning frameworks. The primary contributions of this study are follows.
We introduce a novel and simple method to incorporate given constraints into backpropa- gation by using a Lagrangian function as the objective function. The proposed method is able to address any set of constraints the weights insomuch the constraint functions are mathematically well-defined.
We introduce pseudo-LMi LMM with constraint functions cs (w) that are nondifferentiable at Wm (= arg min (w)) and analyze the kinetics of pseudo-LMM in the continuous time domain.
We introduce optimal (sawtooth-like) constraint functions with gradually vanishing unconstrained-weigle windows and provide guiding principle for the stable co-optimization of weights and Lagrange multipliers in quasi-static fashion.
We evaluate the performance of CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet (pre-trained using backpropagation with full-precision weights) with four different constraints (binary, ternary, one-bit shift, and two-bit shift weight constraints) on ImageNet as proof-of- -roo--ef-concept examples. The results highlight the classification accuracy outperforming the previous state-of sfate-of-the- the--arttt art results.
2 Related work
The simplest approach to weight quantization the quantization of pre-trained weights. Gong et al. (20] proposed several methods for weight quantization and demonstrated that binarizing weights using a sign function degraded the top-1 accuracy on ImageNet by less than 10%. Mellempudi et al. [21] proposed tine-graine quantization algorithm that calculates the optimal thresholds for the ternarization of pre-trained weights. The expectation backpropagation algorithm [22] implements a variational Bayesian approach to weight quantization. It uses binary weights and activations during the inference stage. Zhou et al. (23) proposed the incremental network guantization method (INQ) that iteratively re- re-trains group of weights to compensate for the performance loss caused by the rest of weights which are quantized using pre-set quantization thresholds.
Several methods of weight quantization utilize auxiliary real-valued weights in conjunction with quantized weights during training. The srreight-through-stie
(STE) comprises the conduction of forward and backpropagation using quantized weights but relies the auxiliary real-valued weights for the update of weights 3.. BinaryConnect [66] utilizes weights binarized by a sign function for forward and backpropagation, and the real-valee real-valued weights updated via backpropagation with binary weights. The hinary-weight-ne the real-valued weights using scaling factor, and it exhibits higher classification accuracy than BinaryConnect on ImageNet. The binarized neural nets 24 and XNOR-Nets [7 are extensions of BinaryConnect and BWN, respectively, which utilize binary activations alongside binary weights.
(BWN) [17] identifies the binary weights closest to
Lin et al. 18 proposed TernaryConnect and Ternary- weinrry-wwight-etwoor (TWN), which are similar to BinaryConnect and BWN but use weiggt-ternarizztion methods instead. Trained ternary quantization

---

(TTQ) 125 also uses ternary weights that are quantized using trainable thresholds for quantization. LQ-Nets proposed by Zhang et al. [26] utilize activation- and weight-quantizers considering the
actual distributions of full-precision activation and weight, respectively. DeepShift Network [27] includes the LinearShift and ConvShift operators that replace the multiplications of fully-connected layers and convolution layers, respectively. Qin et al. (28) introduced IR-Nets that feature the use of error decay estimators to approximate sign functions for weight and activation binarization to differentiable forms. Gong et al. [29] and Yang et al. [30] employed functions similar to step functions but differentiable. Pouransari et al. 3I proposed the least squares quantization method that searches proper scale factors for binary quantization. Elthakeb et al. [32) attempted to quantize weights and activations by applying sinusoidal regularization. The regularization involves two hyperparameters that determine weeight-qanttiznn
and bitwidth-reuularizatie strengths.
To take into account the constraint on weight precision, Leng et al. 3 used an augmented Lagrangian function as an objective function, which includes a constraint-agreement sub-function. The method was successfully applied to various DNNs on ImageNet; yet, the method failed to reach the accuracy level for full- fill-precision models even when 3-bit precision was used.
The CBP algorithm proposed in this study also utilizes LMM; but CBP essentially differs from 33 given that (i) the basic differential multiplier method (BDMM), rather than ADMM, is used to apply various constraints on weight precision, (ii) particularly designed constraint functions with gradually vanishing unconstrained- -weight windows are used, and (iii) substantial improvement on the classification accuracy on ImageNet achieved.
3 Optimization method
3.1 Pseudo-Lagrange multiplier method
LMM calculates the maximum or minimum value of differentiable function f under a differentiable constraint cs = 0. (34) Let us assume that a function f (z, y) attains its minimum (or maximum) value m satisfying a given constraint ( i.e., f (I 3mm = m. Further, cs (Im, 3mm) as the constraint is satisfied at this point. In this case, the point of intersection between the graphs of the two functions, ff (x, y) m and cs (I, y) = 0, is (mm: Vm). Because the two functions have a common tangent at the point of intersection, the following equation holds:
V Vayl =-VV As,,ss
at (z Vm), where X is Lagrange multiplier (35).
The Lagrangian function is defined as L (x. v, = f (I.1) + des (I, (I.y). Consider a lo- cal point (x,z) at which the gradient of L (z. )) is zero. Therefore, V V.y.all yr, A) V.. [f (z. y) + Aes (I. y)1 + VA [des v)l. Thus, Vanal following equations.
(I. (a, = equivalent to the
VE. [f (a, v) + Acs (z. y) 0, Va Dess (z. v)] = 0
V Vayaa (I, =0 +
Vanf (x,y) cs (I, v)
-V esycs (x. v)
= =0
This satisfies the condition in Eq. ( as well as the constraint cs (x, y) = 0. Therefore, the local point (x. y) corresponds to the minimum (or maximum) point of the function f under the constraint.
We define pseudo-LMM to address similar minimization tasks but with continuous constraint func- tions that are nondifferentiable at (Im, 3/m) arg ming,y y cs (I, v). Thus, Eq. ( cannot be satisfied at the optimal point. Nevertheless, pseudo-LMM enables us to minimize the function f (zr) subject to the constraint condition es (ar) = 0 using the Lagrangian function L (æx, A).
Definition 3.1 (Pseudo-LMM). Pseudo-LMM is a method to attain the optimal variables æ,, that minimize function / (a)) subject to the constraint condition es (ax) = 0, where the function es (x) is nondifferentiable at a. but reaches the minimum at Fm, i.e., es (xmm) = 0.
Theorem 3.1. Minimizing the Lagrangian function L (æx, A), which is given by L (æx, A) = f (ar) + Acs (z), is equivalent to minimizing the function ff (x) subject to cs (x) 0
3

---

Proof. The Lagrangian function L (a. A) is always differentiable with respect to the Lagrangian multiplier X, so that the equation cs (x) holds at the optimal point mm Thus, we have
minimize
L( X)
minimize L (a. A) + Æ,
(3)
subject to cs (x) (x)= = 0.
When the constraint satisfied, i.e. cs (x) O, the Lagrangian function C (x; A) equals the function (x), so that the task to minimize C (x, A) with respect to x and À corresponds to the task to
minimize (x) subject cs (x) =
Given Theorem 3.1 Dseudo-LMM can attain the optimal point by minimizing the Lagrangian function L. in spite of the nondifferentiabiiity of the constraint function es (æx) at the optimal point mm- Note that not all functions have zero gradients at their minimum points; for instance, the function [a]] attains its minimum at z = 0 but the gradient at the minimum point is not defined. However, all convex functions have zero gradients their minimum points. Thus, LMM to minimize the function f (x) subject to the constraint convex function cs (x) with minimum point 2m is subset of pseudo-LMM. this case, the constraint function has zero gradient at the minimum point, so that Eq. becomes
Vzl (x) = (x, = + { subject to cs (x) 0.
We will consider continuous constraint functions with a few nondifferentiable points their variable domains, including their minimum points. Other than such nondifferentiable points, we will use the gradient descent method to search for the minimum points within the framework of Pseudo-LMM. Hercafter, when the gradient of the Lagrangian L function is remarked, its variable domain excludes such nondifferentiable points.
The optimal solution to Eq. can be found using the basic differential multiplier method (BDMM) (66] that calculates the point at which (æx, X) attains its minimum value by driving toward the constraint subspace E (cs (T) 0). The BDMM updates x and X according to the following relations.
x サ x - 12 Vx,
(5)
and
A x V
(6)
BDMM is cheaper than Newton's 's method in terms of computational cost. Additionally, Eq. (5) is
identical to the solution used in the gradient descent method during backpropagation, except for the use of Lagrangian function instead of loss function. This indicates the compatibility of pseudo-LMM with the optimization framework based on backpropagation.
3.2 Constrained backpropagation using the pseudo-Lagrange multiplier method
We utilize pseudo-LMM to train DNNs with particular sets of nright-constraints We define Lagrangian function L in the context of feedforward DNN using the following relation.
C y
W. >)
C. (1))) (g)) w) Ccss (W).
(k),
c8 = [0s] (WI) ...,) c8n (Wn.))
A Diy.... An
where (*) and ý (8) denote the actual output vector for the kth input data and correct label, respec- tively. The set W denotes a set weight matrices, including n., weights in aggregate, and the function C' denotes a loss function. Each weight w; (1 < < n.w)) ) is given one constraint function cs; (w:) and one multiplier X;
We chose sawtooth-shapel constraint functions. We quantize the real-valued weights into ng values in {q ye where q, qi+1 for all i. We also employ set of the medians of neighboring
the set Q

---

Constraint function
Constraint function
Binary weight constraint, g=2
Ternary weight constraint, g=2
UCS window
UCS
UCS
1
0
0
-2
-2
Weight
Weight
Constraint function
Constraint function
Binary weight constraint, g=10
Ternary weight constraint, g=10
-2
-1
0
-2
-1
1
Weight
Weight
Figure 1:
Binary- and termary-weight constraint functions for g 2 and 10. Blue-filled regions indicate unconstrained-wweihh windows.
values in the set Q: M [m; where mn (gi (:+1) /2. Using Q and M, we define
partial constraint function yı for i 0, 1 S ng. and ng.
W < q -2 (w 41) otherwise,
yo (w)
0
2 ww - my + gi+1 q
if 4: S w < 9++1
Yi (w)
otherwise,
2 2(w (mg) if w 2 On otherwise,
Ung (w)
(8)
The constraint function es is the summation of the partial constraint functions Y (w) E (w), gated by the unconstrained-weit weight window ues (w) parameterized by variable g.
respectively.
Cs (w: Q, 2,M,9) ucs (w) Y (w),
where
Bg -1
Wcs (w) = Σ İ=U
H
(g++1 9:) - |/ - m. + +el) el) 2g
(10)
where c + 0+, and denotes the Heaviside step function. The function ucs (w) realizes the imconstrained-weigee
window function of g (2 1). When g == 1, the function outputs zero for Q1 S w < In merely confining w to the range @1 I w < 9n, without weight quantization, whereas, when g oo, the window vanishes, allowing the constraint function quantize the weight in the entire weight range. Examples of function ucs (w) are shown in Fig.
The inccostrained-wwiieee window variable is initially set to one and updated such that it keeps increasing during training, i.c., the window gradually vanishes. The window gradually vanishing allows sequential weight quantization such that the further the initial weights from their nearest q. the later their weights are subject to quantization, which is otherwise subject to simultancous (abrupt) quantization. It is likely that the further the initial weights from their nearest gr, the larger the increase in loss function C' when they are quantized. Thus, the sequential guantization from the weights close to their 4; likely avoids abrupt increase in the loss. Further, while the closer weights are being quantized, the further weights (not subject to quantization yet) are being updated to reduce the loss given the partially quantized weights. This effect will be discussed in Sections 3.3
For every training batch, the weights are updated following a method similar to conventional back- propagation. Nevertheless, the use of the Lagrangian function in Eq. 8 rather than loss function

---

only, as an objective function constitutes a critical difference. The Lagrange multipliers À are sub- sequently updated using the gradient ascent method in Eq. 6 Updating cross-coupled variables. such as W and X, often experiences difficulties in convergence toward the optimal values hecause of oscillation around the optimal values. feasible solution involves quasi-static update. To this end, we significantly reduce the update frequency of the Lagrange multipliers À compared with weights W.
Weight update: Weights W are updated once every iteration as for the conventional backpropagation but using the Lagrangian function L.
Lagrange multipliers À are conditionally updated once every training epoch. The update is allowed if the summation of all L in given epoch (C is not smaller than f.uu for the previous epoch (Cemm) or the multipliers À have not been updated in the past Pmaz epochs. This achieves the convergence of W for given À in quasi-static manner.
Lagrange multiplier update:
Unconstrained weight window update: Unconstrained-weighh window variable g is updated on the same condition as for the Lagrange multipliers X. Unlike weights W and multipliers X, the variable g (initialized to one) constantly increases when updated such that Ag = 1 when g < 10, Ag 10 for 10 S < 100, and Ag 100 otherwise.
The detailed learning algorithm is shown in the pseudocode in Appendix A.2
3.3 Learning kinetics
Learning with BDMM using the Lagrangian function is better understood in the continuous time domain. We first address the kinetics of learning without the unconstrained weight window ues (w). The change in the Lagrangian function L at given learning step in the discrete time domain is equivalent to the derivative of with time at time in the continuous time domain, which is given by
dC
New
DC
Ocs: Dwi
2
TW
Ow;
+
T
CS
(11)
The Lagrange multiplier X; time is given by
X; (1) - No (0) S. cs.dt.
(12)
The constraint functions cs; approach zero as the weights approach their corresponding quantized values gi, and thus the Lagrange multipliers in Eq. (12) asymptotically converge to their limits.
Eqs. 11 and 12 are derived Appendix A.I
At equilibrium, the Lagrange function is no longer time-dependent, i.e., dC/dt 0. This requires the Lagrange multipliers reaching their limits, which turn requires the weights reaching their corresponding quantized values W leading to 0. For convenience, we define the integration of cs; in Eq. (12) as Acs; (2 0).
Acs, L' cs;dt, if cs; 0 as co.
(13)
Thus, the equilibrium Lagrange multiplier Aie0 can be expressed as X,ce X, (0) + T Aes.
(14)
Therefore, is evident from Eq.
TTT that the equilibrium leads to
ac (A: (0) + - c88
Ocs: Ow:
Vi,
(15)
Ow;
We consider sawtooth constraint functions with slopes Ls, i.e., Ocs;/Ow; =s. where > 0.
Eq. (8) is the case 2. Generally, the Lagrange multiplier is initialized to zero, i.e., X; (0) = 0. Therefore, the gradient of loss function C at the equilibrium point a is given by
ac
Vi, Ow:
= t sAcs; =\
(16)

---

Consider that the loss function C' has the equilibrium point W (= arg min w C). Eq. (16) eluci- dates the increase of loss by attaining the equilibrium point W In this regard, A corresponds to the cost of weight quantization. Assuming the convexity of the loss function C' in domain D including W and W* C (W *c*) keeps increasing as A increases. If the initial pre-trained weights equal their corresponding quantized weights, i.e., W w then Acs; O, and thus X = 0 according to Eq. (14) Eq. (T)) consequently yields ac = O, indicating zero cost of quantization.
Considering the gradually vanishing mncontrraine-wweiht window ues (w) yields the derivative of L with time time L. in the continuous time domain follows.
dC
OC
0Y; Ajucsi
(ucs:Y.)
(17)
TW
Ows
Ows
The derivation of Eq. (77) weight window Dues from the others at a given time i, Eq. (17) can be written by
is given in Appendix A.33 Distinguishing the weights in the unconstrained-
dC = TW W Σ dt
aC
2
ac Ow.
+ Ai OY, Ows
)
Σ
TW
xx Y?
(18)
Owi
we E Draa
The latter term the right-hand side of Eq. (18) indicates that the weights outside the window Dues are being quantized at the cost of increase of loss. However, as indicated by the former term, the weights in the window Ducs are being optimized only to decrease the loss function with partially quantized weights. Compare this gradual quantization with abrupt quantization without the gradually vanishing unconstrained-we weight window, where all weights quantization. The gradual quantization allows the weights in the window to further reduce the loss function regarding the weights that have already been quantized eventual cost of quantization is likely smaller than the simultaneous quantization case.
subject to simultaneous (abrupt)
being quantized, and thus the
4 Experiments
To evaluate the performance of our algorithm, we trained three models (AlexNet, and ResNet- 18 and 50) on the ImageNet dataset [37 with four different weight constraints (binary. ternary, and one-bit, and two-bit shift weight constraints). ImageNet consists of approximately 1.2 million training images and 50 thousands validation images, All training images were pre-processed such that they were randomly cropped and resized to 224 x 224 with mean subtraction and variance division. Additionally, random horizontal flipping and color jittering were applied. For validation, the images were resized to 256 x 256 and their centers in 224 x 224 were cropped. We evaluated the top-1 and top-5 classification accuracies the validation set.
We considered binary, ternary, one-bit shift and two-bits shift weight constraints validate the CBP algorithm as a general weight-quantizatioon framework. For all cases, we introduced layer-wise scaling factors such that (1) (for the (th layer) is given by a w II1 (n) where w and (!()) denote the weight matrix of the /th layer and the number of elements of (!)), respectively. As for [17 and [19], the weight matrices of the first and last layers were not quantized. The quantized weights employed for each constraint case is elaborated as follows.
weight constraint: A set of quantized weights Q is {-a,a}, a).
The other weight constraints:
A set of quantized weights Q is (0. +2 +2-44) D and 2 for the ternary, one-bit shift, and two-bit shift weight constraints. Each ternary weight needs 2- memory while each of one-bit and two-bit shift weight needs 3-bit memory.
d=o- where D 0, 1
We adopted the STE 38 to train the models such that the forward pass is based on quantized weights Wg.
Fly
Σ (q++1 q.) (sign (w m.) /2,
Wy = q1 +
whereas the backward pass uses the real-valued weights w that are subject to quantization, OC/Ow OC aC/Ow,
7

---

For all cases, the DNN was pre-trained using conventional backpropagation with full-precision weights and activations, which was followed by post-learning using CBP. We used the stochastic gradient descent with momentum to minimize the Lagrangian function L with respect to W and Adam [39 maximize L with respect to A. The initial multiplier-learning rate 7) and Peaar were set to 10-a and 20, respectively. The weight-learning rate Vjw decreased to 10-1 times the initial rate when g reached 20 for all cases except GoogLeNet with the binary-weight constraint (the weight- learning rate decayed when y 200). The hyperparameters used are shown in Appendix A.5 which were found using manual searches.
By asymptotically minimizing the Lagrangian function L, the constraint function cs (W) approaches O. The degree of constraint- per weight was evaluated based on the constraint-failum score (CFS), which is defined as
1 Σ CFS =
Y (wo Q,
(19)
Thw
where new denotes the total number of weights. The CBP algorithm was implemented in Python on workstation (CPU: Intel Xeon Silver 41 10 2.10GHz, GPU: Titan RTX).
It should be noted that used CBP post-training method, so that the random seed effect is involved only when organizing the mini-batches. The accuracy deviation is consequently marginal.
Table I: Top- 1/Top5 accuracy of AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet
Algorithm
Binary
Ternary
One-bit shift
Two-bit shift
Full-precision
AlexNet
BWN 17
56.8%/79 4.%
ADMM 33
57.0%
58.2%//0.669
60.0%
LQ-Nets 266
60.5%
60.0%
TTQ 22 CBP
57.5% 7%
58.0% 60%
58.8% /
60.8%/82.co 62.6%
50.9%//2.8%%
ResNet-18
BWN
17
60.8%/83.0%
TWN
19
61.8%/84.2%
INQ
23
66.0%/87.11
68.1%/88.44
ADMM
33
(4.8%//8..4 .%%
67. 77.00%/87.5%
67.5% 97%
68. 98%8
69.6%/89.22%
QN 30
66.5%//87.8 733%
69.1 11/88.889
69.9%
70.4%//89.
IR-Nets
28
66.5%
LQ-Nets
26
68.0% 08.0% 00%
69.3 69.3%
TTQ 25
66.6%//87.2%
DSQ 29
63.71%
LS 31
66. 16/6.5
CBP
66.6% 0176//87.11 1
69.1
1011%%889.00
88.00%
69.6% 89.3%
69.6% /893%
ResNet-50
BWN
17
63.9%.
TWN
19
65.6%
76.0%/
ADMM
33
68.7%//8..
72.5%/990.7%
73.9% 55.5%
74.0% 166
QN 30
72.8%//91.39
75.2%
75.5%
76.2%6/93.29
CBP
74.4% 992.1%
75.1% /92.5%%
76.0% 76.0% 99..9%
76.0% 76.0% /92.9%
GoogLeNet
BWN
17
59.0%//2.44%
TWN
19
61.2%
71.0%/90.8%
ADMM B3
60.3%//8.29%
63.1%/
65.9%//7.39%
66.3%/87.55%
CBP
64.0% 64.0% 86.0%
66.0% /87.3%
69.8%/89.7% 10.1%
4.1 AlexNet
Aet is a simple convolutional networks which consists of five convolutional layers and three fully-connected layers (2. We used AlexNet with batch normalization (40) as in [7 19 (3). The
initial weight-learning rate 7Jw was set to 10 for the binary- and dernary-weighe constraints and

---

10-4 for the other constraints. The batch size was set to 256. We used a weight decay rate (L2- regularization) of 5 x 10-4. The CBP algorithm exhibited state-of- the-art results as listed in Table. The detailed behaviors of networks with binary- and ternary-weight constraints are addressed in Appendix [B] The behaviors highlight asymptotic increases in the top-1 and top-5 recognition accuracy with asymptotic decrease in CFS.. Consequently, the weight distribution bifurcates asymptotically, fulfilling the constraints imposed on the weights.
4.2 ResNet-18 and ResNet-50
We also evaluated our algorithm on ResNet-18 and ResNer-50 L5 which were pre-trained using conventional backpropagation. For 18, the initial weight-learning rate Iiw was set to 10-3 for all constraint cases. The batch size was 256. For ReNet-50, the initial weight-learning rate 'w was set to 10-3 for binary- and ternary-weight constraints and 10 for the other cases. The batch size was set to 128. The weight decay rate (L2-regularization) was set to 10 for both ResNet-18 and ResNet-50. The results are summarized in Table IT] highlighting state-of-the-ar performance compared with previous results. Notably, CBP with the one- and two-bit shift weight constraints almost reaches the performance of the full-precision networks. Particularly, for ResNet-50, CBP with the binght-weight constraint significantly outperforms other methods. The detailed behaviors of weight quantizations for ResNet- 18 and ResNet-50 are addressed in Appendix
4.3 GoogL.eNet
GoogLeNet consists of 22 layers organized with the inception modules 411. We evaluated our algorithm on GoogLeNet which was pre-trained using conventional backpropagation. The weight- learning rate 7h was initially set to 10 for all constraint cases. The batch size, Pmnrr and weight decay rate were set to 256, 10 and 10-4, respectively. For the binary-weigh constraint case, the weight-learnngg rate V/ decreased to 10-1 times the initial rate when g reached 200. The results summarized in [able Notably, CBP significantly outperforms the previous results for all constraint cases. The detailed behaviors of weight quantizations for GoogLeNet are addressed in Appendix B
Discussion
To evaluate the effect of the constraint function on training performance, we considered three different cases of post-training DNN using CBP (i) with and (ii) without the unconstrained-weeig window, and (iii) without the constraint function at all, i.c. conventional backpropagation with STE only. Because all DNNs in this work include STE, the comparison between these three cases highlights the effect the constraint function in addition to STE. For all cases, pre-training using conventional backpropagation preceded the three different post-training schemes. Table 2 addresses the comparison, highlighting accuracy and CFS improvement in Case (i) over Case (iii). This indicates that CBP allows the DNN to learn features while the weights are being quantized by the constraint function with the gradually vanishing unconstrained-weiiee window. On the contrary, CBP without the unconstrained- weight window (Case (ii)) rather degraded the accuracy compared with Case (iii), whereas the improvement on CFS was significant. This may be because the constraint function without unconstrained-weitt weight window strongly forced the weights to be quantized without learning the features.
Table 2: Top-1 accuracy of ResNet- 18 trained in various conditions
Post-training algorithm
Accuracy
CFS
CBP with update of g
66.6% 1% 60.2%
1.19 x10-3
CBP without encconstraine-wwiieee window Backpropagation--TT
1.05 10
64.6%
3.58x10 10-2
We used manual searches for the hyperparameters, weight-learning rate vw, multiplier-learning rate 7), multiplier update scheduling variable Pmaz, and unconstrained-weit
window variable
Ag. We used identical parameters f/,, Pmars and Ag for all four models, each with the four distinct constraints, i.c., 12 cases in total. CBP is unlikely susceptible to the hyperparameters for different

---

models. Therefore, the hyperparameters used in this work may serve as the decent initial values for other models.
CBP needs floating-point operations (FLOPs) for the Lagrange multiplier update addition to FLOPs for the weight update, which causes additional computational complexity. Given that Lagrange multiplier is assigned to each weight, the additional complexity scales with the number of weights. The total computational complexity of CBP exceeds the conventional backpropagation by approximately 2% for ResNet-18 and ResNet-50 whereas by approximately 25% for AlexNet. The complexity estimation is elaborated in Appendix
We used CBP as a post-training method. That is, the networks considered were pre-trained using conventional backpropagation. Applying CBP to untrained networks hardly reached the accuracies of classification listed in [ When efficiency in training is the most important concern, CBP may not be the best choice. However, when efficiency in memory usage is of the most important concern CBP may be the optimal choice with regard to its excellent learning capability with maximum 3-bit weight precision, which almost reaches the classification accuracy of the full-precision networks. The use of one-bit or two-bit shift weights can avoid multiplication operations that consume a considerable amount of power, so that can significantly improve computational efficiency. Additionally, CBP is not a method tailored to particular models. Therefore, our work may have a broader impact on various application domains where memory capacity is limited and/or computational efficiency is of significant concern.
6 Conclusion
In this study, we proposed the CBP algorithm that trains DNNs by simultaneously considering both loss and constraint functions. It enables the implementation of any well-defined set of constraints on weights in a common training framework, unlike previous algorithms for weight quantization, which were tailored to particular constraints. The evaluation of CBP on ImageNet with with different constraint functions (binary, ternary, one-bit shift and two-bit shift weight constraints) demonstrated its high capability, highlighting its state-of-the-a art accuracy of classification.
Acknowledgments and Disclosure of Funding
This work was supported by the Ministry of Trade, Industry & Energy (grant no. 20012002) and Korea Semiconductor Research Consortium program for the development of future semiconductor devices and by National R&D Program through the National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (202IMFF3244010755)
References
[1] Y. Taigman, M. Yang. M. Ranzato, and L. Wolf, "Deepface: Closing the gap to human-level performance in face verification," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1701-1708.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25, 2012, pp. 1097- 1105.
[3] G. Hinton, I.. Deng, D. Yu, G. E. Dahl, A.r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
[4] T. N. Sainath, A. Mohamed, B. Kingsbury, and B. Ramabhadran, "Deep convolutional neural networks for LVCSR," in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013, pp. 8614-8618.
[5] G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
10

---

[6] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. pp. 1735-1780, 1997.
[7] K. Lee, D. Yoo, W. Jeong, and S. Han, "SIMPLE-NN: an efficient package for training and executing neural-network interatomic potentials," Computer Physics Communications, vol. 242, pp. 903, 2019.
[8] G. Kim, V. Kornijeuk, D. Kim, I. Kim, C. Hwang, and S. Jeong, "Artificial neural network for response inference of a nonvolatile resistance-switch array," Micromachines, vol. 10, no. 4, p. 219, 2019.
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in Neural Information Processing Systems 27, 2014, pp. 2672-2680.
[10] A. Radford, L. Metz, and S. Chintala, "Unsupervised representation learning with deep con- volutional generative adversarial networks," in 4th International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2016.
[11] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, "Unrolled generative adversarial networks," in 5th International Conference on Learning Representations, ICLR, Conference Track Proceed- ings, 2017.
[12] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, "InfoGAN: inter- pretable representation learning by information maximizing generative adversarial nets," in Advances in Neural Information Processing Systems 29, 2016, pp. 2172-2180.
[13] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70. PMLR, 2017, pp. 214-223.
[14] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," " in 3rd International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2015.
[15] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[16] M. Courbariaux, Y. Bengio, and J.-P. David, "BinaryConnect: training deep neural networks with binary weights during propagations," in Advances in Neural Information Processing Systems 28, 2015, pp. 3123-3131.
[17] M. Rastegari, V. Ordonez, Redmon, and A. Farhadi, "XNOR-Net. imagenet classification using binary convolutional neural networks," in European Conference on Computer Vision, 2016, pp. 525-542.
[18] Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, "Neural networks with few multiplica- tions," in 4th International Conference on Learning Representations, ICLR, Conference Track Proceedings. 2016.
[19] Li, B. Zhang, and Liu, "Ternary weight networks," 2016,
[20] Y. Gong, L. Liu, M. Yang, and L. Bourdev, "Compressing deep convolutional networks using vector quantization," 2014, arXiv:1412.6111
[21] N. Mellempudi, A. Kundu, D. Mudigere, D. Das, B. Kaul, and P. Dubey, "Ternary neural networks with fine-grained quantization," " 2017, arXin:1705.01462
[22] D. Soudry, I. Hubara, and R. Meir, "Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights," " in Advances in Neural Information Processing Systems 27, 2014, pp. 963-971.
[23] A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, "Incremental network quantization: Towards lossless cnns with low-precision weights," 5th International Conference on Learning Repre- sentations, ICLR, Conference Track Proceedings, 2017.

---

[24] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, "Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -,," 2016, arXiv11602002830.
[25] C. Zhu, S. Han, H. Mao, and W. J. Dally, "Trained ternary quantization," " arXiv preprint 1rXiv:6612.01064, 2016.
[26] D. Zhang. J. Yang. D. Ye, and G. Hua, "Lq-nets: Learned quantization for highly accurate and compact deep neural networks," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 365-382.
[27] M. Elhoushi, Z. Chen, F. F.aaffq, Y. H. Tian, and J. Y. "Deepshift: Towards multiplication-les neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, June 2021, pp. 2359-2368.
[28] H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song, "Forward and backward information retention for accurate binary neural networks," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition; 2020, pp. 2250-2259.
[29] R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan, "Differentiable soft quantiza- tion: Bridging full cull-precision and low-bit neural networks," in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[30] J. Yang, X. Shen, J. Xing, X. Tian, H. Li, B. Deng. J. Huang, and X.-s. Hua, "Quantization networks," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[31] H. Pouransari, Z. Tu, and O. Tuzel, "Least squares binary quantization of neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[32] A. T. Elthakeb, P. Pilligundla, F. Mireshghallah, T. Elgindi, C.-A. Deledalle, and H. Es- maeilzadeh, "Waveq: Gradient-based deep quantization of neural networks through sinusoidal adaptive regularization," arXiv preprint arXiv:2003.00146, 2020.
[33] C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, "Extremely low bit neural network: Squeeze the last bit out with admm," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
[34] D. Bertsekas and W. Rheinboldt, Constrained Optimization and Lagrange Multiplier Methods, ser. Computer science and applied mathematics. Elsevier Science, 2014.
[35] D. Luenberger and Y. Ye, Linear and Nonlinear Programmiing. Springer, 2015.
[36] C. Platt and A. H. Barr, "Constrained differential optimization." " in Proceedings of the 1987 International Conference on Neural Information Processing Systems, 1987, 612-621.
[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision (IJCV), vol. 1 115, no. 3, pp. 211-252, 2015.
[38] Y. Bengio, N. Léonard, and A. Courville, "Estimating propagating gradients through stochas- tic neurons for conditional computation," arXiv preprint 1308.3330 2013.
[39] D. Kingma and Ba, "Adam: A method for stochastic optimization." in 3rd International Conference Learning Representations, ICLR, Conference Track Proceedings, 2015.
[40] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," " in International conference on machine learning. PMLR, 2015, pp. 448-456.
[41] C. Szegedy, W. Liu, Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
12


Reviews:
[{"review_id": "kGO5NCOnaxo", "reviewer": "Reviewer_WFRz", "summary": "The paper introduces a simple and effective way of applying constraints on weight parameters during training. The method is basically an application of pseudo-Lagrange multiplier method to the deep neural networks weight constraint problem. Heuristic methods such as Lagrange multiplier update scheduling and the concept of unconstrained weight window are proposed together to achieve better accuracy. The proposed method achieved the state-of-the-art accuracy on ImageNet classification tasks with multiple network architectures.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The proposed method is simple and effective. \n2. The proposed method is theoretically grounded.\n3. Experiments were performed thoroughly with ablation studies.\n\n**Weakness**\n\nThe proposed method requires additional training process after pre-training using conventional backprop. However, as authors mentioned in the paper, it may not be considered as a significant overhead if the efficiency in the inference stage is more important than the efficiency in the training stage.\n\n**Questions**\n1. What exactly does the \"Initialization;\" in Algorithm 1 refer to?\n2. Based on my understanding, the weight parameters would be almost converged after the pre-training process is finished and the introduction of the constraints will penalize if the weights moves far away from their nearest quantization points. In such situation, how could the weights are updated significantly so that the different quantization points are assigned in inference stage?\n3. Do you expect this method will also be effective on networks that use binary activation function?\n\n**Additional Comments**\n\nIt seems there are more previous works that also suggests novel methods for network training weight constraints. Please check following works and add them to the performance comparison table if they are relevant. \n[1] Gong, Ruihao, et al. \"Differentiable soft quantization: Bridging full-precision and low-bit neural networks.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n[2] Yang, Jiwei, et al. \"Quantization networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper properly included the expected limitation, which is the retraining overhead.\n\nAlthough the societal impact was not mentioned, I could not come up with direct societal impact of this work either. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "c-7P1ZO_mMZ", "reviewer": "Reviewer_ECoB", "summary": "They propose a pseudo-Lagrange multiplier-based learning algorithm to quantize weights. Their contributions include:\n\n1. They propose a pseudo-LMM framework that is capable of addressing constraints on the weights.\n\n2. They introduce sawtooth-like constraint functions with gradually vanishing unconstrained-weight windows, which eases the difficulties of co-optimization.\n\n3. They propose a guiding principle to optimize weights and Lagrange multipliers, and their results outperform the previous state-of-the-art results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a very interesting method to improve the standard post-training weight quantization by solving a constraint optimization problem with a pseudo-Lagrange multiplier-based learning algorithm. Compared to the previous work [1], I think this work has great originality from many perspectives. Also, the paper is mostly well-written with minor typos. The proposed sawtooth-like constraint functions with gradually vanishing unconstrained-weight windows (controlled by g) are useful for the mentioned constraint optimization problem. Although I am concerned about the complexity of the proposed learning algorithm, I believe overall this is a good submission above the acceptance threshold.\n\nMy questions and suggestions are listed below:\n\n1. The authors summarize the learning kinetics with BDMM as a major contribution but they merely mention it once in the main text. It's better to at least discuss and conclude the results of the learning kinetics in the main text as well. \n\n2. In line 141, the authors mention the algorithm excludes those nondifferentiable points in the function. How are they excluded? or those points are not occurred during training most likely?\n\n3. Why use SGD and Adam to optimize W and \\lamda separately? What if using the same optimizer? \n\n4. I am concerned about the cost of parameter selections including learning rate, learning rate schedule, and stopping criteria for p and g. It's good to explain those costs in the appendix. \n\n5. The experiment results look great overall. How about the computational complexity? Are these algorithms compared based on the same number of training epochs? I think more training details are needed.\n\nTypos:\n1. In line 180, missing reference.\n\n\n\n[1] C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, “Extremely low bit neural network: Squeeze the last bit out with admm,” in Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol. 32, no. 1, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the authors do not mention their societal impact in the paper, I do think they have positive impacts by producing high accuracy and low-precision models. I think the limitations include:\n\n1. As mentioned above, the cost of parameter selections is unknown, and probably those parameters are sensitive to different learning tasks.\n\n2. Although it's good to see its improvement for the post-training quantization problems, I think a broader impact would be using the proposed method on low-precision training problems where the precision of weights is limited. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "8SPMOuNjVAY", "reviewer": "Reviewer_BfHB", "summary": "The paper proposed a new approach to learn the quantized weight by adding a constraint on backpropagation using the Lagrange multiplier method. To solve the nondifferentiable issue of the constraint function, the pseudo-LLM approach is proposed. Moreover, for stable co-optimization weights and Lagrange multipliers, the optimal (sawtooth-like) constraint functions are proposed. The new approach is compared with the state-of-the-art approaches and shows a slight improvement in accuracy on the average of benchmarks. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1- The idea behind the constraint (weight precision) backpropagation (CBP) with pseudo-LLM is novel and interesting. \n\n2- The paper is well organized. The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n\n3- Most of the previous work are adequately cited and discussed\n\nWeakness:\n\n1- The approach is evaluated in limited models (2 models). It would be suggested the approach is tested with various models such as MobileNet V2, EfficientNet. \n\n2- The proposed approach did not compare with [1]. It would be suggested the comparison with this work discussed in the paper.\n\n3- The result section is not complete. The author does not explain the reason why the CBP approach is shown to perform better compared to other approaches. The discussion for figure 1 and all figures in Appendix B is not complete.\n\n4- The overhead compared to the previous approach and the limitation of the CBP algorithm are not discussed in the paper.\n  \nReferences:\n\n[1] Pouransari, Hadi, Zhucheng Tu, and Oncel Tuzel. \"Least squares binary quantization of neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\nPost Rebuttal =====\nThanks for the additional results in terms of GoogleNet and overhead complexity. I think the response resolved most of my comments on the clarity and quality part and I raised my score from 5 to 6. Still realizing a way to show the reasoning behind the advantage of CBP over others' approaches is highly recommended. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe limitations and potential negative societal impact of their work are not addressed by the author. For instance, the author can discuss the order of practical applicability there approach [1]. \n\n[1] Nagel, Markus, et al. \"Data-free quantization through weight equalization and bias correction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\nPost Rebuttal =====\nClarification: I suggested looking at the Levels of quantization solutions section in [1] and discuss in the paper, CBP approach is categorized in the which level. You do not need to compare your approach with [1].   ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "51pl1vAj97I", "reviewer": "Reviewer_nXqG", "summary": "This paper proposes the constrained back-propagation algorithm based on a pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. Experimental results suggest promising performance.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel method that can incorporate given constraints into back-propagation with a Lagrangian function as the objective function. The writing quality is good and well-organised for readers to follow. Related works are covered and discussed. The proposed method is introduced with clear algorithm pseudocode and questions to support. Experiments are conducted on a variety of datasets with many well-known constraints to demonstrate performance. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper discusses limitation but no negative impact. \nQuestions:\n1. Can we also see its performance on more datasets, other than image related?\n2. Can it be applied on to other network architecture (for other types of tasks other than image classification)? Such as network with recurrent units.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally felt that the method proposed in this paper was interesting, well-presented, and applicable. Multiple reviewers raised the question of computational complexity, but the authors addressed this adequately in their response (and will address it in a revision). Overall, this seems like a legitimate contribution.
Decision: Accept (Poster)