CBP: backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method

Abstract: Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints — binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post-training method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For most cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6\%, 74.4\%, and 64.0\% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP is publicly available at \url{https://github.com/dooseokjeong/CBP}.


Reviews and author responses:

 I appreciate the authors' responses. I agree with the authors that space complexity of weights is important especially when considering the transmission cost of data. However, the space complexity of acitvations is also important during the inference computations on devices with limited on-chip memory since activations account for most of memory accesses to the main memory which dominates the power consumption of the system. Moreover, computations on quantized activations are more efficient in terms of addition cost than the full-precision one. After carefully reading the other reviewers' comments and the authors' responses, I have decided to keep my original rating unchanged.

This paper proposes a technique for quantization aware training. The idea is to train in full precision using the back-propagation algorithm but add a constraint to the weights that is supposed to mimic the effects of quantization. Essentially, the constraint is a sawtooth function whose minima coincide with the quantization levels of the low bit-width number representation that is desired. Experimentally, the method is shown to achieve state-of-the-art accuracy on ImageNet using AlexNet, ResNet-18, and ResNet-50.  The proposed idea is interesting, but I have some questions/concerns. 

- In section 3.1, the Lagrange multiplier \lambda is made into a learnable parameter whose gradient is required and over which the optimization is carried over. This is odd as in the usual case the Lagrange multiplier acts as a regularizer and its value is a hyperparameter tuned as part of the training process. There is no explanation as to why differentiating the loss function with respect to the Lagrange multiplier is required. Can the authors provide some explanations and/or insights?

- The idea of using a regularization that makes weights fall into the quantization levels has been explored before, using a sinusoidal regularization function. Such a function has the added benefit of being continuous and differentiable at its minima, which coincide with the quantization levels of the number representation. Can the authors compare their work with this one and provide some comments: https://arxiv.org/pdf/2003.00146.pdf

- What is the cost of constrained back-prop in terms of training difficulty? How does the convergence look? Since such a complicated optimization mechanism is being employed, it is to be suspected that training becomes more difficult. Perhaps the authors can provide some data on training difficulty pooled out from the experimental results.

Minor comment: there are several typos and LaTeX errors such as broken cross-references.


Post-Rebuttal Comments:
I wish to thank the reviewers for their thorough response. After reading both the author feedback, as well as the other reviews, I am keeping my original score and review as is. The method proposed in the paper is interesting, but in my opinion, the paper in its current form has significant room for improvement, particularly in its exposition and in the assessment of the overhead of the proposed method. Not Applicable.

 We thank Reviewer for the valuable comments. They help us improve the quality of our work.

1 In section 3.1, the Lagrange multiplier $\lambda$ is made into a learnable parameter whose gradient is required and over which the optimization is carried over. This is odd as in the usual case the Lagrange multiplier acts as a regularizer and its value is a hyperparameter tuned as part of the training process. There is no explanation as to why differentiating the loss function with respect to the Lagrange multiplier is required. Can the authors provide some explanations and/or insights?

Answer: In fact, the whole section 3.1 is dedicated to the explanation of general LMM and our pseudo-LMM. The LMM does not differentiate a loss function (which is not a function of the Lagrange multiplier) with the Lagrange multiplier; instead, the Lagrangian function is differentiated with respect to the Lagrange multiplier.
Generally, the Lagrangian multiplier method (LMM) applies to attain a common tangent at the point of intersection of two functions, $f(x,y)=0$ and $cs(x,y)=0$. For simplicity, we consider two-variable functions. To this end, the LMM uses a Lagrangian function $\mathcal{L}(x,y,\lambda)=f(x,y)+\lambda\cdot cs(x,y)$. The gradient of $\mathcal{L}$ with respect to $x$, $y$, and $\lambda$ is given by

$\nabla_{x, y, \lambda}\mathcal{L} = \left[\left(\dfrac{\partial f}{\partial x}+\lambda\dfrac{\partial cs}{\partial x}\right),\left(\dfrac{\partial f}{\partial y}+\lambda\dfrac{\partial cs}{\partial y}\right) , cs\right]^T$.

Equating this equation to zero, i.e., $\nabla_{x,\lambda}\mathcal{L} = \left[0, 0, 0\right]^T$, give us the following three equations.

$\dfrac{\partial f}{\partial x} = -\lambda\dfrac{\partial cs}{\partial x}$, $\dfrac{\partial f}{\partial y} = -\lambda\dfrac{\partial cs}{\partial y}$, and $cs=0$. This implies that $(x^*, y^*, \lambda^*)$ satisfying $\nabla_{x,y,\lambda}\mathcal{L}(x^*, y^*, \lambda^*) = \left[0, 0, 0\right]^T$ leads to 

$\dfrac{\partial f}{\partial x}(x^*, y^*) = -\lambda^* \dfrac{\partial cs}{\partial x}(x^*, y^*)$, 

$\dfrac{\partial f}{\partial y}(x^*, y^*) = -\lambda^* \dfrac{\partial cs}{\partial y}(x^*, y^*)$, and 

$cs(x^*, y^*)=0$. 

Putting the first two equations together yields

$\nabla_{x,y}f(x^*,y^*)=-\lambda^* \nabla_{x,y}cs(x^*,y^*)$. 

This means that the two functions $f$ and $g$ have a common tangent line at $(x^*,y^*)$ unless $\lambda^*=0$. Therefore, we should find $(x^*,y^*,\lambda^*)$ to find the common tagent line. Note that, clearly, the Lagrange multiplier $\lambda$ here is a variable not a constant. 

In our weight quantization, cs at $(x^*,y^*)$ is nondifferentiable; therefore, we propose the pseudo-LMM in which minimizing $\mathcal{L}$ spontaneously attains minimal $f$ with $cs=0$. 

If the Lagrange multiplier is fixed, this is no longer LMM because the key to LMM is to find the optimal $\lambda$. In this case, the constraint term merely serves as a regularizer as Reviewer mentioned.

Please see the Appendix, where we explain the detailed dynamics of the pseudo-LMM in a continuous time domain. 

2 The idea of using a regularization that makes weights fall into the quantization levels has been explored before, using a sinusoidal regularization function. Such a function has the added benefit of being continuous and differentiable at its minima, which coincide with the quantization levels of the number representation. Can the authors compare their work with this one and provide some comments: https://arxiv.org/pdf/2003.00146.pdf

Answer: WaveQ is the one using a regularizer applying quantization penalties. The terms $\lambda_w$ (weight quantization regularization strength) and $\lambda_{\beta}$ (bitwidth regularization strength) are not Lagrange multipliers but hyper-parameters. We will briefly explain the difference between the LMM and additive regularizers in the revised manuscript to avoid any confusion.    

3 What is the cost of constrained back-prop in terms of training difficulty? How does the convergence look? Since such a complicated optimization mechanism is being employed, it is to be suspected that training becomes more difficult. Perhaps the authors can provide some data on training difficulty pooled out from the experimental results.

Answer: It turned out that the increase of computational complexity due to the constraint term is not large insomuch as, for ResNet, the increase compared to conventional backprop is approximately 2\%. For the computational complexity of CBP in detail, We estimate the computational overhead of CBP in terms of the number of floating point operations (FLOPs). 

\#FLOPs for CBP = (\#FLOPs for weight update) + (\#FLOPs for Lagrange multiplier update), where

\#FLOPs for weight update = (\#FLOPs for loss evaluation) + (\#FLOPs for error-backpropagation).

\#FLOPs for loss evaluation = (\#FLOPs for forward propagation) + (\#FLOPs for constraint contribution calculation $\lambda^T\mathbf{cs}$). 

The number of FLOPs for the latter scales with the number of parameters in total ($n_w$) because each parameter is given a set of $\lambda$ and $cs$. The number of multiplication $\lambda\times cs_i\left(w_i\right)$ is the same as the number of parameters ($n_w$).The calculation of $cs_i$ for a given $w_i$ involves six FLOPs according to Eqs. (8)-(10). Therefore, 

\#FLOPs for loss evaluation = (\#FLOPs for forward propagation) + 6$n_w$.

As for conventional backprop, the number of FLOPs for weight update (using error-backpropagation) approximately equals the number of FLOPs for forward propagation. Therefore,

\#FLOPs for weight update = 2$\times$(\#FLOPs for forward propagation) + 6$n_w$


The Lagrange multiplier update for each multiplier involves one multiplication ($\eta_{\lambda}\times cs_i$) and one addition ($\lambda_i\gets \lambda_i+\eta_{\lambda}cs_i$), but uses $cs_i$ that has been calculated already when calculating the loss function. Therefore,

\#FLOPs for Lagrange multiplier update = 2$n_w$.

It should be noted that the multiplier is updated merely a few times during the entire training period: less than 20 percent of the training epochs, which is parameterized by $p$.

Therefore, we have

\#FLOPs for CBP = 2(\#FLOPs for forward propagation) + 2$(p+3)n_w$

The number of FLOPs for CBP for three models (for $p=0.2$) is shown below.

AlexNet: \#FLOPs for CBP $\approx$ 1.82G, and \#FLOPs for BP $\approx$ 1.45G (i.e., 25\% increase in \#FLOPs)

ResNet18: \#FLOPs for CBP $\approx$ 3.69G, and \#FLOPs for BP $\approx$ 3.62G (i.e., 2\% increase in \#FLOPs)

ResNet50: \#FLOPs for CBP $\approx$ 7.89G, and \#FLOPs for BP $\approx$ 7.74G (i.e., 2\% increase in \#FLOPs)

We will address this complexity estimation in the Discussion section of our revised manuscript.

The number of maximum epochs set for our work is as follows:

(AlexNet) binary: 122, ternary: 113, one-bit shift: 201, two-bit shift: 201

(ResNet18) binary: 151, ternary: 151, one-bit shift: 151, two-bit shift: 151

(ResNet50) binary: 114, ternary: 101, one-bit shift: 141, two-bit shift: 141

The training procedure with training epoch for all models (AlexNet, ResNet18, ResNet50), each with four weight-constraints, is shown in Appendix B. 

Minor comment: there are several typos and LaTeX errors such as broken cross-references.

Answer: We thank Reviewer for the indication.

 We thank Reviewer for the valuable comments. They help us improve the quality of our work.

1 It was only at line number 225 and 235 that it became clear that CBP was applied post full-precision standard BP training! This means that CBP is a post-training method but with a high training complexity (compared to both post and in-training quantization methods) since training proceeds in two phases - standard BP followed by CBP. The paper does not mention how many epochs of CBP training was required. Please provide this information.

Answer: We agree with Reviewer. We will clearly state that CBP is a post-training algorithm from the first place in the revised manuscript. For the computational complexity of CBP in detail, We estimate the computational overhead of CBP in terms of the number of floating point operations (FLOPs). 

\#FLOPs for CBP = (\#FLOPs for weight update) + (\#FLOPs for Lagrange multiplier update), where

\#FLOPs for weight update = (\#FLOPs for loss evaluation) + (\#FLOPs for error-backpropagation).

\#FLOPs for loss evaluation = (\#FLOPs for forward propagation) + (\#FLOPs for constraint contribution calculation $\lambda^T\mathbf{cs}$). 

The number of FLOPs for the latter scales with the number of parameters in total ($n_w$) because each parameter is given a set of $\lambda$ and $cs$. The number of multiplication $\lambda\times cs_i\left(w_i\right)$ is the same as the number of parameters ($n_w$).The calculation of $cs_i$ for a given $w_i$ involves six FLOPs according to Eqs. (8)-(10). Therefore, 

\#FLOPs for loss evaluation = (\#FLOPs for forward propagation) + 6$n_w$.

As for conventional backprop, the number of FLOPs for weight update (using error-backpropagation) approximately equals the number of FLOPs for forward propagation. Therefore,

\#FLOPs for weight update = 2$\times$(\#FLOPs for forward propagation) + 6$n_w$


The Lagrange multiplier update for each multiplier involves one multiplication ($\eta_{\lambda}\times cs_i$) and one addition ($\lambda_i\gets \lambda_i+\eta_{\lambda}cs_i$), but uses $cs_i$ that has been calculated already when calculating the loss function. Therefore,

\#FLOPs for Lagrange multiplier update = 2$n_w$.

It should be noted that the multiplier is updated merely a few times during the entire training period: less than 20 percent of the training epochs, which is parameterized by $p$.

Therefore, we have

\#FLOPs for CBP = 2(\#FLOPs for forward propagation) + 2$(p+3)n_w$

The number of FLOPs for CBP for three models (for $p=0.2$) is shown below.

AlexNet: \#FLOPs for CBP $\approx$ 1.82G, and \#FLOPs for BP $\approx$ 1.45G (i.e., 25\% increase in \#FLOPs)

ResNet18: \#FLOPs for CBP $\approx$ 3.69G, and \#FLOPs for BP $\approx$ 3.62G (i.e., 2\% increase in \#FLOPs)

ResNet50: \#FLOPs for CBP $\approx$ 7.89G, and \#FLOPs for BP $\approx$ 7.74G (i.e., 2\% increase in \#FLOPs)

We will address this complexity estimation in the Discussion section of our revised manuscript.

The number of maximum epochs set for our work is as follows:

(AlexNet) binary: 122, ternary: 113, one-bit shift: 201, two-bit shift: 201

(ResNet18) binary: 151, ternary: 151, one-bit shift: 151, two-bit shift: 151

(ResNet50) binary: 114, ternary: 101, one-bit shift: 141, two-bit shift: 141

2 Related 1., why wasn't CBP applied with a cold-start? Does it not converge or does it converge to a poor solution?

Answer: The convergence is always observable. But the cold-start reduces the classification accuracy by approximately 2-3\%. Therefore, we use CBP as a post-training algorithm.

3 The accuracy numbers reported match but do not significantly exceed state-of-the-art reported numbers. This is probably too much to expect since weight quantization is a mature topic already. Nevertheless, it would have been nice to see CBP come out ahead in some other respect, e.g., computational complexity.

Answer: As explained regarding Reviewer's comment 1, for ResNet, the increase of computational complexity of CBP is approximately 2\% compared to conventional backprop. In fact, we consider that the accuracy enhancement achieved by CBP is significant. For instance, ResNet50 with the two-bit shift weight constraint almost reaches the performance of ResNet50 with full-precision weights, which exceeds the SOTA accuracy (ADMM) by almost 2\%. 

Additionally, CBP can provide a common single weight-quantization algorithm for various types of weight constraints unlike other quantization methods that are tailored to particular weight quantization (except ADMM). Applying different weight constraints can be easily done by using the constraint function introduced in our work. Overall, CBP is an easy-to-use algorithm supporting various weight constraints.   

Limitations And Societal Impact:

The authors have not discussed the limitations and potential negative societal impact of their work.

Answer: we already addressed the limitation of our work in the original manuscript as follows.

``We used CBP as a post-training method in this work. That is, the networks considered were pre-trained using conventional backpropagation. Applying CBP to untrained networks hardly reached the accuracies of classification listed in Tables 1, 2, and 3. When efficiency in training is of the most important concern, CBP may not be the best choice. However, when efficiency in memory usage is of the most important concern, CBP may be the optimal choice with regard to its excellent learning capability with maximum 3-bit weight precision, which almost reaches the classification accuracy of full-precision networks. Additionally, the use of one-bit or two-bit shift weights can avoid multiplication operations that consume a considerable amount of power, so that it can significantly improve calculation efficiency''

However, Reviewer is right that we did not mention the societal impact of our work. As mentioned, CBP is not a method tailored to particular networks or particular datasets. This may cause a broader impact across deep learning application domains where memory capacity is limited. We will address this societal impact in the revised manuscript.

 We thank Reviewer for the positive evaluation and valuable comments.

1 Since the paper addressed shift computations, the proposed method could have been compared with shift networks such as DeepShift Network

Answer: We thank Reviewer for the suggestion. We will address the suggested work in the performance comparison table.

2 Quantization of activations was not considered in this paper.

Answer: It may be hard to apply the current version of CBP to activation quantization. This is because the constraint function for activation varies on inputs unlike weights whose constraint functions are independent of inputs. Computational complexity can further be reduced by quantizing activations; yet, the reduction is hardly significant because most of computational complexity (particularly space complexity) arises from weights. Therefore, we only consider the reduction in space complexity by quantizing weights.   

This paper presents a method to quantize weights of neural networks by imposing constraints using a pseudo-Lagrange multiplier. The proposed method was used to quantize weights of networks to 1 bit, 2 bits and 3 bits. It's shown that the proposed method can quantize AlexNet, ResNet-18 and ResNet-50 successfully on ImageNet dataset.
    Here are the main pros and cons of the paper.
Pros: 
+ Mathematical formulation for the proposed approach
+ SOTA results on ImageNet dataset on different networks

Cons:
- Since the paper addressed shift computations, the proposed method could have been compared with shift networks such as DeepShift Network
- Quantization of activations was not considered in this paper. Yes

The paper describes the use of the Lagrange multiplier method (LMM) to train deep nets with constraints on weight precision incorporated as an intrinsic part of back prop (BP). The proposed method referred to as constrained BP (CBP) allows one to obtain low weight precision networks whose accuracy is close to that of full-precision networks. The paper describes the LMM method and its application to training DNNs. Results on AlexNet, ResNet-18 and ResNet-50 on ImageNet is shown.  This is a very well-written paper. It was enjoyable to read due to its clarity of exposition and simplicity of the concept. The proposed idea viz. to augment the loss function by including weight constraints, is novel and interesting. I was hoping that the proposed method CBP would provide some advantage over existing methods. Unfortunately, this was not the case. The main issues are:

1. It was only at line number 225 and 235 that it became clear that CBP was applied post full-precision standard BP training! This means that CBP is a post-training method but with a high training complexity (compared to both post and in-training quantization methods) since training proceeds in two phases - standard BP followed by CBP. The paper does not mention how many epochs of CBP training was required. Please provide this information. 

2. Related 1., why wasn't CBP applied with a cold-start? Does it not converge or does it converge to a poor solution?

3. The accuracy numbers reported match but do not significantly exceed state-of-the-art reported numbers. This is probably too much to expect since weight quantization is a mature topic already. Nevertheless, it would have been nice to see CBP come out ahead in some other respect, e.g., computational complexity. 

Overall, though I was rooting for this paper due to the elegance of the method, I am hard-pressed to find a tangible benefit ensuing from it. Perhaps, I am missing something. I am willing to adjust my ratings if the authors or the other reviewers are able to shed some light. The authors have not discussed the limitations and potential negative societal impact of their work. 

Meta-review: This paper uses a Lagrange multiplier method to train neural networks with constraints on the values of the weights that allows them to be quantized. While some reviewers appreciated the simplicity and effectiveness of this method, they found that the paper didn't properly explain the additional costs of the method, or justify its practical benefits. There also seems to be some questions about whether or not the baseline methods compared to represent the SOTA for weight quantization. The quality of the writing also seems pretty rough in places, based on my own quick scan. While I think this paper may contain a solid contribution to the field, the draft could use some work.

Decision: accept