Title: Loss-aware Binarization of Deep Networks
Authors: Lu Hou
Quanming Yao
James T. Kwok
Venue: ICLR 2017
Keywords: Deep learning
Applications
Optimization

Abstract: Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximations and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.


Review discussion:
Do the binarized activations in the BPN2 setting affect the training procedure? Or does the training procedure remain the same as for BPN?
What is the main source of regularization: binarization or good optimization? (i.e. is the training error of BPN higher or lower than the training error of BWN?) Does learning reach convergence in all cases (BPN, BWN, etc.), or is there an implicit early-stopping regularization due to the maximum number of epochs?
Thanks for your feedback and questions.
1. The training procedure for BPN2 is almost the same as BPN, with the following changes:
- In feedforward propagation, the activations are binarized with the sign function (after step 7 in Algorithm 1).
- In backward propagation, the gradient for the sign function is computed using straight-through estimator as in "BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1".
2. For the FNN experiments, both BPN and BWN converge to near-zero training loss.
MNIST:     	full-precision: 0.000	BinaryConnect: 0.000	BWN: 0.000	BPN: 0.000
CIFAR-10: 	full-precision: 0.000	BinaryConnect: 0.000	BWN: 0.000	BPN: 0.000
SVHN:      	full-precision: 0.000	BinaryConnect: 0.000	BWN: 0.000	BPN: 0.000
For the RNN experiments, BPN has lower training loss than BWN.
War and Peace:  full-precision: 1.095 	BinaryConnect: 3.003	BWN: 1.220	BPN: 1.203
Linux Kernel:   full-precision: 0.968	BinaryConnect: 3.493	BWN: 1.081 	BPN: 1.062
Note that Adam is used as the optimization solver for all the models.
Hence, the difference in performance is due to the different binarization schemes used.
Convergence is reached for  the full-precision network, BWN and BPN (specifically, the change of training and validation losses in successive iterations is smaller than 10^{-5}).
However, BinaryConnect cannot converge in the RNN experiments. This is due to the exploding gradient problem as discussed section 3.2 and also supported by experimental results in section 3.4.
Please let us know if you have further questions.  Thanks again.
Taking into account the loss in the binarization step through a proximal Newton algorithm is a nice idea. This is at least one approach to bringing in the missing loss in the binarization step, which has recently gone from a two step process of train and binarize to a single step simultaneous train/compress. Performance on a few small tasks show the benefit. It would be nice to see some results on substantial networks and tasks which really need compression on embedded systems (a point made in the introduction).
Is it necessary to discuss exploding/vanishing gradients when the RNN experiments are carried out by an LSTM, and handled by the cell error carousel? We see the desire to tie into proposition 2, but not clear that the degradation we see in the binary connect is related. Adam is used in the LSTM optimization, was gradient clipping really needed, or is the degradation of binary connect simply related to capacity? For proposition 3. 1, theorem 3. 1 and proposition 3. 2 put the pointers to proofs in appendix.
Thanks for your review and suggestions, for the questions:
1. "Is it necessary to discuss exploding/vanishing gradients when the RNN experiments are carried out by an LSTM, and handled by the cell error carousel? We see the desire to tie into proposition 2, but not clear that the degradation we see in the binary connect is related."
- LSTM can handle the vanishing gradient problem, but does not address the the exploding gradient problem explicitly (see "On the difficulty of training recurrent neural networks." ICML 2013).
- The gradients of the LSTM are provided in Appendix E. As can be seen, the backpropagated gradients of LSTM still take the form of a      product of the Jacobian matrices like a vanilla RNN, and may explode when the scale of the weight matrix is large.
- Proposition 2 suggests that a binary weight matrix (as produced by BinaryConnect) is more likely to suffer from the exploding gradient problem.
- Empirically, for LSTM (with TS=100) on the "War and Peace" dataset, the gradient values obtained by the full-precision net, BWN and BPN are usually in the range [10^{-6}, 10^{-3}].
However, the gradient values obtained by BinaryConnect are [10^{-4}, 10^{-1}] when the gradients are clipped after each time step(Figure 1.(b)), and in [10^{28}, 10^{32}]when the gradients are clipped at the end of each back pass(Figure 2 in Appendix D.1).
Hence, the binary LSTM produced by BinaryConnect suffers from the exploding gradients problem.
- We also did a preliminary experiment that scales the binarized matrix (from BinaryConnect) with a scaling factor \alpha < 1. The following shows the testing cross-entropy values on the "War and Peace" dataset.
\alpha=1(original): 2.942
\alpha=0.1:	    1.543
\alpha=0.2:         1.422
\alpha=0.3:         1.441
As can be seen, using an appropriate scaling can improve performance.
On the other hand, the proposed binarization scheme can learn this \alpha automatically (section 3.2), and leads to an even better testing cross-entropy value of 1.291.
2. "Adam is used in the LSTM optimization, was gradient clipping really needed, or is the degradation of binary connect simply related to capacity?"
- All the network models share the same architecture and so have the same capacity.
- We also tried varying the clipping threshold for BinaryConnect. Results can be found in Appendix D.1. As can be seen from Table 5 there, this brings little improvement to the performance of BinaryConnect.
Thanks again for your time and efforts.
The paper presents a second-order method for training a neural networks while ensuring at the same time that weights (and activations) are binary. Through binarization, the method aims to achieve model compression for subsequent deployment on low-memory systems. The method is abbreviated BPN for "binarization using proximal Newton algorithm".
The method incorporates the supervised loss function directly in the binarization procedure, which is an important and desirable property. (Authors mention that existing weight binarization methods ignore the effect of binarization to the loss.) The method is clearly described and related analytically to the previously proposed weight binarization methods.
The experiments are extensive with multiple datasets and architectures, and demonstrate the generally higher performance of the proposed approach.
A minor issue with the feed-forward network experiments is that only test errors are reported. Such information does not really give evidence for the higher optimization performance. (see also comment "RE: AnonReviewer3's questions" stating that all baselines achieve near perfect training accuracy.) Making the optimization problem harder (e.g. by including an explicit regularizer into the training objective, or by using a data extension scheme), and monitoring the training objective instead of the test error could be a more direct way of demonstrating superior optimization performance.
The superiority of BPN is however becoming more clearly apparent in the subsequent LSTM experiments.
Thanks for your review and suggestions, we will add the suggested experiments in the future.
This paper proposed a proximal (quasi-) Newton’s method to learn binary DNN. The main contribution is to combine pre-conditioning with binarization in a proximal framework. It is interesting to have a proximal Newton’s method to interpret the different DNN binarization schemes. This gives a new interpretation of existing approaches.
However, the theoretical analysis is not very convincing or useful. The formulated optimization problem (3)-(4) is essentially a mixed integer programming.
Even though the paper treats the integer part as a constraint and address it in proximal operators, the constraint set is still discrete and there is no guarantee that the proximal Newton algorithm could converge under practically useful conditions.
In practice it is hard to verify the assumption [d_t^t]_k > \beta in Theorem 3.1. This relation could be hard to hold in DNN as the loss surface could be extremely complicated.
Thank you for your thoughts.
1. "the constraint set is still discrete and there is no guarantee that the proximal Newton algorithm could converge under practically useful conditions.
In practice it is hard to verify the assumption [d_t^l]_k > \beta in Theorem 3.1."
- Recall that \ell is assumed to have Lipschitz-continuous gradient with modulus \beta. If \beta is known, we can adjust \eta^t in step   21 so that "[d_t^l]_k > \beta" holds.
- In practice, \beta may not be known.
However, as deep networks are highly nonlinear, it is not unusual that convergence proofs need to make some assumptions, and our assumption can be considered as quite weak.
For example, in the theoretical analysis of ADAM, the cost function is assumed to be convex (which is not possible in deep networks).
- In practice, the proposed algorithm converges. As can be seen from the plots in https://www.dropbox.com/s/wkys181colu6yta/iclr_rebuttal.pdf?dl=0,  the training loss converges on all the data sets.
I agree with the authors on the challenge of theoretical analysis for DNN.
However, the authors should place more emphasis on the development of algorithm instead of the theoretical contribution.
Overall, the paper makes solid contribution and additional experiments/plots are convincing and I decide to raise the score.

Meta-review:
role: PC
data: comment: It's a simple contribution supported by empirical and theoretical analyses. After some discussion, all reviewers viewed the paper favourably.

Decision: Accept (Poster)