Demystifying and Generalizing BinaryConnect

1 Introduction . Scaling up to extremely large datasets and models has been a main ingredient for the success of deep learning . Indeed , with the availability of big data , more computing power , convenient software , and a bag of training tricks as well as algorithmic innovations , the size of models that we routinely train in order to achieve state-of-the-art performance has exploded , e.g. , to billions of parameters in recent language models [ 9 ] . However , high memory usage and computational cost at inference time has made it difficult to deploy these models in real-time or on resource-limited devices [ 28 ] . The environmental impact of training and deploying these large models has also been recognized [ 40 ] . A common approach to tackle these problems is to compress a large model through quantization , i.e. , replacing high-precision parameters with lower-precision ones . For example , we may constrain a subset of the weights to be binary [ 1 , 5 , 11 , 29 , 33 , 37 , 45 ] or ternary [ 26 , 53 ] . Quantization can drastically decrease the carbon footprint of training and inference of neural networks , however , it may come at the cost of increased bias [ 21 ] . One of the main methods to obtain quantized neural networks is to encourage quantized parameters during gradient training using explicit or implicit regularization techniques , however , other methods are possible [ 16–18 , 20 , 25 , 32 , 51 ] . Besides the memory benefits , the structure of the quantization can speed up inference using , for example , faster matrix-vector products [ 18 , 23 ] . Training and inference can be made even more efficient by also quantizing the activations [ 37 ] or gradients [ 52 ] . Impressive performance has been achieved with quantized networks , for example , on object detection [ 44 ] and natural language processing [ 43 ] tasks . The theoretical underpinnings of quantized neural networks , such as when and why their performance remains reasonably well , have been actively studied [ 3 , 13 , 22 , 41 , 46 ] . ∗Work done during an internship at Huawei Noah ’ s Ark Lab . Correspondence to tim.dockhorn @ uwaterloo.ca . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . BinaryConnect [ BC , 11 ] and its many variations [ 10 , 37 , 53 ] are considered the gold standard for neural network quantization . Compared to plain ( stochastic ) gradient descent , BC does not evaluate the gradient at the current iterate but rather at a ( close-by ) quantization point using the Straight Through Estimator [ 6 ] . Despite its empirical success , BC has largely remained a “ training trick ” [ 1 ] and a rigorous understanding of its inner workings has yet to be found , with some preliminary steps taken in Li et al . [ 27 ] for the convex setting and in Yin et al . [ 45 ] for a particular quantization set . As pointed out in Bai et al . [ 5 ] , BC only evaluates gradients at the finite set of quantization points , and therefore does not exploit the rich information carried in the continuous weights network . Bai et al . [ 5 ] also observed that BC is formally equivalent to the dual averaging algorithm [ 31 , 42 ] , while some similarity to the mirror descent algorithm was found in Ajanthan et al . [ 1 ] . The main goal of this work is to significantly improve our understanding of BC , by connecting it with well-established theory and algorithms . In doing so we not only simplify and improve existing results but also obtain novel generalizations . We summarize our main contributions in more details : • In Section 2 , we show that existing gradient-based quantization algorithms are surprisingly similar to each other : the only high-level difference is at what points we evaluate the gradient and perform the update . • In Section 3 , we present a principled theory for constructing proximal quantizers . Our results unify previous efforts , remove tedious calculations , and bring theoretical convenience . We illustrate our theory by effortlessly designing a new quantizer that can be computed in one-pass , works for different quantization sets ( binary or multi-bit ) , and includes previous attempts as special cases [ 5 , 11 , 45 , 53 ] . • In Section 4 , we significantly extend the observation of Bai et al . [ 5 ] that the updates of BC are the same as the dual averaging algorithm [ 31 , 42 ] : BC is a nonconvex counterpart of dual averaging , and more importantly , dual averaging itself is simply the generalized conditional gradient algorithm applied to a smoothened dual problem . The latter fact , even in the convex case , does not appear to be widely recognized to the best of our knowledge . • In Section 5 , making use of the above established results , we propose ProxConnect ( PC ) as a family of algorithms that generalizes BC and we prove its convergence properties for both the convex and the nonconvex setting . We rigorously justify the diverging parameter in proximal quantizers and resolve a discrepancy between theory and practice in the literature [ 1 , 5 , 45 ] . • In Section 6 , we verify that PC outperforms BC and ProxQuant [ 5 ] on CIFAR-10 for both finetuning pretrained models as well as end-to-end training . On the more challenging ImageNet dataset , PC yields competitive performance despite of minimal hyperparameter tuning . 2 Background . We consider the usual ( expected ) objective ` ( w ) = E ` ( w , ξ ) , where ξ represents random sampling . For instance , ` ( w ) = 1n ∑n i=1 ` i ( w ) where ξ is uniform over n training samples ( or minibatches thereof ) , w are the weights of a neural network and ` i may be the cross-entropy loss ( of the i-th training sample ) . We denote a sample ( sub ) gradient of ` at w and ξ as ∇̃ ` ( w ) = ∇ ` ( w , ξ ) so that E∇̃ ` ( w ) = ∇ ` ( w ) . Throughout , we use the starred notation w∗ for continuous weights and reserve w for ( semi ) discrete ones . We are interested in solving the following ( nonconvex ) problem : min w∈Q ` ( w ) , ( 1 ) where Q ⊆ Rd is a discrete , nonconvex quantization set . For instance , on certain low-resource devices it may be useful or even mandatory to employ binary weights , i.e. , Q = { ±1 } d. Importantly , our goal is to compete against the non-quantized , continuous weights network ( i.e . Q = Rd ) . In other words , we do not necessarily aim to solve ( the hard , combinatorial ) problem ( 1 ) globally and optimally . Instead , we want to find discrete weights w ∈ Q that remain satisfactory when compared to the non-quantized continuous weights . This is how we circumvent the difficulty in ( 1 ) and more importantly how the structure of ` could come into aid . If ` is reasonably smooth , a close-by quantized weight of a locally , or even globally , optimal continuous weight will likely yield similar performance . Tremendous progress has been made on quantizing and compressing neural networks . While it is not possible to discuss all related work , below we recall a few families of gradient-based quantization algorithms that directly motivate our work ; more methods can be found in recent surveys [ 15 , 36 ] . BinaryConnect ( BC ) . Courbariaux et al . [ 11 ] considered binary networks where Q = { ±1 } d and proposed the BinaryConnect algorithm : w∗t+1 = w ∗ t − ηt∇̃ ` ( P ( w∗t ) ) , ( 2 ) where P is a projector that quantizes the continuous weights w∗t either deterministically ( by taking the sign ) or stochastically . Note that the ( sample ) gradient is evaluated at the quantized weights wt : = P ( w ∗ t ) , while its continuous output , after scaled by the step size ηt , is added to the continuous weights w∗t . For later comparison , it is useful to break the BC update into the following two pieces : wt = P ( w ∗ t ) , w ∗ t+1 = w ∗ t − ηt∇̃ ` ( wt ) . ( 3 ) Other choices of P [ 1 , 45 ] and Q [ 44 , 53 ] in this framework have also been experimented with . ProxQuant ( PQ ) . Bai et al . [ 5 ] applied the usual proximal gradient to solve ( 1 ) : wt+1 = P ( wt − ηt∇̃ ` ( wt ) ) , ( 4 ) which can be similarly decomposed into : wt = P ( w ∗ t ) , w ∗ t+1 = wt − ηt∇̃ ` ( wt ) . ( 5 ) Thus , the only high-level difference between BC and PQ is that the former updates the continuous weights w∗t while the latter updates the quantized weights wt . This seemingly minor difference turns out to cause drastically different behaviors of the two algorithms . For example , choosing P to be the Euclidean projection to Q works well for BinaryConnect but not at all for ProxQuant . Reversing BinaryConnect . The above comparison naturally suggests a reversed variant of BC : wt = P ( w ∗ t ) , w ∗ t+1 = wt − ηt∇̃ ` ( w∗t ) , ( 6 ) which amounts to switching the continuous and quantized weights in the BC update . Similar to PQ , the choice of P is critical in this setup . To the best of our knowledge , this variant has not been formally studied before . Reversing BinaryConnect may not be intuitive , however , it serves as a starting point for a more general method ( see Section 5 ) . Post-Training Quantization . Lastly , we can also rewrite the naive post-training quantization scheme in a similar form : wt = P ( w ∗ t ) , w ∗ t+1 = w ∗ t − ηt∇̃ ` ( w∗t ) , ( 7 ) where we simply train the continuous network as usual and then quantize at the end . Note that the quantized weights wt do not affect the update of the continuous weights w∗t . 3 What Makes a Good Quantizer ? . As we have seen in Section 2 , the choice of the quantizer P turns out to be a crucial element for solving ( 1 ) . Indeed , if P = PQ is the projector onto the discrete quantization set Q , then BC ( and ProxQuant ) only evaluate the gradient of ` at ( the finite set of ) points in Q . As a result , the methods will not be able to exploit the rich information carried in the continuous weights network , which can lead to non-convergence [ Fig . 1b , 5 ] . Since then , many semi-discrete quantizers , that turn continuous weights into more and more discrete ones , have been proposed [ 1 , 5 , 33 , 45 ] . In this section , we present a principled way to construct quantizers that unifies previous efforts , removes tedious calculations , and also brings theoretical convenience when it comes down to analyzing the convergence of the algorithms in Section 2 . Our construction is based on the proximal map Pµr : Rd ⇒ Rd of a ( closed ) function r : Pµr ( w ∗ ) = arg min w 1 2µ‖w −w ∗‖22 + r ( w ) , ( 8 ) where µ > 0 is a smoothing parameter . The proximal map is well-defined as long as the function r is lower bounded by a quadratic function , in particular , when r is bounded from below . If r is proper and ( closed ) convex , then the minimizer on the right-hand side of ( 8 ) is uniquely attained , while for general ( nonconvex ) functions the proximal map may be multi-valued ( hence the notation ⇒ ) . If r = ιQ is the indicator function ( see ( 15 ) below ) , then Pµr reduces to the familiar projector PQ ( for any µ ) . Remarkably , a complete characterization of such maps on the real line is available : Theorem 3.1 ( [ Proposition 3 , 48 ] ) . A ( possibly multi-valued ) map P : R ⇒ R is a proximal map ( of some function r ) iff it is ( nonempty ) compact-valued , monotone and has a closed graph . The underlying function r is unique ( up to addition of constants ) iff P is convex-valued , while r is convex iff P is nonexpansive ( i.e . 1-Lipschitz continuous ) . The sufficient and necessary conditions of Theorem 3.1 allow one to design proximal maps P directly , without needing to know the underlying function r at all ( even though it is possible to integrate a version of r from P ) . The point is that , as far as quantization algorithms are concerned , having P is enough , and hence one is excused from the tedious calculations in deriving P from r as is typical in existing works . For example , the ( univariate ) mirror maps constructed in Ajanthan et al . [ Theorem 1 , 1 ] , such as tanh , are proximal maps according to Theorem 3.1 : In our notation , the ( Mirror Descent ) updates of Ajanthan et al . [ 1 ] are the same as ( 3 ) with P = ( ∇Φ ) −1 for some mirror map Φ . Since Φ is taken to be strictly convex in Ajanthan et al . [ 1 ] , it is easy to verify that P satisfies all conditions of Theorem 3.1.2 The next result allows us to employ stochastic quantizers , where in each iteration we randomly choose one of Pi to quantize the weights3 . We may also apply different quantizers to different layers of a neural network . Theorem 3.2 . Let Pi : Rd ⇒ Rd , i = [ k ] , be proximal maps . Then , the averaged map P : = ∑k i=1 αiPi , where αi ≥ 0 , ∑k i=1 αi = 1 , ( 9 ) is also a proximal map . Similarly , the product map P : = P1 × P2 × · · · × Pk , w∗ = ( w∗1 , . . . , w∗k ) 7→ ( P1 ( w ∗ 1 ) , . . . , Pk ( w ∗ k ) ) ( 10 ) is a proximal map ( from Rdk to Rdk ) . ( The proof of Theorem 3.2 and all other omitted proofs can be found in Appendix A . ) Example 3.3 . Let Q be a quantization set ( e.g . Q = { −1 , 0 , 1 } ) . Clearly , the identity map id and the projector PQ are proximal maps . Therefore , their convex combination Pµ = id+µPQ 1+µ , µ ≥ 0 , is also a proximal map , which is exactly the quantizer used in Yin et al . [ 45 ] . Lastly , we mention that it is intuitively desirable to have Pµ → PQ when µ increases indefinitely , so that the quantizer eventually settles on bona fide discrete values in Q . This is easily achieved by letting minimizers of the underlying function r , or the ( possibly larger set of ) fixed points of Pµ , approach Q . We give one such construction by exploiting Theorem 3.1 and Theorem 3.2 .

Summary: The paper presents a refined theoretical understanding of binary connect via relating the dual-averaging method to generalized conditional gradient descent. Then, a general proximal connect algorithm is presented. Experiments on image classification datasets show marginal improvements over the comparable methods.