Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network

Recently , Frankle & Carbin ( 2019 ) demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network . However , finding these high performing trainable subnetworks is expensive , requiring iterative process of training and pruning weights . In this paper , we propose ( and prove ) a stronger Multi-Prize Lottery Ticket Hypothesis : A sufficiently over-parameterized neural network with random weights contains several subnetworks ( winning tickets ) that ( a ) have comparable accuracy to a dense target network with learned weights ( prize 1 ) , ( b ) do not require any further training to achieve prize 1 ( prize 2 ) , and ( c ) is robust to extreme forms of quantization ( i.e. , binary weights and/or activation ) ( prize 3 ) . This provides a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full precision neural networks . We also propose an algorithm for finding multi-prize tickets ( MPTs ) and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets . Empirical results indicate that as models grow deeper and wider , multi-prize tickets start to reach similar ( and sometimes even higher ) test accuracy compared to their significantly larger and full-precision counterparts that have been weight-trained . Without ever updating the weight values , our MPTs-1/32 not only set new binary weight network state-of-the-art ( SOTA ) Top-1 accuracy – 94.8 % on CIFAR-10 and 74.03 % on ImageNet – but also outperform their full-precision counterparts by 1.78 % and 0.76 % , respectively . Further , our MPT-1/1 achieves SOTA Top-1 accuracy ( 91.9 % ) for binary neural networks on CIFAR-10 . Code and pre-trained models are available at : https : //github.com/chrundle/biprop . 1 INTRODUCTION . Deep learning ( DL ) has made a significant breakthroughs in a wide range of applications ( Goodfellow et al. , 2016 ) . These performance improvements can be attributed to the significant growth in the model size and the availability of massive computational resources to train such models . Therefore , these gains have come at the cost of large memory consumption , high inference time , and increased power consumption . This not only limits the potential applications where DL can make an impact but also have some serious consequences , such as , ( a ) generating huge carbon footprint , and ( b ) creating roadblocks to the democratization of AI . Note that significant parameter redundancy and a large number of floating-point operations are key factors incurring the these costs . Thus , for discarding the redundancy from DNNs , one can either ( a ) Prune : remove non-essential connections from an existing dense network , or ( b ) Quantize : constrain the full-precision ( FP ) weight and activation values to a set of discrete values which allows them to be represented using fewer bits . Further , one can exploit the complementary nature of pruning and quantization to combine their strengths . Although pruning and quantization1 are typical approaches used for compressing DNNs ( Neill , 2020 ) , it is not clear under what conditions and to what extent compression can be achieved without sacrificing the accuracy . The most extreme form of quanitization is binarization , where weights and/or activations can only have two possible values , namely−1 ( 0 ) or +1 ( the interest of this paper ) . In addition to saving memory , binarization results in more power efficient networks with significant computation acceleration since expensive multiply-accumulate operations ( MACs ) can be replaced by cheap XNOR and bit-counting operations ( Qin et al. , 2020a ) . In light of these benefits , it is of interest to question if conditions exists such that a binarized DNN can be pruned to achieve accuracy comparable to the dense FP DNN . More importantly , even if these favourable conditions are met then how do we find these extremely compressed ( or compact ) and highly accurate subnetworks ? Traditional pruning schemes have shown that a pretrained DNN can be pruned without a significant loss in the performance . Recently , ( Frankle & Carbin , 2019 ) made a breakthrough by showing that dense network contain sparse subnetworks that can match the performance of the original network when trained from scratch with weights being reset to their initialization ( Lottery Ticket Hypothesis ) . Although the original approach to find these subnetworks still required training the dense network , some efforts ( Wang et al. , 2020b ; You et al. , 2019 ; Wang et al. , 2020a ) have been carried out to overcome this limitation . Recently a more intriguing phenomenon has been reported – a dense network with random initialization contains subnetworks that achieve high accuracy , without any further training ( Zhou et al. , 2019 ; Ramanujan et al. , 2020 ; Malach et al. , 2020 ; Orseau et al. , 2020 ) . These trends highlight good progress being made towards efficiently and accurately pruning DNNs . In contrast to these positive developments for pruning , results on binarizing DNNs have been mostly negative . To the best of our knowledge , post-training schemes have not been successful in binarizing pretrained models without retraining . Even with training binary neural networks ( BNNs ) from scratch ( though inefficient ) , the community has not been able to make BNNs achieve comparable results to their full precision counterparts . The main reason being that network structures and weight optimization techniques are predominantly developed for full precision DNNs and may not be suitable for training BNNs . Thus , closing the gap in accuracy between the full precision and the binarized version may require a paradigm shift . Furthermore , this also makes one wonder if efficiently and accurately binarizing DNNs similar to the recent trends in pruning is ever feasible . In this paper , we show that a randomly initialized dense network contains extremely sparse binary subnetworks that without any weight training ( i.e. , efficient ) have comparable performance to their trained dense and full-precision counterparts ( i.e. , accurate ) . Based on this , we state our hypothesis : Multi-Prize Lottery Ticket Hypothesis . A sufficiently over-parameterized neural network with random weights contains several subnetworks ( winning tickets ) that ( a ) have comparable accuracy to a dense target network with learned weights ( prize 1 ) , ( b ) do not require any further training to achieve prize 1 ( prize 2 ) , and ( c ) is robust to extreme forms of quantization ( i.e. , binary weights and/or activation ) ( prize 3 ) . Contributions . First , we propose the multi-prize lottery ticket hypothesis as a new perspective on finding neural networks with drastically reduced memory size , much faster test-time inference and 1A detailed discussion on related work on pruning and quantization is provided in Appendix F. lower power consumption compared to their dense and full-precision counterparts . Next , we provide theoretical evidence of the existence of highly accurate binary subnetworks within a randomly weighted DNN ( i.e. , proving the multi-prize lottery ticket hypothesis ) . Specifically , we mathematically prove that we can find an ε-approximation of a fully-connected ReLU DNN with width n and depth ` using a sparse binary-weight DNN of sufficient width . Our proof indicates that this can be accomplished by pruning and binarizing the weights of a randomly weighted neural network that is a factor O ( n3/2 ` /ε ) wider and 2 ` deeper . To the best of our knowledge , this is the first theoretical work proving the existence of highly accurate binary subnetworks within a sufficiently overparameterized randomly initialized neural network . Finally , we provide biprop ( binarize-prune optimizer ) in Algorithm 1 to identify MPTs within randomly weighted DNNs and empirically test our hypothesis . This provides a completely new way to learn BNNs without relying on weight-optimization . Results . We explore two variants of multi-prize tickets – one with binary weights ( MPT-1/32 ) and other with binary weights and activation ( MPT-1/1 ) where x/y denotes x and y bits to represent weights and activation , respectively . MPTs we find have 60 − 80 % fewer parameters than the original network . We perform a series of experiments on on small and large scale datasets for image recognition , namely CIFAR-10 ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) . On CIFAR-10 , we test the performance of multi-prize tickets against the trend of making the model deeper and wider . We found that as models grow deeper and wider , both variants of multi-prize tickets start to reach similar ( and sometimes even higher ) test accuracy compared to the dense and full precision original network with learned weights . In other words , the performance of multiprize tickets improves with the amount of redundancy in the original network . We also carry out experiments with state-of-the-art ( SOTA ) architectures on CIFAR-10 and ImageNet datasets with an aim to investigate their redundancy . We find that within most randomly weighted SOTA DNNs reside extremely compact ( i.e. , sparse and binary ) subnetworks which are smaller than , but match the performance of trained target dense and full precision networks . Furthermore , with minimal hyperparameter tuning , our MPTs achieve Top-1 accuracy comparable to ( or higher than ) SOTA BNNs . The performance of MPTs is further improved by allowing the parameters in BatchNorm layer to be learned . Finally , on both CIFAR-10 and ImageNet , MPT-1/32 subnetworks outperform their significantly larger and full-precision counterparts that have been weight-trained . 2 MULTI-PRIZE LOTTERY TICKETS : THEORY AND ALGORITHMS . We first prove the existence of MPTs in an overparameterized randomly weighted DNN . For ease of presentation , we state an informal version of Theorem 2 which can be found in Appendix B . We then explore two variants of tickets ( MPT-1/32 and MPT-1/1 ) and provide an algorithm to find them . 2.1 PROVING THE MULTI-PRIZE LOTTERY TICKETS HYPOTHESIS . In this section we seek to answer the following question : What is the required amount of overparameterization such that a randomly weighted neural network can be compressed to a sparse binary subnetwork that approximates a dense trained target network ? Theorem 1 . ( Informal Statement of Theorem 2 ) Let ε , δ > 0 . For every fully-connected ( FC ) target network with ReLU activations of depth ` and width n with bounded weights , a random binary FC network with ReLU activations of depth 2 ` and width O ( ( ` n3/2/ε ) + ` n log ( ` n/δ ) ) contains with probability ( 1− δ ) a binary subnetwork that approximates the target network with error at most ε . Sketch of Proof . Consider a FC ReLU network F ( x ) = W ( ` ) σ ( W ( ` −1 ) · · ·σ ( W ( 1 ) x ) ) , where σ ( x ) = max { 0 , x } , x ∈ Rd , W ( i ) ∈ Rki×ki−1 , k0 = d , and i ∈ [ ` ] . Additionally , consider a FC network with binary weights given by G ( x ) = B ( ` ′ ) σ ( B ( ` ′−1 ) · · ·σ ( B ( 1 ) x ) ) , where B ( i ) ∈ { −1 , +1 } k ′ i×k ′ i−1 , k′0 = d , and i ∈ [ ` ′ ] . Our goal is to determine a lower bound on the depth , ` ′ , and the widths , { k′i } ` ′ i=1 , such that with probability ( 1− δ ) the network G ( x ) contains a subnetwork G̃ ( x ) satisfying ‖G̃ ( x ) − F ( x ) ‖ ≤ ε , for any ε > 0 and δ ∈ ( 0 , 1 ) . We first establish lower bounds on the width of a network of the form g ( x ) = B ( 2 ) σ ( B ( 1 ) x ) such that with probability ( 1 − δ′ ) there exists a subnetwork g̃ ( x ) of g ( x ) s.t . ‖g̃ ( x ) − σ ( Wx ) ‖ ≤ ε′ , for any ε′ > 0 and δ′ ∈ ( 0 , 1 ) . This process is carried out in detail in Lemmas 1 , 2 , and 3 in Appendix B . We have now approximated a single layer FC real-valued network using a subnetwork of a two-layer FC binary network . Hence , we can take ` ′ = 2 ` and Lemma 3 provides lower bounds on the width of each intermediate layer such that with probability ( 1 − δ ) there exists a subnetwork G̃ ( x ) of G ( x ) satisfying ‖G̃ ( x ) − F ( x ) ‖ ≤ ε . This is accomplished in Theorem 2 in Appendix B . To the best of our knowledge this is the first theoretical result proving that a sparse binary-weight DNN that can approximate a real-valued target DNN . As it has been established that real-valued DNNs are universal approximators ( Scarselli & Tsoi , 1998 ) , our result carries the implication that sparse binary-weight DNNs are also universal approximators . In relation to the first result establishing the existence of real-valued subnetworks in a randomly weighted DNN approximating a realvalued target DNN ( Malach et al. , 2020 ) , the lower bound on the width established in Theorem 2 is better than their lower bound of O ( ` 2n2 log ( ` n/δ ) /ε2 ) .

Summary: The authors propose a stronger lottery ticket hypothesis in this paper – the multi-prize lottery ticket hypothesis. In particular, the new hypothesis seeks answer to the required amount of over-parameterization for a randomly initialized network to become able to compress to a sparse untrained binary subnetwork with on-par accuracy. The authors prove the existence of such subnetwork and show the bounds on over-parameterization. The paper proposes new methods to get the binary-weight tickets and the binary-activation tickets, where binary-weight tickets are subnetworks with weights as binary, and binary-activation tickets have the activation function in the forward propagation as binary. As binary networks can largely reduce the computational complexity for inference, this work has practical importance especially for applications with constraints for memory and power. The paper has many simulation results to support the theoretical guarantees, and the proposed approach on binary-weight networks has advantages over existing methods.