Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network

Recently , Frankle & Carbin ( 2019 ) demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network . However , finding these high performing trainable subnetworks is expensive , requiring iterative process of training and pruning weights . In this paper , we propose ( and prove ) a stronger Multi-Prize Lottery Ticket Hypothesis : A sufficiently over-parameterized neural network with random weights contains several subnetworks ( winning tickets ) that ( a ) have comparable accuracy to a dense target network with learned weights ( prize 1 ) , ( b ) do not require any further training to achieve prize 1 ( prize 2 ) , and ( c ) is robust to extreme forms of quantization ( i.e. , binary weights and/or activation ) ( prize 3 ) . This provides a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full precision neural networks . We also propose an algorithm for finding multi-prize tickets ( MPTs ) and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets . Empirical results indicate that as models grow deeper and wider , multi-prize tickets start to reach similar ( and sometimes even higher ) test accuracy compared to their significantly larger and full-precision counterparts that have been weight-trained . Without ever updating the weight values , our MPTs-1/32 not only set new binary weight network state-of-the-art ( SOTA ) Top-1 accuracy – 94.8 % on CIFAR-10 and 74.03 % on ImageNet – but also outperform their full-precision counterparts by 1.78 % and 0.76 % , respectively . Further , our MPT-1/1 achieves SOTA Top-1 accuracy ( 91.9 % ) for binary neural networks on CIFAR-10 . Code and pre-trained models are available at : https : //github.com/chrundle/biprop . 1 INTRODUCTION . Deep learning ( DL ) has made a significant breakthroughs in a wide range of applications ( Goodfellow et al. , 2016 ) . These performance improvements can be attributed to the significant growth in the model size and the availability of massive computational resources to train such models . Therefore , these gains have come at the cost of large memory consumption , high inference time , and increased power consumption . This not only limits the potential applications where DL can make an impact but also have some serious consequences , such as , ( a ) generating huge carbon footprint , and ( b ) creating roadblocks to the democratization of AI . Note that significant parameter redundancy and a large number of floating-point operations are key factors incurring the these costs . Thus , for discarding the redundancy from DNNs , one can either ( a ) Prune : remove non-essential connections from an existing dense network , or ( b ) Quantize : constrain the full-precision ( FP ) weight and activation values to a set of discrete values which allows them to be represented using fewer bits . Further , one can exploit the complementary nature of pruning and quantization to combine their strengths . Although pruning and quantization1 are typical approaches used for compressing DNNs ( Neill , 2020 ) , it is not clear under what conditions and to what extent compression can be achieved without sacrificing the accuracy . The most extreme form of quanitization is binarization , where weights and/or activations can only have two possible values , namely−1 ( 0 ) or +1 ( the interest of this paper ) . In addition to saving memory , binarization results in more power efficient networks with significant computation acceleration since expensive multiply-accumulate operations ( MACs ) can be replaced by cheap XNOR and bit-counting operations ( Qin et al. , 2020a ) . In light of these benefits , it is of interest to question if conditions exists such that a binarized DNN can be pruned to achieve accuracy comparable to the dense FP DNN . More importantly , even if these favourable conditions are met then how do we find these extremely compressed ( or compact ) and highly accurate subnetworks ? Traditional pruning schemes have shown that a pretrained DNN can be pruned without a significant loss in the performance . Recently , ( Frankle & Carbin , 2019 ) made a breakthrough by showing that dense network contain sparse subnetworks that can match the performance of the original network when trained from scratch with weights being reset to their initialization ( Lottery Ticket Hypothesis ) . Although the original approach to find these subnetworks still required training the dense network , some efforts ( Wang et al. , 2020b ; You et al. , 2019 ; Wang et al. , 2020a ) have been carried out to overcome this limitation . Recently a more intriguing phenomenon has been reported – a dense network with random initialization contains subnetworks that achieve high accuracy , without any further training ( Zhou et al. , 2019 ; Ramanujan et al. , 2020 ; Malach et al. , 2020 ; Orseau et al. , 2020 ) . These trends highlight good progress being made towards efficiently and accurately pruning DNNs . In contrast to these positive developments for pruning , results on binarizing DNNs have been mostly negative . To the best of our knowledge , post-training schemes have not been successful in binarizing pretrained models without retraining . Even with training binary neural networks ( BNNs ) from scratch ( though inefficient ) , the community has not been able to make BNNs achieve comparable results to their full precision counterparts . The main reason being that network structures and weight optimization techniques are predominantly developed for full precision DNNs and may not be suitable for training BNNs . Thus , closing the gap in accuracy between the full precision and the binarized version may require a paradigm shift . Furthermore , this also makes one wonder if efficiently and accurately binarizing DNNs similar to the recent trends in pruning is ever feasible . In this paper , we show that a randomly initialized dense network contains extremely sparse binary subnetworks that without any weight training ( i.e. , efficient ) have comparable performance to their trained dense and full-precision counterparts ( i.e. , accurate ) . Based on this , we state our hypothesis : Multi-Prize Lottery Ticket Hypothesis . A sufficiently over-parameterized neural network with random weights contains several subnetworks ( winning tickets ) that ( a ) have comparable accuracy to a dense target network with learned weights ( prize 1 ) , ( b ) do not require any further training to achieve prize 1 ( prize 2 ) , and ( c ) is robust to extreme forms of quantization ( i.e. , binary weights and/or activation ) ( prize 3 ) . Contributions . First , we propose the multi-prize lottery ticket hypothesis as a new perspective on finding neural networks with drastically reduced memory size , much faster test-time inference and 1A detailed discussion on related work on pruning and quantization is provided in Appendix F. lower power consumption compared to their dense and full-precision counterparts . Next , we provide theoretical evidence of the existence of highly accurate binary subnetworks within a randomly weighted DNN ( i.e. , proving the multi-prize lottery ticket hypothesis ) . Specifically , we mathematically prove that we can find an ε-approximation of a fully-connected ReLU DNN with width n and depth ` using a sparse binary-weight DNN of sufficient width . Our proof indicates that this can be accomplished by pruning and binarizing the weights of a randomly weighted neural network that is a factor O ( n3/2 ` /ε ) wider and 2 ` deeper . To the best of our knowledge , this is the first theoretical work proving the existence of highly accurate binary subnetworks within a sufficiently overparameterized randomly initialized neural network . Finally , we provide biprop ( binarize-prune optimizer ) in Algorithm 1 to identify MPTs within randomly weighted DNNs and empirically test our hypothesis . This provides a completely new way to learn BNNs without relying on weight-optimization . Results . We explore two variants of multi-prize tickets – one with binary weights ( MPT-1/32 ) and other with binary weights and activation ( MPT-1/1 ) where x/y denotes x and y bits to represent weights and activation , respectively . MPTs we find have 60 − 80 % fewer parameters than the original network . We perform a series of experiments on on small and large scale datasets for image recognition , namely CIFAR-10 ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) . On CIFAR-10 , we test the performance of multi-prize tickets against the trend of making the model deeper and wider . We found that as models grow deeper and wider , both variants of multi-prize tickets start to reach similar ( and sometimes even higher ) test accuracy compared to the dense and full precision original network with learned weights . In other words , the performance of multiprize tickets improves with the amount of redundancy in the original network . We also carry out experiments with state-of-the-art ( SOTA ) architectures on CIFAR-10 and ImageNet datasets with an aim to investigate their redundancy . We find that within most randomly weighted SOTA DNNs reside extremely compact ( i.e. , sparse and binary ) subnetworks which are smaller than , but match the performance of trained target dense and full precision networks . Furthermore , with minimal hyperparameter tuning , our MPTs achieve Top-1 accuracy comparable to ( or higher than ) SOTA BNNs . The performance of MPTs is further improved by allowing the parameters in BatchNorm layer to be learned . Finally , on both CIFAR-10 and ImageNet , MPT-1/32 subnetworks outperform their significantly larger and full-precision counterparts that have been weight-trained . 2 MULTI-PRIZE LOTTERY TICKETS : THEORY AND ALGORITHMS . We first prove the existence of MPTs in an overparameterized randomly weighted DNN . For ease of presentation , we state an informal version of Theorem 2 which can be found in Appendix B . We then explore two variants of tickets ( MPT-1/32 and MPT-1/1 ) and provide an algorithm to find them . 2.1 PROVING THE MULTI-PRIZE LOTTERY TICKETS HYPOTHESIS . In this section we seek to answer the following question : What is the required amount of overparameterization such that a randomly weighted neural network can be compressed to a sparse binary subnetwork that approximates a dense trained target network ? Theorem 1 . ( Informal Statement of Theorem 2 ) Let ε , δ > 0 . For every fully-connected ( FC ) target network with ReLU activations of depth ` and width n with bounded weights , a random binary FC network with ReLU activations of depth 2 ` and width O ( ( ` n3/2/ε ) + ` n log ( ` n/δ ) ) contains with probability ( 1− δ ) a binary subnetwork that approximates the target network with error at most ε . Sketch of Proof . Consider a FC ReLU network F ( x ) = W ( ` ) σ ( W ( ` −1 ) · · ·σ ( W ( 1 ) x ) ) , where σ ( x ) = max { 0 , x } , x ∈ Rd , W ( i ) ∈ Rki×ki−1 , k0 = d , and i ∈ [ ` ] . Additionally , consider a FC network with binary weights given by G ( x ) = B ( ` ′ ) σ ( B ( ` ′−1 ) · · ·σ ( B ( 1 ) x ) ) , where B ( i ) ∈ { −1 , +1 } k ′ i×k ′ i−1 , k′0 = d , and i ∈ [ ` ′ ] . Our goal is to determine a lower bound on the depth , ` ′ , and the widths , { k′i } ` ′ i=1 , such that with probability ( 1− δ ) the network G ( x ) contains a subnetwork G̃ ( x ) satisfying ‖G̃ ( x ) − F ( x ) ‖ ≤ ε , for any ε > 0 and δ ∈ ( 0 , 1 ) . We first establish lower bounds on the width of a network of the form g ( x ) = B ( 2 ) σ ( B ( 1 ) x ) such that with probability ( 1 − δ′ ) there exists a subnetwork g̃ ( x ) of g ( x ) s.t . ‖g̃ ( x ) − σ ( Wx ) ‖ ≤ ε′ , for any ε′ > 0 and δ′ ∈ ( 0 , 1 ) . This process is carried out in detail in Lemmas 1 , 2 , and 3 in Appendix B . We have now approximated a single layer FC real-valued network using a subnetwork of a two-layer FC binary network . Hence , we can take ` ′ = 2 ` and Lemma 3 provides lower bounds on the width of each intermediate layer such that with probability ( 1 − δ ) there exists a subnetwork G̃ ( x ) of G ( x ) satisfying ‖G̃ ( x ) − F ( x ) ‖ ≤ ε . This is accomplished in Theorem 2 in Appendix B . To the best of our knowledge this is the first theoretical result proving that a sparse binary-weight DNN that can approximate a real-valued target DNN . As it has been established that real-valued DNNs are universal approximators ( Scarselli & Tsoi , 1998 ) , our result carries the implication that sparse binary-weight DNNs are also universal approximators . In relation to the first result establishing the existence of real-valued subnetworks in a randomly weighted DNN approximating a realvalued target DNN ( Malach et al. , 2020 ) , the lower bound on the width established in Theorem 2 is better than their lower bound of O ( ` 2n2 log ( ` n/δ ) /ε2 ) .

Summary: The paper proposes an innovate method based on lottery ticket hypothesis to prune a BNN (parameters are only -1(0) and +1, it can be viewed as an extreme case of quantization) from a dense NN. It focuses on learning a mask to prune the NN instead of the traditional method (pruning on an already trained network). In addition, not only experiments but theortical proof are given and have a highly brief result.