Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks

1 INTRODUCTION . Many impressive successes in machine learning are reached through neural network architectures with a huge number of trainable parameters . Consequently , substantial research in the field aims at reducing the size of such networks while maintaining good accuracy , e.g . for deployment in resource constrained devices ( Yang et al. , 2017 ) . A major empirical fact of such endeavour is the contrast between the initial model overparametrization , which appears necessary for effective training , and the extent to which the size of the resulting model can be reduced through compression techniques . Among the latter , pruning methods appear as a mature and efficient way of achieving significant compression , often without incurring any accuracy loss ( Blalock et al. , 2020 ) . Recently , the aforementioned contrast between the initial and final number of parameters has been addressed by the lottery ticket hypothesis ( Frankle & Carbin , 2019 ) , or LTH for short , which states that any randomly initialized network contains lottery tickets , that is , sparse subnetworks that can be trained just once and reach the performance of the fully-trained original network . This hypothesis was first verified experimentally , leveraging pruning methods to identify the lottery tickets ( Frankle & Carbin , 2019 ; Lee et al. , 2019 ) . ( Ramanujan et al. , 2020 ) then proposed a stronger version of the hypothesis , named strong lottery ticket hypothesis ( SLTH ) by ( Pensia et al. , 2020 ) : it stipulates that a network with random weights contains , with high probability , sub-networks that can approximate any given sufficiently-smaller neural network . In other words , a sufficiently large and randomly initialized network that can be successfully trained for a task , could instead be suitably pruned to obtain a network that , even without training , achieves good accuracy . Experimental support for this stronger version were reported by Ramanujan et al . ( 2020 ) ; Zhou et al . ( 2019 ) ; Wang et al . ( 2020 ) , making use of pruning methods to find lottery tickets in a range of architectures , including convolutional neural networks ( CNNs ) . A first rigorous proof of the SLTH has been given by Malach et al . ( 2020 ) for the case of dense networks ( i.e . consisting of fully connected layers ) . Pensia et al . ( 2020 ) and Orseau et al . ( 2020 ) successively improved this result by showing that logarithmic over-parametrization is sufficient . Their results are also restricted to dense networks and they leave as an open problem to extend it to CNNs . Our contributions . We extend and complete the proof of the SLTH ( and thus , also , of the LTH ) , for classical network architectures which can combine convolutional and fully connected layers . More precisely , we prove that any CNN with given weights can be approximated by pruning a CNN with random weights ( random CNN for short ) , with the latter being larger than the former by a logarithmic factor . We also provide basic experiments showing that starting from a random CNN which is roughly 30 times larger than LeNet5 , it is possible to compute in few hours a pruning mask that allows to approximate the trained convolutional layers of LeNet5 with relative error 10−3 , even when ignoring some hypothesis of our theoretical result . Our theoretical analysis follows the approach of Malach et al . ( 2020 ) and make use of two layers to approximate one . We borrow from Pensia et al . ( 2020 ) the use of random subset sum ( RSS ) ( Lueker , 1998 ) to approximate a given weight via the sum of a subset of a sample of random weights , and carefully design instances of RSS via a combination of two convolutional layers . By controlling the error accumulated by each layer with Young ’ s convolution inequality , we establish the following result . Informal version of Theorem 1 . Given ε , δ > 0 , any CNN with k parameters and ` layers , and kernels with ` 1 norm at most 1 , can be approximated within ε by pruning a random CNN withO ( k log ( k ` min { ε , δ } ) ) parameters and 2 ` layers with probability at least 1− δ . This result generalizes those by Pensia et al . ( 2020 ) , Orseau et al . ( 2020 ) and Malach et al . ( 2020 ) , as the dense-network case is retrieved as a special case of our theorem , when the kernel size matches the input size . Roadmap . After discussing related work in the next section , we state our theoretical results alongside a high-level idea of the proofs . Successively , we report our experimental results . Finally , in Section 4 , we provide detailed proofs of our statements . 1.1 RELATED WORK . Pruning methods are classical neural network compression strategies that date back to the 80 ’ s ( LeCun et al. , 1989b ; Mozer & Smolensky , 1988 ) . We recommend the recent survey Blalock et al . ( 2020 ) for an overview of the current state of research on these techniques . As for the lottery ticket hypothesis , Lange ( 2020 ) provides a survey which summarizes the progress on the topic until the results by Malach et al . ( 2020 ) . In the following we briefly mention works which are not discussed in Lange ( 2020 ) . Cosentino et al . ( 2019 ) shows that lottery tickets can be adversarially trained , yielding sparse and robust neural networks . Soelen & Sheppard ( 2019 ) shows that lottery tickets are transferable , in the sense of showing remarkable accuracy for tasks other than the original one for which they have been found . Sabatelli et al . ( 2021 ) further shows that minimal retraining on a new task allows lottery tickets to often achieve better generalization than models trained ad-hoc for the task . ( Yu et al. , 2020 ) empirically supports that the LTH holds also in the context of reinforcement learning and natural language processing . ( Diffenderfer & Kailkhura , 2021 ) shows that lottery tickets are also robust against extreme quantization of the weights . Aladago & Torresani ( 2021 ) provides a method to train networks where each initial weight is restricted to few possible random values . An extreme case of the latter is to share only a single ( random ) value among all weights , and focus the training solely on finding the best architecture ( Gaier & Ha , 2019 ) . Our work also relates to recent papers investigating properties of random CNNs , such as Ulyanov et al . ( 2020 ) which observes that random CNNs already seem to capture some natural image statistics required for tasks such as de-noising and inpainting . 2 THEORETICAL RESULTS . We start by introducing some of our notation . The rest of it follows Goodfellow et al . ( 2016 ) with minor modifications , so we defer a full description to Section 4.1 . Given n ∈ N , we denote the set { 1 , . . . , n } by [ n ] . The symbol ∗ represents the convolution operation , represents the element-wise ( Hadamard ) product , and σ represents ReLU activation function . Finally , the notation ‖·‖1 refers to the sum of the absolute values of each entry in a tensor while ‖·‖max denotes the maximum norm : the maximum among the absolute value of each entry . Those are akin to vector norms and should not be confused with operator norms . We restrict our setting to convolutional neural networks f : [ 0 , 1 ] D×D×c0 → RD×D×c ` of the form f ( X ) = K ` ∗ σ ( K ` −1 ∗ · · ·σ ( K1 ∗ X ) ) , where Ki ∈ Rdi×di×ci−1×ci , and the convolutions have no bias and are suitably padded with zeros . The restrictions on tensor sizes and the exclusion of bias terms1 aim only to ease presentation . Our initial goal is to approximate a convolution with a single kernel , as depicted in Figure 1 , using convolutions with ( pruned ) random kernels . We achieve this by the means of the structure presented in Figure 2 , using two convolutions with random tensors . Lemma 1 asserts that , with high probability , we can prune this structure to approximate the output of a convolution with any given kernel as long as the amount of kernels is large enough . Lemma 1 ( Single kernel ) . Let D , d , c , n ∈ N , and ε , C ∈ R > 0 , where n ≥ C log d 2c ε , U ∈ Rd×d×c×n , V ∈ R1×1×n×1 , and S ∈ { 0 , 1 } d×d×c×n . Let us further assume that the entries of U and V are i.i.d . random variables following a Unif ( [ −1 , 1 ] ) distribution , and define g ( X ) = V ∗ σ ( U ∗ X ) and gS ( X ) = V ∗ σ ( ( U S ) ∗ X ) thus yielding a random CNN g : [ 0 , 1 ] D×D×c → RD×D×1 and its pruned version gS w.r.t . the mask S. Then , we can choose constant C independently from other parameters so that , with probability at least 1− ε , for all K ∈ [ −1 , 1 ] d×d×c×1 with ‖K‖1 ≤ 1 , there exists a pruning mask S such that sup X∈ [ 0,1 ] D×D×c ∥∥K ∗ X− gS ( X ) ∥∥max < ε . Proof idea . We leverage the absence of negative entries in the input and an initial pruning of U to bypass the ReLU non-linearity . This allows us to virtually replace the operations in g by a single 1If biases are present , the structures used in the proofs also puts them in a RSS configuration . Thus the results can be readily adapted by replacing the d2i terms by d 2 i + 1. convolution with a random kernel obtained by combining U and V. Each entry of this resulting kernel is the sum of n random variables , where we can choose to include/exclude each term in the sum by choosing to keep/prune the relevant weights . We finish the proof by applying Theorem 2 to conclude that n variables suffice to approximate all entries , simultaneously , with enough precision to ensure the thesis . We now extend Lemma 1 to an entire layer . As before , a detailed proof is provided in Section 4.3 . Lemma 2 ( Convolutional Layer ) . Let D , d , c0 , c1 , n ∈ N , and ε , C ∈ R > 0 , where n ≥ C c1 log d2c0c1 ε , U ∈ Rd×d×c0×n , V ∈ R1×1×n×c1 , S ∈ { 0 , 1 } d×d×c0×n and T ∈ { 0 , 1 } 1×1× [ n ] × [ c1 ] . Let us further assume that the entries of U and V are i.i.d . random variables following a Unif ( [ −1 , 1 ] ) distribution , and define g ( X ) = V ∗ σ ( U ∗ X ) and gT , S ( X ) = ( V T ) ∗ σ ( ( U S ) ∗ X ) thus yielding a random CNN g : [ 0 , 1 ] D×D×c0 → RD×D×c1 and its pruned version gT , S ( X ) w.r.t . the masks S and T. Then , we can choose constant C independently from other parameters so that , with probability at least 1− ε , for all K ∈ [ −1 , 1 ] d×d×c0×c1 with ‖K‖1 ≤ 1 , there exist masks S and T such that sup X∈ [ 0,1 ] D×D×c0 ‖K ∗ X− gT , S ( X ) ‖max < ε . Proof Idea . The lemma follows by applying Lemma 1 to each kernel independently , so that all of them are approximated by a factor at most εc1 ; the latter approximation allows to apply the union bound so that the desired approximation holds simultaneously for all c1 output kernels with probability at least 1− ε . We are now ready to extend the previous approximation result from a single layer to the entire network , thus proving our main result . A detailed proof is given in Section B. Theorem 1 ( Convolutional Network ) . Let D , d , c0 , c1 , ` ∈ N , and ε , C ∈ R > 0 . For each i ∈ [ ` ] , let L2i−1 and L2i be tensors with size di × di × ci−1 × ( Cci log ci−1cid 2 i ` min { ε , δ } ) and 1 × 1 × ( Cci log ci−1d 2 i ci ` min { ε , δ } ) ×ci , respectively . Furthermore , for each i ∈ [ 2 ` ] , let S i ∈ { 0 , 1 } size ( Li ) . Finally , let us assume that the entries of L1 , ... , L2 ` are i.i.d . random variables following a Unif ( [ −1 , 1 ] ) distribution , and define g ( X ) = L2 ` ∗ σ ( L2 ` −1 ∗ · · ·σ ( L1 ∗ X ) ) and gS1 , ... , S2 ` ( X ) = ( L 2 ` S2 ` ) ∗ σ [ ( L2 ` −1 S2 ` −1 ) ∗ · · ·σ [ ( L1 S1 ) ∗ X ] ] thus yielding a random 2 ` -layer CNN g : [ 0 , 1 ] D×D×c0 → RD×D×c1 and its pruned version gS1 , ... , S2 ` ( X ) w.r.t . masks S 1 , ... , S2 ` . Let F be the class of functions from [ 0 , 1 ] D×D×c0 to RD×D×c ` such that , for each f ∈ F f ( X ) = K ` ∗ σ ( K ` −1 ∗ · · ·σ ( K1 ∗ X ) ) , where , for each i ∈ [ ` ] , Ki ∈ [ −1 , 1 ] di×di×ci−1×ci and ‖Ki‖1 ≤ 1 . Then , we can choose constant C independently from other parameters so that , with probability at least 1− δ , the following holds for every f ∈ F : inf ∀i∈ [ 2 ` ] , Si∈ { 0,1 } size ( Li ) sup X∈ [ 0,1 ] D×D×c0 ∥∥∥f ( X ) − gS1 , ... , S2 ` ( X ) ∥∥∥ max < ε . Proof Idea . The proof leverages Lemma 2 in an analogous way to how the latter relied on Lemma 1 ; namely , we apply Lemma 2 by requiring an approximation factor that guarantees , with sufficient probability , that a suitable approximation is reached across all layers simultaneously . The latter requirement is responsible for the ` factor which appears in the logarithms of the dimensions of each random tensor Li .

Summary: The authors provide theoretical analysis on the lottery ticket hypothesis for convolutional neural networks. The techniques of the random subset sum are used to prove theorems. The authors start with introducing the case of a single convolutional kernel and then generalize to the case of the whole convolutional neural networks. By specifying multiple constraints and assumptions on the models' parameters and inputs, the error between the approximated and the true convolution results can be limited to a small number.