On the Existence of Universal Lottery Tickets

1 INTRODUCTION . Deep learning has achieved major breakthroughs in a variety of tasks ( LeCun et al. , 1990 ; Schmidhuber , 2015 ) , yet , it comes at a considerable computational cost ( Sharir et al. , 2020 ) , which is exaggerated by the recent trend towards ever wider and deeper neural network architectures . Reducing the size of the networks before training could therefore significantly broaden the applicability of deep learning , lower its environmental impact , and increase access ( Dhar , 2020 ) . However , such sparse representations are often difficult to learn , as they may not enjoy the benefits associated with over-parameterization ( Belkin et al. , 2019 ) . Frankle & Carbin ( 2019 ) provided a proof of concept that sparse neural network architectures are well trainable if initialized appropriately . Their lottery ticket hypothesis states that a randomlyinitialized network contains a small subnetwork that can compete with the performance of the original network when trained in isolation . Further , Ramanujan et al . ( 2020 ) conjectured the existence of strong lottery tickets , which do not need any further training and achieve competitive performance at their initial parameters . These tickets could thus be obtained by pruning a large randomly initialized deep neural network . Unfortunately , existing pruning algorithms that search for ( strong ) lottery tickets have high computational demands , which are often comparable to or higher than training the original large network . However , Morcos et al . ( 2019 ) posited the existence of so-called universal lottery tickets that , once identified , can be effectively reused across a variety of settings . Contributions .. • In this paper , we formalize the notion of universality , and prove a strong version of the original universal lottery ticket conjecture . Namely , we show that a sufficiently overparameterized , randomly initialized neural network contains a subnetwork that qualifies as a universal lottery ticket without further training of its parameters . Furthermore , it is adapted to a new task only by a linear transformation of its output . This view can explain some empirical observations regarding the required size of universal lottery tickets . • Our proof relies on the explicit construction of basis functions , for which we find sparse neural network representations that benefit from parameter sharing , as it is realized by convolutional neural networks . The fact that these representations are sparse and universal is the most remarkable insight . • To show that they can also be obtained by pruning a larger randomly initialized neural network , we extend existing subset sum results ( Lueker , 1998 ) and develop a proof strategy , which might be of independent interest , as it improves current bounds on pruning for general architectures by making the bounds depth dependent . Accordingly , the width of the large random network can scale as n0 ≥ O ( ntLt/L0 log ( ntL0/ ( Ltϵ ) ) ) to achieve a maximum error ϵ , where Lt denotes the depth and nt the width of the target network , and L0 the depth of the large network . • In support of our existence proofs , we adapt standard parameter initialization techniques to a specific non-zero bias initialization and show in experiments that pruning is feasible in the proposed setting under realistic conditions and for different tasks . Related work . The lottery ticket hypothesis ( Frankle & Carbin , 2019 ) and its strong version ( Ramanujan et al. , 2020 ) have inspired the proposal of a number of pruning algorithms that either prune before ( Wang et al. , 2020 ; Lee et al. , 2019 ; Tanaka et al. , 2020 ) or during and after training ( Frankle & Carbin , 2019 ; Savarese et al. , 2020 ) . Usually , they try to find lottery tickets in the weak sense , with the exception of the edge-popup algorithm ( Ramanujan et al. , 2020 ) that identifies strong lottery tickets , albeit at less extreme sparsity . In general , network compression is a problem that has been studied for a long time and for good reasons , see , e.g. , Lin et al . ( 2020 ) for a recent literature discussion . Here we focus specifically on lottery tickets , whose existence has been proven in the strong sense , thus , they can be derived from sufficiently large , randomly initialized deep neural networks by pruning alone . To obtain these results , recent work has also provided lower bounds for the required width of the large randomly initialized neural network ( Malach et al. , 2020 ; Pensia et al. , 2020 ; Orseau et al. , 2020 ; Fischer & Burkholz , 2021a ; b ) . In addition , it was shown that multiple candidate tickets exist that are also robust to parameter quantization ( Diffenderfer & Kailkhura , 2021 ) . The significant computational cost associated with finding good lottery tickets has motivated the quest for universal tickets that can be transferred to different tasks ( Morcos et al. , 2019 ; Chen et al. , 2020 ) . We prove here their existence . 1.1 NOTATION . For any d-dimensional input x = ( x1 , . . . , xd ) T , let f ( x ) be a fully connected deep neural network with architecture n̄ = [ n0 , n1 , ... , nL ] , i.e. , depth L and widths nl for layers l = 0 , ... , L , with ReLU activation function ϕ ( x ) : = max ( x , 0 ) . An input vector x ( 0 ) is mapped to neurons x ( l ) i as : x ( l ) = ϕ ( h ( l ) ) , h ( l ) = W ( l ) x ( l−1 ) + b ( l ) , W ( l ) ∈ Rnl−1×nl , b ( l ) ∈ Rnl , ( 1 ) where h ( l ) i is called the pre-activation of neuron i , W ( l ) the weight matrix , and b ( l ) the bias vector of layer l. We also write θ for the collection of all parameters θ : = ( ( W ( l ) , b ( l ) ) ) L l=1 and indicate a dependence of f on the parameters by f ( x|θ ) . We also use 1-dimensional convolutional layers , for which the width nl refers to the number of channels in architecture n̄ . For simplicity , we only consider 1-dimensional kernels with stride 1 . Larger kernels could simply be pruned to that size and higher strides could be supported as they are defined so that filters overlap . The purpose of such convolutional layers is to represent a univariate function , which is applied to each input component . Typically , we distinguish three different networks : 1 ) a large ( usually untrained ) deep neural network f0 , which we also call the mother network , 2 ) a smaller target network f , and 3 ) a close approximation , our lottery ticket ( LT ) fϵ , which will correspond to a subnetwork of f0 . fϵ is obtained by pruning f0 , as indicated by a binary mask B = ( bi ) i∈ { 0,1 } |θ0| that specifies for each parameter θi = biθi,0 whether it is set to zero ( bi = 0 ) or inherits the parameter of f0 by θi = θi,0 ( for bi = 1 ) . We usually provide approximation results with respect to the l1-norm ∥x∥ : =∑i |xi| but they hold for any p-norm with p ≥ 1 . C generally stands for a universal constant that can change its value from equation to equation . Its precise value can be determined based on the proofs . Furthermore , we make use of the notation [ n ] : = { 0 , ... , n } for n ∈ N , and [ n ] k for a k-dimensional multi-index with range in [ n ] . 2 UNIVERSAL LOTTERY TICKETS . Before we can prove the existence of strong universal LTs , we have to formalize our notion of what makes a strong LT universal . First of all , a universal LT can not exist in the same way as a strong LT , which is hidden in a randomly initialized deep neural network and is identified by pruning , i.e. , setting a large amount of its parameters to zero while the rest keep their initial value . For a ticket to be universal and thus applicable to a variety of tasks , some of its parameters , if not all , need to be trained . So which parameters should that be ? In deep transfer learning , it is common practice to only train the top layers ( close to the output ) of a large deep neural network . The bottom layers ( close to the input ) are reused and copied from a network that has been already trained successfully to perform a related task . This approach saves significant computational resources and often leads to improved training results . It is therefore reasonable to transfer it to LTs ( Morcos et al. , 2019 ) . Independently from LTs , we discuss conditions when this is a promising approach , i.e. , when the bottom layers of the deep neural network represent multivariate ( basis ) functions , whose linear combination can represent a large class of multivariate functions . The independence of the functions is not required and could be replaced by dictionaries , but the independence aids the compression of the bottom layers and thus our objective to find sparse LTs . This view also provides an explanation of the empirically observed phenomenon that universal tickets achieve good performance across a number of tasks only at moderate sparsity levels and become more universal when trained on larger datasets ( Morcos et al. , 2019 ; Chen et al. , 2020 ) . Including a higher number of basis functions naturally reduces the sparsity of a LT but also makes it adaptable to richer function families . 2.1 HOW UNIVERSAL CAN A LOTTERY TICKET BE ? . A trivial universal ticket . A trivial solution of our problem would be to encode the identity function by the first layers , which would only require 2d or d neurons per layer or even 1-2 neurons per convolutional layer . This would be an extremely sparse ticket , yet , pointless as the ticket does not reduce the hardness of our learning task . In contrast , it can not leverage the full depth of the neural network and needs to rely on shallow function representations . How could we improve the learning task ? The next idea that comes to our mind is to reduce the complexity of the function that has to be learned by the upper layers . For instance , we could restrict it to learn univariate functions . To explore this option , our best chance of success might be to utilize the following theorem . The Kolmogorov-Arnold representation theorem states that every multivariate function can be written as the composition and linear combination of univariate functions . In particular , recent results based on Cantor sets C promise potential for efficient representations . Thm . 2 in ( SchmidtHieber , 2021 ) shows the existence of only two univariate functions g : C → R and ψ : [ 0 , 1 ] → C so that any continuous function f : [ 0 , 1 ] d → R can be written as f ( x ) = g ( ∑d i=1 3 1−iψ ( xi ) ) . Furthermore , only g depends on the function f , while ψ is shared by all functions f and is hence universal . Could ∑d i=1 3 1−iψ ( xi ) be our universal LT ? Unfortunately , it seems to be numerically infeasible to compute for higher input dimensions d > 10 . In addition , the resulting representation of f seems to be sensitive to approximation errors . On top of this , the outer function g is relatively rough even though it inherits some smoothness properties of the function f ( cf . Schmidt-Hieber , 2021 ) and is difficult to learn . Thus , even restricting ourselves to learning an univariate function g in the last layers does not adequately simplify our learning problem . To make meaningful progress in deriving a notion of universal LTs , we therefore need a stronger simplification .

Summary: This paper proves that large enough randomly initialized ReLU networks can contain subnetworks, which can act as universal lottery tickets, that is, only by tuning the last layer of these subnetworks, they can approximate any (reasonably well-behaved) function. The paper does this by showing that large enough randomly initialized ReLU networks contain subnetworks such that their last hidden layer can act as the basis for a class of functions, and hence the output layer can simply take a linear combination of the neurons of the last hidden layer to approximate any function in that class. The paper proves this for polynomial and Fourier basis. Another key contribution of this paper is that it demonstrates that large depth can be leveraged to reduce the width needed for the strong lottery ticket hypothesis to hold.