Reducing the number of neurons of Deep ReLU Networks based on the current theory of Regularization

We introduce a new Reduction Algorithm which makes use of the properties of ReLU neurons to reduce significantly the number of neurons in a trained Deep Neural Network . This algorithm is based on the recent theory of implicit and explicit regularization in Deep ReLU Networks from ( Maennel et al , 2018 ) and the authors . We discuss two experiments which illustrate the efficiency of the algorithm to reduce the number of neurons significantly with provably almost no change of the learned function within the training data ( and therefore almost no loss in accuracy ) . 1 INTRODUCTION . 1.1 MOTIVATION . In this work , we investigate a particular type of deep neural network . Its architecture ( see section 2 ) can be better understood , thanks to the previous work on wide shallow neural networks : Neyshabur et al . ( 2014 ) ; Ongie et al . ( 2019 ) ; Savarese et al . ( 2019 ) ; Williams et al . ( 2019 ) ; Maennel et al . ( 2018 ) ; Heiss et al . ( 2019 ) and unpublished work of the authors on deep neural networks ( with arbitrarily many inputs and outputs ) . These results state that ` 2 weight regularization on parameter space is equivalent to L1-typed Pfunctionals on function space under certain conditions . This implies that the optimal function could also be represented by finitely many neurons ( Rosset et al. , 2007 ) . With the knowledge of these properties , we were able to design a reduction algorithm which can reduce infinitely wide ( in practice : arbitrarily wide ) layers in our architecture to much smaller layers . This allows us to reduce the number of neurons by 90 % to 99 % without introducing sparsity ( which allows more efficient GPU-implementation ( Gale et al. , 2020 ) ) and with almost no loss in accuracy . This can be of interest for deploying neural networks on small devices or for making predictions which are computationally less costly and less energy consuming . 1.2 LITERATURE / LINK TO OTHER RESEARCH . Many papers have been written on the subject of reducing neural networks . There is the approach of weight pruning , by removing the least salient weights ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Han et al. , 2015 ; Tanaka et al. , 2020 ) . A different technique is pruning neurons Mariet & Sra ( 2015 ) ; He et al . ( 2014 ) ; Srinivas & Babu ( 2015 ) , which does not introduce sparsity in the network by removing single weights , but reduces the number of neurons . For CNNs there are ways to prune the filters ( Li et al. , 2016 ) . In transfer learning , one can prune the weights with decreasing magnitude ( Sanh et al. , 2020 ) . All these techniques require the same steps : train a large network , prune and update remaining weights or neurons , retrain . And for too much pruning , the accuracy of the pruned models drops significantly , also it might not always be useful to fine-tune the pruned models ( Liu et al. , 2018 ) . Another approach is knowledge distillation ( Hinton et al. , 2015 ; Ba & Caruana , 2014 ) where one establishes a teacher/student relation between a complex and simpler network . The lottery ticket hypothesis ( Frankle & Carbin , 2018 ) states that ” a randomly-initialized , dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations ” . In this work , the method can be related to neuron pruning , in that we are working directly on a large already-trained network . We are , however , trying to preserve the learned function contrary to the cited techniques which focus on the loss function and where pruning results in a different learned function . Therefore , in our algorithm , the neurons are not only pruned but rather condensed , put together into new neurons which contain all the information learned during training . Our method , hence , does not require retraining . But it is beneficial to further retrain the network , and reduce it again , in an iterative process . 2 DESCRIPTION OF THE ARCHITECTURE . Starting from a traditional Shallow Neural Network with ReLU activation function ( see fig . 1 ) ( which contains a single hidden layer ) and is ` 2 regularized , we will define two variants of a One Stack Network . First , by adding a direct ( or skip ) connection between the input layer and the output layer ( see fig . 2 ) , one can obtain the simplified One Stack Network . Second , by adding a layer in the middle of this direct ( skip ) connection , one can get a One Stack Network ( see fig . 3 ) . This new layer contains neurons with a linear activation function ( the identity function multiplied by a constant ) . It contains as many neurons as the minimum between the number of neurons in the input layer and the number of neurons in the output layer , it also has no bias . Furthermore , we call it the affine layer and the new weights before and after it the affine weights . These new weights can also be ` 2 regularized , but typically by a different hyperparameter than the non-linear weights . The architecture that we are going to study can be described as a sequence of stacks , or a Deep Stack Network . We repeat the pattern described above ( see figs . 4 and 5 ) . Since the output layer is at the end of the architecture , we call all intermediate layers related to the output layers as introduced earlier ( typically containing few neurons dj ) : bottlenecks . The bottlenecks contain neurons with a linear ( identity ) activation function . For every stack , all parameters are regularized except for the biases in the bottleneck . We recommend using a higher learning rate for the affine weights . In our experiments , we used a ten-times higher learning rate for the affine weights . For every stack j , the weights and biases connecting the previous input to the ( infinitely-wide nj → ∞ ) ReLU layer are written as V ( j ) ∈ Rnj×dj−1 and b ( j ) ∈ Rnj , the ones connecting to the bottleneck are written as W ( j ) ∈ Rdj×nj and c ( j ) ∈ Rdj . Remark 2.1 . ( ordinary fully connected deep neural network ) Note that our algorithm could also be applied to an ordinary fully connected deep ReLU neural network and would output a neural network with skip connections . But if every second layer is much wider ( dj nj ) , we have a much better theoretical understanding of what is happening and typically we can get better test accuracy with fewer neurons . We don ’ t use ReLUs on the bottlenecks for easier interpretation . The compression-rate would not suffer if we apply our algorithm on architectures without affine weights . 3 CONSEQUENCES FROM P-FUNCTIONAL REGULARIZATION THEORY . If we choose the numbers nj of hidden nodes sufficiently large for each stack while we keep finite bottleneck dimensions dj the authors have shown in an unpublished paper that the solution of the optimization problem NN θ∗λ̃ with θ ∗ , λ̃ ∈ argmin θ ( L ( NN θ ) + λ̃ ‖θ‖22 ) ( 1 ) can be characterized as the solution of a much easier to interpret optimization problem NN θ∗λ̃ ∈ argmin f ( L ( f ) + λP ( f ) ) , ( 2 ) where we optimize over all continuous functions f so we do not have to care about if f is representable as a neural network.1 According to the unpublished paper by the authors ( similarly to the work by Neyshabur et al . ( 2014 ) ; Ongie et al . ( 2019 ) ; Savarese et al . ( 2019 ) ; Williams et al . ( 2019 ) ; Heiss et al . ( 2019 ) ) , we can obtain the P -functional P of the deep stacked neural networks from the P -functional Pj of a single stacks , P ( f ) = inf ( f1 , ... f # stacks ) , s.t . f=f # stacks◦···◦f1 ( P1 ( f1 ) + P2 ( f2 ) + · · ·+ P # stacks ( f # stacks ) ) . ( 3 ) 1For some functions P ( f ) = ∞ . E.g . idR2 : R2 → R2 : x 7→ x results in P ( idR2 ) = ∞ if one of the bottlenecks has dimension less than two i.e . ∃j ∈ { 1 , . . . , # stacks− 1 } : dj = 1 ) For a single wide shallow neural network we get2 Pj ( fj ) : = min ϕ∈T , c∈Rdj s.t . fj= ∫ S dj−1−1 ϕs ( 〈s , ·〉 ) ds +c ∫ Sdj−1−1 ∫ R ∥∥∥ϕs ( r ) ′′∥∥∥ 2 g ( r ) dr ds+ ‖c‖22 , where ( 4 ) T : = { ϕ ∈ C ( R , Rdj ) S dj−1−1 ∣∣∣∣∀s ∈ Sd−1 : limr→−∞ϕs ( r ) = 0 and limr→+∞ ∂∂rϕs ( r ) = 0 } handles the boundary condition , Sd−1 denoted the ( d − 1 ) -dimensional unit sphere and we have a weighing function3 g ( r ) = 1√ r2+1 . For the other types of stacks shown in Figures 2 and 3 the corresponding P -functionals would only need minor modifications . If one would not regularize the affine weights ( skip connections ) the theory would drastically change because P would assign 0 to many highly oscillating functions if # stacks ≥ 2 , but in practice implicit regularization would weaken this effect . For this paper , the most important aspect of eqs . ( 3 ) and ( 4 ) is that there is ‖·‖2 ( instead of ‖·‖ 2 2 ) 4 inside the integral , which reminds very much of an L1-norm . Therefore the function optimizing eq . ( 2 ) can be represented by a finite number of neurons ( Rosset et al. , 2007 ) . Similarily Maennel et al . ( 2018 ) found that also without explicit regularization gradient decent favours solutions which could be represented by much smaller networks as well . In the following we observe that this finite numbers nj can be very low in practice . 4 REDUCTION ALGORITHM . We will now consider a one-stack network according to the description of the architecture given above ( see fig . 3 ) . Therefore , we can fix j and write W : =W ( j ) for example in this case . Then , the reduction can be applied similarly on every stack . 4.1 REMOVING AND REPLACING OUTSIDE NEURONS . The main idea of this step is to combine the information of all the non-linear ( ReLU ) neurons whose kink positions are outside the convex hull of the previous stack ’ s representation of the training data into the affine neurons . We define the outside neurons as those neurons that will always return a zero or always return strictly positive numbers for all training data points . Therefore we find two cases . Either the activation function of the outside neuron acts as a zero function in which case we can simply remove it from the ReLU layer as it makes no contribution to the learned function , or the activation function of the outside neuron acts as an identity function . In the latter case , the contribution of the neuron to the learned function is affine and we can add it to the affine weights and remove it from the ReLU layer . In order to do this , we will define Waffine and caffine : 2Since ϕs ( r ) ′′ can be a distribution instead of an function we use a generalized notion of integration—e.g . ∫ R ∥∥ ( α , β ) > δ0 ( r ) ∥∥2 dr = √α2 + β2 . More generally we redefine these integrals∫ A h ( ‖f ( r ) ‖2 ) dr : = limε→0+ ∫ A h ( ∥∥f ( r ) ∗ 1 2ε 1 [ −ε , ε ] ( r ) ∥∥ 2 ) dr with the help of a convolution which in- finitesimally smoothens the function or distribution f. This results for example in ∫ R ‖δ0 ( r ) ‖ 2 2 dr = ∞ and∫ R ∥∥∥∥∥wk max ( 0 , 〈 vk , vk ‖vk‖2 r 〉 +bk ) ′′∥∥∥∥∥ 2 g ( r ) dr = ‖vk‖2‖wk‖2 g ( −bk ‖vk‖2 ) . 3If the biases b ( j ) are not regularized one could omit the weighting function in theory , but because of implicit regularization one would still observe the qualitative phenomena that second derivative gets more expensive far away from the origin . 4If V and b were fixed randomly and only W ( and c ) were trained , there would be a square in the integral ( Heiss et al. , 2019 ) . Waffine = ∑ k∈M wk vk caffine = ∑ k∈M bk wk , ( 5 ) where M is the set of outside neurons whose activation function is positive for all training data points . Note that we can remove the neurons whose activation function acts as a zero function without any replacement since they do not contribute to the learned function . Note as well that wk are column vectors and vk are row vectors and their multiplication in that order gives a matrix . Therefore , Waffine is the result of a sum of matrices and caffine is a vector , both summarize the contributions of all outside neurons to the learned function . 5 We are now finally able to remove the outside neurons and add their contributions to the affine weights . The question we ask ourselves now is : how do we determine which neuron is an outside neuron ? We decided to loop through every ReLU neuron , then check their prediction for every training data point . If the prediction was always zero or always strictly positive , we could be sure that we found an outside neuron . Of course such a method is also computationally expensive , and would hence be very impractical if we had many neurons and many training data points . But we observed empirically that it was not necessary to loop through every training data point ( more on this in the experimentation chapter ) . Remark 4.1 . Exactly speaking , the learned function will not be the same as before . In fact , it will be different outside of the convex hull of the training data . But these changes further smoothen the extrapolation behavior of the learned function without affecting training accuracy . Removing the outside neurons does not change the function on the training data and it does not change the stack ’ s function inside the convex hull of the previous stack ’ s representation of the training data .

Summary: The paper suggests a pruning technique specific to ReLU networks by taking advantage of activation patterns and the separating hyperplanes. The technique consists of three steps: : (1) remove neurons that are never active and combine neurons that are always active, (2) remove neurons with little contribution to the output, and (3) use weighted k-means to combine other neurons. The method is evaluated on specific toy problems and the MNIST dataset.