Recurrent Parameter Generators

1 INTRODUCTION . Deep learning has achieved great success with increasingly more training data and deeper & larger neural networks : A recently developed NLP model , GPT-3 ( Brown et al. , 2020 ) , has astonishingly 175 billion parameters ! While the model performance generally scales with the number of parameters ( Henighan et al. , 2020 ) , with parameters outnumbering training data , the model is significantly over-parameterized . Tremendous effort has been made to reduce the parameter redundancy from different perspectives , including neural network pruning ( LeCun et al. , 1990 ; Han et al. , 2016 ; Liu et al. , 2018 ) , efficient network design spaces ( Howard et al. , 2017 ; Iandola et al. , 2016 ; Sandler et al. , 2018 ) , parameter regularization ( Wan et al. , 2013 ; Wang et al. , 2020a ; Srivastava et al. , 2014 ) , model quantization ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Louizos et al. , 2019 ) , neural architecture search ( Zoph & Le , 2017 ; Cai et al. , 2018 ; Wan et al. , 2020 ) , recurrent models ( Bai et al. , 2019 ; 2020 ; Wei et al. , 2016 ) , multi-task feature encoding ( Ramamonjisoa & Lepetit , 2019 ; Hao et al. , 2021 ) , etc . One of the most prominent approaches in this direction is the pruning-based model compression , which dates back to the late 80s or early 90s ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ) and has enjoyed a resurgence ( Han et al. , 2016 ; Blalock et al. , 2020 ) recently . These pruning methods seek to remove the unimportant parameters from a pre-trained large neural network and can frequently achieve an enormous model-compression ratio . Though sharing a similar motivation to reduce the parameter redundancy , we explore an entirely different territory of model parameter reduction : rather than compressing a large model , we define an arbitrarily large model based on a fixed set of parameters to maximize the model capacity . In this work , we propose to define many different layers in a deep neural network based on a fixed amount of parameters , which we call recurrent parameter generator ( RPG ) . That is , we differentiate the number of model parameters and degrees of freedom ( DoF ) . Traditionally , model parameters are treated independently of each other ; the total number of parameters is the number of DoF . However , by tapping into how a core set of free parameters can be assigned to the neural network model , we can develop a large model of many parameters with a small degree of freedom . In other words , there is excess capacity in neural network models independent of how and where the parameters are used in the network . Even at the level of individual scalar values , parameters can be reused in another arbitrary location of the deep network architecture without significantly impacting model performance . Surprisingly , backpropagation training of a deep network is able to cope with that the same parameter can be assigned to multiple random locations in the network without significantly impacting model performance . Through extensive experiments , we show that a large neural network does not need to be over overparameterized to achieve competitive performance . Particularly , a ResNet18 can be implemented with the number of weights equivalent to one convolution layer in a conventional ResNet ( 4.72× parameter reduction ) and still achieve 67.2 % ImageNet top-1 accuracy . The proposed method is also extremely flexible in reducing the model parameters . In some sense , the proposed RPG method can be viewed as an automatic model parameter reduction technique , which explores the optimal accuracy-parameter trade-off . When we reduce the model parameter , RPG shows graceful performance degradation , and its compression results are frequently on par with the SOTA pruning methods besides the flexibility . Even if we reduce the ResNet18 backbone parameters to 36K , which is about 300× reduction , ResNet18 can still achieve 40.0 % ImageNet top-1 accuracy . Notably , we choose a destructive parameter sharing method ( Cheung et al. , 2019 ) for RPG in this work , which discourages any potential representation sharing from layer to layer . Compared to other recurrent weight-sharing methods , e.g. , convolutional pose machine ( CPM ) or multi-scale deep equilibrium models ( MDEQ ) , our method achieves competitive performance on various benchmarks . Further , we show RPG can be quantized and pruned to improve FLOPs and run time with very tiny accuracy drops . This makes RPG a strong and practical baseline for probing whether there is nontrivial representation sharing within any recurrent network . To summarize , we make the following contributions : 1 . This work provides a new perspective towards automatic model parameter reduction : we can define a neural network with certain DoF constraint and let gradient descent optimization automatically find the best model under the desired constraint . 2 . We propose the recurrent parameter generator ( RPG ) , which decouples the network architecture and the network DoF . Given a certain neural network architecture , we can flexibly choose any DoF to construct the network . 3 . By separating the network architecture from the parameter generator , RPG becomes a tool for us to understand the relationship between the model DoF and the network performance . We observe an empirical log-linear DoF-Accuracy relationship . 2 RELATED WORK . There are many important efforts to compress neural networks or to reduce the redundancy in neural network parameters . We discuss each of the approaches and their relationships to our work . Model Pruning , Neural Architecture Search , and Quantization . Model pruning seeks to remove the unimportant parameters in a trained model . Recently , it ’ s proposed to use neural architecture search as a coarse-grained model pruning ( Yu et al. , 2018 ; Dong & Yang , 2019 ) . Another related effort is neural network quantization ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Louizos et al. , 2019 ) , which seeks to reduce the bits used for each parameter and can frequently reduce the model size by 4× with minimal accuracy drop . More recently , Dollár et al . ( 2021 ) presents a framework for analyzing model scaling strategies that considers network properties such as FLOPs and activations . Parameter Regularization and Priors . Another highly related direction is parameter regularization . Regularization has been widely used to reduce model redundancy ( Krogh & Hertz , 1992 ) , alleviate model overfitting ( Srivastava et al. , 2014 ; Wan et al. , 2013 ) , and ensure desired mathematical regularity ( Wang et al. , 2020a ) . RPG can be viewed as a parameter regularization in the sense that weight sharing poses many equality constraints to weights and regularizes weights to a low-dimensional space . HyperNeat ( Stanley et al. , 2009 ) and CPPNs ( Stanley , 2007 ) use networks to determine the weight between two neurons as a function of their positions . Karaletsos et al . ( 2018 ) and Karaletsos & Bui ( 2020 ) introduced a similar idea by providing a hierarchical prior for network parameters . Recurrent Networks and Deep Equilibrium Models . Recurrence and feedback have been shown in psychology and neuroscience to act as modulators or competitive inhibitors to aid feature grouping ( Gilbert & Sigman , 2007 ) , figure-ground segregation ( Hupé et al. , 1998 ) and object recognition ( Wyatte et al. , 2012 ) . Recurrence-inspired mechanisms also achieve success in feed-forward models . There are two main types of employing recurrence based on if weights are shared across recurrent modules . ResNet ( He et al. , 2016 ) , a representative of reusing similar structures without weight sharing , introduces parallel residual connections and achieves better performance by going deeper in networks . Similarly , some works ( Szegedy et al. , 2015 ; Srivastava et al. , 2015 ) also suggest iteratively injecting thus-far representations to the feed-forward network useful . Stacked inference methods ( Ramakrishna et al. , 2014 ; Wolpert , 1992 ; Weiss & Taskar , 2010 ) are also related while they consider each output in isolation . Several works find sharing weights across recurrent modules beneficial . They demonstrate applications in temporal modelling ( Weiss & Taskar , 2010 ; Xingjian et al. , 2015 ; Karpathy & Fei-Fei , 2015 ) , spatial attention ( Mnih et al. , 2014 ; Butko & Movellan , 2009 ) , pose estimation ( Wei et al. , 2016 ; Carreira et al. , 2016 ) , and so on ( Li et al. , 2016 ; Zamir et al. , 2017 ) . Such methods usually shine in modeling long-term dependencies . In this work , we recurrently share weights across different layers of a feedback network to reduce network redundancy . Given stacking weight-shared modules improve the performance , researchers consider running even infinite depth of such modules by making the sequential modules converge to a fixed point ( LeCun et al. , 1988 ; Bai et al. , 2019 ) . Employing such equilibrium models to existing networks , they show improved performance in many natural language processing ( Bai et al. , 2019 ) and computer vision tasks ( Bai et al. , 2020 ; Wang et al. , 2020b ) . One issue with deep equilibrium models is that the forward and backward propagation usually takes much more iterations than explicit feed-forward networks . Some work ( Fung et al. , 2021 ) improves the efficiency by making the backward propagation Jacobian free . Another issue is that infinite depth and fixed point may not be necessary or even too strict for some tasks . Instead of achieving infinite depth , our model shares parameters to a certain level . We empirically compare with equilibrium models in Section 5 . Efficient Network Space and Matrix Factorization . Convolution is an efficient and structured matrix-vector multiplication . Arguably , the most fundamental idea in building efficient linear systems is matrix factorization . Given the redundancy in deep convolutional neural network parameters , one can leverage the matrix factorization concept , e.g. , factorized convolutions , and design more efficient network classes ( Howard et al. , 2017 ; Iandola et al. , 2016 ; Tan & Le , 2019 ; Sandler et al. , 2018 ) . 3 RECURRENT PARAMETER GENERATOR . We define recurrent parameter generators and show a certain kind of generating matrices that leads to destructive weight sharing . For better parameter capacity , we introduce an even sampling strategy . Recurrent Parameter Generator . Assuming we are constructing a deep convolutional neural network , which contains L different convolution layers . Let K1 , K2 , . . . , KL be the corresponding L convolutional kernels 1 . Rather than using separate sets of parameters for different convolution layers , we create a single set of parameters W ∈ < N and use it to generate the corresponding parameters for each convolution layer : Ki = Ri ·W , i ∈ { 1 , . . . , L } ( 1 ) where Ri is a fixed predefined generating matrix , which is used to generate Ki from W. We call { Ri } and W the recurrent parameter generator ( RPG ) . In this work , we always assume that the size of W is smaller than the total parameters of the model , i.e. , |W| ≤ ∑ i |Ki| . This means an element of W will generally be used in more than one layer of a neural network . Additionally , the gradient of W is a linear superposition of the gradients from each convolution layer . During the neural network training , let ’ s assume convolution kernel Ki receives gradient ∂ ` ∂Ki , where ` is the loss function . Based on the chain rule , it is clear that the gradient of W is : ∂ ` ∂W = L∑ i=1 RTi · ∂ ` ∂Ki ( 2 ) Generating Matrices . There are many different ways to create the generating matrices { Ri } . In this work , we primarily explore the destructive generating matrices , which tend to prevent different kernels from sharing the representation during weight sharing . Destructive Weight Sharing . For easier discussion , let us first look at a special case , where all of the convolutional kernels have the same size and are used in the same shape in the corresponding convolution layers . In other words , { Ri } are square matrices , and the spatial sizes of all of the convolutional kernels have the same size , din × dout ×w× h , and the input channel dimension din is always equal to the output channel dimension dout . In this case , a filter f in a kernel can be treated as a vector in < dwh . Further , we choose Ri to be a block-diagonal matrix Ri = block-diag { Ai , Ai , . . . , Ai } , where Ai ∈ O ( dwh ) is an orthogonal matrix that rotates each filter from the kernel Ki in the same fashion . Similar to the Proposition 2 in ( Cheung et al. , 2019 ) , we show in the Appendix B that : if Ai , Aj are sampled from the O ( M ) Haar distribution and fi , fj are the same filter ( generated by Ri , Rj respectively from W ) from Ki , Kj respectively , then we have E [ 〈fi , fj〉 ] = 0 and E [ 〈 fi‖fi‖ , fj ‖fj‖ 〉 2 ] = 1M . Since M is usually large , the same filter from Ki , Kj are close to orthogonal and generally dissimilar . This shows that even when { Ki } are generated from the same W , they are not sharing the representation . Even though { Ai } are not updated during the training , the size of Ai can be quite large in general . In practice , we can use permutation p ∈ P ( M ) and element-wise random sign reflection b ∈ B ( M ) to construct a subset of the orthogonal group O ( M ) , i.e . we choose Ai ∈ { b ◦ p| b ∈ B ( M ) , p ∈ P ( M ) } .2 Since pseudo-random numbers are used , it takes only two random seeds to store a random permutation and an element-wise random sign reflection . In this we work , we generalize the usage of Ri beyond the block-diagonal generating matrices described above . { Ki } may have different sizes , which can be chosen even larger than the size of W. When Ki ∈ < Ni is larger than W ∈ < N , the corresponding generating matrix Ri is a tall matrix . There are many ways to efficiently create the generating matrices . We use random permutations P ( Ni ) and element-wise random sign reflections B ( Ni ) to create Ri : Ri ∈ { b ◦ p| b ∈ B ( Ni ) , p ∈ P ( Ni ) } , i = 1 , . . . , L ( 3 ) { Ri } tend to lead to destructive weight sharing and lead to better utilization of the parameter capacity . Even Parameter Distribution for Different Layers . While it is easy to randomly sample elements from the W when generating parameters for each layer , it may not be optimal as some elements in 1In this paper , we treat each convolutional kernel as a vector . When the kernel is used to do the convolution , it will be reshaped into the corresponding shape . 2Permutations and element-wise random sign reflection conceptually are subgroups from the orthogonal group , but we shall never use them in the matrix form for the obvious efficiency purpose . W may never be used , and some elements may be used more than average . We use an equalization technique to guarantee all elements of W are evenly sampled . Suppose the size of the W is N , and the total size of parameters of layers to be generated is M , M > N . We first generate b∗cMN arrays { x|x = 1 , .. , N } and concatenate them with ( M mod N ) elements randomly sampled from array { x|x = 1 , .. , M } . We call the concatenated array index array u ∈ < M . We randomly shuffle all elements in u . When initializing each layer ’ s parameter , we sequentially get indices of chosen elements from the shuffled index array u . In this way , each layer ’ s parameters are randomly and evenly sampled from W. We refer to W as model rings since elements are recurrently used in a loop . For data saving efficiency , we just need to save two random seed numbers ( one for sampling ( M mod N ) elements and one for shuffling ) instead of saving the large index array u. Batch Normalization . Model performance is relatively sensitive to the batch normalization parameters . For better performance , each of the convolution layers needs to have its own batch normalization parameters . In general , however , the size of batch normalization is relatively negligible . Yet when W is extremely small ( e.g. , 36K parameters ) , the size of batch normalization should be considered .

Summary:  This paper uses parameter generation to generate parameters for different hidden layers in neural networks. There are several significant strengths and weaknesses in this paper. Especially, the practicability of the proposed method is questionable. It would be good to see my detailed comments below.