Recurrent Parameter Generators

1 INTRODUCTION . Deep learning has achieved great success with increasingly more training data and deeper & larger neural networks : A recently developed NLP model , GPT-3 ( Brown et al. , 2020 ) , has astonishingly 175 billion parameters ! While the model performance generally scales with the number of parameters ( Henighan et al. , 2020 ) , with parameters outnumbering training data , the model is significantly over-parameterized . Tremendous effort has been made to reduce the parameter redundancy from different perspectives , including neural network pruning ( LeCun et al. , 1990 ; Han et al. , 2016 ; Liu et al. , 2018 ) , efficient network design spaces ( Howard et al. , 2017 ; Iandola et al. , 2016 ; Sandler et al. , 2018 ) , parameter regularization ( Wan et al. , 2013 ; Wang et al. , 2020a ; Srivastava et al. , 2014 ) , model quantization ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Louizos et al. , 2019 ) , neural architecture search ( Zoph & Le , 2017 ; Cai et al. , 2018 ; Wan et al. , 2020 ) , recurrent models ( Bai et al. , 2019 ; 2020 ; Wei et al. , 2016 ) , multi-task feature encoding ( Ramamonjisoa & Lepetit , 2019 ; Hao et al. , 2021 ) , etc . One of the most prominent approaches in this direction is the pruning-based model compression , which dates back to the late 80s or early 90s ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ) and has enjoyed a resurgence ( Han et al. , 2016 ; Blalock et al. , 2020 ) recently . These pruning methods seek to remove the unimportant parameters from a pre-trained large neural network and can frequently achieve an enormous model-compression ratio . Though sharing a similar motivation to reduce the parameter redundancy , we explore an entirely different territory of model parameter reduction : rather than compressing a large model , we define an arbitrarily large model based on a fixed set of parameters to maximize the model capacity . In this work , we propose to define many different layers in a deep neural network based on a fixed amount of parameters , which we call recurrent parameter generator ( RPG ) . That is , we differentiate the number of model parameters and degrees of freedom ( DoF ) . Traditionally , model parameters are treated independently of each other ; the total number of parameters is the number of DoF . However , by tapping into how a core set of free parameters can be assigned to the neural network model , we can develop a large model of many parameters with a small degree of freedom . In other words , there is excess capacity in neural network models independent of how and where the parameters are used in the network . Even at the level of individual scalar values , parameters can be reused in another arbitrary location of the deep network architecture without significantly impacting model performance . Surprisingly , backpropagation training of a deep network is able to cope with that the same parameter can be assigned to multiple random locations in the network without significantly impacting model performance . Through extensive experiments , we show that a large neural network does not need to be over overparameterized to achieve competitive performance . Particularly , a ResNet18 can be implemented with the number of weights equivalent to one convolution layer in a conventional ResNet ( 4.72× parameter reduction ) and still achieve 67.2 % ImageNet top-1 accuracy . The proposed method is also extremely flexible in reducing the model parameters . In some sense , the proposed RPG method can be viewed as an automatic model parameter reduction technique , which explores the optimal accuracy-parameter trade-off . When we reduce the model parameter , RPG shows graceful performance degradation , and its compression results are frequently on par with the SOTA pruning methods besides the flexibility . Even if we reduce the ResNet18 backbone parameters to 36K , which is about 300× reduction , ResNet18 can still achieve 40.0 % ImageNet top-1 accuracy . Notably , we choose a destructive parameter sharing method ( Cheung et al. , 2019 ) for RPG in this work , which discourages any potential representation sharing from layer to layer . Compared to other recurrent weight-sharing methods , e.g. , convolutional pose machine ( CPM ) or multi-scale deep equilibrium models ( MDEQ ) , our method achieves competitive performance on various benchmarks . Further , we show RPG can be quantized and pruned to improve FLOPs and run time with very tiny accuracy drops . This makes RPG a strong and practical baseline for probing whether there is nontrivial representation sharing within any recurrent network . To summarize , we make the following contributions : 1 . This work provides a new perspective towards automatic model parameter reduction : we can define a neural network with certain DoF constraint and let gradient descent optimization automatically find the best model under the desired constraint . 2 . We propose the recurrent parameter generator ( RPG ) , which decouples the network architecture and the network DoF . Given a certain neural network architecture , we can flexibly choose any DoF to construct the network . 3 . By separating the network architecture from the parameter generator , RPG becomes a tool for us to understand the relationship between the model DoF and the network performance . We observe an empirical log-linear DoF-Accuracy relationship . 2 RELATED WORK . There are many important efforts to compress neural networks or to reduce the redundancy in neural network parameters . We discuss each of the approaches and their relationships to our work . Model Pruning , Neural Architecture Search , and Quantization . Model pruning seeks to remove the unimportant parameters in a trained model . Recently , it ’ s proposed to use neural architecture search as a coarse-grained model pruning ( Yu et al. , 2018 ; Dong & Yang , 2019 ) . Another related effort is neural network quantization ( Hubara et al. , 2017 ; Rastegari et al. , 2016 ; Louizos et al. , 2019 ) , which seeks to reduce the bits used for each parameter and can frequently reduce the model size by 4× with minimal accuracy drop . More recently , Dollár et al . ( 2021 ) presents a framework for analyzing model scaling strategies that considers network properties such as FLOPs and activations . Parameter Regularization and Priors . Another highly related direction is parameter regularization . Regularization has been widely used to reduce model redundancy ( Krogh & Hertz , 1992 ) , alleviate model overfitting ( Srivastava et al. , 2014 ; Wan et al. , 2013 ) , and ensure desired mathematical regularity ( Wang et al. , 2020a ) . RPG can be viewed as a parameter regularization in the sense that weight sharing poses many equality constraints to weights and regularizes weights to a low-dimensional space . HyperNeat ( Stanley et al. , 2009 ) and CPPNs ( Stanley , 2007 ) use networks to determine the weight between two neurons as a function of their positions . Karaletsos et al . ( 2018 ) and Karaletsos & Bui ( 2020 ) introduced a similar idea by providing a hierarchical prior for network parameters . Recurrent Networks and Deep Equilibrium Models . Recurrence and feedback have been shown in psychology and neuroscience to act as modulators or competitive inhibitors to aid feature grouping ( Gilbert & Sigman , 2007 ) , figure-ground segregation ( Hupé et al. , 1998 ) and object recognition ( Wyatte et al. , 2012 ) . Recurrence-inspired mechanisms also achieve success in feed-forward models . There are two main types of employing recurrence based on if weights are shared across recurrent modules . ResNet ( He et al. , 2016 ) , a representative of reusing similar structures without weight sharing , introduces parallel residual connections and achieves better performance by going deeper in networks . Similarly , some works ( Szegedy et al. , 2015 ; Srivastava et al. , 2015 ) also suggest iteratively injecting thus-far representations to the feed-forward network useful . Stacked inference methods ( Ramakrishna et al. , 2014 ; Wolpert , 1992 ; Weiss & Taskar , 2010 ) are also related while they consider each output in isolation . Several works find sharing weights across recurrent modules beneficial . They demonstrate applications in temporal modelling ( Weiss & Taskar , 2010 ; Xingjian et al. , 2015 ; Karpathy & Fei-Fei , 2015 ) , spatial attention ( Mnih et al. , 2014 ; Butko & Movellan , 2009 ) , pose estimation ( Wei et al. , 2016 ; Carreira et al. , 2016 ) , and so on ( Li et al. , 2016 ; Zamir et al. , 2017 ) . Such methods usually shine in modeling long-term dependencies . In this work , we recurrently share weights across different layers of a feedback network to reduce network redundancy . Given stacking weight-shared modules improve the performance , researchers consider running even infinite depth of such modules by making the sequential modules converge to a fixed point ( LeCun et al. , 1988 ; Bai et al. , 2019 ) . Employing such equilibrium models to existing networks , they show improved performance in many natural language processing ( Bai et al. , 2019 ) and computer vision tasks ( Bai et al. , 2020 ; Wang et al. , 2020b ) . One issue with deep equilibrium models is that the forward and backward propagation usually takes much more iterations than explicit feed-forward networks . Some work ( Fung et al. , 2021 ) improves the efficiency by making the backward propagation Jacobian free . Another issue is that infinite depth and fixed point may not be necessary or even too strict for some tasks . Instead of achieving infinite depth , our model shares parameters to a certain level . We empirically compare with equilibrium models in Section 5 . Efficient Network Space and Matrix Factorization . Convolution is an efficient and structured matrix-vector multiplication . Arguably , the most fundamental idea in building efficient linear systems is matrix factorization . Given the redundancy in deep convolutional neural network parameters , one can leverage the matrix factorization concept , e.g. , factorized convolutions , and design more efficient network classes ( Howard et al. , 2017 ; Iandola et al. , 2016 ; Tan & Le , 2019 ; Sandler et al. , 2018 ) . 3 RECURRENT PARAMETER GENERATOR . We define recurrent parameter generators and show a certain kind of generating matrices that leads to destructive weight sharing . For better parameter capacity , we introduce an even sampling strategy . Recurrent Parameter Generator . Assuming we are constructing a deep convolutional neural network , which contains L different convolution layers . Let K1 , K2 , . . . , KL be the corresponding L convolutional kernels 1 . Rather than using separate sets of parameters for different convolution layers , we create a single set of parameters W ∈ < N and use it to generate the corresponding parameters for each convolution layer : Ki = Ri ·W , i ∈ { 1 , . . . , L } ( 1 ) where Ri is a fixed predefined generating matrix , which is used to generate Ki from W. We call { Ri } and W the recurrent parameter generator ( RPG ) . In this work , we always assume that the size of W is smaller than the total parameters of the model , i.e. , |W| ≤ ∑ i |Ki| . This means an element of W will generally be used in more than one layer of a neural network . Additionally , the gradient of W is a linear superposition of the gradients from each convolution layer . During the neural network training , let ’ s assume convolution kernel Ki receives gradient ∂ ` ∂Ki , where ` is the loss function . Based on the chain rule , it is clear that the gradient of W is : ∂ ` ∂W = L∑ i=1 RTi · ∂ ` ∂Ki ( 2 ) Generating Matrices . There are many different ways to create the generating matrices { Ri } . In this work , we primarily explore the destructive generating matrices , which tend to prevent different kernels from sharing the representation during weight sharing . Destructive Weight Sharing . For easier discussion , let us first look at a special case , where all of the convolutional kernels have the same size and are used in the same shape in the corresponding convolution layers . In other words , { Ri } are square matrices , and the spatial sizes of all of the convolutional kernels have the same size , din × dout ×w× h , and the input channel dimension din is always equal to the output channel dimension dout . In this case , a filter f in a kernel can be treated as a vector in < dwh . Further , we choose Ri to be a block-diagonal matrix Ri = block-diag { Ai , Ai , . . . , Ai } , where Ai ∈ O ( dwh ) is an orthogonal matrix that rotates each filter from the kernel Ki in the same fashion . Similar to the Proposition 2 in ( Cheung et al. , 2019 ) , we show in the Appendix B that : if Ai , Aj are sampled from the O ( M ) Haar distribution and fi , fj are the same filter ( generated by Ri , Rj respectively from W ) from Ki , Kj respectively , then we have E [ 〈fi , fj〉 ] = 0 and E [ 〈 fi‖fi‖ , fj ‖fj‖ 〉 2 ] = 1M . Since M is usually large , the same filter from Ki , Kj are close to orthogonal and generally dissimilar . This shows that even when { Ki } are generated from the same W , they are not sharing the representation . Even though { Ai } are not updated during the training , the size of Ai can be quite large in general . In practice , we can use permutation p ∈ P ( M ) and element-wise random sign reflection b ∈ B ( M ) to construct a subset of the orthogonal group O ( M ) , i.e . we choose Ai ∈ { b ◦ p| b ∈ B ( M ) , p ∈ P ( M ) } .2 Since pseudo-random numbers are used , it takes only two random seeds to store a random permutation and an element-wise random sign reflection . In this we work , we generalize the usage of Ri beyond the block-diagonal generating matrices described above . { Ki } may have different sizes , which can be chosen even larger than the size of W. When Ki ∈ < Ni is larger than W ∈ < N , the corresponding generating matrix Ri is a tall matrix . There are many ways to efficiently create the generating matrices . We use random permutations P ( Ni ) and element-wise random sign reflections B ( Ni ) to create Ri : Ri ∈ { b ◦ p| b ∈ B ( Ni ) , p ∈ P ( Ni ) } , i = 1 , . . . , L ( 3 ) { Ri } tend to lead to destructive weight sharing and lead to better utilization of the parameter capacity . Even Parameter Distribution for Different Layers . While it is easy to randomly sample elements from the W when generating parameters for each layer , it may not be optimal as some elements in 1In this paper , we treat each convolutional kernel as a vector . When the kernel is used to do the convolution , it will be reshaped into the corresponding shape . 2Permutations and element-wise random sign reflection conceptually are subgroups from the orthogonal group , but we shall never use them in the matrix form for the obvious efficiency purpose . W may never be used , and some elements may be used more than average . We use an equalization technique to guarantee all elements of W are evenly sampled . Suppose the size of the W is N , and the total size of parameters of layers to be generated is M , M > N . We first generate b∗cMN arrays { x|x = 1 , .. , N } and concatenate them with ( M mod N ) elements randomly sampled from array { x|x = 1 , .. , M } . We call the concatenated array index array u ∈ < M . We randomly shuffle all elements in u . When initializing each layer ’ s parameter , we sequentially get indices of chosen elements from the shuffled index array u . In this way , each layer ’ s parameters are randomly and evenly sampled from W. We refer to W as model rings since elements are recurrently used in a loop . For data saving efficiency , we just need to save two random seed numbers ( one for sampling ( M mod N ) elements and one for shuffling ) instead of saving the large index array u. Batch Normalization . Model performance is relatively sensitive to the batch normalization parameters . For better performance , each of the convolution layers needs to have its own batch normalization parameters . In general , however , the size of batch normalization is relatively negligible . Yet when W is extremely small ( e.g. , 36K parameters ) , the size of batch normalization should be considered .

Summary: The paper proposes recurrent parameter generator (RPG) that is able to generate (ideally) arbitrarily large model based on a fixed set of inputs $\mathbf{W}$. Unlike common pruning or compressing techniques, the authors argue that RPG decouples the expressivity with the degree of freedom of a model, and that we can dynamically generate model parameters on the fly (while taking advantage of the pseudo-random seed) with a simple mechanism. Experiments show that this new way of generating model parameters is able to perform on par with, or better than, many existing compressive or pruning approaches.