Year: 2021

Abstract:
We present a generic method for recurrently using the same parameters for many different convolution layers to build a deep network. Specifically, for a network, we create a recurrent parameter generator (RPG), from which the parameters of each convolution layer are generated. Though using recurrent models to build a deep convolutional neural network (CNN) is not entirely new, our method achieves significant performance gain compared to the existing works. We demonstrate how to build a one-layer-size neural network to achieve similar performance compared to other traditional CNN models on various applications and datasets. We use the RPG to build a ResNet18 network with the number of weights equivalent to one convolutional layer of a conventional ResNet and show this model can achieve 67.2% ImageNet top-1 accuracy. Additionally, such a method allows us to build an arbitrarily complex neural network with any amount of parameters. For example, we build a ResNet34 with model parameters reduced by more than 400 times, which still achieves 41.6% ImageNet top-1 accuracy. Furthermore, the RPG can be further pruned and quantized for better run-time performance in addition to the model size reduction. We provide a new perspective for model compression. Rather than shrinking parameters from a large model, RPG sets a certain parametersize constraint and uses the gradient descent algorithm to automatically find the best model under the constraint. Extensive experiment results are provided to demonstrate the power of the proposed recurrent parameter generator.

heading: 1 INTRODUCTION
text: Deep learning has achieved great success with increasingly more training data and deeper & larger neural networks: A recently developed NLP model, GPT-3 (Brown et al., 2020), has astonishingly 175 billion parameters! While the model performance generally scales with the number of parameters (Henighan et al., 2020), with parameters outnumbering training data, the model is significantly over-parameterized. Tremendous effort has been made to reduce the parameter redundancy from different perspectives, including neural network pruning (LeCun et al., 1990; Han et al., 2016; Liu et al., 2018), efficient network design spaces (Howard et al., 2017; Iandola et al., 2016; Sandler et al., 2018), parameter regularization (Wan et al., 2013; Wang et al., 2020a; Srivastava et al., 2014), model quantization (Hubara et al., 2017; Rastegari et al., 2016; Louizos et al., 2019), neural architecture search (Zoph & Le, 2017; Cai et al., 2018; Wan et al., 2020), recurrent models (Bai et al., 2019; 2020; Wei et al., 2016), multi-task feature encoding (Ramamonjisoa & Lepetit, 2019; Hao et al., 2021), etc.
One of the most prominent approaches in this direction is the pruning-based model compression, which dates back to the late 80s or early 90s (Mozer & Smolensky, 1989; LeCun et al., 1990) and has enjoyed a resurgence (Han et al., 2016; Blalock et al., 2020) recently. These pruning methods seek to remove the unimportant parameters from a pre-trained large neural network and can frequently achieve an enormous model-compression ratio.
Though sharing a similar motivation to reduce the parameter redundancy, we explore an entirely different territory of model parameter reduction: rather than compressing a large model, we define an arbitrarily large model based on a fixed set of parameters to maximize the model capacity. In this work, we propose to define many different layers in a deep neural network based on a fixed amount of parameters, which we call recurrent parameter generator (RPG). That is, we differentiate the number of model parameters and degrees of freedom (DoF). Traditionally, model parameters are treated independently of each other; the total number of parameters is the number of DoF. However, by tapping into how a core set of free parameters can be assigned to the neural network model, we can develop a large model of many parameters with a small degree of freedom. In other words,
there is excess capacity in neural network models independent of how and where the parameters are used in the network. Even at the level of individual scalar values, parameters can be reused in another arbitrary location of the deep network architecture without significantly impacting model performance. Surprisingly, backpropagation training of a deep network is able to cope with that the same parameter can be assigned to multiple random locations in the network without significantly impacting model performance. Through extensive experiments, we show that a large neural network does not need to be over overparameterized to achieve competitive performance. Particularly, a ResNet18 can be implemented with the number of weights equivalent to one convolution layer in a conventional ResNet (4.72× parameter reduction) and still achieve 67.2% ImageNet top-1 accuracy. The proposed method is also extremely flexible in reducing the model parameters. In some sense, the proposed RPG method can be viewed as an automatic model parameter reduction technique, which explores the optimal accuracy-parameter trade-off. When we reduce the model parameter, RPG shows graceful performance degradation, and its compression results are frequently on par with the SOTA pruning methods besides the flexibility. Even if we reduce the ResNet18 backbone parameters to 36K, which is about 300× reduction, ResNet18 can still achieve 40.0% ImageNet top-1 accuracy. Notably, we choose a destructive parameter sharing method (Cheung et al., 2019) for RPG in this work, which discourages any potential representation sharing from layer to layer. Compared to other recurrent weight-sharing methods, e.g., convolutional pose machine (CPM) or multi-scale deep equilibrium models (MDEQ), our method achieves competitive performance on various benchmarks. Further, we show RPG can be quantized and pruned to improve FLOPs and run time with very tiny accuracy drops. This makes RPG a strong and practical baseline for probing whether there is nontrivial representation sharing within any recurrent network.
To summarize, we make the following contributions: 1. This work provides a new perspective towards automatic model parameter reduction: we can
define a neural network with certain DoF constraint and let gradient descent optimization automatically find the best model under the desired constraint.
2. We propose the recurrent parameter generator (RPG), which decouples the network architecture and the network DoF. Given a certain neural network architecture, we can flexibly choose any DoF to construct the network.
3. By separating the network architecture from the parameter generator, RPG becomes a tool for us to understand the relationship between the model DoF and the network performance. We observe an empirical log-linear DoF-Accuracy relationship.

heading: 2 RELATED WORK
text: There are many important efforts to compress neural networks or to reduce the redundancy in neural network parameters. We discuss each of the approaches and their relationships to our work.
Model Pruning, Neural Architecture Search, and Quantization. Model pruning seeks to remove the unimportant parameters in a trained model. Recently, it’s proposed to use neural architecture search as a coarse-grained model pruning (Yu et al., 2018; Dong & Yang, 2019). Another related effort is neural network quantization (Hubara et al., 2017; Rastegari et al., 2016; Louizos et al., 2019), which seeks to reduce the bits used for each parameter and can frequently reduce the model size by 4× with minimal accuracy drop. More recently, Dollár et al. (2021) presents a framework for analyzing model scaling strategies that considers network properties such as FLOPs and activations.
Parameter Regularization and Priors. Another highly related direction is parameter regularization. Regularization has been widely used to reduce model redundancy (Krogh & Hertz, 1992), alleviate model overfitting (Srivastava et al., 2014; Wan et al., 2013), and ensure desired mathematical regularity (Wang et al., 2020a). RPG can be viewed as a parameter regularization in the sense that weight sharing poses many equality constraints to weights and regularizes weights to a low-dimensional space. HyperNeat (Stanley et al., 2009) and CPPNs (Stanley, 2007) use networks to determine the weight between two neurons as a function of their positions. Karaletsos et al. (2018) and Karaletsos & Bui (2020) introduced a similar idea by providing a hierarchical prior for network parameters.
Recurrent Networks and Deep Equilibrium Models. Recurrence and feedback have been shown in psychology and neuroscience to act as modulators or competitive inhibitors to aid feature grouping (Gilbert & Sigman, 2007), figure-ground segregation (Hupé et al., 1998) and object recognition (Wyatte et al., 2012). Recurrence-inspired mechanisms also achieve success in feed-forward models. There are two main types of employing recurrence based on if weights are shared across recurrent modules. ResNet (He et al., 2016), a representative of reusing similar structures without weight sharing, introduces parallel residual connections and achieves better performance by going deeper in networks. Similarly, some works (Szegedy et al., 2015; Srivastava et al., 2015) also suggest iteratively injecting thus-far representations to the feed-forward network useful. Stacked inference methods (Ramakrishna et al., 2014; Wolpert, 1992; Weiss & Taskar, 2010) are also related while they consider each output in isolation. Several works find sharing weights across recurrent modules beneficial. They demonstrate applications in temporal modelling (Weiss & Taskar, 2010; Xingjian et al., 2015; Karpathy & Fei-Fei, 2015), spatial attention (Mnih et al., 2014; Butko & Movellan, 2009), pose estimation (Wei et al., 2016; Carreira et al., 2016), and so on (Li et al., 2016; Zamir et al., 2017). Such methods usually shine in modeling long-term dependencies. In this work, we recurrently share weights across different layers of a feedback network to reduce network redundancy.
Given stacking weight-shared modules improve the performance, researchers consider running even infinite depth of such modules by making the sequential modules converge to a fixed point (LeCun et al., 1988; Bai et al., 2019). Employing such equilibrium models to existing networks, they show improved performance in many natural language processing (Bai et al., 2019) and computer vision tasks (Bai et al., 2020; Wang et al., 2020b). One issue with deep equilibrium models is that the forward and backward propagation usually takes much more iterations than explicit feed-forward networks. Some work (Fung et al., 2021) improves the efficiency by making the backward propagation Jacobian free. Another issue is that infinite depth and fixed point may not be necessary or even too strict for some tasks. Instead of achieving infinite depth, our model shares parameters to a certain level. We empirically compare with equilibrium models in Section 5.
Efficient Network Space and Matrix Factorization. Convolution is an efficient and structured matrix-vector multiplication. Arguably, the most fundamental idea in building efficient linear systems is matrix factorization. Given the redundancy in deep convolutional neural network parameters, one can leverage the matrix factorization concept, e.g., factorized convolutions, and design more efficient network classes (Howard et al., 2017; Iandola et al., 2016; Tan & Le, 2019; Sandler et al., 2018).

heading: 3 RECURRENT PARAMETER GENERATOR
text: We define recurrent parameter generators and show a certain kind of generating matrices that leads to destructive weight sharing. For better parameter capacity, we introduce an even sampling strategy.
Recurrent Parameter Generator. Assuming we are constructing a deep convolutional neural network, which contains L different convolution layers. Let K1,K2, . . . ,KL be the corresponding L convolutional kernels 1. Rather than using separate sets of parameters for different convolution layers, we create a single set of parameters W ∈ <N and use it to generate the corresponding parameters for each convolution layer: Ki = Ri ·W, i ∈ {1, . . . , L} (1) where Ri is a fixed predefined generating matrix, which is used to generate Ki from W. We call {Ri} and W the recurrent parameter generator (RPG). In this work, we always assume that the size of W is smaller than the total parameters of the model, i.e., |W| ≤ ∑ i |Ki|. This means an element of W will generally be used in more than one layer of a neural network. Additionally, the gradient of W is a linear superposition of the gradients from each convolution layer. During the neural network training, let’s assume convolution kernel Ki receives gradient ∂`∂Ki , where ` is the loss function. Based on the chain rule, it is clear that the gradient of W is:
∂`
∂W = L∑ i=1 RTi · ∂` ∂Ki (2)
Generating Matrices. There are many different ways to create the generating matrices {Ri}. In this work, we primarily explore the destructive generating matrices, which tend to prevent different kernels from sharing the representation during weight sharing.
Destructive Weight Sharing. For easier discussion, let us first look at a special case, where all of the convolutional kernels have the same size and are used in the same shape in the corresponding convolution layers. In other words, {Ri} are square matrices, and the spatial sizes of all of the convolutional kernels have the same size, din × dout ×w× h, and the input channel dimension din is always equal to the output channel dimension dout. In this case, a filter f in a kernel can be treated as a vector in <dwh. Further, we choose Ri to be a block-diagonal matrix Ri = block-diag{Ai,Ai, . . . ,Ai}, where Ai ∈ O(dwh) is an orthogonal matrix that rotates each filter from the kernel Ki in the same fashion. Similar to the Proposition 2 in (Cheung et al., 2019), we show in the Appendix B that: if Ai, Aj are sampled from the O(M) Haar distribution and fi, fj are the same filter (generated by Ri, Rj respectively from W) from Ki, Kj respectively, then we have E [〈fi, fj〉] = 0 and E [ 〈 fi‖fi‖ , fj ‖fj‖ 〉 2 ] = 1M . Since M is usually large, the same filter from Ki, Kj are close to orthogonal and generally dissimilar. This shows that even when {Ki} are generated from the same W, they are not sharing the representation.
Even though {Ai} are not updated during the training, the size of Ai can be quite large in general. In practice, we can use permutation p ∈ P (M) and element-wise random sign reflection b ∈ B(M) to construct a subset of the orthogonal group O(M), i.e. we choose Ai ∈ {b ◦ p| b ∈ B(M), p ∈ P (M)}.2 Since pseudo-random numbers are used, it takes only two random seeds to store a random permutation and an element-wise random sign reflection.
In this we work, we generalize the usage of Ri beyond the block-diagonal generating matrices described above. {Ki} may have different sizes, which can be chosen even larger than the size of W. When Ki ∈ <Ni is larger than W ∈ <N , the corresponding generating matrix Ri is a tall matrix. There are many ways to efficiently create the generating matrices. We use random permutations P (Ni) and element-wise random sign reflections B(Ni) to create Ri:
Ri ∈ {b ◦ p| b ∈ B(Ni), p ∈ P (Ni)}, i = 1, . . . , L (3) {Ri} tend to lead to destructive weight sharing and lead to better utilization of the parameter capacity. Even Parameter Distribution for Different Layers. While it is easy to randomly sample elements from the W when generating parameters for each layer, it may not be optimal as some elements in
1In this paper, we treat each convolutional kernel as a vector. When the kernel is used to do the convolution, it will be reshaped into the corresponding shape.
2Permutations and element-wise random sign reflection conceptually are subgroups from the orthogonal group, but we shall never use them in the matrix form for the obvious efficiency purpose.
W may never be used, and some elements may be used more than average. We use an equalization technique to guarantee all elements of W are evenly sampled. Suppose the size of the W is N , and the total size of parameters of layers to be generated is M , M > N . We first generate b∗cMN arrays {x|x = 1, .., N} and concatenate them with (M mod N) elements randomly sampled from array {x|x = 1, ..,M}. We call the concatenated array index array u ∈ <M . We randomly shuffle all elements in u. When initializing each layer’s parameter, we sequentially get indices of chosen elements from the shuffled index array u. In this way, each layer’s parameters are randomly and evenly sampled from W. We refer to W as model rings since elements are recurrently used in a loop. For data saving efficiency, we just need to save two random seed numbers (one for sampling (M mod N) elements and one for shuffling) instead of saving the large index array u.
Batch Normalization. Model performance is relatively sensitive to the batch normalization parameters. For better performance, each of the convolution layers needs to have its own batch normalization parameters. In general, however, the size of batch normalization is relatively negligible. Yet when W is extremely small (e.g., 36K parameters), the size of batch normalization should be considered.

heading: 4 RECURRENT PARAMETER GENERATOR AT MULTIPLE SCALES
text: In the previous section, we discuss the general idea of superposition where only one RPG is created and shared globally across all layers. We could also create several local RPGs, and each of them is shared at certain scales, such as blocks and sub-networks. Such super-positions may be useful for certain applications such as recurrent modeling.
RPGs at Block-Level. Researchers propose network architectures that reuse the same design of network blocks multiple times for higher learning capacity, as discussed in the related work. Instead of using one global RPG for the entire network, we could alternatively create several RPGs that are shared within certain network blocks. We take ResNet18 (He et al., 2016) as a concrete example (Fig.3). ResNet18 has four building blocks. Every block has 2 residual convolution modules. To superpose ResNet18 at block scale, we create four local RPGs. Each RPG is shared within the corresponding building block, where the size of the RPG is flexible and can be determined by users.
RPGs at Sub-Network-Level. Reusing sub-networks, or recurrent networks have achieved success in many tasks as they iteratively refine and improves the prediction. Usually, weights are shared when reusing the sub-networks. This may not be optimal as sub-networks at different stages iteratively improve the prediction, and shared weights may limit the learning capacity to adapt for different stages. On the other hand, not sharing weights at all greatly increases the model size. We superpose different sub-networks with one or more RPGs. Superposition sub-networks could have a much smaller model size, while parameters for different sub-networks undergo destructive changes instead of directly copy-paste. We show applications of superpose sub-networks for pose estimation and multitask regression (Section 5.3 and 5.4).

heading: 5 EXPERIMENTAL RESULTS
text: We evaluate the performance of RPG with various tasks illustrated in Fig.2. We refer to model DoF as number of parameters or parameter size for convenience, although their differences have been discussed in Introduction. For classification, RPG was used for the entire network except for the last fully connected (fc) layer. Thus, we discuss reduction in backbone parameters. For example, Res18 has 11M backbone parameters and 512K fc parameters, and RPG was applied to reduce 11M backbone parameters only. Experiments are conducted on NVIDIA GeForce GTX 2080Ti GPUs.

heading: 5.1 CIFAR CLASSIFICATION
text: Implementation Details. All CIFAR experiments use batchsize of 128, weight decay of 5e-4, and initial learning rate of 0.1 with gamma of 0.1 at epoch 60, 120 and 160. We use Kaiming initialization (He et al., 2015) with adaptive scaling. Specifically, shared parameters are initialized with a particular variance and scale the parameters for each layer to make it match the Kaiming initialization.
Compared to Deep Equilibrium Models. As a representative of implicit models, deep equilibrium models (Bai et al., 2019) can reduce model redundancy by finding fix points via additional optimizations. We compare the image classification accuracy on CIFAR10 and CIFAR100 datasets, as well as the inference time on CIFAR100 (Table 1). Following the settings of MDEQ (Bai et al., 2020), an image was sequentially fed into the initial convolutional block, the multi-scale deep equilibrium block (dubbed as MS block), and the classification head. MDEQ (Bai et al., 2020) achieves infinite MS blocks by finding the fixed point of the MS block. We reuse the MS block two to four times without increasing the number of parameters. Our RPG achieves 3.4% - 5.8% gain on CIFAR10 and 3% - 5.9% gain on CIFAR100. Our inference time is 15 - 25 times smaller than MDEQ since MDEQ needs additional time to solve equilibrium during training.
Global RPG with Varying # Parameters. We create one global RPG to generate parameters for convolution layers of ResNet and refer to it as ResNet-RPG. We report CIFAR100 top-1 accuracy of ResNet-RPG18 and ResNet-RPG34 at different number of parameters (Fig.4 and Table 3). Compared to ResNet, ResNet-RPG achieves higher accuracy at the same parameter size. Specifically, we achieve 36% CIFAR100 accuracy with only 8K backbone parameters. Furthermore, ResNet34-RPG achieves higher accuracy than ResNet18-RPG, indicating increasing time complexity gives performance gain.
Local RPGs at the Block-Level. In the previous ResNet-RPG experiments, we use one global RPG (Fig.3-Upper).We also evaluate the performance when RPGs are shared locally at a block level, as shown in Fig.3-Lower. In Table 2, compared to plain ResNet18 at the same number of parameters, our block-level RPG network gives 1.0% gain. In contrast, our ResNet-RPG (parameters are evenly distributed) gives a 1.4% gain. Using one global RPG where parameters of each layer are evenly distributed is 0.4% higher than multiple RPGs.
Comparison to Baselines. Table 2 compares RPG and other baseline parameter reduction methods including random weight sharing, weight sharing with the hashing trick (Chen et al., 2015) and weight sharing with Lego filters Yang et al. (2019). At the same number of parameters, our RPG outperforms all other baselines, demonstrating the effectiveness of the proposed method.

heading: 5.2 IMAGENET CLASSIFICATION
text: Implementation Details. All ImageNet experiments use batch size of 256, weight decay of 3e-5, and an initial learning rate of 0.3 with gamma of 0.1 every 75 epochs and 225 epochs in total. Our schedule is different from the standard schedule as the weight-sharing mechanism requires different training dynamics. We tried a few settings and found this one to be the best for RPG.
RPG with Varying # Parameters. We use one RPG with different number of parameters for ResNet and report the top-1 accuracy (Table 3 and Fig.1(Right)). ResNet-RPGs consistently achieve higher performance compared to ResNets under the same number of parameters. Specifically, ResNetRPG34 achieves the same accuracy 73.4% as ResNet34 with only half of ResNet34 backbone parameters. ResNet-RPG18 also achieves the same accuracy as ResNet18 with only half of ResNet18 backbone parameters. Further, we find RPG networks have higher generalizability (Section 5.6).
Power Law. Empirically, accuracy and number of parameters follow a power law, when RPG model size is lower than 50% original plain ResNet model size. The exponents of the power laws are the same for ResNet18-RPG and ResNet34-RPG on ImageNet, when comparing with ResNet34 accuracies. The scaling law may be useful for estimating the network performance without training the network. Similarly, (Henighan et al., 2020) also identifies a power law for performance and model size of transformers. The proposed RPG enables under-parameterized models for large-scale datasets such as ImageNet, which may unleash more new studies and findings.

heading: 5.3 POSE ESTIMATION
text: Implementation Details. We superpose sub-networks for pose estimation with a globally shared RPG. We use hourglass networks (Newell et al., 2016) as the building backbone. The input image is first fed to an initial convolution block to obtain a feature map. The feature map is then fed to multiple stacked pose estimation sub-networks. Each sub-network outputs a pose estimation prediction, which is penalized by the pose estimation loss. Convolutional pose machine (CPM) (Wei et al., 2016) share all the weights for different sub-networks. We create one global RPG and generate parameters for each sub-network. Our model size is set to be the same as CPM. We also compare with larger models where parameters of sub-networks are not shared.
We evaluate on MPII Human Pose dataset (Andriluka et al., 2014), a benchmark for articulated human pose estimation, which consists of over 28K training samples over 40K people with annotated body joints. We use the hourglass network (Newell et al., 2016) as backbone and follow all their settings.
Results and Analysis. We report the Percentage of Correct Key-points at 50% threshold (PCK@0.5) of different methods in Table 4. CPM (Wei et al., 2016) share all parameters for different sub-
networks. We use one RPG that is shared globally at the same size as CPM. For reference, we also compare with the no-sharing model as the performance ceiling. Adding the number of recurrences leads to performance gain for all methods. At the same model size, RPG achieves higher PCK@0.5 compared to CPM. Increasing the number of parameters by not sharing sub-network parameters also leads to some performance gain.

heading: 5.4 MULTI-TASK REGRESSION
text: Implementation Details. We superpose sub-networks for multi-task regression with multiple RPGs at the building-block level. We focus on predicting depth and normal maps from a given image. We stack multiple SharpNet (Ramamonjisoa & Lepetit, 2019), a network for monocular depth and normal estimation. Specifically, we create multiple RPGs at the SharpNet building-block level. That is, parameters of corresponding blocks of different sub-networks are generated from the same RPG.
We evaluate the monocular depth and normal prediction performance on Standford 3D indoor scene dataset (Armeni et al., 2017). It contains over 70K images with corresponding depths and normals covering over 6,000 m2 indoor area. We follow all settings of SharpNet (Ramamonjisoa & Lepetit, 2019), a state-of-the-art monocular depth and normal estimation network.
Results and Analysis. We report the mean square errors for depth and normal estimation in Table 5. Compared to one-time inference without recurrence, our RPG network gives 3% and 2% gain for depth and normal estimation, respectively. Directly sharing weights but using new batch normalization layers decrease the performance by 1.2% and 0.3% for depth and normal. Sharing weights and normalization layers further decrease the performance by 0.7% and 0.9% for depth and normal.

heading: 5.5 PRUNING RPG
text: Fine-Grained Pruning. Fine-grained pruning methods aim at reducing the model parameters by sparsifying weight matrices. Such methods usually do not reduce the inference speed, although custom algorithms (Gale et al., 2020) may improve the speed. At the same number of parameters, RPG outperforms state-of-the-art fine-grained pruning method IMP (Frankle et al., 2019). Accuracy drops of RPG and IMP are similar, both around 2% (Table 6). It is worth noting that IMP could have faster inference speed with sparse GPU kernels (Gale et al., 2020).
Coarse-Grained Pruning. While RPG is not designed to reduce FLOPs, it can be combined with coarse-grained pruning to reduce FLOPs. We prune RPG filters with lowest `1 norms. Table 7 shows that the pruned RPG achieves on-par performance as state-of-the-art coarse-grained pruning method Knapsack (Aflalo et al., 2020) at the same FLOPs.

heading: 5.6 ANALYSIS
text: Comparison to Pruning Methods. We report our ResNet18-RPG performance with different number of parameters on ImageNet and some baseline pruning methods in Fig.1(Right). Our RPG networks outperform SOTA pruning methods such as (Aflalo et al., 2020; Dong & Yang, 2019; He et al., 2019; 2018; Dong et al., 2017; Khetan & Karnin, 2020). Specifically, at the same number of parameters, our RPG network has 0.6% gain over the knapsack pruning (Aflalo et al., 2020), a method that achieves the best ImageNet pruning accuracy.
Generalizability. We report the performance gap between training and validation set on ImageNet (Table 8(a)) and MPII pose estimation (Table 8(b)). CPM (Wei et al., 2016) serves as the baseline pose estimation method. RPG models consistently achieve lower gaps between training and validation set, indicating the RPG model suffers less from over-fitting.
We also report the out-of-distribution performance of RPG models. ObjectNet (Barbu et al., 2019) contain 50k images with 113 classes overlapping with ImageNet. Previous models are reported to
have a large performance drop on ObjectNet. We directly evaluate the performance of ImageNettrained model on ObjectNet without any fine-tuning (Table 8(c)). With the same number of backbone parameters, our ResNet-RPG achieves a 3.1% gain compared to ResNet18. With the same network architecture design, our ResNet-RPG achieves 0.5% gain compared to ResNet34. This indicates our RPG networks have higher out-of-distribution performance even with smaller model sizes.
Quantization. Network quantization can reduce model size with minimal accuracy drop. It is of interest to study if RPG models, whose parameters have been shrunk, can be quantized. After 8-bit quantization, the accuracy of ResNet18-RPG (5.6M parameters) only drop 0.1 percentage point on ImageNet, indicating RPG can be quantized for further size reduction. Details are in Appendix A.
Security. Permutation matrices generated by the random seed can be considered as security keys to decode the model. In addition, only random seeds to generate transformation matrices need to be saved and transferred, which is efficient in terms of size.

heading: 5.7 ABLATION STUDIES
text: We conduct ablation studies to understand the functions of permutation and reflection matrices (Fig.5). We evaluate ResNet-RPG34 with 2M backbone parameters. With permutation and reflection matrices leads to 76.5% accuracy, with permutation matrices only leads to 75.8%, with reflection matrices only leads to 71.1%, and with no permutation and reflection matrices leads to 70.7%. This suggests permutation and reflection matrices are useful for RPGs.

heading: 6 DISCUSSION
text: The common practice in machine learning is to search for the best model in large model space with many parameters or degrees of freedom (DoF), and then shrink the optimal model for deployment. Our key insight is that a direct and opposite approach might work better: We start from a lean model with a small DoF, which can be unpacked into a large model with many parameters. Then we can let the gradient descent automatically find the best model under this DoF constraint. Our work is a departure from mainstream approaches towards model optimization and parameter reduction. We show how the model DoF and actual parameter size can be decoupled: We can define a large model of arbitrary number of parameters with a small DoF.
We limit our scope to linear destructive weight sharing for different convolutional layers. However, in general, there might also exist nonlinear RPGs and efficient nonlinear generation functions to create convolutional kernels from a shared model ring W. Further, although RPG focuses on reducing model DoF, it can be quantized and pruned to further reduce the FLOPs and run time.
To sum up, we develop an efficient approach to build an arbitrarily complex neural network with any amount of DoF via a recurrent parameter generator. On a wide range of applications, including image classification, pose estimation and multitask regression, we show RPG networks consistently achieve higher performance at the same model size. Further, analysis shows that such networks are less possible to overfit and have higher performance on out-of-distribution data.
RPG can be added to any existing network flexibly with any amount of DoF at the user’s discretion. It provides new perspectives for recurrent models, equilibrium models, and model compression. It also serves as a tool for understanding the relationship between network properties and network DoF by factoring out the network architecture.
Reproducibility: We provide our code in supplementary materials.

Review summaries:
The paper proposes recurrent parameter generator (RPG) that is able to generate (ideally) arbitrarily large model based on a fixed set of inputs $\mathbf{W}$. Unlike common pruning or compressing techniques, the authors argue that RPG decouples the expressivity with the degree of freedom of a model, and that we can dynamically generate model parameters on the fly (while taking advantage of the pseudo-random seed) with a simple mechanism. Experiments show that this new way of generating model parameters is able to perform on par with, or better than, many existing compressive or pruning approaches.

The paper presents a technique for parameter reduction in neural networks. It follows the motivation of parameter redundancy in over-parameterized networks and aims to parameterize large networks with a smaller and shared set of weights. More specifically, a set of parameters based on a predetermined reduction factor is randomly assigned to various layers of a network and trained jointly. It is a simple technique that can be applied to any network architecture as the parameter generation is decoupled from the underlying architecture. The proposed method is evaluated in several tasks from classification to segmentation by using different network architectures. The experiments show improved or competitive performance when trained with similar or fewer parameters compared to the baselines.

 This paper uses parameter generation to generate parameters for different hidden layers in neural networks. There are several significant strengths and weaknesses in this paper. Especially, the practicability of the proposed method is questionable. It would be good to see my detailed comments below. 

In order to reduce size of deep models, this work propose one parameter sharing method for different convolutional layers. With the help of one sharing set of parameters, all convolutional kernels can be generated from the sharing parameters. They show the effeciveness of this method in classification, pose estimation tasks.