Recurrent Parameter Generators
Authors: Jiayun Wang
Yubei Chen
Stella Yu
Brian Cheung
Yann LeCun
Venue: ICLR 2022 Submitted (2022)
Keywords: recurrent
parameters
degrees of freedom

Abstract: We present a generic method for recurrently using the same parameters for many different convolution layers to build a deep network. Specifically, for a network, we create a recurrent parameter generator (RPG), from which the parameters of each convolution layer are generated. Though using recurrent models to build a deep convolutional neural network (CNN) is not entirely new, our method achieves significant performance gain compared to the existing works. We demonstrate how to build a one-layer-size neural network to achieve similar performance compared to other traditional CNN models on various applications and datasets. We use the RPG to build a ResNet18 network with the number of weights equivalent to one convolutional layer of a conventional ResNet and show this model can achieve $67.2\%$ ImageNet top-1 accuracy. Additionally, such a method allows us to build an arbitrarily complex neural network with any amount of parameters. For example, we build a ResNet34 with model parameters reduced by more than $400$ times, which still achieves $41.6\%$ ImageNet top-1 accuracy. Furthermore, the RPG can be further pruned and quantized for better run-time performance in addition to the model size reduction. We provide a new perspective for model compression. Rather than shrinking parameters from a large model, RPG sets a certain parameter-size constraint and uses the gradient descent algorithm to automatically find the best model under the constraint. Extensive experiment results are provided to demonstrate the power of the proposed recurrent parameter generator.


Under review as a conference paper at ICLR 2022
RECURRENT PARAMETER GENERATORS
Anonymous authors Paper under double-blind review
ABSTRACT
We present a generic method for recurrently using the same parameters for many different convolution layers to build a deep network. Specifically, for a network, we create a recurrent parameter generator (RPG), from which the parameters of each convolution layer are generated. Though using recurrent models to build a deep convolutional neural network (CNN) is not entirely new, our method achieves significant performance gain compared to the existing works. We demonstrate how build one-layer-size neural network to achieve similar performance compared to other traditional CNN models on various applications and datasets. We use the RPG to build a ResNet18 network with the number of weights equivalent to one convolutional layer of conventional ResNet and show this model can achieve 67.2% ImageNet top-1 accuracy. Additionally, such a method allows us to build an arbitrarily complex neural network with any amount of parameters. For example, we build a ResNet34 with model parameters reduced by more than 400 times, which still achieves 41.6% ImageNet top-1 accuracy. Furthermore, the RPG can be further pruned and quantized for better run-time performance in addition to the model size reduction. We provide a new perspective for model compression. Rather than shrinking parameters from a large model, RPG sets a certain parameter- size constraint and uses the gradient descent algorithm to automatically find the best model under the constraint. Extensive experiment results are provided to demonstrate the power of the proposed recurrent parameter generator.
INTRODUCTION
Deep learning has achieved great success with increasingly more training data and deeper & larger neural networks: A recently developed NLP model, GPT-3 (Brown et al. 2020), has astonishingly 175 billion parameters! While the model performance generally scales with the number of parameters (Henighan et al.) 2020), with parameters outnumbering training data, the model is significantly over-parameterized. Tremendous effort has been made to reduce the parameter redundancy from different perspectives, including neural network pruning (LeCun et al. 1990, Han et al.) 2016 Liu
ct al. 2018), efficient network design spaces (Howard et al. [017]] landola et al. [2016] Sandler et al. 2018), parameter regularization (Wan et al.) [2013] Wang et al. [2020a]] Srivastava et al.
2014), model
quantization (Hubara et al. 2017] Rastegari et | 2021610Lon Louizos et al. 2019), neural architecture
search (Zoph & Le| 2017] Cai et al. 2018 Wan et al. [2220), recurrent models (Bai et al.
2019 2020
Wei et al.) 2016), mulli-task feature encoding [Ramamonjisoa & Lepetit 2019 Hao et al. 2022 etc.
One of the most prominent approaches in this direction the pruning- model compression, which dates back to the late 80s or early 90s (Mozer & Smolensky 1989 [eCCun et al. 1990) and has
enjoyed resurgence (Han et al. 2016; Blalock et al.] (2020) recently. These pruning methods seek to remove the unimportant parameters from a pre-trained large neural network and can frequently achieve an enormous compe-sooonessinn ratio.
Though sharing a similar motivation to reduce the parameter redundancy, we explore an entirely different territory of model parameter reduction: rather than compressing large model, we define an arbitrarily large model based on a fixed set of parameters to maximize the model capacity. In this work, we propose to define many different layers deep neural network based on a fixed amount of parameters, which we call recurrent parameter generator (RPG). That is, we differentiate the number of model parameters and degrees of freedom (DoF). Traditionally, model parameters are treated independently of each other; the total number of parameters is the number of DoF. However, by tapping into how a core set of free parameters can be assigned to the neural network model, we can develop a large model of many parameters with small degree of freedom. In other words,

---

Under review as a conference paper at ICLR 2022
75
70
Model Ring
imaggetes erOuRN 050
65
Gencuder
-X-- R54-1PG (curs)
RIB--PFHH [B3]
R1B SFP (62)
RIB-CCOL [63]
Model Parameters
40
R18 Prunehuc
10
10'
Number bacibione porameterr
Figure I: We propose a recurrent parameter generator (RPG) that shares a fixed set of parameters in ring and use them to generate parameters of different parts of a neural network, whereas in the standard neural network, all the parameters are independent of each other, so the model gets bigger as it gets deeper. Left: The third section of the model starts to overlap with the first section in the model ring, and all later layers share generating parameters for possibly multiple times. Right: Employing the Recurrent Parameter Generator (RPG) for ResNet could reduce the model parameters any size. Specifically, with only half ResNet34 backbone parameters, we achieve the same ImageNet top-1 accuracy. We also outperform model compression methods such Knapsack (Aflalo et 2020).
there is excess capacity neural network models independent of how and where the parameters are used in the network. Even at the level of individual scalar values, parameters can be reused in another arbitrary location of the deep network architecture without significantly impacting model performance. Surprisingly, backpropagation training of deep network is able cope with that the same parameter can be assigned multiple random locations in the network without significantly impacting model performance. Through extensive experiments, does not need to be over overparameterized to achieve competitive performance. Particularly, a ResNet18 can be implemented with the number of weights equivalent conventional ResNet (4.72x parameter reduction) and still achieve 67. 67.2% ImageNet top-1 accuracy. The proposed method is also extremely flexible in reducing the model parameters. some sense, the proposed RPG method can be viewed as an automatic model parameter reduction technique, which explores the optimal accuracy-parametty trade-off. When we reduce the model parameter, RPG shows graceful performance degradation, and its compression results are frequently on par with the SOTA pruning methods besides the flexibility. Even if we reduce the ResNet18 backbone parameters to 36K, which is about x reduction, ResNet18 can still achieve 40.0% ImageNet top-1 accuracy.
show that large neural network
convolution layer in
Notably, we choose a destructive parameter sharing method (Cheung et al. 2019) for RPG this work, which discourages any potential representation sharing from layer to layer. Compared to other recurrent weight-sharing methods, c.g., convolutional pose machine (CPM) or multi-scale deep equilibrium models (MDEQ). method achieves competitive performance various benchmarks. Further, we show RPG can be quantized and pruned to improve FLOPs and run time with very tiny accuracy drops. This makes RPG strong and practical baseline for probing whether there is nontrivial representation sharing within any recurrent network.
To summarize, make the following contributions:
This work provides a new perspective towards automatic model parameter reduction: we can define neural network with certain DoF constraint and let gradient descent optimization automatically find the best model under the desired constraint.
Iospard
depth
image convl conv2 LoavL
image
pose
image
Figure 2: We demonstrate the effectiveness of RPGs various applications including image classification (Left), human pose estimation (Middle), and multitask regression (Right). A network either have global RPG or multiple local RPGs that shared within blocks or sub-networks
2

---

Under review as a conference paper at ICLR 2022
2. We propose the recurrent parameter generator (RPG), which decouples the network archi- tecture and the network DoF. Given a certain neural network architecture, we can flexibly choose any DoF to construct the network.
3. By separating the network architecture from the parameter generator, RPG becomes a tool for us to understand the relationship between the model DoF and the network performance. We observe an empirical oog-linear DoF-Accuracy relationship.
2 RELATED WORK
There many important efforts compress neural networks or reduce the redundancy in neural network parameters. We discuss each of the approaches and their relationships to our work.
Model Pruning, Neural Architecture Search, and Quantization. Model pruning seeks to remove the unimportant parameters in a trained model. Recently, it's proposed to use neural architecture search as a coarse- grained model pruning (Yu et al.. 2018 Dong & Yang 2019)) Another related effort is neural network quantization (Hubara et al. 2017 Rastegari et al. | which seeks reduce the bits used for each parameter and can frequently reduce the model size by 4x with minimal accuracy drop. More recently, [Doláar et al. (2021) presents a framework for analyzing model scaling strategies that considers network properties such as FLOPs and activations.
Louizos et al. 2019).
Parameter Regularization and Priors. Another highly related direction is parameter regularization. Regularization has been widely used to reduce model redundancy (Krogh Hertz 1992), alleviate model overfitting (Srivastava et al. (2044) Wan et al.] 2013). and ensure desired mathematical regularity (Wang et al., 2020a). RPG can be viewed as a parameter regularization in the sense that weight sharing poses many equality constraints to weights and regularizes weights to a -ondimensional space. HyperNeat (Stanley et al. 2009) and CPPNs (Stanley 2007) use networks to determine the weight between two neurons a function of their positions. Karaletsos et al. 2018) and Karaletsos & Bui (020) introduced a similar idea by providing a hierarchical prior for network parameters.
Recurrent Networks and Deep Equilibrium Models. Recurrence and feedback have been shown in psychology and neuroscience to act as modulators or competitive inhibitors to aid feature grouping Gilbert & Sigman 2007). figure-ground segregation (Hupé al.. 1998) and object recognition (Wyatte et al. 2012 Recurrence-inspee inspired mechanisms also achieve success in feed-forward models. There are two main types of employing recurrence based on if weights are shared across recurrent modules. ResNet (He et al., 2016), a representative of reusing similar structures without weight sharing, introduces parallel residual connections and achieves better performance by going deeper in networks. Similarly, some works (Szegedy et al.) [2015] Srivastava et al. 2015) also suggest iteratively injecting thus-far representations to the feed-forward network useful. Stacked inference methods (Ramakrishna et al. 2014] Wolperi 1992. Weiss & Taskar 2010) are also related while they consider each output in isolation. Several works find sharing weights across recurrent modules beneficial. They demonstrate applications in temporal modelling (Weiss & Taskar 2010 Xingjian al. 2015
Karpathy & Fei-Fei 2015), spatial attention (Mnih et al.] 2014 Butko & Movellan, 2009)] pose (Li et al. 2016 Zamir al. (017).
estimation (Wei al.] [2166] Carreira et al.; 2016), and
Such methods usually shine in modeling long-term dependencies. In this work, we recurrently share weights across different layers of feedback network to reduce network redundancy.
Given stacking weight-shared modules improve the performance, researchers consider running even infinite depth of such modules by making the sequential modules converge fixed point (LeCun) et al. 1988; Bai et al. 2019). Employing such equilibrium models to existing networks, they show improved performance in many natural language processing (Bai et al.) 2019) and computer vision tasks (Bai et al.) 2020 Wang et al. 2020b)). One issue with deep equilibrium models is that the forward and backward propagation usually takes much more iterations than explicit feed-forward networks. Some work (Fung et al. 2021) improves the efficiency by making the backward propagation Jacobian free. Another issue is that infinite depth and fixed point may not be necessary or even too strict for some tasks. Instead of achieving infinite depth, our model shares parameters to a certain level. We empirically compare with equilibrium models in Scction
Efficient Network Space and Matrix Factorization. Convolution is an efficient and structured natrix-vector multiplication. Arguably, the most fundamental idea in building efficient linear systems is matrix factorization. Given the redundancy in deep convolutional neural network parameters, one can leverage the matrix factorization concept, e.g., factorized convolutions, and design more efficient
network classes HHwwadd et al. 2017 landola et al. 2016 Tan & Le| 2019 Sandler et al. 2018).
3

---

Under review as a conference paper at ICLR 2022
3 RECURRENT PARAMETER GENERATOR
We define recurrent parameter generators and show a certain kind of generating matrices that leads to destructive weight sharing. For better parameter capacity, we introduce an even sampling strategy.
Recurrent Parameter Generator. Assuming we are constructing deep convolutional neural network, which contains L different convolution layers. Let K,, K K,. be the corresponding convolutional kernels | Rather than using separate sets of parameters for different convolution layers, we create a single set of parameters W E R and use to generate the corresponding parameters for each convolution layer:
(1) where R. is fixed predefined generating matrix, which is used to generate K, from W. We call {R.} } and W the recurrent parameter generator (RPG). In this work, we always assume that the size of W is smaller than the total parameters of the model, i.e., |W| S Σ, K This means an element of W will generally be used in more than one layer of a neural network. Additionally, the gradient of W is linear superposition of the gradients from each convolution layer. During the neural network training, let's assume convolution kernel K; receives gradient of
= Wii (I......I D
where f is the
JK,
loss function. Based on the chain rule, it is clear that the gradient of W is:
Oe
de
Σ
OK;
Generating Matrices. There are many different ways to create the generating matrices {R.). In this work, we primarily explore the destructive generating matrices, which tend to prevent different kernels from sharing the representation during weight sharing.
Destructive Weight Sharing. For easier discussion, let us first look at a special case, where all of the convolutional kernels have the same size and are used in the same shape the corresponding convolu- tion layers. In other words. } } are square matrices, and the spatial sizes of all of the convolutional kernels have the same size, din dout X w x h, and the input channel dimension din is always equal to the output channel dimension dout- In this case, a filter f in kernel can be treated as vector in Rdwhh Further, we choose R. to be a block-diagonal matrix R. = block-diag where A, E O(dwh) is an orthogonal matrix that rotates each filter from the kernel K; in the same fashion. Similar to the Proposition 2 in (Cheung et al., 2019). we show in the Appendix B that: if A,, Ay are sampled from the (M) Haar distribution and f,, f, are the same filter (generated by R. R2 respectively from W) from K,, K, respectively, then we have E [f,, f,)] = 0 and
( (A, A.,..
).).
E
(
f. f ( [ | = 1
M Since M is usually large, the same filter from K,, Kj are close to orthogo-
nal and generally dissimilar. This shows that even when {K,} are generated from the same W, they are not sharing the representation.
Even though } are not updated during the training, the size of A, can be quite large in general. In practice, we can use permutation p E P(M) and element-wise random sign reflection b E ((M) to construct a subset of the orthogonal group O(M), i.c. we choose A, E {b pl b E B(M), P e P(M) Since pseudo-random numbers are used, it takes only two random seeds to store random permutation and an element- wiie random sign reflection.
In this we work, we generalize the usage of R3 beyond the block-diagonal generating matrices described above, {K,} may have different sizes, which can be chosen even larger than the size of W. When K, E RRN; is larger than W E RAV the corresponding generating matrix R; is a tall matrix. There are many ways to efficiently create the generating matrices. We use random permutations P(N)) and element-wise random sign reflections B(N,) to create R,;
R4 {b o pl E B(N.). p E P(N+),, i i.....,,
(3)
(R.) tend to lead to destructive weight sharing and lead to better utilization of the parameter capacity.
Even Parameter Distribution for Different Layers. While it is easy to randomly sample elements from the W when generating parameters for each layer, it may not be optimal as some elements in
In this paper, we treat each convolutional kernel as vector. When the kernel is used do the convolution, will he reshaped into the corresponding shape.
Perrmutations and element- random sign reflection conceptually are subgroups from the orthogonal group, but we shall never them in the matrix form for the obvious efficiency purpose.
4

---

Under review as a conference paper at ICLR 2022
plobal RPG
Imago
Tx7 comn 64
pool
ooat Seit
mi oxtec Eil
plail
FCR CIEEX 125
FTR CORE 124
piot
2.56
CONT 256
pasl
CODE S12
FER evIES 512
local RPG
peeal RPG
Oca RPG
Socal
RPG
Imago
1x7 comm 64
posl
COIV S4
FCR oitY
pool
CIET 125
TES COOE
124
piLE
OOEY 256
TOL COET
256
posl
CORY SE2
TO OOe
512
POLT
Figure 3: Recurrent parameter generators at multiple scales.
Upper: A global RPG is used for
generating convolution kernels for the entire ResNet18, Lower: Four local RPGs are each responsible for generating convolution kernels within each corresponding building block of the ResNet18.
W may never be used, and some elements may be used more than average. We use an equalization technique to guarantee all elements of W are evenly sampled. Suppose the size of the W is N, and the total size of parameters of layers to be generated is M, M > N. We first generate |* arrays {a|ar 1, ,.., N} and concatenate them with (M mod N) elements randomly sampled from array {æ| = .,, }. . We call the concatenated array index array u E RRM We randomly shuffle all elements in When initializing each layer's parameter, we sequentially get indices of chosen elements from the shuffled index array u. In this way, each layer's parameters are randomly and evenly sampled from W. We refer to W as model rings since elements are recurrently used in a loop. For data saving efficiency, we just need to save two random seed numbers (one for sampling (M mod N) elements and one for shuffling) instead of saving the large index array u.
Batch Normalization. Model performance is relatively sensitive to the batch normalization parame- ters. For better performance, each of the convolution layers needs to have its own batch normalization parameters. In general, however, the size of batch normalization is relatively negligible. Yet when W extremely small (e.g., 36K parameters), the size of batch normalization should be considered.
4 RECURRENT PARAMETER GENERATOR AT MULTIPLE SCALES
In the previous section, we discuss the general idea of superposition where only one RPG is created and shared globally across all layers. We could also create several local RPGs, and each of them is shared at certain scales, such blocks and sub-networks. Such super-positions may be useful for certain applications such as recurrent modeling.
RPGs at Block-Level. Researchers propose network architectures that reuse the same design of network blocks multiple times for higher learning capacity, as discussed in the related work. Instead of using one global RPG for the entire network, we could alternatively create several RPGs that are shared within certain network blocks. We take ResNet18 (He et al. 2016) as concrete example 3). ResNet18 has four building blocks. Every block has residual convolution modules. To superpose ResNer18 at block scale, we create four local RPGs. Each RPG is shared within the corresponding building block, where the size of the RPG is flexible and can be determined by users.
RPGs at Sub-Network-Leve Reusing sub-networks, or recurrent networks have achieved success in many tasks as they iteratively refine and improves the prediction. Usually, weights are shared when reusing the sub-networks. This may not be optimal sub-network, at different stages iteratively improve the prediction, and shared weights may limit the learning capacity to adapt for different stages. On the other hand, not sharing weights at all greatly increases the model size. We superpose different sub-networks with one or more RPGs. Superposition sub-networks could have a much smaller model size, while parameters for different sub-networks undergo destructive changes instead of directly copy-paste. We show applications of superpose sub-networks for pose estimation and multitask regression (Section 5.3 and
5 EXPERIMENTAL RESULTS
We evaluate the performance of RPG with various tasks illustrated in Fig 2 We refer to model DoF as number of parameters parameter size for convenience, although their differences have been discussed in Introduction. For classification, RPG was used for the entire network except for the last fully connected (fc) layer. Thus, discuss reduction in backbone parameters. For example, Res18 has 11M backbone parameters and 512K fo parameters, and RPG was applied to reduce 11M backbone parameters only. Experiments are conducted on NVIDIA GeForce GTX 2080Ti GPUs.
5

---

Under review as a conference paper at ICLR 2022
Reeshet 18-4PG -X-
28.
Reshiet34
CIFAR100 WWw.ncc Kccunry (5)
23
24.2
23.5
20
DPAR000 Errer
10-
100
10
Number of bace parameters
Xermr
V purm
Figure 4: CIFARIOU accuracy versus the backbone parameter size for plain ResNet and RPG, RPG only has 0.2% drop with 50% Res34 parameters.
Figure 5: Ablation studies of permutation and re- flection matrices Res34-RPG, Having both ma- trices gives the highest performance.
5.1 CIFAR CLASSIFICATION
Implementation Details. All CIFAR experiments use batchsize of 128, weight decay of 5e-4, and initial learning rate of 0.1 with gamma of 0.1 at epoch 60, 120 and 160. We use Kaiming initialization (He et al. 2013) with adaptive scaling. Specifically, shared parameters are initialized with a particular variance and scale the parameters for each layer to make it match the Kaiming initialization.
Compared to Deep Equilibrium Models. As a representative of implicit models, deep equilibrium models (Bai et al. 2019) can reduce model redundancy by finding fix points via additional optimiza- tions. We compare the image classification accuracy on CIFAR 10 and CIFAR 100 datasets, as well the inference time CIFAR 100 (Table D Following the settings of MDEQ (Bai et al. (020)). an image was sequentially fed into the initial convolutional block, the multi-scale deep equilibrium block (dubbed MS block), and the classification head. MDEQ (Bai al. 2020) achieves infinite MS blocks by finding the fixed point of the MS block. We reuse the MS block two to four times without increasing the number of parameters. Our RPG achieves 3.4% 5.8% gain on CIFAR 10 and 3% 5.9% gain on CIFARIOU 100. Our inference time is 15 25 times smaller than MDEQ since MDEQ needs additional time to solve equilibrium during training.
Global RPG with Varying # Parameters. We create one global RPG to generate parameters for convolution layers of ResNet and refer to it as ResNer-RPG. We report CIFAR100 top- accuracy of ResNet-RPG18 and Res at different number of parameters (Fig Table 3 Compared to ResNet, ResNet-RPG achieves higher accuracy at same parameter size. Specifically, we achieve 36% CIFARIOG 100 accuracy with only 8K backbone parameters. Furthermore, ResNet34-RPC achieves higher accuracy than ResNet18 8-PGG, indicating increasing time complexity gives performance gain.
Local RPGs at the Level-Level. In the previous ResNet-RPG experiments, (Fig Upper).WWe.. We also evaluate the performance when RPGs are shared locally at block level, as shown in Fig Lower. In Table compared to plain ResNet18 the same number of parameters, our bloek-level RPG network gives 1.0% gain. In contrast, our ResNet-RPG (parameters are evenly distributed) gives 1.4% gain. Using one global RPG where parameters of each layer are evenly distributed is 0.4% higher than multiple RPGs.
global RPG
Table 2: At the same number of backbone pa- rameters on CIFARIO0 using local RPGs at block-level improves accuracy. Using a global RPG further improves the accuracy. RPG also outperforms baseline methods.
Table Our method compared with multiscale deep equilibrium models (Bai et al., 2020) on CIFAR 10 and CIFAR classification. At the same number of model parameters, we achieve 3% 6% improvement with 15 25x less inference time. Inference time is measured by milliseconds per image.
#Param
Acc. (%)
Res18
11M
77,5
Our RPG
Res34-RPG.blk
11M
78.5
Accuracy (%)
MDEQ
2x MS blk
3x MS blk
4x MS blk
Res34-RPG
11M
78.9
CI
85.1
88.5
90.1
90.9
Rex34-random weight share
11M
74.9
CIFARIOK
59.8
62.8
64.7
65.7
Res34-Hash (Chen et al. 2015
11M
75.6
Inference (ms)
3.15
0.12
0.18
0.22
Res34-Lego Yang et 2019
11M
78.4
21M
79.1

---

Under review as a conference paper at ICLR 2022
Table 3: ImageNet and CIFARI00 top-1 classification accuracy versus the number back-bone parameters for our ResNet-RPG and plain ResNet. Our ResNet-RPG consistently achieves higher performance at the same number of parameters.
ResNet18
ResNet34
Res18-RPG
Res34-RPG
# Purameters
11M
21M
45K
2M
45K
2M
11M
ImageNet acc. (%)
69.8
73.4
40.0
67.2
41.6
69.1
73.4
CIFAR 100 acc. (5)
77.6
79.1
60.2
75.6
61.7
76.5
78,9
Comparison to Baselines. T compares RPG and other baseline parameter reduction methods including random weight sharing, weight sharing with the hashing trick (Chen et al.] (0155) and weight sharing with Lego filters Yang et al. (2019). At the same number of parameters, our RPG outperforms all other baselines, demonstrating the effectiveness of the proposed method.
5.2 IMAGENET CLASSIFICATION
Implementation Details. All ImageNet experiments use batch size of 256, weight decay of 3e-5, and initial learning rate of 0.3 with gamma of 0. every 75 epochs and 225 epochs in total. Our schedule different from the standard schedule as the weight-sharing mechanism requires different training dynamics. We tried a few settings and found this one be the best for RPG.
RPG with Varying # Parameters. We use one RPG with different number of parameters for ResNet and report the top-1 accuracy aad Fig Right)). ResNet-RPGs consistently achieve higher performance compared to ResNets under the same number of parameters. Specifically, ResNet- RPG34 achieves the same accuracy 73.4% as ResNet34 with only half of ResNet34 backbone parameters. ResNet-RPG18 also achieves the same accuracy as ResNet18 8 with only half of ResNet18 backbone parameters. Further, we find RPG networks have higher generalizability (Section (56).
Power Law. Empirically, accuracy and number of parameters follow power law, when RPG model size is lower than 50% original plain ResNet model size. The exponents of the power laws are the same for ResNet18-Ree accuracies. The scaling law may be useful for estimating the network performance without training the network. Similarly, (Henighan al., 2020) also identifies a power law for performance and model size transformers. The proposed RPG enables under-pante parameterized models for large-scale datasets such ImageNet, which may unleash more studies and findings.
and ResNec34-RPG on ImageNet, when comparing with ResNet34
5.3 POSE ESTIMATION
Implementation Details. We superpose sub-networks for pose estimation with a globally shared RPG. We hourglass networks (Newell et al. 2016) the building backbone. The input image is first fed to an initial convolution block to obtain a feature map. The feature map is then fed to multiple stacked pose estimation sub-networks. Each sub-network outputs pose estimation prediction, which is penalized by the pose estimation loss. Convolutional pose machine (CPM) (Wei et al., 2016) share all the weights for different sub-networks. We create one global RPG and generate parameters for each sub-network. Our model size is set to be the same as CPM. We also compare with larger models where parameters of sub-networks are not shared.
We evaluate on MPII Human Pose dataset (Andriluka et al. 2014). benchmark for articulated human pose estimation, which consists of over 28K training samples over 40K people with annotated body joints. We use the hourglass network (Newell et al. 2016) backbone and follow all their settings,
Results and Analysis. We report the Percentage Correct Key-points at 50% threshold (PCK @0.5) of different methods in Table CPM (Wei et al.) 2016) share all parameters for different sub-
Table Multitask regression errors on S3DIS with sub-net architecture as [Ramamonjisoa & Lepetit (2019). Lower is better. Number of parameters for all methods are the same. Sub-net reused once.
Table 4: Pose estimation performance (pa- rameter size) on MPII human pose com- pared with CPM (Wei et al. 2016). The
metric is NCKh@0.5.
Depth RMSE
Nornal RMSE
CPM
No
No reusing the sub-net
25.5%
41.0%
1x sub-net
84.7 (3.3M)
Reuse sub-net
24.7%
40.3%
2x sub-nets
86.1 (3,3M)
86.5 (3,3M)
87.1 (6.7M)
Reuse new BN
24.0%
39.4%
4x sub-nets
86.5 (3.3M)
87.3 (3.3M)
88.0 (13.3M)
Reuse & new BN perm. reflect.
22.8%
39.1
7

---

Under review as a conference paper at ICLR 2022
networks. We one RPG that is shared globally the same size as CPM. For reference, also compare with the no-sharing model as the performance ceiling. Adding the number of recurrences leads to performance gain for all methods. At the same model size, RPG achieves higher PCK@0.5 compared to CPM. Increasing the number of parameters by not sharing sub-network parameters also leads to some performance gain.
5.4 MUL.II-AASK REGRESSION
Implementation Details. We superpose sub-networks for multi-task regression with multiple RPGs the building-block level. We focus on predicting depth and normal maps from a given image. We stack multiple SharpNet (Ramamonjisoa & Lepetit 2019). network for monocular depth and normal estimation. Specifically, we create multiple RPGs the SharpNet building-blos blongk-bockkk level. That is, parameters of corresponding blocks of different sub-netwer networks are generated from the same RPG.
We evaluate the monocular depth and normal prediction performance on Standford 3D indoor scene dataset (Armeni et al.. 2017). It contains over 70K images with corresponding depths and normals covering over 6,000 m² indoor area. We follow all settings of SharpNet (Ramamonjisoa Lepetit 2019), state-of-the-at art monocular depth and normal estimation network.
Results and Analysis. We report the mean square errors for depth and normal estimation Table 5 Compared one-time inference without recurrence, our RPG network gives 3% and 2% gain for depth and normal estimation, respectively. Directly sharing weights but using new batch normalization layers decrease the performance by 1.2% and 0.3% for depth and normal. Sharing weights and normalization layers further decrease the performance by 0.7% and 0.9% for depth and normal.
5.5 PRUNING RPG
Table 6: Comparison with fine-grained pruning for reducing model size. Compared with IMP (Frankle) ct al. 2019) on CIFAR10, RPG achieves higher
Table 7: Comparison with coarse-grained prun- ing for reducing FLOPs and inference speed. RPG achieves similar performance as Knapsack (Aflalo et al., 2020) on ImageNet classification.
pruned accuracy and similar accuracy drops.
Ace before
Ace after
Params
Acc drop
Params 274k
# Params before praning Pruned acc. FLOPS
Res18-IMP
92.3
90.3
Res18-Knapsade
11,2M
69.35%
L09E9
Res18-RPG
95.0
93.0
2.0
274k
Pruned Res18-RPG
3.6M
69.10%
1993
Fine-Grained Pruning. Fine-grained pruning methods aim at reducing the model parameters by sparsifying weight matrices. Such methods usually do not reduce the inference speed, although custom algorithms (Gale et al.,, 2020p) may improve the speed. At the same number of parameters, RPG outperforms state-of-the-art fine-grained pruning method IMP (Frankle et al. 2019). Accuracy drops of RPG and IMP similar, both around 2% (Table 6). It is worth noting that IMP could have faster inference speed with sparse GPU kernels (Gale et al. 2020).
Coarse-Graines Pruning. While RPG is not designed to reduce FLOPs, it can be combined with coarse-grained pruning to reduce FLOPs. We prune RPG filters with lowest L norms. Table shows that the pruned RPG achieves on-par performance state-of-the-art coarse-grained pruning method Knapsack (Aflalo et al., 2020) at the same FLOPs.
5.6 ANALYSIS
Comparison to Pruning Methods. We report our ResNet18- 8-RRGG performance with different number of parameters on ImageNet and some baseline pruning methods in Right. Our RPG networks outperform SOTA pruning methods such as (Aflalo et al. 2020 Dong & Yang 2019 He et al. 2019: 2018 Dong et al., 2017, Khetan & Karnin, 2020). Specifically, at the same number of parameters, our RPG network has 0.6% gain over the knapsack pruning (Aflalo et al. 2020). a method that achieves the best ImageNet pruning accuracy.
Generalizability. We report the performance gap between training and validation set on ImageNet (Table a)) and MPII pose estimation (Table (()).. CPM (Wei et al., 2016) serves the baseline pose estimation method. RPG models consistently achieve lower gaps between training and validation set, indicating the RPG model suffers less from over-fitting
We also report the contain 50k images with 1 113 classes overlapping with ImageNet. Previous models are reported to
performance of RPG models. ObjectNet (Barbu et al. 2019)
8

---

Under review as a conference paper at ICLR 2022
Table 8: RPG increases the model generalizability. (a) ResNet with RPG has the lower gap between training and validation set on ImageNet classification. The metric is training accuracy minus validation accuracy. Lower is better. (b) Using RPG for pose estimation also decreases the training and validation performance GAP. The metric is training PCK @0.5 minus validation PCK @0.5. Lower better. (c) ResNet with RPG has higher performance on out-of-distributoon dataset ObjectNet 2019). The model is trained on ImageNet only and directly evaluated on ObjectNet.
[Barbu et al.
(a)
(b)
(c)
ResNet
RFG
no shared w
shared W
RPG
RI8
R14
R3
conv
-0.7
-2.
2x sub-nets sub-nets
1.15
1.13
0.64
Params
11M
11M
21M
34
II
-23
1.88
1.70
1.15
Acc. (%)
13.4
16.5
16.0
have large performance drop on ObjectNet. We directly evaluate the performance of ImageNet- trained model on ObjectNet without any fine-tuning (Table&jc). With the same number of backbone parameters, our ResNet-RPG achieves 3.1%% gain compared to ResNet18. With the same network architecture design, our ResNet-RPP -PN achieves 0.5% gain compared to ResNet34. This indicates our RPG networks have higher cut-of-distribution performance even with smaller model sizes.
Quantization. Network quantization can reduce model size with minimal accuracy drop. It is of interest to study if RPG models, whose parameters have been shrunk, can be quantized. After 8-bit quantization, the accuracy of DesNet18-RPG (5.6M parameters) only drop 0. percentage point on ImageNet, indicating RPG can be quantized for further size reduction. Details are in Appendix
Security. Permutation matrices generated by the random seed can be considered as security keys to decode the model. In addition, only random seeds generate transformation matrices need to be saved and transferred, which is efficient in terms of size.
5.7 ABLATION STUDIES
We conduct ablation studies to understand the functions of permutation and reflection matrices (Fig We evaluate ResNet-RPP34 with 2M backbone parameters. With permutation and reflection matrices leads to 76.5% accuracy, with permutation matrices only leads to 75.8%, with reflection matrices only leads to 71.1%, 1%, and with no permutation and reflection matrices leads to 70.7%. This suggests permutation and reflection matrices are useful for RPGs.
DISCUSSION
The common practice in machine learning is to search for the best model in large model space with many parameters or degrees of freedom (DoF), and then shrink the optimal model for deployment. Our key insight is that direct and opposite approach might work better: We start from lean model with a small DoF, which can be unpacked into a large model with many parameters. Then we can let the gradient descent automatically find the best model under this DoF constraint. Our work is departure from mainstream approaches towards model optimization and parameter reduction. We show how the model DoF and actual parameter size can be decoupled: We can define large model of arbitrary number of parameters with a small DoF.
We limit our scope to linear destructive weight sharing for different convolutional layers. However, in general, there might also exist nonlinear RPGs and efficient nonlinear generation functions to create convolutional kernels from a shared model ring W. Further, although RPG focuses on reducing model DoF, it can be quantized and pruned to further reduce the FLOPs and run time.
To sum up, we develop an efficient approach to build an arbitrarily complex neural network with any amount of DoF via a recurrent parameter generator. On a wide range of applications, including image classification, pose estimation and multitask regression, we show RPG networks consistently achieve higher performance the same model size. Further, analysis shows that such networks are less possible to overfit and have higher performance on out-of- dif-iitribnttin data.
RPG can be added to any existing network flexibly with any amount of DoF the user's discretion. It provides new perspectives for recurrent models, equilibrium models, and model compression. also serves tool for understanding the relationship between network properties and network DoF by factoring out the network architecture.
Reproducibility: We provide our code in supplementary materials.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik. Knapsack pruning with inner distillation. arXiv preprint arXiv:2002.08258, 2020.
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693 2014.
Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint orXiv:/702.01 1050000000000000 2017.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32:690-701, 2019.
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 33, 2020.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale seale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32:9453-9463, 2019.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Proceedings of Machine Learning and Systems, 2020.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few- shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Nicholas J Butko and Javier R Movellan. Optimal scanning for faster object detection. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2009.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2018
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4733-4742, 2016.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference machine learning, pp. 2285-2294. PMLR, 2015.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In Advances in neural information processing systems, 2019.
Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 224-932, 2021.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gareett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5840-5848, 2017.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: /903.016/1 2019.
10

---

Under review as a conference paper at ICLR 2022
Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Fixed point networks: Implicit depth models with jacobian-fres backprop. arXiv preprint arXiv:2103.12803 2021.
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse GPU kernels for deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, 2020.
Charles D Gilbert and Mariano Sigman. Brain states: top-down influences in sensory processing. Neuron, 54(5):677-696, 2007.
Song Han, Huizi Mao, and William J Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proceedings of the International Conference Learning Representations, 2016.
Yongchang Hao, Shilin He, Wenxiang Jiao, Zhaopeng Tu, Michael Lyu, and Xing Wang. Multi-task learning with shared encoder for con-eutoreer machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3989-39999999 1399, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference Artificial Intelligence, pp. 2234 2240,220,220 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340 4349, 2019.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010. 14701, 2020.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprini arXiv: 01..74444.444/ 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, (8)))6886-6888, 2017.
JM Hupé, AC James, BR Payne, SG Lomber, P Girard, and J Bullier. Cortical feedback improves discrimination between figure and background by v1, v2 and v3 neurons. Nature, 394(5))5): 784-787, 1998.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 1602.0736) 2016.
Theofanis Karaletsos and Thang D. Bui. Hierarchical gaussian process priors for bayesian neural network weights. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2020.
Theofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani. Probabilistic metaarrppresentations of neural networks. arXiv preprint arXiv: 1810.00555 2018.

---

Under review as a conference paper at ICLR 2022
Andrej Karpathy and Li Fei-Fei. Deep visual-semanti aligmments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3112 2015.
Ashish Khetan and Zohar Karnin. Prunenet: Channel pruning via global importance. arXiv preprint arXiv:2005. 1282, 2020.
Anders Krogh and John Hertz. simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Yann LeCun, D Touresky, G Hinton, and Sejnowski. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pp. 21-28, 1988.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Ke Li, Bharath Hariharan, and Jitendra Malik. Iterative instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3659-3667, 2016.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
C Louizos, M Reisser, T Blankevoort, E Gavves, and M Welling. Relaxed quantization for dis- cretized neural networks. In International Conference on Learning Representattons. International Conference on Learning Representations, ICLR, 2019.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, 2014.
Michael C Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science, 1(1):3-16, 1989.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pp. 483-499. Springer, 2016.
Varun Ramakrishna, Daniel Munoz, Martial Hebert, James Andrew Bagnell, and Yaser Sheikh. Pose machines: Articulated pose estimation via inference machines. In European Conference on Computer Vision, pp. 33-47. Springer, 2014.
Michael Ramamonjisoa and Vincent Lepetit. Sharpnet: Fast and accurate recovery of occluding contours in monocular depth estimation. The IEEE International Conference on Computer Vision (ICCV) Workshops, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pp. 525-542. Springer, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1999-1958 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv: 150:::00000038 2015.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines. 8(2):131-62, 2007.
Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypercube-baie based encoding for evolving large-scale neural networks. Artificial life, 1552)21 2009.
12

---

Under review as a conference paper at ICLR 2022
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14, 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 12965-12974, 2020.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference machine learning, pp. 1058-1066. PMLR, 2013.
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11505- 2020a.
Tiancai Wang, Xiangyu Zhang, and Jian Sun. Implicit feature pyramid network for object detection. arXiv preprint arXiv:2012.13556 2020b.
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 4724 2016.
David Weiss and Benjamin Taskar. Structured prediction cascades. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 916-923. JMLR Workshop and Conference Proceedings, 2010.
David H Wolpert. Stacked generalization. Neural networks, 5(2): 1992.
Dean Wyatte, Tim Curran, and Randall O'Reilly. The limits of feedforward vision: Recurrent processing promotes robust object recognition when objects are degraded. Journal of Cognitive Neuroscience, 24(11))2248-2261, 2012.
Shi Xingjian, Zhourong Chen, Hao Wang, Dit- Din-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting, In Advances in neural information processing systems, pp. 802-810, 2015.
Zhaohui Yang, Yunhe Wang, Chuanjian Liu, Hanting Chen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Legonet: Efficient convolutional neural networks with lego filters. In International Conference on Machine Learning, pp. 7005-7014. PMLR, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018
Amir Zamir, Te-Lin Wu, Lin Sun, William Shen, Bertram Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1308-1317, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
13


Reviews:
[{"review_id": "hGYCB_bEzZ7", "reviewer": "Reviewer_W9rN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n(Neutral) The parameters of each convolution layer are generated by the generator. Could the authors discuss it with \"Parameter prediction for unseen deep architecture\"?\n\n(Positive) Though the novelty of this paper is limited, the proposed method achieves significant performance gain compared to the existing works.\n\n(Negative) My main concern is that although the proposed method can build an arbitrarily complex neural network with any amount of parameters, the computational complexity is not reduced. It would be good for the authors to rethink the practical use of the proposed method. For example, although the parameters are saved, the FLOPS are not reduced. To some extent, the proposed method is related to network pruning. But all in all, the proposed technique is far less practical than network pruning methods.\n\n(Negative) I understand that the proposed method shares some merits of convolution in weight sharing. But sharing weights in spatial is different from sharing weights in depths. Sharing weights in the spatial can (i) save parameters, (ii) can ease optimization, and (iii) can accelerate the computation by using matrix multiplication. But sharing weights in depths (i) will hurt model performances and (ii) cannot reduce computational cost. Therefore, the proposed method might not be able to be regarded as \"Efficient.\"\n\n(Positive) Equation (2) is elegant and precise.\n\n(Positive) The following property is elegant: \"Since M is usually large, the same filter from Ki, Kj are close to orthogonal and generally dissimilar.\"\n\n(Negative) Due to the existence of matrix {A_i}, there would be additional computation cost in the generator, which further increases the computation complexity. In other words, although the parameters are saved, the computational cost might increase.\n\n(Positive) It is reasonable to have a separate BN. It is reasonable to RPGs at Block-Level and RPGs at Sub-Network-Level.\n\n(Negative) It would be good if the authors could apply the proposed method to the recent ViTs. Actually, based on my rich experience, vision transformers might have a more severe problem in parameter redundancy in the axis of depth. Maybe the proposed method can do well in ViTs. It would be nice if the authors could provide such studies to prove the generability of the proposed method.\n\n(Negative) The inference time should also be added to Table 2, Table 3, and Figure 4 to show whether the proposed method is at a disadvantage.\n\n(Positive) The observation of the Power Law is insightful, valuable, and useful, which benefits the community.\n\n(Negative) From Table 7, we can see that even combined with the network pruning method, the proposed method does not hold an advantage, not to mention that SOTA network pruning methods are not compared with.\n\n(Negative) Although the paper compares the proposed method with network pruning methods in Fig. 1 (right), the comparison is quite strange. Most typical network pruning methods focus on FLOPS pruning or the combination of FLOPs and parameters, but not merely parameters.\n\n\n(Positive) The following result is valuable: \"RPG models consistently achieve lower gaps between training and validation set, indicating the RPG model suffers less from over-fitting.\" The result on OOD data is also promising and valuable. It would be better to have an explanation.\n", "summary_of_the_paper": "\nThis paper uses parameter generation to generate parameters for different hidden layers in neural networks. There are several significant strengths and weaknesses in this paper. Especially, the practicability of the proposed method is questionable. It would be good to see my detailed comments below.\n", "summary_of_the_review": "\n\nBalancing the strengths and weaknesses of the proposed method, I would like to recommend a rating of weak rejection for this paper unless the authors provide a persuasive response to the above negative comments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aWXyldRXj0U", "reviewer": "Reviewer_cgCS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:  \n1. This paper proposes a method of weight sharing. The authors show that re-utilization of parameters generated by their recurrent parameter generator introduces diversity among kernel parameters within a single model. By reusing weights, the model size is reduced greatly. \n\nWeaknesses:  \n1. This method is weak in creativity. In contrast to conventional vector quantization or just quantization methods, the RPG is not better.    \n2. In Section.3, they refer to the batch normalization. In the stage of inference for normal CNNs, we usually fuse the convolutional layer and batch normalization layer to one conv layer. However, the proposed method is not easy to do this. If use fusion method, extra computations will be executed.   \n3. In the paper \"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding\", the compression ratio is neraly 2%. I can not see any comparison in the experiments.   \n4. The description of \"Destructive Weight Sharing\" is not easy to understand.", "summary_of_the_paper": "In order to reduce size of deep models, this work propose one parameter sharing method for different convolutional layers. With the help of one sharing set of parameters, all convolutional kernels can be generated from the sharing parameters. They show the effeciveness of this method in classification, pose estimation tasks.", "summary_of_the_review": "Inherently, this recurrent parameter generator (RPG) is one kind of vector quantization methods, which have alread studied in some papers. Such as the citiation \"LegoNet\" in the paper and \"Quantized Convolutional Neural Networks for Mobile Devices\" in CVPR2016.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PcKSG2ZSM9", "reviewer": "Reviewer_HpYd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThe proposed technique is simple yet effective. The evaluations show that it can be applied to different types of network architectures and in different tasks. I particularly find the generalization performance valuable. While the proposed model could be expected to achieve a lower gap between the training and the validation performance due to the smaller DoF, the out-of-distribution performance improves as well. \nThe proposed technique also achieves better performance by using permutations of the weights in the subsequent iterations, if the base architecture follows an iterative refinement by passing the output of a layer/sub-network to itself. In my opinion, this could be a more interesting contribution claim than the parameter reduction. \nThe proposed work enables a more efficient parameterization by reusing a smaller set of weights. It reduces the storage space. However, the actual model size and computational complexity remain the same. \n\nCons and questions:\n- Could the authors clarify the difference between the proposed method with the random weight assignment?  If my understanding is correct, the permutation P(M) results in a similar random assignment effect. The “even sampling” and the random sign reflections seem to be the main difference in this case. If so, why the proposed technique works significantly better than the random assignment? Unfortunately, this comparison is available only in Tab. 2 (i.e., “Res34-random weight share”). If the permutation P(M) takes away any structure in the parameter selection matrices R_i, is the model diagram in Fig.3 (upper) accurate anymore? \n\n- This is a follow-up to my previous comment. In the pose estimation task (section 5.3), the CPM shares weights for the sub-networks and the network size remains constant as the number of sub-networks increases. RPG assigns the same weights to these sub-networks but shuffles them. Could it be concluded that using a different permutation of the same weights in the subsequent iterations improves the performance? Could the authors clarify and comment on this?\n- Although parameter reduction is one of the major contribution claims, only in one experiment (Tab. 2), parameter reduction baselines (i.e., -hash and -lego entries) are used. \n\n- It is not discussed in the paper. How does the proposed reduction technique affect the training convergence? \n\n- The plot in Fig.1 is not clear. Only the top-right corner of the plot could be illustrated. Similarly, Fig. 4 is hard to interpret with the caption. \n\n- I could not get the contribution claim in the security paragraph in section 5.6. Doesn’t it hold for any network? \n\n- The following work could be discussed in the related work:\nDehghani, Mostafa, et al. \"Universal transformers.\" arXiv preprint arXiv:1807.03819 (ICLR 19’).\n\n\n-- Post-rebuttal update --\n\nI thank the authors for their clarifications and additional experiments. After reading other reviews and responses, I decided to keep my score. As stressed by the authors, I agree that the contribution of this paper is not limited to compression. However, the experiments focus on the evaluation of the proposed method mainly in the parameter reduction tasks rather than providing insights on why and how it works. The authors addressed our concerns, yet it is still unclear why random sign flipping and even weight assignment are so effective. In my opinion, a comprehensive comparison of the proposed destructive weight-sharing approach with different concepts could improve the paper's contribution. For example, the RPG at block or sub-network levels introduces some locality. The \"LegoNet\" paper proposes a more structured parameter sharing approach. Comparing various techniques from fully random to slightly structured could be more insightful. \nHowever, I still find the performance in the generalization and the OOD experiments and the applicability of the proposed approach on different architectures valuable. Hence, this paper can be considered for acceptance. ", "summary_of_the_paper": "The paper presents a technique for parameter reduction in neural networks. It follows the motivation of parameter redundancy in over-parameterized networks and aims to parameterize large networks with a smaller and shared set of weights. More specifically, a set of parameters based on a predetermined reduction factor is randomly assigned to various layers of a network and trained jointly. It is a simple technique that can be applied to any network architecture as the parameter generation is decoupled from the underlying architecture. The proposed method is evaluated in several tasks from classification to segmentation by using different network architectures. The experiments show improved or competitive performance when trained with similar or fewer parameters compared to the baselines.", "summary_of_the_review": "The paper presents interesting results and shows that the proposed technique offers efficient parameter utilization. Applying the proposed technique in the subsequent iterations of the same network outperforms the original model. While parameter reduction is one of the main contribution claims of the paper, the experiments lack direct baselines focusing on this task. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BJLElRWLygz", "reviewer": "Reviewer_xUeP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I find this work interesting, as RPG provides a simple (in the sense of how $\\mathbf{K}_i$'s are generated from $\\mathbf{W}$) but performant strategy of parameterizing deep neural networks. \n\nStrengths:\n1. The method itself is simple in nature, although the actual design, such as the motivation for using permutations and sign reflections, remains somewhat unclear (e.g., why not use other ways to create orthogonal matrix?). It is surprising to me how well this method works.\n2. The empirical results are relatively thorough, and the improvement over baselines is substantial.\n\nWeaknesses and questions:\n1. After reading this paper I still don't understand why RPG works. For example, compared to the conventional hypernetworks (which can be considered as a sort of input-based parameter generator, and so somewhat more reasonable), what is the new and key ingredient that RPG brings? From this point of view, while the exact form of RPG is novel, using a learnable module to generate parameters **is not**. While this is an empirical paper that presents a surprising finding, I feel that some discussions of why RPG is sensible/reasonable is lacking in the current version of the paper. \n2. I have some doubts over whether some experiment settings may bias towards the RPG-based networks. For instance, typically ResNets are trained on ImageNet for 100 epochs, and generally more epochs bring (diminishingly) better results. With 100 epochs, ResNet-18/-34 are actually able to reach very good performance already. While the authors acknowledge that they found \"this one [setting] to be the best for RPG\", I wonder if we might want to do the same for the baseline models. For instance, the numbers reported in the original ResNet paper [1]  seem already higher than in Table 3.\n3. Although the method is extensively compared with other pruning or compression methods like IMP and Knapsack, the RPG method itself can also be seen as a standalone model that differs from pure compressive/pruning efforts. After all, there is nothing \"compressed\" or \"sparse\" about the modeling. The fixed generating matrix, while unlearned, is actually still **parameters of the model**, which just happens can be efficiently stored using the seed trick because of the way pseudo-random number generators work. But from a modeling point of view, the parameters $\\theta$ of $f(\\cdot, \\theta)$ definitely still contains $\\mathbf{R}_i$. This means ResNet-RPG is only **one instantiation** of the RPG framework; and I'd be very interested in seeing whether RPG works for other architectures (e.g., Transformers) that are structurally very different. Implementation wise it should be pretty straightforward; you just need a new `SuperConv1d` or `SuperLinear` module.\n4. I like that the paper gets into a lot of discussions (e.g., quantization, security, log-linear DoF-accuracy relationship), but it mostly just briefly touched the surface of these topics. These are all important questions to investigate further, and the current paper hasn't been able to provide much insight other than the good empirical performance with ResNets.\n5. (This is a question, not a weakness.) For the comparison with MDEQ, do you use an RPG to generate the multiscale deep equilibrium layer's parameters (which creates multi-level feature map)? Or is it also just ResNet-RPG?\n\nMinor:\n- Table 9 format is slightly messed up.\n- In Sec. 5.2, the paper claims \"ResNet-RPG18 ... backbone parameters\". Probably you want to point of Appendix A here.\n\n\n[1] https://arxiv.org/pdf/1512.03385.pdf\n", "summary_of_the_paper": "The paper proposes recurrent parameter generator (RPG) that is able to generate (ideally) arbitrarily large model based on a fixed set of inputs $\\mathbf{W}$. Unlike common pruning or compressing techniques, the authors argue that RPG decouples the expressivity with the degree of freedom of a model, and that we can dynamically generate model parameters on the fly (while taking advantage of the pseudo-random seed) with a simple mechanism. Experiments show that this new way of generating model parameters is able to perform on par with, or better than, many existing compressive or pruning approaches.", "summary_of_the_review": "As mentioned in the main review, I'm not exactly satisfied with the lack of many potential discussions and further analysis of why and how RPG works. There clearly are a lot of things that can be studied here, and mere empirical evidence may not be the only thing that we need. Also, while the results are good, the method is not too different from the canonical hypernetwork idea which is also trained end-to-end. On the other hand, I feel the paper has a strong set of empirical evidence demonstrating the effectiveness of RPG and I'm surprised by the results.\n\n\n-------------------\n\nPost-rebuttal: See my comment below. I keep my score of 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta Review for Recurrent Parameter Generators

This work investigates a method for reducing the parameters of a deep CNN by having a recurrent parameter generator (RPG) produce the weights, in effect achieving this compression via parameter sharing across layers (similar to earlier works, such as the 2016 Hypernetworks paper, as discussed in between xUeP and the author during the review period). But unlike previous work, this work conducts extensive empirical experiments on classification and even pose estimate tasks, and proposes an additional method, such as the use of pseudo-random seed to perform element-wise random sign reflection in the weight sharing. The novelty and experimental results are clearly displayed in this work, and shows a lot of promise, but after much discussion, I currently cannot recommend acceptance for ICLR 2022.

In my assessment, and also looking at reviewers and discussion, I believe this work is a great workshop paper at present, but there are a few items that would make it much stronger. There are outstanding issues in the paper that need to be improved. In particular, during discussions, reviewers noted that the paper has a problem with the design and presentation of the experiments. It somehow shifts the reader’s focus to the compression task (3 of the 4 reviewers raised concerns about the compression performance and questioned the baselines). In their rebuttal, the authors emphasized that their contribution is not limited to compression but is rather more fundamental, and the authors propose an approach for understanding the relationship between the model DoF and the network performance. But if that's the main narrative of the paper, rather than the compression aspects, the authors need to clearly articulate why decoupling the DoF from the underlying architecture is advantageous (and also make the narrative more clear in the writing). While there are novel innovations in the method proposed, the authors also need to explain clearly why their method works well, why the even weight assignment and random sign flipping are so effective?

There is discussion between the authors and reviewers about what constitutes vector quantization, and I believe the author has clarified their position effectively (with regard to cgCS's review), and I believe this will be explained in great clarity in future revisions. But even with that disagreement out of the way, we still believe that this work needs improvement to meet the bar of ICLR 2022. Reviewers, including myself, do acknowledge the novelty and are excited about the method proposed, and we look forward to seeing an updated version of this work published or presented at a future journal or conference. Good luck!
Decision: Reject