Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes recurrent parameter generator (RPG) that is able to generate (ideally) arbitrarily large model based on a fixed set of inputs $\mathbf{W}$. Unlike common pruning or compressing techniques, the authors argue that RPG decouples the expressivity with the degree of freedom of a model, and that we can dynamically generate model parameters on the fly (while taking advantage of the pseudo-random seed) with a simple mechanism. Experiments show that this new way of generating model parameters is able to perform on par with, or better than, many existing compressive or pruning approaches.
main_review: Overall I find this work interesting, as RPG provides a simple (in the sense of how $\mathbf{K}_i$'s are generated from $\mathbf{W}$) but performant strategy of parameterizing deep neural networks. 

Strengths:
1. The method itself is simple in nature, although the actual design, such as the motivation for using permutations and sign reflections, remains somewhat unclear (e.g., why not use other ways to create orthogonal matrix?). It is surprising to me how well this method works.
2. The empirical results are relatively thorough, and the improvement over baselines is substantial.

Weaknesses and questions:
1. After reading this paper I still don't understand why RPG works. For example, compared to the conventional hypernetworks (which can be considered as a sort of input-based parameter generator, and so somewhat more reasonable), what is the new and key ingredient that RPG brings? From this point of view, while the exact form of RPG is novel, using a learnable module to generate parameters **is not**. While this is an empirical paper that presents a surprising finding, I feel that some discussions of why RPG is sensible/reasonable is lacking in the current version of the paper. 
2. I have some doubts over whether some experiment settings may bias towards the RPG-based networks. For instance, typically ResNets are trained on ImageNet for 100 epochs, and generally more epochs bring (diminishingly) better results. With 100 epochs, ResNet-18/-34 are actually able to reach very good performance already. While the authors acknowledge that they found "this one [setting] to be the best for RPG", I wonder if we might want to do the same for the baseline models. For instance, the numbers reported in the original ResNet paper [1]  seem already higher than in Table 3.
3. Although the method is extensively compared with other pruning or compression methods like IMP and Knapsack, the RPG method itself can also be seen as a standalone model that differs from pure compressive/pruning efforts. After all, there is nothing "compressed" or "sparse" about the modeling. The fixed generating matrix, while unlearned, is actually still **parameters of the model**, which just happens can be efficiently stored using the seed trick because of the way pseudo-random number generators work. But from a modeling point of view, the parameters $\theta$ of $f(\cdot, \theta)$ definitely still contains $\mathbf{R}_i$. This means ResNet-RPG is only **one instantiation** of the RPG framework; and I'd be very interested in seeing whether RPG works for other architectures (e.g., Transformers) that are structurally very different. Implementation wise it should be pretty straightforward; you just need a new `SuperConv1d` or `SuperLinear` module.
4. I like that the paper gets into a lot of discussions (e.g., quantization, security, log-linear DoF-accuracy relationship), but it mostly just briefly touched the surface of these topics. These are all important questions to investigate further, and the current paper hasn't been able to provide much insight other than the good empirical performance with ResNets.
5. (This is a question, not a weakness.) For the comparison with MDEQ, do you use an RPG to generate the multiscale deep equilibrium layer's parameters (which creates multi-level feature map)? Or is it also just ResNet-RPG?

Minor:
- Table 9 format is slightly messed up.
- In Sec. 5.2, the paper claims "ResNet-RPG18 ... backbone parameters". Probably you want to point of Appendix A here.


[1] https://arxiv.org/pdf/1512.03385.pdf

summary_of_the_review: As mentioned in the main review, I'm not exactly satisfied with the lack of many potential discussions and further analysis of why and how RPG works. There clearly are a lot of things that can be studied here, and mere empirical evidence may not be the only thing that we need. Also, while the results are good, the method is not too different from the canonical hypernetwork idea which is also trained end-to-end. On the other hand, I feel the paper has a strong set of empirical evidence demonstrating the effectiveness of RPG and I'm surprised by the results.


-------------------

Post-rebuttal: See my comment below. I keep my score of 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a technique for parameter reduction in neural networks. It follows the motivation of parameter redundancy in over-parameterized networks and aims to parameterize large networks with a smaller and shared set of weights. More specifically, a set of parameters based on a predetermined reduction factor is randomly assigned to various layers of a network and trained jointly. It is a simple technique that can be applied to any network architecture as the parameter generation is decoupled from the underlying architecture. The proposed method is evaluated in several tasks from classification to segmentation by using different network architectures. The experiments show improved or competitive performance when trained with similar or fewer parameters compared to the baselines.
main_review: Pros:
The proposed technique is simple yet effective. The evaluations show that it can be applied to different types of network architectures and in different tasks. I particularly find the generalization performance valuable. While the proposed model could be expected to achieve a lower gap between the training and the validation performance due to the smaller DoF, the out-of-distribution performance improves as well. 
The proposed technique also achieves better performance by using permutations of the weights in the subsequent iterations, if the base architecture follows an iterative refinement by passing the output of a layer/sub-network to itself. In my opinion, this could be a more interesting contribution claim than the parameter reduction. 
The proposed work enables a more efficient parameterization by reusing a smaller set of weights. It reduces the storage space. However, the actual model size and computational complexity remain the same. 

Cons and questions:
- Could the authors clarify the difference between the proposed method with the random weight assignment?  If my understanding is correct, the permutation P(M) results in a similar random assignment effect. The “even sampling” and the random sign reflections seem to be the main difference in this case. If so, why the proposed technique works significantly better than the random assignment? Unfortunately, this comparison is available only in Tab. 2 (i.e., “Res34-random weight share”). If the permutation P(M) takes away any structure in the parameter selection matrices R_i, is the model diagram in Fig.3 (upper) accurate anymore? 

- This is a follow-up to my previous comment. In the pose estimation task (section 5.3), the CPM shares weights for the sub-networks and the network size remains constant as the number of sub-networks increases. RPG assigns the same weights to these sub-networks but shuffles them. Could it be concluded that using a different permutation of the same weights in the subsequent iterations improves the performance? Could the authors clarify and comment on this?
- Although parameter reduction is one of the major contribution claims, only in one experiment (Tab. 2), parameter reduction baselines (i.e., -hash and -lego entries) are used. 

- It is not discussed in the paper. How does the proposed reduction technique affect the training convergence? 

- The plot in Fig.1 is not clear. Only the top-right corner of the plot could be illustrated. Similarly, Fig. 4 is hard to interpret with the caption. 

- I could not get the contribution claim in the security paragraph in section 5.6. Doesn’t it hold for any network? 

- The following work could be discussed in the related work:
Dehghani, Mostafa, et al. "Universal transformers." arXiv preprint arXiv:1807.03819 (ICLR 19’).


-- Post-rebuttal update --

I thank the authors for their clarifications and additional experiments. After reading other reviews and responses, I decided to keep my score. As stressed by the authors, I agree that the contribution of this paper is not limited to compression. However, the experiments focus on the evaluation of the proposed method mainly in the parameter reduction tasks rather than providing insights on why and how it works. The authors addressed our concerns, yet it is still unclear why random sign flipping and even weight assignment are so effective. In my opinion, a comprehensive comparison of the proposed destructive weight-sharing approach with different concepts could improve the paper's contribution. For example, the RPG at block or sub-network levels introduces some locality. The "LegoNet" paper proposes a more structured parameter sharing approach. Comparing various techniques from fully random to slightly structured could be more insightful. 
However, I still find the performance in the generalization and the OOD experiments and the applicability of the proposed approach on different architectures valuable. Hence, this paper can be considered for acceptance. 
summary_of_the_review: The paper presents interesting results and shows that the proposed technique offers efficient parameter utilization. Applying the proposed technique in the subsequent iterations of the same network outperforms the original model. While parameter reduction is one of the main contribution claims of the paper, the experiments lack direct baselines focusing on this task.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper uses parameter generation to generate parameters for different hidden layers in neural networks. There are several significant strengths and weaknesses in this paper. Especially, the practicability of the proposed method is questionable. It would be good to see my detailed comments below.

main_review: 

(Neutral) The parameters of each convolution layer are generated by the generator. Could the authors discuss it with "Parameter prediction for unseen deep architecture"?

(Positive) Though the novelty of this paper is limited, the proposed method achieves significant performance gain compared to the existing works.

(Negative) My main concern is that although the proposed method can build an arbitrarily complex neural network with any amount of parameters, the computational complexity is not reduced. It would be good for the authors to rethink the practical use of the proposed method. For example, although the parameters are saved, the FLOPS are not reduced. To some extent, the proposed method is related to network pruning. But all in all, the proposed technique is far less practical than network pruning methods.

(Negative) I understand that the proposed method shares some merits of convolution in weight sharing. But sharing weights in spatial is different from sharing weights in depths. Sharing weights in the spatial can (i) save parameters, (ii) can ease optimization, and (iii) can accelerate the computation by using matrix multiplication. But sharing weights in depths (i) will hurt model performances and (ii) cannot reduce computational cost. Therefore, the proposed method might not be able to be regarded as "Efficient."

(Positive) Equation (2) is elegant and precise.

(Positive) The following property is elegant: "Since M is usually large, the same filter from Ki, Kj are close to orthogonal and generally dissimilar."

(Negative) Due to the existence of matrix {A_i}, there would be additional computation cost in the generator, which further increases the computation complexity. In other words, although the parameters are saved, the computational cost might increase.

(Positive) It is reasonable to have a separate BN. It is reasonable to RPGs at Block-Level and RPGs at Sub-Network-Level.

(Negative) It would be good if the authors could apply the proposed method to the recent ViTs. Actually, based on my rich experience, vision transformers might have a more severe problem in parameter redundancy in the axis of depth. Maybe the proposed method can do well in ViTs. It would be nice if the authors could provide such studies to prove the generability of the proposed method.

(Negative) The inference time should also be added to Table 2, Table 3, and Figure 4 to show whether the proposed method is at a disadvantage.

(Positive) The observation of the Power Law is insightful, valuable, and useful, which benefits the community.

(Negative) From Table 7, we can see that even combined with the network pruning method, the proposed method does not hold an advantage, not to mention that SOTA network pruning methods are not compared with.

(Negative) Although the paper compares the proposed method with network pruning methods in Fig. 1 (right), the comparison is quite strange. Most typical network pruning methods focus on FLOPS pruning or the combination of FLOPs and parameters, but not merely parameters.


(Positive) The following result is valuable: "RPG models consistently achieve lower gaps between training and validation set, indicating the RPG model suffers less from over-fitting." The result on OOD data is also promising and valuable. It would be better to have an explanation.

summary_of_the_review: 

Balancing the strengths and weaknesses of the proposed method, I would like to recommend a rating of weak rejection for this paper unless the authors provide a persuasive response to the above negative comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In order to reduce size of deep models, this work propose one parameter sharing method for different convolutional layers. With the help of one sharing set of parameters, all convolutional kernels can be generated from the sharing parameters. They show the effeciveness of this method in classification, pose estimation tasks.
main_review: 
Strengths:  
1. This paper proposes a method of weight sharing. The authors show that re-utilization of parameters generated by their recurrent parameter generator introduces diversity among kernel parameters within a single model. By reusing weights, the model size is reduced greatly. 

Weaknesses:  
1. This method is weak in creativity. In contrast to conventional vector quantization or just quantization methods, the RPG is not better.    
2. In Section.3, they refer to the batch normalization. In the stage of inference for normal CNNs, we usually fuse the convolutional layer and batch normalization layer to one conv layer. However, the proposed method is not easy to do this. If use fusion method, extra computations will be executed.   
3. In the paper "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding", the compression ratio is neraly 2%. I can not see any comparison in the experiments.   
4. The description of "Destructive Weight Sharing" is not easy to understand.
summary_of_the_review: Inherently, this recurrent parameter generator (RPG) is one kind of vector quantization methods, which have alread studied in some papers. Such as the citiation "LegoNet" in the paper and "Quantized Convolutional Neural Networks for Mobile Devices" in CVPR2016.
review_title: null

Meta-review: Meta Review for Recurrent Parameter Generators

This work investigates a method for reducing the parameters of a deep CNN by having a recurrent parameter generator (RPG) produce the weights, in effect achieving this compression via parameter sharing across layers (similar to earlier works, such as the 2016 Hypernetworks paper, as discussed in between xUeP and the author during the review period). But unlike previous work, this work conducts extensive empirical experiments on classification and even pose estimate tasks, and proposes an additional method, such as the use of pseudo-random seed to perform element-wise random sign reflection in the weight sharing. The novelty and experimental results are clearly displayed in this work, and shows a lot of promise, but after much discussion, I currently cannot recommend acceptance for ICLR 2022.

In my assessment, and also looking at reviewers and discussion, I believe this work is a great workshop paper at present, but there are a few items that would make it much stronger. There are outstanding issues in the paper that need to be improved. In particular, during discussions, reviewers noted that the paper has a problem with the design and presentation of the experiments. It somehow shifts the reader’s focus to the compression task (3 of the 4 reviewers raised concerns about the compression performance and questioned the baselines). In their rebuttal, the authors emphasized that their contribution is not limited to compression but is rather more fundamental, and the authors propose an approach for understanding the relationship between the model DoF and the network performance. But if that's the main narrative of the paper, rather than the compression aspects, the authors need to clearly articulate why decoupling the DoF from the underlying architecture is advantageous (and also make the narrative more clear in the writing). While there are novel innovations in the method proposed, the authors also need to explain clearly why their method works well, why the even weight assignment and random sign flipping are so effective?

There is discussion between the authors and reviewers about what constitutes vector quantization, and I believe the author has clarified their position effectively (with regard to cgCS's review), and I believe this will be explained in great clarity in future revisions. But even with that disagreement out of the way, we still believe that this work needs improvement to meet the bar of ICLR 2022. Reviewers, including myself, do acknowledge the novelty and are excited about the method proposed, and we look forward to seeing an updated version of this work published or presented at a future journal or conference. Good luck!
Decision: Reject