Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors try to improve the parameter efficiency of neural networks by using recursively shared filter basis. To address the gradient vanishing/exploding problem in naive recursively parameter sharing, orthogonality regularization is proposed and shows effectiveness in ablation studies.
main_review: - The motivation of this paper is very clear. Parameter efficiency is more important in resource-constrained devices like Jetson TX2.
- The proposed parameter sharing approach shows reasonable performance on different benchmarks including ImageNet classification and COCO detection.
- Ablations and analysis are provided to show the effectiveness of proposed orthogonality regularization.
limitations_and_societal_impact: - The position of shared filter basis used in different networks is kind of confusing. It would be great if the authors can explain the reason to not share every convolutions in the blocks (as there're still some non-shared elements in shared conv), maybe also with some ablation studies.
- The parameter reduction in compact models is not very impressive.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors introduce a new parameter sharing scheme in CNNs by learning orthogonal filter bases, and representing each convolutional filters as a linear combination of the bases. 
Notably, the authors show both empirically and theoretically that orthogonal regularization imposed on the learned filter bases improves the stability of gradients and helps achieve better overall performance.
main_review: While learning shared filter basis for parameter reduction in deep neural networks is not a new idea, the discussion on the orthogonality of the bases and its advantages on gradient flow is interesting and constructive. 
The introduced method is well presented and the paper is easy to follow, with source code provided for reproduction. 
Wide range of experiments including image classification, object detection, and instance segmentation, are appreciated. 

I have following concerns:
- As discussed in L81, the authors claim gradient issues cause the performance drop in [1], any support on this?

- Following the previous question, the main contribution of this paper remains a bit unclear to me. If the orthogonal regularization is indeed a principle way of stabilizing the gradient and improve the performance of parameter sharing in CNNs, can it be apply to other methods for improved performance? Or it can only improve the performance of the proposed method?

- How are all the hyperparameters determined? Further ablation study is expected, e.g. the additional orthogonality regularization inevitably introduces new hyperparameter for tuning.  

- Missing references and comparisons [2, 3, 4]. These are all published parameter sharing methods for networks and need to be discussed in the paper. Comparisons against [1] are also expected. 

- The term 'Deeply' in the proposed method is a bit confusing to me. It seems that the authors are trying to emphasize 'deep', then how is the proposed method different from the previous parameter sharing methods in terms of 'deep'? 

[1] Learning implicitly recurrent CNNs through parameter sharing, ICLR 2019
[2] Legonet: Efficient convolutional neural networks with lego filters, ICML 2019
[3] Structured convolutions for efficient neural network design, NeurIPS 2020
[4] HyperNetworks, ICLR 2017

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a method to share parameters between iterations of a residual block.  Rather than sharing all parameters or none, a filter basis W_basis and mixture coefficients alpha are defined; the basis is shared between layers, while the coefficients are unshared.  A theoretical argument shows a potential cause of training instability due to repeated application of a shared weights term; an orthonormality regularizer is applied to the weights basis, to help alleviate vanishing/exploding gradients.  Measurements are performed on ImageNet, CIFAR and COCO, applied to a variety of resnets and mobilenet.  The resulting model uses about half as many parameters (though similar number of FLOPs) as a baseline model, and achieves similar or slightly better accuracy.

main_review: The system is similar to Savarese and Maire 2019, which is described in the related work.  The differences here appear to be (a) the analysis of repeated weights that may cause large/small gradients, (b) the orthonormality regularizer, and (c) the use of small numbers of unshared weights in addition to basis-generated weights.

While the theoretical argument seemed persuading, and the final system in showed good results in the experiments, I think additional experimental baselines and comparisons would make the benefits of the method more convincing.  In particular, there are no comparisons for using no unshared elements (U=0), or the naive approach of fully sharing weights within residual groups.  (Savarese and Maire is also not compared directly, though a configuration of this system without regularizer may be equivalent).

Likewise, the effect of the orthogonality regularizer is demonstrated to create larger gradients (if I'm interpreting the plot correctly, see below additional comments), but the final effect on CIFAR-100 is about 1% accuracy gain in the best case, which is good but seems a little small compared to the motivation of unstable gradients --- might there be other effects, e.g. on the number of steps needed in the training schedule?

Overall, the system improves upon similar parameter sharing schemes, with good results.  But it's unclear how much it helps relative to a few other simple baselines, described above.  Including these would strengthen the experimental case.



Additional comments and questions:


- Steerable weights in related work:  There is some work on steerable cnns that I feel is related and might be discussed; these systems use transformed weights for enforced equivariance, e.g. https://arxiv.org/pdf/1612.08498.pdf, https://openaccess.thecvf.com/content_cvpr_2018/CameraReady/3214.pdf


- What is the effect of orthgonality regularizer when training on ImageNet?  The difference in classification error from this term is only shown for a subset of the cifar experiments (resnet34 on cifar100 and resent56 on cifar10).  What happens if orthogonality regularizer is not used on ImageNet?  More comprehensive experiments on this would be more convincing.

- What is the effect of lambda in the loss?  To me it looks it would be relatively unsensitive to the exact value, and it should be set fairly low, around 0.01 or lower.  But would be good to confirm by showing an experiment for this.  


- Compare to naively shared weights (where successive block iters use the same weights):  I didn't see this comparison in the paper --- in addition to unshared baseline, it would be good to see param count and accuracy drop of the baseline model with weights shared in each stage, to confirm its accuracy loss.


- Effect of U0 vs U1:  The text says that a small number of unshared weights help performacne, but I didn't see this evaluated; the smallest number of unshared basis elements is 1.  What about 0 (all shared)?



- sec 4.4 Fig 3:  I don't understand how to interpret this plot, more explanation may be needed.  What does the varying lightness of the light blue bars mean --- are there multiple overlapping translucent bars, and why?  Also, what does a value of 0.0075 mean --- relative to what?  Are these per-element gradients, and if so, what is this value relative to the weight, and why is this value good?  What about the step size, incorporating the optimizer learning rate, momentum etc?  If gradients are larger or better conditioned, can the training schedule be reduced?


- sec 4.4 Fig 4:  What are the similarity matrices for groups 1 and 4?  Showing only 2 and 3 seems incomplete to me.  Do they show the same pattern?  And if not, what might be a cause for difference?


- eq. 5 and text:  Since the alpha coefficients don't appear here explicitly, and are part of the repeated net (with unshared weights), I think it could be useful to explain where they are here (as part of d x^j / d a^j), and relate to the paragraph on l.156 on what influences their stability


- Appendix fig A.1:  u=0 is missing from these curves, so it's hard to conclude that having one unshared element helps, as the text says.  Also would be good to have error bars over a few different random inits, as the errors are jumping around 1% out of 22%.  This is a nice plot for the supplement, though, I thought it was interesting to see.



- l.148 "If Wbasis is orthogonal ... ":  this sentence should probably also use "orthonormal"

- l.227: resnet50 does not appear in the table






limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a recursive convolutional block design for compact CNNs with efficient training. The convolutional filters (across all layers) are decomposed into shared basis filters and non-shared expanding coefficients, both of which are trained in an end-to-end manner. To avoid vanishing/exploding gradients, orthogonality regularization is imposed, and the efficacy of which is shown through both theoretical analysis and empirical verification. The proposed architecture consistently outperforms the corresponding baseline in terms of accuracy with a reduced model size.
main_review: ** Pros **
1. The paper is very well-written and well-organized. It is easy to read.
2. Imposing orthogonality constraint on the shared basis is interesting, and its effectiveness is explained both theoretically and empirically.
3. A small portion of non-shared layer-specific basis is also used to maintain expressiveness of the model.
4. Extensive numerical experiments have been conducted to demonstrate the effectiveness of the model.

** Cons **
1. Missing citation: the idea of filter decomposition into shared basis and non-shared coefficients is not new, e.g., it is also proposed in Qiu et al., "DCFNet: Deep Neural Network with Decomposed Convolutional Filters", ICML 2018.
2. Figure 3 reports the effectiveness of regularization in preventing vanishing gradients. Is there a visual illustration demonstrating how the regularization can prevent exploding gradients?
3. The choice of lambda is rarely mentioned in the manuscript. An ablation study on lambda should be included in section 4.4.

Overall it is an interesting paper (even though the idea of filter decomposition has been widely studied and adopted), but I would love to see the feedbacks from the authors.

** Review after author feedback **
The authors have partially addressed my concerns. I am raising my rating to 6.
limitations_and_societal_impact: No comment.
review_title: null

Meta-review: Authors propose to use recursively shared filter basis for compact CNNs. I agree with most reviewers that the idea of shared filter basis in CNN is not new and similar systems have been discussed before. However, ideas such as imposing orthogonality constraint on the shared basis seem interesting, and the effectiveness is well explained. Extensive experiments have been conducted to show improvement. 

3 out of 4 reviewers are inclined to accept the paper. 
Decision: Accept (Poster)