Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the problem of obtaining more compact CNNs by a parameter sharing method. The authors propose to represent a weight filter in a low-rank subspace (represented as a linear combination of low-rank filter basis) plus a set of non-shared low-rank filter basis (per-layer). In this way, the shared low-rank filter basis is reused across several layers, and the non-shared ones per layer are used to enhance model generalization ability. Experiments are performed on CIFAR and ImageNet datasets, using some popular CNN structures for evaluation.

This paper suffers from following issues.

---Novelty issue.

To the best of my knowledge, representing a weight filter (in a specific convolutional layer) as a linear combination of a low-rank filter basis was explored in many works, such as “Efficient and Accurate Approximations of Nonlinear Convolutional Networks”, in CVPR 2015, and “Speeding up Convolutional Neural Networks with Low Rank Expansions”, in BMVC 2014. Additionally, even  the first two parts of Figure 1 are very similar to Figure 1 of CVPR 2015 paper. However, they are either missed or not discussed/compared.

Compared with existing low-rank approximation related works, the main contribution of this paper is sharing low-rank filter basis across several layers (in the same group or block of a CNN), while retaining a non-shared low-rank filter basis per layer. To me, the motivation is not clear enough, e.g., such kind of recursive design may easily lead to more high computational cost; it is usually not useful to get obvious benefits. Furthermore, from the definition in the method part, sharing low-rank filter basis across layers needs weight filters have the same shape size. Such a strong constraint also limits its applications.  Additionally, how to determine shared/non-shared low-rank filter basis for different CNNs and their building blocks is not clear enough, and they are manually tuned one by one. 

The authors claim that parameter sharing in this way can address “… a shared filter basis can cause vanishing gradients and exploding gradients problems”.  However, there is no convincing experiments to support this claim, e.g., how this happens in the experiments and how this is alleviated?

--- Experiments issue.

Comparison with related works such as existing low-rank methods and recursive parameter sharing methods are completely missing.  

Real wall-clock speed comparison is also missing.

On small dataset like CIFAR, experiments are run only once. Due to random effects, it may easily lead to different conclusions. For fair comparison, the authors should at least run experiments for several runs (e.g., 3/5) and report mean accuracies.

Also, on CIFAR-100 dataset, the authors use ResNet34/50/MobileNetV2. However, these CNNs are defined on ImageNet (larger image size 224x224), which have different numbers of down-sampling layers compared to those (ResNet32/56/110) for CIFAR dataset (small image size 32x32). What kind of modifications did the authors make?  Why not using more standard CNNs like  those used on CIFAR-10 to CIFAR-100?
review_title: Ok but not good enough 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary]
This method proposes to decompose convolutional filters using a low rank filter basis where the convolutional operation in a layer consists of shareable filter basis and non-shareable layer coefficients. This is developped to save computational costs whilst maintaining performance. To regularise against vanishing/exploding gradients and promoting more useful representations, the authors seek orthogonal filter basis'. This filter basis is shared across layers in contrast to other works who look at recursive sharing.

[Main Comments]
1. I like the idea to decompose convolutions into shared weights and layer-specific components. This builds in nicely with a lot of work in multi-task learning about learning which weights to share and so I wonder about how your method might generalise to other problems - could the authors comment on this?
2. The authors compared their method against ResNet, MobileNet and DenseNet. I would like to see comparisons against more similar techniques such as [Savarese & Maire 2019] and [Guo et al. 2019] to better appreciate the performance of the method. The results against ResNet/MobileNet are indeed promising and good to see.
3. The presentation of the results (Table 1 and Table 2) is cumbersome. I would suggest reworking this section to make it easier to appreciate the results.
4. The results in general seem good but due to presentation issues, it is difficult to discern what is going on. In particular, the nomenclature for ResNetL-SsUu is confusing.

[Other]
1. Could you add an extra parameter to the filter basis, to control dilation of the convolution and thus learn when it is needed to increase the receptive field in addition to sharing weights?
2. There is a typo in in the sentence above Equation (5) ; reusrive --> recursive
3. I would recommend a figure to display the differences between vanilla ResNet and your version
4. Was it possible to analyse empirically the gradient magnitude to show that the orthogonality regularisation helped?
5. For Figure 2 - I would potentially make a heatmap style figure with $u$ and $s$ as the x-y axis to help display results
6. How does orthogonality regularisation work on a 1x1 conv?
7. The work of Ioannou et al. (https://arxiv.org/abs/1605.06489) might be of interest as this seeks structured sparsity in ConvNets to decrease computation without loss of performance
review_title: Nice solution to decrease computation complexity in CNNs. Needs more work on results section (writing + experiments) 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proposed a parameter-sharing method among repetitive convolution layers, where typical filters are decomposed into a set of resuable filter bases and coefficients. The experimental results show some improvement about the number of parameters and FLOPs. Generally, the paper is well written and the method is presented clearly. The authors also claimed that orthogonality regularization can reduce the potential vanishing/exploding gradients problem in weight sharing training. 
Here are my concerns:
(1) The experiments mainly compared the proposed method with non-shared net designs. However, the comparison results or methods/algorithms with other weight sharing paper are lacking. (e.g., Jastrzebski et al.,2018; Köpüklü et al., 2019 mentioned in the paper). Therefore, it's not easy to judge the novelty of this incremental work. 
(2) Some of the non-shared baseline accuracy is not as good as reported in previous works (e.g., CIFAR100 accuracy-parameter, MobileNetV2 accuracy.). Some comparative results are not fair enough, for example, in the ImageNet experiments, MobileNetV2-Shared model is trained for 300 epochs but the baseline is only trained for 140 epochs.
(3) The author should, I suggest, focus more on the actual performance such as memory and time overheads during network training and deployment inference, rather than theoretical MACs and FLOPs, (also discussed in ShuffleNet_v2 paper ) which can contribute more to the compact NN community.


review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: To compress a DNN model, the proposed method improves traditional kernel sharing methods by sharing kernel bases obtained from decomposing a convolution layer. Experiments on ImageNet show that the method reduces 10.01M parameters of ResNet-34 with almost the same error rate, i.e., the method is more effective for DNN architectures without using separable convolution.

pros.
1. In Equations (1)~(3), the authors clearly and formally explain one of their motivations that a convolution layer with high computing complexity can be replaced by a convolution layer with lower computing complexity and a liner layer.
2. A classic problem, exploding/vanishing gradients, is alleviated by keeping a variable W_basis close to orthogonal. Moreover, in Section 4.1.3, experiments without orthogonality and with orthogonality are qualitatively and quantitatively compared.
3. In Table 3, the proposed method reduces 10.01M parameters of ResNet-34 with almost the same error rate.

cons.
1. The novelty/contribution is not quite sufficient. The main novely of this paper is to simultaneously train convolution decomposing and weight sharing. The strategy of convolution decomposing (as shown in Equation 3) in this paper is similar with separable convolution used in MobileNet. For weight sharing, the authors propose to share weight across layers which is also similar with previous weight sharing papers.
2. The experiments are not sufficient. In Section Experiments, the proposed method is only compared with classic ResNet-34 and MobileNet-V2. However, tons of traditional DNN compressiong papers work on decomposing a convolutional layer, matrix decomposing, or sharing weights. The proposed method should be compared with these traditional methods.
review_title: The proposed method achives outstanding performance on ResNet-34, while I have concerns on main contribution and experiments

Meta-review: I agree with the concerns raised by the reviewers. In particular, the issues of novelty and experimental evaluation (mentioned in the revision summary) remain the major weak points of the paper. My impression is that the changes made in the revision represent a significant experimental addition to the paper, one which might merit a full pass through peer review, and one which in any event did not alter the reviewers' scores. While I think this paper has something to contribute (and the empirical results suggest the method may outperform competitors), I think it would be improved by a rewrite (and possibly a restructure) that makes the part that is your contribution much more clear. For example, in the abstract, it's only in the sentence "We show both theoretically
and empirically that potential vanishing/exploding gradients problems can be mitigated by enforcing orthogonality to the shared filter bases" that we actually get to the part that is really novel about this contribution (the "enforcing orthogonality"): that would ideally be much earlier in the abstract.
Decision: Reject