Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a scheme for partially sharing CNN parameter across layers.  Specifically, convolutional layer parameters are factored into the product of layer-specific filter atoms (which define spatial filter structure) and a set of shared coefficient that mix features across channels.  Training CNNs reparameterized in this manner yields improvements along the accuracy-parameter tradeoff curve, compared to baseline models on image classification tasks.
main_review: The factorization approach for convolutional layers is exactly that proposed in DCFNet [Qui et al., 2018].  The difference is in terms of choosing to share channel mixing coefficients between multiple layers.  However, [Qui et al., 2018] focus on another means of parameter reduction -- namely, choosing a fixed basis for the filter atoms, and report accuracy-parameter efficiency improvements.

As the paper makes use of the same factorization as DCFNet [Qui et al., 2018], I would expect an experimental comparison between DCFNet and the proposed approached.  The paper also cites [Li et al., 2019] as another previous work that uses a filter basis factorization.  Moreover, Section 4.2 of [Li et al., 2019] proposes sharing the same basis across multiple convolutional layers.  Given this similarity in terms of ideas, it also seems necessary to experimentally compare to [Li et al., 2019] and provide further discussion of how the proposed approach and/or experimental results differ from these two prior works.

Furthermore, though the paper does compare results to LegoNet [Yang et al, 2019] in Table 2, that comparison is uninformative as it applies the filter parameterization scheme of LegoNet to VGG, while applying its proposed scheme to Wide ResNet.  VGG vs Wide ResNet is a significant change in the base network architecture, which is a confound for the question of which filter parameterization is better; each method could be applied to either base architecture and the comparison should be made by choosing the same base architecture.

summary_of_the_review: This paper proposes a cross-layer parameter sharing scheme for CNNs that demonstrates some benefits on image classification tasks.  However, the proposed method is similar to filter factorization approaches in several previous published works, for which experimental comparison is missing or incomplete.

Update (after rebuttal):
The author response addresses some of the experimental concerns from my initial review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel method to decrease redundancy of parameters in a CNN through a joint subspace view. The authors use a low-rank decomposition of the convolutional kernel (A * D) into shared coefficients (A) and atoms (D). The authors propose a variety of proposal methods for their method termed Atom Coefficient Decomposed Convolution. Notably by sharing A across all layers, sharing A across groups of layers and by utilising the idea of filter groups. The authors demonstrate that ACDC-x outperforms baselines at moderate to significant reductions in model size. Further, they show that their method can alleviate issues of model degradation as deeper networks are used.
main_review: [Strengths]

* The idea to use a low-rank decomposition of convolutional kernels and to consequently use the coefficients as sharable weights throughout a network is really interesting. This is especially true for the filter groups networks as the authors show a significant reduction in parameters and improvement in model performance (Table 1). Performance of the method when compared against a good number of _light_ CNN architectures on CIFAR-10/1000 + TinyImageNet is good and further demonstrates the performance.

[Weaknesses]

* The main motivation for the joint subspace (Section 2.2) is quite confusing in the opinion of this reviewer. It is difficult to understand why exactly one can exploit "intrinsic, low-rank structure...across layers" from the explanation and the pilot MNIST experiment. This is an important section that needs to be reworked.

* The idea to share filter basis' across layers is not entirely new some and _shares_ similarites to https://arxiv.org/pdf/2006.05066.pdf. (Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks). In that paper, they also learn a filter basis that can be shared across layers despite a different methodology. 

[Other]

* In the filter groups model, how exactly does ACDC work? Is there a limitation that you need to have equal amounts of filter groups at every layer as the dimension of the shared coefficient is contingent on the groups _g_?

* The decomposition method that is based on Qiu et al. 2018 is presented in Section 2.1. How much does this differ? Why not just use Qiu et al?

* I would double check the notation for all subscripts as some $\times$ symbols seem to be missing
summary_of_the_review: The idea to learn a joint subspace view to share weights across layers in a CNN is interesting especially with filter groups. They show good results on a variety of datasets against a solid number of baselines. Despite this, it is to my feeling that the idea is not entirely novel as i) low rank decomposition of convolution kernels has already been performed and ii) the idea to share filter basis/weights across layers has also been proposed. Further, the motivation/pilot experiment for the insight to sharing shared coefficients over layers needs to be worked on. I therefore propose 6 for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work first shows empirical observations that there exist obvious correlations in features across layers within a CNN after proper linear transformations. Accordingly, the authors propose coefficient sharing strategies across layers or across groups of filters, which aim to decompose convolution filters for reducing model sizes. The experiments are conducted on four image benchmarks using several CNN backbones, and the results show the effectiveness of the proposed ACDC on image classification and few-shot image classification tasks.
main_review: Strengths:
(1)	The idea of coefficient sharing across different settings (e.g., the whole networks, the convolution stages, convolution layers, and groups of filters) seem interesting.
(2)	The experimental results show the proposed ACDC is superior or competitive to state-of-the-art methods.
(3)	The paper is clear written.

Weaknesses:
(1)	My first concern is the settings of coefficient sharing. For CNN models, different convolution stages capture various kinds of features. For example, the preceding layers or blocks extract the fundamental cues (e.g., color or boundaries), while the latter layers or blocks learn the high-level semantic features. Although there exist some correlations in features across layers, correlations among different-level settings may be clearly different. Therefore, the best settings (e.g., the whole networks, the convolution stages, convolution layers, and groups of filters) of coefficient sharing may vary in various backbones and tasks. So, could the authors give the guideline or develop an adaptive method?
(2)	As shown in Table 1, coefficient sharing across groups of filters have great effect on the final performance. Could the authors give more detailed discussion? Furthermore, the original backbone models with only groups of filters could be compared as baselines for verifying the effectiveness of ACDC-g. How about the performance of ACDC-g on ImageNet?
(3)	The authors claim that the proposed method can provide better interpretation for CNN. However, it seems not very clear for me. Could the authors give more detailed discussion?  

Other Comments:

1.	Compared to 2D convolutions, 3D convolutions need more optimization due to high computation complexity. Therefore, do the proposed method adopt to 3D convolutions (spatial and temporal information are coupled)?  Could the authors provide some tiny examples about 3D convolutions?
2.	Some typos, e.g., 13.49-> 13.49M and 8.57 -> 8.57M in Table 3.

summary_of_the_review: The idea of coefficient sharing across different settings seem interesting, and a ok experimental results. However, there exist some weaknesses in this work (see main comments above). Therefore, my current rating is borderline accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a joint subspace view between the layers of convolutional neural network. Since the important features across the layers in deep CNN is maintained, the filters can be decomposed into shared coefficients and layer-specific coefficients. Motivated by multi-scale decomposition, they compose the shared coefficient in order to exploit correlations between features maximally. Also, by using group convolution, they reduce the amount of parameters as well as maintain the performance. This method is easily compatible to modern CNN architectures.
main_review: Pros
1.	Motivation is well defined. By suggesting motivating experiment, the method of the paper could be verified theoretically as well as experimentally.
2.	The method is simple but very strong. The method is just decomposing the shared atom coefficient of the layers in CNN, then they convolute images only with layer-specific coefficients followed after matrix-vector multiplication with respect to shared coefficients. Furthermore, by using group convolution, they accelerate the speed of the algorithm as in Tab.4. Even with fast computation, it shows good performance at network reduction, with at least maintaining or improving the error ratio.
3.	The method can be compatible with various case. Since there are additional methods to be combined with the proposed such as block (section 2.3.) and group (section 2.4.) approaches, it can have a lot of variants.

Cons
1.	There is a lack of reasons for using maximum value of C_in and C_out in the section 2.3. If you already have ablation study on that, it would be better to have in text, somewhere.

Minors
1.	The caption of Figure 1 is same as the third paragraph of the Introduction (replica). If the authors provide additional description of the figure, it will be clearer.
2.	The authors need to explain about the bottom block in table 2. The description of third paragraph of section 3.1 has no referred table or figure. (The former is likely results about your methods and the latter seems to be an analysis about table 2.)  Also, the author needs to clarify the meaning of y-axis in figure 5.
3.	There is a typo in Fig. 4 : D_{n_1}\in\mathbb{R}^{k\times l\times l} -> D_{n_1}\in\mathbb{R}^{m\times l\times l}
4.	The organization of the manuscript is uncommon; a section of related work is located after result section.

summary_of_the_review: Overall, the paper is hard to follow the logic because of unusual organization. However, the novelty of the proposed method is fully convinced with supported experiments and their explanations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To reduce the parameters of the deep networks, this paper proposes a joint subspace view to convolutional filters across network layers. Experimental results verify the effectiveness of the proposed method.
main_review: Depending on the low-rank structure observation, this paper proposes to jointly model filter subspace across different layers by enforcing cross-layer shared subspace coefficients. The key idea is that all convolutional filters are decomposed into a shared coefficient submatrix and special filter atoms. 

+ves: 
+ The idea of the shared subspace coefficient for reducing the parameter redundancy seems interesting. 
+ This paper is well-written and is easy to follow.

Concerns: 

- The novelty of this paper is weak. In fact, the idea that the convolutional filter is considered a multiplication of two small (low-dimensional) submatrixes has been proposed in [ref-1]. Of course, this paper proposed a shared coefficient submatrix. However, this novelty is not enough for the standard of ICLR.
- The experimental results are not surprising. For the small datasets (e.g., CIFAR-10, CIFAR-100, and TinyImageNet), they only need a small deep model essentially although the proposed method is to reduce a large number of the parameters. For the large datasets (e.g., ImageNet), the proposed method is to reduce a small number of the parameters. Moreover, many related works (e.g., ShuffleNet-V2,MobleNet-V2) are missing to compared with the proposed method on ImageNet.
-I am sincerely hoping that the authors deeply think more important problems in the topic of speeding up deep networks.

[ref-1] Speeding up Convolutional Neural Networks with Low Rank Expansions, BMVC 2014.

summary_of_the_review: I recommend the borderline rejection due to the limited novelty and unsurprising experimental results.
review_title: null

Meta-review: The paper eventually got 5 "marginally above the threshold" after rebuttal. Such scores testify to that the paper is a borderline one. By reading the post-rebuttal comments, it is evident that most of the reviewers still deemed that the novelty is incremental. One of the reviewer (vUb9) raised the score simply to "encourage the authors to think more important problems", rather than acknowledging the merits of the paper. The AC also read through the paper and had the following opinions:
1. The paper is actually about DNN compression, based on the "new finding" that the weights across layers are low-rank. However, the authors would not write the paper in the way of DNN compression, but put more emphasis on the "new finding", which has no theoretical support at all (only some heuristic reasoning). The AC would deem that the "new finding" is only an assumption.
2. Actually the "new finding" is not new at all. For example,

[*] Zhong et al., ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition, Neural Networks, 2019, 

used a shared core tensor (which could be regarded as the common dictionary) across all layers for higher compression rates. More recent references that use tensors and consider shared information across layers for compression can be easily found as well.

So the AC thanked the authors for preparing the rebuttals carefully, but regretfully the paper is not good enough for ICLR.
Decision: Reject