Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This works focuses on CNN-based model compression. It first views the kernel in the standard convolution as an ordered set of vectors, where each vector has smaller dimension than the inputs. Each vector can then be treated as a linear combination of another group of basis vectors. By limiting the number of the basis vectors, it approximates the standard convolutional kernel. The proposed operator is called block-wise separable convolutions (BlkSConv) and can be implemented as a combination of point-wise and group-wise convolutions. The paper also proposes a PCA-based method to determine the dimension and the number of basis vectors. By replacing some standard convolutions with the proposed BlkSConv in CNNs, the authors demonstrate the effectiveness of BlkSConv in CNN-based model compression.
strengths_and_weaknesses: Strengths:
The paper provides a unique view of convolutional kernels and derives the proposed BlkSConv. Meanwhile, the authors find an easy way to implement BlkSConv. It also proposes a systematic way to determine hyperparameters in BlkSConv. All of these make the proposed method easy and practical to use.

Weaknesses:
1. The notations in figures do not match those in texts. It makes the paper less clear.
2. The experiments only compare with regular ResNet. Comparisons with other model compression works are necessary.

questions: Does the fact that BlkSConv can be implemented as a combination of point-wise and group-wise convolutions indicate some insights on how BlkSConv works from another point of view?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes block-wise factorization for approximating convolutions. Given a k x k x M convolutional kernel, it factorizes it into many smaller blocks k x k x t, and then use PCA to select a subset of these blocks to approximate the original more expensive computation.  Overall, kernel factorization is a somewhat common way to reduce ConvNet computations (previous  depthwise/pointwise/group convs are more common), but the new PCA based block selection is kind of new and interesting. Results on CIFAR and ImageNet show the proposed approach can achieve better accuracy-computation trade-offs with various ResNet networks.
strengths_and_weaknesses: ==== Strengths ====
1. Interesting idea on using PCA to choose the most important top-k sub blocks. To my knowledge, this seems to be quite new and different from previous factorization.
2. Well written paper with promising results.

==== Weaknesses ====
1. No latency is reported: Table 3 and 4 show impressive reduction in parameters and MAdds, but it is unclear how fast are these models in real hardwares. Would be nice to include latency measurements.
2. Limited experiments: most experiments are based on ResNets, would be better to explore other models  (especially those more FLOPs-efficient models such as EfficientNets or MobileNetV3).
3. Some details are not very clear: for example, it requires both M and t to be the form of 2^k, but what if the original network has irregular channel sizes (such as 40)?
questions: see above.
limitations: No concern on social impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a novel convolution factorization for efficient image recognition. The formalization proceeds as follows: First, the authors rewrite the standard convolution (on a single output channel) as the sum of contributions of the application of different kernels to different groups of input channels (t groups). Then, every kernel for every group is formalized as a low-rank approximation achieved by means of a linear combination of s basis kernels. As this approach effectively reduces the number of trainable parameters but does not affect (or even increases) the number of operations, the authors showcase an efficient implementation. Specifically, exploiting the linearity of all parts involved (both the convolution and the decomposition of the kernel into basis vectors), they show the factorized convolution can be espressed in terms of standard operations such as point-wise convolution, reshaping, and group convolutions. Finally, the authors propose a PCA-based technique to automatically search for suitable values of t and s, that enable a good tradeoff between accuracy and computation. Experiments are carried out with ResNets variants on 5 datasets.
strengths_and_weaknesses: PROs

- The proposed low-rank formulation of convolutional kernels seems novel to me. Moreover, the authors show how it is easily implementable as a combination of standard neural network operations.
- The paper is clear for most parts, and the authors succeed in explaining all the operations involved in a clear way. To the best of my judgment, the technical contribution is sound and correct.
- The experimental section showcases good performances in all the tested settings, with the proposed convolution factorization being able to reach performance within 1-3 points of accuracy of the original model (or even surpassing its performance) at a fraction of the computational cost. Remarkably, Table 5 illustrates how optimizing a ResNet with the proposed factorization is a better strategy that using a smaller version of the architecture (e.g. BlkSConv-ResNet-18 or BlkSConv-ResNet-26 are better than ResNet-10 on ImageNet).
- The authors release their code as part of the supplementary material, which is considerable.


CONs

- The main issue with the paper is that in the experimental section it does not provide any baseline for the proposed factorization. It is true that the block-wise separable convolution provides a much better accuracy/efficiency tradeoff with respect to standard convolutions. However, several other methods have been popularized in the past to save computations. For instance, how would group convolutions, depthwise separable convolutions, low-rank factorization methods [4,5] or channel pruning [6] perform in the same settings the authors use? Without reporting at least a subset of them as baselines, it is very hard to quantify whether the proposed factorization, despite being beneficial, would be something interesting and impactful for the community
- The paper misses the discussion of model compression prior works that rely on low-rank factorization of weight matrices. For instance, [4,5] are seminal approaches that use SVD decomposition to factorize convolutional kernels as a product of lower rank matrices. Since here a low-rank factorization is also achieved, I think a discussion of those models would improve the paper's quality.
- Formalizing a convolutional kernel as a linear combination of a set of basis has recently raised a lot of interest, especially in the context of conditional computation. For instance, models such as [1,2,3] predict the combination coefficients based on the input feature itself. Although this is somewhat different with what is proposed here, I think the paper misses a discussion of this branch of related works, for completeness.
- The proposed implementation described in Sec 3.2 incurs into a higher memory overhead	and involves some heavy memory-bound operations such as reshaping and copying. I therefore expect that, despite the reduction in MAdds, the actual on device latency of the proposed convolution is similar to (or even higher than) the original one. I think this should be verified and discussed in the manuscript.
- Fig 1 is helpful but could be made more clear:
  - Fig. 1 left should specify, for clarity, that it represents the special case for s=1.
  - The color coding (green) suggests that the kernel is shared among the N output channels. To my understanding, that's not the case.
  
Typos:
- line 138/148: "top of Figure 1" / "bottom of Figure 1" -> "Figure 1 left" / "Figure 1 right"

In summary, I think this is a good paper with the potential of being impactful, yet it needs to report some baseline performances to put the results into perspective.

[1] Chen, Yinpeng, et al. "Dynamic convolution: Attention over convolution kernels." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Li, Yunsheng, et al. "Revisiting dynamic convolution via matrix decomposition." ICLR. 2021.

[3] Zhang, Mingda, et al. "Basisnet: Two-stage model synthesis for efficient inference." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Zhang, Xiangyu, et al. "Accelerating very deep convolutional networks for classification and detection." IEEE transactions on pattern analysis and machine intelligence 38.10 (2015): 1943-1955.

[5] Jaderberg, Max, Andrea Vedaldi, and Andrew Zisserman. "Speeding up convolutional neural networks with low rank expansions." arXiv preprint arXiv:1405.3866 (2014).

[6] He, Yihui, Xiangyu Zhang, and Jian Sun. "Channel pruning for accelerating very deep neural networks." Proceedings of the IEEE international conference on computer vision. 2017.

questions: N/A
limitations: - The authors briefly mention memory overhead as a potential limitation. I encourage them to discuss about on time latency measurements as well, in the case the proposed model is not beneficial in that respect.
- No potential negative societal impact that I can think of.
review_title: null

Meta-review: metareview: This paper proposes a factorized convolution operation, which can be implemented with a combination of group wise convolution and point wise convolution. The reviewers in general agreed that this a valid idea, and found the PCA based hyper-parameter search method interesting. However, several concerns were raised on the experimental side, including lack of fair comparison with other efficient convolution baselines, as well as degraded latency and memory cost compared to simple ResNets. Given the practical nature of the work, the AC agrees that this work needs substantial improvements on the empirical evaluations to pass the bar for publication. 
confidence: Certain
Decision: Reject