Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors of this paper offer a method to compute the singular values (SVs) of convolutional layers efficiently. They propose a method to decompose the matrix $\mathcal{K}$ representing a convolutional layer to smaller matrices $\mathcal{K}^{(1)} \mathcal{K}^{(2)} \mathcal{K}^{(3)}$ and show that the SVs of $\mathcal{K}$ can be expressed by the union of the SVs of some smaller matrices related to $ \mathcal{K}^{(2)}$.
strengths_and_weaknesses: ## Strengths
- The decomposition of the matrix for convolutional layers is neat.
- The experimental results on clean and robust accuracy are improved with the proposed method.

## Weaknesses
- The assumptions of the lemma and theorems are not specified clearly. Do they apply to all kinds of convolutional layers?
- It seems that Theorem 2 requires $n \equiv 0\ (\text{mod}\ s)$, which means the spatial size $n$ has to be divisible by the stride $s$. Does it mean the stride equals the kernel size (i.e., $s = k$)? Many convolutional layers may not satisfy such a requirement.
- The experiments of computational complexity are too limited. A detailed comparison (in terms of, for example, running time) between the proposed method and prior methods should be added.
- The correctness of the singular values (SVs) using the proposed method should be validated empirically. For example, the SVs computed by the proposed method can be compared to those from [1]. Note that the method from [1] may only apply to convolutional layers with a stride equaling 1. For other strides, an alternative comparison is [2] by using the inequalities of matrix norms: $r\|A\|_\alpha\leq\|A\|_\beta\leq s\|A\|_\alpha$, i.e., by empirically showing that the largest singular value from the proposed method can be bounded by the $\ell_1$ norm or $\ell_\infty$ norm from [2].

[1] Sedghi H, Gupta V, Long PM. The singular values of convolutional layers. arXiv preprint arXiv:1805.10408. 2018 May 26.

[2] Liang Y, Huang D. Large norms of CNN layers do not hurt adversarial robustness. InProceedings of the AAAI Conference on Artificial Intelligence 2021 May 18 (Vol. 35, No. 10, pp. 8565-8573).
questions: Please see the Weaknesses above for the questions.
limitations: The limitations are not sufficiently discussed. For example, the assumptions of the lemma and theorems are not specified clearly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The work proposed a less expansive approach to calculating the singular values of convolutional layers. The proposed approach builds on the assumption that modern neural networks are highly overparametrized in such a way that a sparser surrogate network could achieve similar performance and the method of Tensor-Train (TT) decomposition. The work further empirically shows that a network's robustness could be improved by adding additional orthogonality constraint terms derived from the proposed approach and clipping calculated singular values. 
strengths_and_weaknesses: Strengths:
The overall presentation of the work is clear. The relationships with prior arts are carefully discussed. And the experimental part clearly demonstrates the benefits of applying the proposed method and also discusses many technical details.

Weaknesses:
The reviewer found that the title of the work is not well aligned with the actual content. The title indicates that a more efficient approach for calculating singular values of convolutional layers will be proposed, one would expect to directly use the proposed method to calculate singular values of some given network (like the algorithm proposed in [1]). However, the calculation described in this work heavily relies on the decomposition (K = K1 ∘  K2 ∘  K3) and it is not clear to the reviewer how to decompose a given convolutional kernel to such form (please correct me if I'm wrong). Rather, the reviewer believes that the major accomplishment of the work lies in designing networks/constraints to improve the robustness of networks.
Besides, the experiment section seems a little bit weak to only compare with SOC only (Table 1) and the vanilla setting (Table 2). The effect of the proposed approaches can be better demonstrated If more experiments can be added. 

[1] Hanie Sedghi, Vineet Gupta, and Philip M. Long. The singular values of convolutional layers, 2019

questions: My major concerns are discussed in the Weaknesses section. 

As an additional question, I'm wondering why the constraint discussed in Section 4.2 does not include a term to deal with K2 and why is the constraint useful without controlling K2.

Lastly, I would suggest expanding the clipping part discussed in Section 5.2 to a more comprehensive comparison between the proposed method and [1] (perhaps adding some comparisons when [1] is tractable in order to show both the efficiency and accuracy of the proposed method). 
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an efficient method to compute eigenvalues of convolutional layers based on a sparsity-driven approach. The proposed work achieves by proposing a novel framework that compresses the layer representation (section 3.2) based on a tensor decomposition. The authors provide theoretical justification in Lemma 1 and Theorem 1 of their proposed framework. In addition, empirical evidence is provided to demonstrate how effective their tensor decomposition across a wide range of architecture and metrics. 

The contributions are listed as the following:
1). Proposes a novel framework Tensor-Train (TT) decomposition to compute singular values that reduce computational complexity.
2). Extend previous work on the exact computation of singular values to strided convolutions
3). Demonstrated significance of  proposed framework with varying control over the singular values on a few CNN architectures by comparing  their method to the baseline across some important metrics (accuracry, generalization error, and  adversarial robustness).
strengths_and_weaknesses: Strengths-
1). Works on important problems to the ML/AI community.
2). Approves upon previous work by reducing the computation complexity and extending work to strided convolution. 
3). Convincing empirical evidence that this framework is effective.

Weakness-
1). Theoretical claims could be better explained/justified. It's not obvious upon reading this how all the theoretical contributions are related to the work. Theorem 1 and lemma 1 are related to TT framework, and theorem 2 is about explicit singular value formula for strided convolution. To help clear some ambiguity, it might be helpful to put the lemma in appendix and expand up the technical details of theorem 1 and the importance of it for the framework. Then explain how all the theoretical contributions fit into algorithm (1).

2). I believe the paper would benefit from additional experiment, similar to the one in Figure (1), which would include speed up versus some performance metrics (accuracy, ECE). This would be very useful from a practitioner's standpoint

questions: 1). Could you please provide some context w/ respect to Figure 1? Such as if you have speed up of x amount, could that lead to requiring less memory in GPU compute? For example, say a researcher only has access to 8 GB GPU could they compress the network by a certain factor to fit onto their GPU. 

2). Strided convolutions are a large part of Generative networks in GANs, do you think your work could be built upon to study mode collapse in GAN's based on the singular value composition? As well, did you consider study these types of models in your paper? 

limitations: Limitations are in section 6, assumptions are reasonable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This article proposed a framework based on the Tensor Train (TT) decomposition to decrease the complexity of computing a given convolutional layer's singular values (SVs). The key idea is to decompose the original convolution layers into low-rank factors and calculate the SVs of the factors with smaller sizes instead of the original kernel tensor. The main paper and supplementary materials give detailed proofs of the lemma and theorems. The framework is evaluated on accuracy, robustness, and calibration. The experiments are conducted on LipConv and WideResNet-28-10, using CIFAR-10 as the dataset.
strengths_and_weaknesses: Strength
1. The lemma and theorems are clearly stated, and the proofs are detailed.

Weakness
1. The originality of the proposed framework lacks significance, since using tensor decomposition to decompose a CONV layer into sequential smaller ones is not novel.
2. The related work contains too many topics, which instead blurs the key point of this article. It makes the research gap and the motivation of the proposed framework unclear.
3. This work does not give a clear strategy to find the proper TT ranks, which is of great significance for TT decomposition.
4. The experiments are not comprehensive and lack discussion and ablation study.

questions: 1. Section 2. As mentioned in the weakness part, too many topics are mentioned in this section. Though the scope of topics mentioned in this section is enormous, each of them only has short content, which makes the critical point unclear. Especially for the last two (viz. Robustness and Calibration), I think they are metrics to evaluate the framework in the experiments, thus it is not necessary to mention them in the related work section. I suggest making Section 2 more concise and the research gap more obvious.
2. About the novelty. Including TT, Tucker-2, CP, and Tensor Ring (TR) are also decomposition methods widely utilized to compress the CONV layers. I wonder why TT is employed in this work but not other decomposition methods? 
3. About the choice of TT ranks. How to choose proper ranks is a vital research topic for tensor decomposition, but it is not discussed in this article. I wonder how the ranks are determined in this paper. The selection is based on experimental results? If yes, there should be an ablation study about it. 
4. The experiments are insufficient. I think experiments on two models using CIFAR-10 are not enough. I think the results will be more convincing if more models (e.g., ResNet-18/50) and larger dataset (at least CIFAR-100) are considered.
5. The results lack analysis. Sections 5.1 & 5.2 are too short, and the results are listed in the Table without further discussion. It will be better if more analysis is provided. Besides, I think the first column in the right part of Table 1 should be “ACC.” instead of “AC.”? The definition of COMP. is also missing, it is not clear how to calculate it and the meaning of the number.
6. line 288 – 321. I think it is better to move this part to the appendix and use the space to add more experiments, discussion of the results, ablation study of rank, etc.

limitations: None
review_title: null

Meta-review: metareview: This paper introduced a tensor decomposition, and associated theory, which allows for the control of singular values in convolutional layers.

Based upon the reviews, rebuttal, and reviewer discussion, I recommend paper acceptance. All reviewers recommend acceptance. The rebuttal was effective, with one reviewer who initially recommended rejection raising their score.

The authors should be sure to follow through, and update the paper to include changes discussed during the review period. Especially, it seems as if the framing of the paper shifted during the review period from centering on practically computing singular values to practically controlling singular values. From my understanding of the work, I agree this second framing makes more sense.
confidence: Certain
Decision: Accept