Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on CNN compression using low-rank factorizations. In particular, the authors propose a new tensor decomposition called SeKron, which generalizes traditional TT, TR, CP and Tucker decomposition. The authors establish a SVD-based algorithm to decompose a given tensor into the SeKron format. Due to the special structure of SeKron, convolutional operations can be conducted without reconstructing the full tensor, which saves memory and FLOPs when doing inference in CNNs.

The authors conduct experiments on WideResNet/ResNet for classification and SRResNet/EDSR for super-resolution. They compare with several decomposition and pruning methods. The proposed model shows similar compression rates, but achieves better performance.
strength_and_weaknesses: **Strengths**

1. The authors propose a new tensor decomposition method based on SVD, which generalizes CP, Tucker and TT/TR.
2. The authors show how to parameterize kernels in CNN by the proposed SeKron format. The proposed method allows convolutional operations by operating each core tensors, without computing the full tensor. Therefore, the proposed method is efficient to do CNN inference.


**Weaknesses and Questions**

1. I do not catch how the authors do the compression. Do they use end-to-end training for the given SeKron format or just apply Algorithm 1 on pretrained CNNs? Is there any finetuning after the decomposition?
What devices do the authors use in experiments? Why do the authors only list CPU time? What about FLOPs and GPU time for SeKron and baselines?
2. The authors consider addressing the architecture limitations of tensor decomposition by proposing the flexible SeKron. However, there exists some work to tackle the problem by exploring many potential tensor networks. I think a good reference is [1], since it also focuses on the CNN compression by using genetic algorithms to search tensor network structures. Moreover, [2] proposes a permutation and rank search for TR format, which can be also applied if the CNN is not trained end-to-end.
3. As SeKron is developed based on the sum of Kronecker structure proposed by Hammed et al. (2022), can the authors provide more intuitions and advantages about using this sequence of sum of Kronecker in SeKron? Moreover, why did not the authors compare SeKron with the sum of Kronecker in experiments? I think it is an important baseline. Currently, the advantages of SeKron are not clear to me. 
4. If I understand correctly, in Algorithm 2, the contraction or projection must be done sequentially for $i= S, \dots, 1$. Why do the authors say the projection can be performed independently? Moreover, will this sequential procedure increase computational time compared with traditional convolutional layers which are highly parallelized?

[1]. Hayashi, K., Yamaguchi, T., Sugawara, Y., & Maeda, S. I. (2019). Exploring unexplored tensor network decompositions for convolutional neural networks. Advances in Neural Information Processing Systems, 32.

[2]. Li, C., Zeng, J., Tao, Z., & Zhao, Q. (2022, June). Permutation Search of Tensor Network Structures via Local Sampling. In International Conference on Machine Learning (pp. 13106-13124). PMLR.

clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is clearly written.

**Quality:** The derivations seem to be correct.

**Novelty:** Medium. The authors propose a new tensor decomposition for convolutional layers that generalizes many previous models.

**Reproducibility:** The authors do not provide code. The hyperparameters and details are not listed either.
summary_of_the_review: This paper proposes a new tensor decomposition method which can be used for CNN compression. The proposed method is flexible, which is potential for some future applications. However, the intuition and advantage is not clear to me currently. I would like to raise my score if the authors could address my concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new tensor decomposition method for neural network compression. The main idea is to devise a decomposition structure that is composed of a sequence of Kronecker products, which generalizes most of the well-known tensor decompositions. Existing decompositions mostly rely on specific structures (of decomposition), which might not be best suited for network compression purposes. The method itself is quite simple (applying successive greedy binary Kronecker decompositions), which is a good point for the main goal. Experiments show that the proposed method achieves state-of-the-art performance for CIFAR-10, ImageNet, and super-resolution experiments.
strength_and_weaknesses: The main idea is quite simple and intuitive (i.e., representing any possible form of tensor products with a series of Kronecker products), but it is definitely a worthy contribution. What the authors have proposed is basically a general tensor decomposition structure that is easy to manipulate. I was personally surprised by the fact that nobody has attempted this (which now seems to be a natural choice for network compression) until this moment, and I also think that this reflects the importance of this contribution. What existing tensor decompositions have focused on was mostly things like having good algebraic properties or finding the best approximation with respect to some criteria. On the other hand, the proposed method might not be particularly better (or might be even worse in some aspects) for the above properties, but it is definitely more flexible in determining the shapes of the factors. For network compression purposes, this can be a better choice.

In fact, I was one of the reviewers of this paper in another venue. I see that some of my previous concerns have been addressed in this paper. I asked about the binary Kronecker decomposition in the experimental comparison, and I see that it has been added in this version. However, the "direction of expansion" in successive SVD-based approximation has not been addressed in this version. (In the paper, the successive SVD approximation is performed in one direction, i.e., from 1 to S. But this is not the only direction possible.) I understand that investigating all the possible directions can be difficult, but I suggest adding this discussion with a few other example directions, which can motivate readers to try other possible combinations.

clarity,_quality,_novelty_and_reproducibility: Another thing that has been brought up in the previous review (mentioned by other reviewers) is the position of this paper in the context of tensor decomposition literature. It has been criticized that the SVD-based approximation technique is not new and this might cause confusion in this context. The claim that TT / TR are special cases of SeKron (shown by a constructive proof) was also criticized in that review. These points were not clearly addressed in this version. I was personally somewhat against these criticisms in that they may be valid in terms of tensor decomposition literature, but they entirely miss the point (and contribution) of this paper, i.e., proposing a flexible decomposition structure that is efficient for neural-net compression. Existing tensor decomposition may also be used for neural-net compression, but definitely not with the flexibility SeKron has. The decomposition format as well as the solver is important in the tensor decomposition context, but for neural-net compression, I believe that the former can be much more important.

As such, I suggest the authors limit the scope of the paper and more clearly describe the contributions. The authors might want to tone down regarding the generality of SeKron but rather emphasize the flexibility that suits the purpose of neural-net compression. In other words, SeKron may not be new in many aspects of the existing tensor decomposition context, except for the flexible decomposition format. The successive SVD-based approximation can be one of these aspects, i.e., it is not entirely new in the literature, but it is rather the simplest example solver that can calculate SeKron easily. I also suggest limiting the discussion regarding the comparison of formats between different decomposition structures strictly to the neural-net compression perspective. Accordingly, I give a review score of 6 and will determine the final score after seeing the rebuttal.

I believe that the experiments in the paper are reproducible.
summary_of_the_review: I believe that the authors propose a flexible decomposition structure that is efficient for neural-net compression. SeKron might not add better characteristics with respect to the traditional tensor decomposition context, but it is surely more efficient for network compression. The previous criticism regarding the position of SeKron in the existing tensor decomposition literature is still not clearly resolved in this version, I suggest toning down and limiting the scope of discussion.

[After rebuttal] I'm satisfied with the answers and raise my score to 8. The authors have appropriately revised the paper, i.e., toned down regarding the generality of SeKron, focusing more on the paper's main focus, i.e., neural net compression.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper considers compression of CNN weight tensors by using a kind of tensor decomposition. The purpose of this is to reduce the number of parameters and speed up the convolution computation.

It is quite clear that the authors have compressed various white spaces in the paper (e.g., around equations, in captions, around floats). This, by itself, is grounds for rejection. In addition to this, the paper is confusing in many places, and key weaknesses of the approach aren't discussed.

### Update after rebuttal ###

I leave my score as it is due, to the following reasons:
- The original submission violated the page limit due to substantial alterations of white spaces.
- The paper doesn't sufficiently address the limitations with how dimensions of the factors have to be chosen. For a method like this to be useful, I think it should be relatively insensitive to the data tensor dimensions. For example, if the tensor to be decomposed is of size $1024 \times 1024 \times 1024$, then SeKron has a great deal of flexibility. In particular, since $1024=2^{10}$, the number of factors $S$ can be chosen to be anywhere from 1 to 10 and the sizes $a_1^{(k)}, a_2^{(k)}, a_3^{(k)}$ for $k=1,\ldots,S$ can satisfy the requirement $\prod_{k=1}^S a_i^{(k)} = 1024$ in many different ways. But if this is changed just slightly to a tensor of size $1021 \times 1021 \times 1021$, then all of a sudden there is no flexibility at all. In particular, since 1021 is a prime number, you have to choose $S=1$ and $a_i^{(1)} = 1021$ for each $i =1,2,3$. This sensitivity in the behavior of the model to the input tensor size is, in my opinion, a substantial weakness. This could potentially be alleviated with appropriate zero padding. But this isn't done in the paper, and the authors don't even acknowledge the weakness of their current model. 
strength_and_weaknesses: --- Strengths ---

S1. Sections 1-2 provide a nice introduction to the area. The overview in Section 2 is particularly good. I also appreciated the clear introduction to CNN layers at the start of Section 3.1.

--- Weaknesses ---

W1. The authors have clearly reduced whitespace throughout the paper; equations are crammed together, captions are too close to the figures. This by itself is grounds for rejection since it effectively violates the 9-page paper limit.

W2. An important weakness that is not mentioned anywhere is that the factors $A^{(k)}$ in Eq (8) must have dimensions that factorize the dimensions of $W$. For example, they must satisfy $\prod_{k=1}^S a_j^{(k)} = w_j$. So what is hailed as *greater flexibility* of the proposed model in the caption of Fig 1 is in fact a *limitation*. For example, if the dimensions of $W$ are prime numbers, then for each mode of $W$, only a single tensor $A^{(k)}$ can have a non-singleton dimension in that same mode. This may be fixable with appropriate zero padding, but this has to at least be discussed and highlighted in the paper.

W3. The 2nd point in the list of contributions in Sec 1 claims that the paper provides a means of finding the *best approximation* in the proposed format. In fact, it is easy to see that this claim is likely to be false: The decomposition corresponds to a difficult non-convex optimization problem, and it is therefore unlikely that a simple algorithm with a finite number of steps could solve it optimally.

W4. SeKron is claimed to generalize various other decompositions. But it is not clear that the proposed algorithm could ever reproduce those decompositions. For example, since there is no SVD-based algorithm for CP decomposition, I strongly suspect that the proposed algorithm (which is SVD-based) cannot recreate the decomposition that, say, an alternating least squares based approach for CP decomposition would achieve.

W5. The paper is unclear and poor notation is used in multiple places. For examples:
- Subscripts are sometimes used to denote indices (e.g., Eq (5)), sometimes to denote sequences of tensors (e.g., Eqs (7), (8)), and sometimes used to denote both at the same time (e.g., Thm 3, Eq (35))! This is very confusing.
- It is unclear how Eq (7) follows from Eq (5). The confusing indices exacerbate this.
- In Thm 1, $A^{(k)}$ are tensors, so it's unclear what you mean by "$R_i$ are ranks of intermediate matrices".
- In Alg 1, you apply SVD to a 3-way tensors. This operation is not defined. If you mean batched SVD, you need to specify that.
- The $W_{r_1 \cdots r_{k-1}}^{(k)}$ tensors in Eq (10) haven't been defined.
- The definition of Unfold below Eq (13) is ambiguous. Similarly, you say that Mat reformulates a tensor to a matrix, but list the output space as $R^{d_1 \cdots d_N}$, i.e., indicating that the output is a vector.
- Below Eq (15) you discuss "projection". This is not an appropriate term to use, since these aren't projections; projection is a term with a specific meaning in linear algebra.
- In Eq (16), the $r_k$ indices appear on the right-hand side but not on the left-hand side.
clarity,_quality,_novelty_and_reproducibility: --- Clarity/Quality ---

The paper is unclear in many respects. In addition to the confusing points listed under weaknesses, some further points follow below.
- You should be more specific about the difference between high- and low-level computer vision tasks in the introduction/throughout the paper.
- In Fig 1 (a), the tensors in SeKron are incorrectly numbered.
- What are the sizes of the factor tensors used in the experiments? This is particularly relevant to state since there is a requirement that the dimensions of the factor tensors factorize the corresponding dimensions of the full tensor.

--- Novelty ---

It's unclear how the paper is different from previous work like that by Hameed et al. (2022).

--- Reproducibility ---

In its current form, it's not reproducible due to the lack of details, poor notation, and lack of code.
summary_of_the_review: The authors have reduced white spaces of the paper, effectively violating the 9-page limit. In addition to this, there are issues with notation  that make the paper hard to follow. Important limitations haven't been discussed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes a tensor decomposition method for neural network compression. The main idea is to devise a decomposition structure that is composed of a sequence of Kronecker products, which generalizes most of the well-known tensor decompositions.  Experiments show that the proposed method achieves state-of-the-art performance for CIFAR-10, ImageNet, and super-resolution experiments.

Strengths: The authors propose a new tensor decomposition method based on SVD, which generalizes CP, Tucker and TT/TR. The proposed method allows convolutional operations by operating each core tensors, without computing the full tensor. Therefore, the proposed method is efficient to do CNN inference. It is definitely more flexible in determining the shapes of the factors. For network compression purposes, this can be a better choice.

Weakness: SeKron is claimed to generalize various other decompositions. But it is not clear that the proposed algorithm could ever reproduce those decompositions. The clarity of the paper needs to further improved regarding the notations.  The novelty of the proposed method is relatively weak since the sum of Kronecker structure proposed by Hammed et al. (2022) and tensor decomposition for NN compression is widely studied. It lacks strong motivation and deep insight why SeKron has significant impacts on NN compression. 
justification_for_why_not_higher_score: Theorem 1 and 2 do not provide clear and significant findings, which can not support the proposed method well.  The paper doesn't sufficiently discuss the limitations of the Kronecker product structure. The clarity of paper could be further improved.
Decision: Reject