Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this study the authors investigate post-training compression of CNN-based image classification neural networks. One method to compress a CNN-based NN is tensor decomposition. Their goal is to learn how much the approximation error of tensor decomposition is predictive of the performance of the compressed CNN-based NN classifier. For example, they ask if the choice of tensor decomposition method can be guided by this or the level of compression. This would be more efficient than using the final task performance of each candidate compressed model. Another question, for example, they investigate is if finetuning after compression does disturb or improve the correlation between approximation error and final task performance of the NN. 
strength_and_weaknesses: S1. The paper investigates a clear objective that is relevant for potential follow-up work and has useful practical implications.

S2. The paper is well written and well structured. 

S3. The paper does give a clear picture and considers many practically relevant questions. 


W1. It is unclear how the increased complexity due to the proposed methods and the level of control on the final task performance do compare against a distillation approach.  

W2. How much does the randomness of many of the TD methods due to their initialization influence to robustness of the approach.
clarity,_quality,_novelty_and_reproducibility: The paper was clear, succint. The experiments are reproducible and all their components, down to the exact correlation that were used have been clearly described. Apart from the two weakness mentioned I have no other concerns.
summary_of_the_review: The paper is well written, focused and has enough experiments to educate the reader about their findings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically evaluates the correlation between different hyperparameters of tensor compressive layers and the classification performance. Their results show that the error is indicative of model performance, even when comparing multiple TD methods, though useful correlation only occurs at the higher compression levels. 
strength_and_weaknesses: [Strengths]

S1: This is the first paper that systematic studies on how well the approximation error relates to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels. 

S2: The experimental results support the conclusion that there is a positive correlation between the relative approximation error on the weights and the resulting performance error of the model for a wide range of TD choices, including layers, methods, and compression levels.

[Weaknesses]

W1: To test Assumption 1, this work reports some experimental phenomena/results based on extensive experiments. However, it doesn't go far enough in the sense that possible reasons behind these phenomena (e.g., the impact of compression levels on the rank correlation) are not sufficiently investigated. 

W2: Although showing the correlation between weight (tensor) approximation error and the classification performance is interesting, it is more significant if some practical suggestions are given for designing compressed deep models based on tensor decompositions. However, the empirical findings of this work seem of limited guiding value for a deep learning practitioner who needs to decide which layer to compress, which tensor decomposition to use, and which compression level to choose for both efficiency and effectiveness.  

W3: The writing is not always satisfactory due to typos like "no works investigate if the make TD decomposition choices using specific approximation errors are well suited", "previous works often only includes", and "Shown are averages and standard deviation over runs We observe that a higher compression". There are also confusing notions used, e.g., "rank" in "the correlation is calculated based on measurements of all combinations of layer, decomposition method, and rank once". 


clarity,_quality,_novelty_and_reproducibility: [Clarify score 4/10]
This paper is in general moderately well written. The clarity is not always satisfactory. See Weakness W3. 

[Quality score 5/10]

(1) This is the first systematic study on the influences of the approximation error to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels. The experimental findings are new. 

(2) Although this work presents extensive experimental results, it lacks deeper explanations for the empirical findings and sufficiently practical suggestions for deep learning practitioners in designing compressed models. 

[Novelty score 4/10]

(1) This paper does not propose new models, algorithms, or theory. 

(2) It empirically tests the validity of Assumption 1 which is motivated by Liebenwein et al. (2021).

[Reproducibility score 4/10] 
The conclusion of this paper relies so heavily on the empirical results. Although the authors gave some detailed descriptions on the experimental settings, there are still so many details not well explained in the current version for reproduction (especially without shared code). For example, what are the detailed model & algorithmic hyperparameters in using CP, Tucker, and TT? 
summary_of_the_review: Due to the evaluation of "Clarity (4/10), Quality (5/10), Novelty (4/10) And Reproducibility (4/10)", I suggest "Reject".

-----------After rebuttal------------
After reading the authors' feedback and other reviewers' comments, I think this is a nice paper which has empirical significance. However, there is still much room for improvement in making this empirical evaluation paper much clearer. So, I decided only to change my suggestion from "3 reject" to "5 borderline below".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper performs an empirical study on the relationship between the approximation error of a model’s weight tensor and the model performance after compression. Multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression are tested on several models and datasets.
strength_and_weaknesses: Strength:
- The paper does a great amount of experiments to validate the assumption, namely there is a positive correlation between the relative approximation error on the weights and the model performance for a wide range of TD choices, including layers, methods, and compression levels.

Weakness:
- The contribution is not significant since the results are not surprised (e.g., relative weight approximation error is better than using absolute weight and rescaled weight), though I appreciate the engineering efforts the author spent on this work.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, and is reproducible. The major concern is that the contribution is not significant.
summary_of_the_review: As discussed above, I am close to a rejection due to its limited novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the use of tensor decomposition (TD) for compression of the weight tensors in CNNs. It considers the problem of choosing the best compression hyperparameters (which layer to compress, which type of TD to use) for a given level of compression. In particular, it investigates if the decomposition error for a given choice can be used as an indicator of how well that particular decomposition choice will perform both without and with fine-tuning when the CNN performs its intended task. If the decomposition error does provide such an indication, it would simplify the task of choosing compression hyperparameters since it would be enough to consider the weight tensor in isolation, rather than how it interacts with the rest of the network. 


strength_and_weaknesses: --- Strengths ---

S1. The paper provides a good overview of previous works that use TD for compression of NNs.

S2. The problem the paper tries to solve is relevant in practice if someone wants to use TD to compress a CNN, but doesn't have any prior insight into what type of TD would work well for that particular network.

S3. Although some details are hard to understand, the paper is overall well-written and the different experiments are interesting.

--- Weaknesses ---

W1. Some details are hard to understand. I list these below.

a. In Fig 2: What is the difference between Relative Weights and Scaled Weights? Similarly, what is the difference between Relative Features and Scaled Features? Based on Table 1, isn't the relative measures the same as the scaled measures but with the particular choice $n_W = ||W||$ and $n_F = ||F||$? If this is correct, then how is $n_W$ and $n_F$ different for the scaled measures? Also, Table 1 is confusing since $n_W$ and $n_F$ aren't introduced anywhere.

b. In the paragraph "Comparison of approximation error measures" on page 7, in the sentence "In Figure 2, the correlation is calculated ... and rank once.": By "rank", do you mean compression level (i.e., one of the three choices {50%, 75%, 90%})? Calling this rank is a bit confusing, since there's many choices of rank that potentially could yield a certain compression ratio, but only three compression ratios under consideration. 

c. Related to the point b above: With 3 types of decomposition and 3 levels of compression and 5 convolutional layers for ResNet-18, shouldn't there be 5 * 3 * 3 = 45 different measurements rather than 40? Similarly, since GaripovNet has 8 layers to choose from, shouldn't there be 8 * 3 * 3 = 72 different measurements for that network rather than 50? Please clarify.

d. In Sec 3, the iterative approach is a bit hard to understand. For example, if my network has 10 layers and I want to compress 5 of them, do I run 5 iterations of the method, compressing one layer each iteration out of the remaining uncompressed layers? Does the iterative method have a target compression, and how does that impact the number of layers to compress? Perhaps you can add a more detailed algorithm for how this works in the supplement?
clarity,_quality,_novelty_and_reproducibility: --- Clarity, Quality ---

The paper is well-written overall, although there are a few places that are unclear. In addition to those points listed under weaknesses above, a few more minor comments follow below.

- The sentence "Across decomposition methods, the correlation before fine-tuning is comparable for all decomposition choices." on page 9 doesn't make sense.
- "Rank" is used both as in tensor decomposition rank, and when discussion the rank correlation throughout the paper. To avoid confusion, I would avoid using "rank" in the second sense, i.e., in reference to the correlation measure. 
- In the supplement, I think you should add a definition of the notation $\times_n$ as well as a definition of the Tucker decomposition when the first 2 modes aren't contracted with the core tensor, just so it's clear what "standard" Tucker looks like.
- The colons in the subscripts in Eq (4) are confusing, since this is usually used to denote all indices along a certain mode. It would better if you just explain that $C$ and $T$ are in fact just 2-way tensors here and remove the colons.
- The figures are a bit blurry/pixelated. I would recommend using proper vector graphics.
- The usage of the term "quantization" in Sec 5 is confusing since it's different from how it's typically used in ML; see e.g. https://pytorch.org/docs/stable/quantization.html. I think it would be better to just call it "reshaping" to avoid confusion.

Some minor typos:
- In the 1st sentence of the 2nd paragraph on page 2: The work "make" should be removed. Also "TD decomposition" sounds strange since this reads "tensor decomposition decomposition".
- Last sentence before start of Sec 3.1: TDs -> TD
- There's a period missing in the Fig 1 caption.

--- Novelty ---

The paper doesn't present any novel method, but it does provide a nice empirical investigation of the problem.

--- Reproducibility ---

If the code is released as promised, the paper will be reproducible.
summary_of_the_review: I like this paper. Although it doesn't introduce any new methods, it provides a nice empirical investigation into the problem of how to choose an appropriate decomposition when using tensor decomposition to compress CNN layers. It is also overall well-written. There are some details that are hard to understand. I listed these four points under weaknesses (a-d under weakness W1). It is particularly important that the reviewers address these four points.

### Update after rebuttal ###

The reviewers addressed the concerns I had. I would have preferred to raise the score from 6 to 7. However, since there is no option for 7, I'm raising it to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper performs an empirical study on the relationship between the approximation error of a model’s weight tensor and the model performance after compression. Multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression are tested on several models and datasets. Although it doesn't introduce any new methods, it provides a nice empirical investigation into the problem of how to choose an appropriate decomposition when using tensor decomposition to compress CNN layers. It is also overall well-written.

Strengths:  The paper investigates a clear objective that is relevant for potential follow-up work and has useful practical implications. This is the first paper that systematic studies on how well the approximation error relates to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels.

Weakness: Although this work presents extensive experimental results, it lacks deeper explanations for the empirical findings and sufficiently practical suggestions for deep learning practitioners in designing compressed models.
justification_for_why_not_higher_score: This is a nice paper which has empirical significance. However, there is still much room for improvement in making this empirical evaluation paper much clearer.  This paper does not propose new models, algorithms, or theory. 
justification_for_why_not_lower_score: The rebuttal do have resolved several major concerns raised by reviewers.  The impressive empirical significance of this work is well received.
Decision: Accept: poster