How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?
Authors: Jetze Schuurmans
kim batselier
Julian Kooij
Venue: ICLR 2023 poster (2023)
Keywords: Tensor Decomposition
Convolutional Neural Networks
Compression

Abstract: Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model. Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression. To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is. We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account. We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning. Basing the approximation error on the features does not improve the correlation significantly. While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning. When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices. This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.

Published as a conference paper at ICLR 2023
How INFORMATIVE IS THE APPROXIMATION ERROR FROM TENSOR DECOMPOSITION FOR NEURAL NET- WORK COMPRESSION?
Jetze Schuurmans+ 1.2, Kim Batselier", Julian F. P. Kooij*
Cognitive Robotics Department (CoR), Delft Center for Systems and Control (DCSC) 3mE Faculty, TU Delft, The Netherlands {i fj.t. schuurmans, k haatseeier, j.f. p.....oooijj J.f.p..ko.jeetudel.fttt
ABSTRACT
Tensor decompositions have been successfully applied to compress neural net- works. The compression algorithms using tensor decompositions commonly min- imize the approximation emrr the weights. Recent work assumes the ap- proximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model. Surprisingly, lit- tle research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression. To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is. We include the approximation error on the features resulting from a compressed layer in our analysis to test this provides a better proxy, it explicitly takes the data into account. We find the approximation er- ror on the weights has a positive correlation with the performance error, before as well as after fine-tuning Basing the approximation eтrоr the features does not improve the correlation significantly. While scaling the approximation error com- monly is used account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.c. layers, decompositions, and level of compression) before fine-tuning. When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices. This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.
INTRODUCTION
Tensor Decompositions (TD) have shown potential for compressing pre-trained models, such as con- volutional neural networks, by replacing the optimized weight tensor with a low-rank multi-linear approximation with fewer parameters (Jaderberg et al., 2014; Lebedev et al., 2015; Kim et al., 2016; Garipov et al., 2016; Kossaifi et al., 2019a). Common compression procedures (Lebedev et al., 2015; Garipov et al., 2016; Hawkins et al., 2021) work by iteratively applying TD on selected weight tensor, where each time several decomposition choices have to be made regarding (i) the layer to compress, (ii) the type of decomposition, and (iii) the compression level. Selecting the best hyperparameters for these choices at a given iteration requires costly re-evaluating the full model for each option. Recently, Liebenwein et al. (2021) suggested comparing the approximation errors on the decomposed weights as a more efficient alternative, though they only considered matrix de- compositions for which analytical bounds the resulting performance exist. These bounds rely on the Eckhart-Young-We Young-Mirsky theorem. For TD, no equivalent theorem is possible (Vannieuwenhoven al., 2014). While theoretical bounds are not available for more general TD methods, the same concept could still be practical when considering TDs too. We summarize this as the following general assumption:
Assumption 1. A lower TD approximation error on a model's 's weight tensor indicates better overall model performance after compression.

---

Published as a conference paper at ICLR 2023
While this assumption appears intuitive and reasonable, we observe several gaps in the existing liter- ature: First, most existing TD compression literature only focuses on a few decomposition choices, e.g. fixing the TD method (Lebedev et al., 2015; Kim et al., 2016). Although various error measures and decomposition choices have been studied in separation, no prior work systematically compares different decomposition errors across multiple decomposition choices. Second, different decom- position errors with different properties have been used throughout the literature (Jaderberg et al., 2014), and unclear if some error measure should be preferred. Third, benefit of TTD that no training data is needed for compression, though if labeled data is available, more recent methods combine TD with subsequent fine-tuning step. Is the approximation error equally valid for the model performance with and without fine-tuning?
Overall, to the best of the authors' knowledge, no prior work investigates if and which decom- position choices for TD network compression can be made using specific approximation errors. This paper studies empirically to what extent a single decomposition error correlates with the com- pressed model's performance across varied decomposition choices, identifying how existing pro- cedures could be improved, and providing support for specific practices. Our contributions are as follows:
A first empirical study is proposed on the correlation between the approximation error on the model weights that result from compression with TD, and the performance of the compressed model!. Studied decomposition choices include the layer, multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression. Measurements els and datasets. We show that the error is indicative of model performance, even when comparing multiple TD methods, though useful correlation only occurs the higher compression levels.
made using several mod-
Different formulations for the approximation error measure are compared, including measuring the error on the features as motivated by the work Jaderberg et al. (2014); Denil et al. (2014) which considers the data distribution. We further study how using training labeled data for additional fine-tuning affects the correlation.
RELATED WORK
There is currently no systematic study on how well the approximation error relates to a compressed neural network's performance across multiple choices of network layers, TD methods, and com- pression levels. We here review the most similar and related studies where we distinguish works with theoretical versus empirical validation, different approximation error measures, and the role of fine-tuning after compression.
The relationship between the approximation error on the weights and the performance of the model was studied by theoretical analysis for matrix decompositions. Liebenwein et al. (2021) derive bounds the model performance for SVD-based compression the convolutional layers, and thus motivate that the SVD approximation error is good proxy for the compressed model perfor- mance. Arora al. (2018) derive bounds on the generalization error for convolutional layers based on a compression error from their matrix projection algorithm. Baykal et al. (2019) show how the amount of sparsity introduced in model's layers relates its generalization performance. While these works show that some theoretical bounds can be found for specific compression methods, such bounds are not available for TDs general. Other works, therefore, study the relationship for TD empirically. For instance, Lebedev et al. (2015) show how CP decomposition rank affects the approximation error, and the resulting accuracy drop as the rank is decreased. Hawkins et al. (2022) observe that, for networks with repeated layer blocks, the approximation error depends on the convolutional layers within the block.
When considering the model's final task performance, the approximation error on the weights might not be the most relevant measure. To consider the effect on the actual data distribution, Jaderberg et al. (2014) instead propose compute an emrrr the approximated output features of a layer after its weights have been compressed. They found that compressing weights by minimizing the error on features, rather than the error on the weights, results in smaller loss in classification accuracy. However, Jaderberg et al. (2014) do not fine-tune the decomposed model, and only use a toy model with few layers. Denil al. (2014) try to capture the information from the data via the empirical
The code for our experiments available https: //g thtbb womm/uSchuurmas/taotoo
2

---

Published as a conference paper at ICLR 2023
covariance matrix. Although this method eliminates the need for multiple passes the data during the compression step, it is limited to a wwo-dimensional case. Eo et al. (2021) forego looking at compression error altogether, and selects the rank based the accuracy on the validation set. This requires the labels to be present and a forward pass through the whole network, even the compressed layer is near the input.
Several norms have been used in the literature to quantify the approximation error, which is the difference between the pretrained weights and the compressed weights. The works that decompose pretrained layers (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al.. 2015; Novikov et al., 2015: Kim et al., 2016) explicitly minimize the Frobenius norm. Hawkins et al. (2021); Liebenwein et al. (2021) calculate the relative Frobenius norm, i.e., the norm of the error proportional to the norm of the pretrained weights, to compare the error for layers of different sizes. Still, remains unclear which error measure is most informative for the compressed model's final performance.
In practice, when training data is still available for compression, fine-tuning for the target task after compressing weights could recover some of the lost performance (Denton et al., 2014; Lebedev et al., 2015; Kim al., 2016). Adding fine-tuning results in a three-step process: pretrain, compress and fine-tune. Optimization thus alternates between minimizing the error of respectively the features, the weights, and finally the features again. While Denton et al. (2014); Kim et al. (2016) compare compressed model performance before and after fine-tuning, they do not investigate how the fine- tuned network performance relates to the weight compression error. Lebedev et al. (2015) does study the compression error for CP decomposition, but only reports performance with fine-tuning.
METHODOLOGY
We consider the task of compressing a pretrained neural network with TD. While TD is a general technique that could be applied many types of layers, the focus will be on convolutional layers. Due to their ubiquity and suitability to compare different types of higher-order decompositions, as the layer weights are four-dimensional tensors.
Generally, a compression procedure will iteratively apply TD to the weights of selected layers, making several choices on how and what weight tensor to decompose, while ideally maintaining as much of the original network's 's performance. In its original uncompressed form, the full-rank weight tensors W E RC*H HCCCHXX xW xT' of a layer represent a local optimum the network's parameter space with respect to the training data and loss, where C' is the number of input channels, H and W are the height and width of the convolutional kernel, and is the number of output channels. When a TD is applied to the weights of specific layer, this results in factorized structure W composed of multiple smaller tensor multiplications, which replaces the original weights in the network. Each time TD is applied, several decompasition choices need to be evaluated:
Layer: The layer from the set of network layers {1,2,-.. .L decompose.
2. Method: The type of TD method m E M. = {CP, Tucker, Tensor Train). The decomposition determines the factorized structure of W.
3. Compression: The compression level C for the selected layer. Here C C (0, 1] is some finite set of testable levels, and с = 0.75 means the number of parameters is reduced by 75% and the factorized layer contains only 25% of the parameters. A given compression level is achieved by decomposing the tensor to some rank R, depending on the selected TD method (see Section 3.3).
We will refer to H L M C as the set with possible hyperparameter values for (l, m, c) to consider. Note that compression procedures the literature might only consider a subset of these choices. For example, procedure might fix the layer for given iteration only consider single TD method. In practice, it is computationally infeasible to evaluate the compressed network's performance for every possible hyperparameter choice at every compression iteration, especially when optimizing for performance after fine-tuning. Instead, automated compression procedures will
efficiently compare an approximation error a; = eWW, W) between the original and decomposed weights using particular choice of hyperparameters h; E H. In doing so one relies on Assumption that a lower approximation error is indicative of better compressed performance p.. If annotated training data is available, additional network fine-tuning on the decomposed structure could result in improved performance pt > Pi.
3

---

Published as a conference paper at ICLR 2023
In this work, we propose to focus single iteration and investigate Assumption in isolation of any specific compression procedure. Our aim is thus to assess how well computing an approximation error can predict the optimal compression choice from some hypothesis set HH, i.c. the choice that results in the lowest compressed network performance error p. or even performance after fine-tuning p". We will study the correlation between approximation error and model performance empirically in our experiments, using the procedure and correlation metric explained in Section 3.1. Details on the different approximation errors that we will explore are covered in Section 3.2. Finally, the considered TD methods are explained in Section 3.3.
3.1 EMPIRICAL EVALUATION OF
CORRELATION
Our proposed empirical evaluation procedure will evaluate a large set of hyperparameters H = [h1, fi..... multiple convolutional neural networks and datasets (see Section 4) for differ- ent options of approximation error metric (see Section 3.2). For given model, dataset, and approx- imation error metric e, the procedure evaluates for each set of hyperparameter choices h2 E HH the approximation error a; e(W,, W). the model performance error p; on the validation split, and the model performance error P after additional fine-tuning on the training data. We thus obtain sets of measurements A {a1, a2: -. J. {PI, P2. -.. and IP* {pi, P2. for Hi.
When comparing two sets of hyperparameters h; E H and h2 E IHI, we want to establish if the set with the smaller approximation error results in a smaller performance error of the model. In other words, the concordance of pairs of measurements needs to be established. Concordant pairs have a larger (smaller) performance еттог when the approximation error is larger (smaller) between two sets of hyperparameter choices, i.e. i and j are concordant if and p; < Py. and discordant otherwise. Kendall's 1938) or ordinal association between two order sets, in our case between approximation errors and a model performances IP (or P*). To avoid confusion with the concept of tensor rank, we will refer to Kendall's simply as correlation. For this correlation measure, the difference between the number of concordant pairs (k) and discordant pairs (d) is scaled with the binomial coefficient m m(m 1)/2 to account for the different ways two measurements can be sampled from total of m measurements:
aj and p; > P3 or if a, < Oj a measure for the rank correlation (Kendali,
T = 2(k d)/(m(m - 1))-
Kendall's 7 can be interpreted as follows: T = indicates perfect positive rank correlation, T = 0
no correlation, and -1 a strong negative correlation. For a set of hyperparameters Hi, a useful approximation error e would thus result in a 7 close to +1, indicating it is predictive of the model's performance. Note that Kendall's r does not depend on assumptions about the underlying distri- bution, whereas Pearson correlation assumes linear relationship between the two measurements. Kendall's т used over Spearman's s because the interpretation of con- and discordance pairs for Kendall's 's is closely related to our use case of choosing between hyperparameter sets.
3.2 APPROXIMATION ERRORS
We now discuss various measures e(W, W) to quantify the approximation error. The basis is to compute some norm on the difference between these tensors, in this work we use the Frobenius norm as is common in the literature (Lebedev et al., 2015; Hawkins et al., 2022). We shall consider three options to scale the norm, which could help make the error more robust when comparing hypotheses with different layers. Additionally, we can consider two options to compute the error on, namely either directly the weights or on the features. In total, we shall thus explore six different approximation errors in this work. An overview is presented in Table I.
Normalization The norm between the difference of the weights is referred to as absolute norm and is used in the objective function when decomposing the pretrained weights. The relative norm used TD literature to compare errors between different layers (Lebedev et al., 2015; Hawkins et al., 2022), it is invariant to the size of the weights. Alternatively, the norm of the difference can be scaled to account for the number of parameters, while keeping the distance from the weights.
Target tensor The most common option is compute approximation error the decomposed layer's weights, W. However, Jaderberg et al. (2014) achieved promising results basing the decom- position the approximation error of the features. Errors in some elements of the weight tensor
4

---

Published as a conference paper at ICLR 2023
might be more permissible if they do not affect the resulting feature space. We therefore also sider the expected error on the features = X W, which is the output tensor the layer after convolving its input with weights W on input data X. Likewise, approximated weights W result in an upproximated F. In practice, computing the feature space requires input data and is computa- tionally more demanding than computing the weight approximation error. However, it is potentially more representative of an approximation's effect the output, plus unlike later fine-tuning step, it could even be used if only data without labels is available.
Table 1: Overview of the approximation errors eWW, W) considered in our evaluation. Rows show the target tensor to evaluate the error on, and columns the different options for error normalization. nw and mp are the number of elements in the weight tensor W or feature tensor F respectively.
Absolute
Relative
Scaled
Weight
IW
W|l
|
|| W|//ow
Feature
Ex
FF
FI]
Ex
||F
Ex
||
FI1/mF
3.3 TENSOR DECOMPOSITION METHODS
Our experiments shall consider three popular decomposition methods for convolutional layers, namely CP (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2015), Tucker (Kim et al., 2016), and Tensor Train (TT) (Garipov et al., 2016). During the decomposition step the decomposed weights are found by minimizing the approximation error between the pretrained weights and the estimated decomposition: arg ming I|W- Wll. For CP this done with ALS (Carroll & Chang, 1970; Harshman, 1972), for Tucker with HOSVD (De Lathauwer et al., 2000), and TT-SVD (Os- eledets, 2011) for Tensor Train. The ALS algorithm requires random initialization. We sample from uniform distribution (0.1) using Tensorly (Kossaifi et al., 2019b). The desired compres- sion level is achieved by finding the corresponding rank, using the package Tensorl T y-oorro ch (Kossaifi et al., 2019b). The ranks used for CP, Tucker, and TT are given in Appendix A.1. For completeness, we list all considered decompositions with a 4-way tensor W.
CP decomposition rank- R CP decomposition (Hitchcock, 1927) sums R rank-one tensors:
R
WCP C.y.a,t
Σ Yu,
(2)
Tucker decomposition A Tucker decomposition (Tucker, 1966) is distinct from CP by the Tucker core G E RR Xxxxxxxxxxxxxcee The Tucker rank defined as the four-tuple ( (Ri, R2, Rs, R4). Since the dimensions of the convolutional weights are small with respect to the width and height dimensions, it is computationally more efficient to contract these with the Tucker core G and form a new H-
G X2 Y X3 X, where X, the n-mode product (Appendix A.2):
R2
R4
W Tuackrr o,y.m,
ΣΣΣ>
G...a.ra.ra Casr Y X r. T Te
Σ
H H Can Te.ra-
(3)
Tensor Train decomposition Another alternative is the Tensor Train decomposition (Oseledets, 2011), which decomposes a given tensor as linear chain of 2-way and 3-way tensors, where the first and last tensors are 2-way. The TT-rank in our four-dimensional case is the 3-tuple (R1, R2, Ra):
R
R₂
Rs
WTT c.g.a.t
Σ
Σ
Can Y Ymug.ry X Try.t.
(4)
n=
m=
EXPERIMENTS
This section provides the implementation details and discusses the results of our empirical approach.
5

---

Published as a conference paper at ICLR 2023
4.1 EXPERIMENTAL SETUP
Datasets The experiments are run on the datasets CIFAR-I0 (Krizhevsky, 2009) and Fashion- MNIST (Xiao et al., 2017). These datasets are common classification benchmarks for testing TD in CNNs (Cheng et al., 2021; Denil et al., 2014; Wu et al., 2020; Garipov et al., 2016; Hawkins al., 2021). For both datasets, the original training sets are split into the set used for training and validation. The split is made such that equal class distributions are maintained. The details specific to the datasets are as follows: CIFAR-10 has 10 classes distributed equally across 60,000 images of 32 x 32 pixels with 3 color channels. After our validation split, there are 45,000 images in the training set and 5,000 in the validation set. The test set of 10,000 images remains unchanged. Fashion-MNIST has 10 classes distributed equally across 70,000 grayscale images of 28 x 28 pixels. After our validation split, there are 55,000 images in the training set and 5,000 in the validation set.
Model architecture and training The models used are ResNet-18 (He et al., 2016) and GaripowNet (Garipov et al., 2016). ResNet a well-performing ofate-of-the-art convolutional neural network. GaripovNet is a 7-layer convolutional neural network proposed in Garipov et al. (2016) and used by Hawkins al. (2022) for image classification. These models enable comparison with other works within and beyond TD for deep learning (Gusak et al., 2019; Garipov et al., 2016; Hawkins et al., 2022; Chu & Lee, 2021; Kossaifi et al., 2020). The following hyperparameters are used: ResNet-18 is trained with batch size 128, for 300 epochs, with Adam optimizer and a learning rate of 10-3 At epochs 100 and 150 the learning rate multiplied by 0.1. GaripoyNet is trained with the same settings as the original paper Garipov et al. (2016). The model is trained with Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a learning rate of 0.1, multiplied by 0.1 at epochs 30, 60, and 90.
The validation set is used for early stopping and the selection of the training hyperparameters, i.e. learning rate, schedule, level of annealing, batch size, and optimizer. Training data is augmented with a random crop (padding with 4 pixels and cropping to the original size) and a random horizontal flip. All images are standardized based on the training mean and standard deviation per channel overall training samples. Early stopping is applied for both training the baseline and fine-tuning the decomposed model. The classification error the test is used for performance errors P. To Fine-tune after decomposition and obtain performance errors P", the ResNet-18 is optimized for another 25 epochs, and GaripovNet for 10 epochs, using the last learning rate from the training.
Decomposition choices We now explain the considered values for the decomposition choices H explained in Section 3. For both neural network models, neither the first nor the last layer will be de- composed, as these layers already contain a relatively small amount of parameters. For GaripovNet the other five layers are part of L. For ResNet- II contains selection of eight convolutional lay- ers, details of which can be found in Appendix A.3. The set of TD methods that will be considered is M = {CP, Tucker, Tensor Train], which were discussed in Section 3.3. The set of compression levels is C = { f10%, 25%, 50%, 75%, 90%). J. Multiple levels of compression are considered as each neural network layer can have different efficiency-perfommin the experiments, we evaluate ResNet-18 8 on CIFAR- 10, GaripowNet on CIFAR-10, and GaripovNet on F-MNIST. We exclude ResNet-18 on F-MNIST the dataset is not sufficiently challenging for this model, and compressing one layer does not lead to a viable impact on the performance due to the model's size and skip-connections.
trade-offs (Lebedev et al., 2015). In
Variance The process of decomposing and fine-tuning is repeated for five independent runs for each choice of layer, decomposition method, and compression level to assess and report variance in the results. Note that due to the stochasticity of the ALS algorithms, the random initializations can result in different CP decomposition estimates. The variance in correlation shown in the plots without fine-tuning results from the randomness in the CP initialization. Fine-tuning adds addi- tional variance through its use of batched SGD. The observed variance with fine-tuning accounts for both the randomness from CP initialization as well as from fine-tuning, thereby representing all sources of randomness in our methodology. All runs for given model and dataset are based on the same pretrained weights, so this is not a source of reported variance. In total, this results in 600 measurements for ResNet-18 18 and 375 measurements for GaripovNet per dataset.

---

Published as a conference paper at ICLR 2023
tuning
tuniig
1.0
0.8
0.6
Compression
Correlation
0.4
25%
0.2
75%
0.0
44
-0.2
ResNet-19 & CIPAR-10
Garipoviitt CIFAR-10
GarppsyNee & F-MNIST
ResMet-18 & CiFAR-1o
Garipovliet CIPAR-10
Gampoviiet F-MNIST
Correlation (r) layers and decompositions, grouped by the level of compression (color), using the Relative Weights approximation error. Shown in the bars are averages and standard deviation over runs. We observe that a higher compression results in larger correlation.
Figure 1:
4.2 EXPERIMENTAL RESULTS
Impact of compression levels on correlation We start by calculating the correlation across the layers and decomposition methods for multiple runs and calculate the averages grouped by compres- sion levels. This is presented in Figure 1, where the bars are the average correlation т between the Relative Weights and the classification error. The correlation only based on the Relative Weights, as this is the most common metric in recent literature (Lebedev et al., 2015; Hawkins et al., 2022; Liebenwein et al., 2021). The correlations are grouped by the different levels of compression and represented by different colors. The eπror bars +1 standard deviation, representing the variance from multiple runs.
In Figure 1, it can be that the larger the compression, the higher the correlation is. This is positive result for our use case. In the end, we are interested in making decomposition choices when compressing. The more compress, the higher the correlation and therefore the more certain are that basing our choice on the approximation error results in the optimal choice. It can also be noted that certain level of compression needed be able to make choices based the approximation error. For both models and datasets, the correlation is small when the compression only 10% and 25%. The variance in the correlation at smaller compression levels larger than higher compression levels. When the compression is too small the effect the performance of the model too small compared the observed variance, especially after fine-tuning. the remainder of the experiments, we therefore focus on compression levels of C (50%, 75%, 90%).
Comparison of approximation used a single approximation error, e.g. Relative Weights, to identify which layer to compress next, implicitly assuming that relative errors between layers are indicative of the relative model perfor- mance differences. We here compare the various approximation eтror measures, testing the corre- lation with performance over all decomposition choices. In Figure 2, the correlation is calculated based on measurements of all combinations of layer, decomposition method, and compression level once. The correlations are averaged over rums, well as the +1 standard deviation is calculated over the runs.
measures Works such Liehenwein al. (2021) have
Figure 2 shows that the correlations are generally positive and significantly different from zero. This means that the decomposition choices can (to some extent) be based on the approximation er- ror. There one exception where the correlation close to zero, namely for the Absolute Weights measure ResNet-18. The difference in correlations can be explained by the difference in approx- imation error between layers, a detailed explanation provided in Appendix A.4. These results suggest that using Absolute-based approximation errors, while they may show high correlations in some cases, are not generally reliable indicator for future model performance, and that normalized measures should be used instead.

---

Published as a conference paper at ICLR 2023
fine-tuning
With buning
1.0
0.8
Approximation Error
0.6
Relative Weights
Correlation
Relative Features
Scaled Weights
0.4
Scaled festures
plute Weights
Absollute Features
0.0
1e CIFAR-10
CaripouNeet CIFAR 20
GaripovNet & F-MINET
RewNor-19 & CIFAR-10
GaripouNer & CIFAR-10
Garipo & F-MNIST
Figure 2: Correlation (r) calculated over layers, decompositions, and compression levels, averaged (and standard deviation) over runs, for different approximation errors (colors). The Relative Weights approximation error provides the highest correlation, both before and after fine-tuning.
Comparing the different approximation error metrics, we observe the highest correlation with the performance for Relative Weights in all tested cases. Interestingly, the magnitude of the correlation for Feature-based measures is similar to smaller than the correlation for Weight-base based measures. Although the findings of Jaderberg et al. (2014) would suggest a stronger correlation for the features, at least before fine-tuning, we do not observe benefits for basing decomposition choices the approximation error of the features rather than the weights. Possibly pretraining already ensures all the elements in the weight tensor are equally important for the target data distribution, thus Weight-based error already reliably reflects resulting errors on the output features. Comparing the error bars with and without fine-tuning, the randomness from fine-tuning has a larger impact on the variance in correlation than the randomness from CP initialization. In summary, our results support the use of the Relative Weights approximation error to make decomposition choices.
Impact of fine-tuning Most works use fine-tuning to recover some of the lost performance (Den- ton et al., 2014; Lebedev et al., 2015; Kim et al., 2016). The right subfigure of Figure 2 shows the mean and standard deviation of five correlations, per model and per dataset after fine-tuning. After fine-tuning, the correlation between the approximation error and the performance error is smaller than before fine-tuning for GaripovNet, as additional training adapts the model and reduces the per- formance gap between the different choices, but this effect is not observed for ResNet- 18 where the correlation was already lower. However, for both models, there is still clear positive correlation between the approximation error and the performance after fine-tuning. This means that decompo- sition choices can still be based on the approximation error when intending to perform fine-tuning later, even though different hyperparameters might be optimal without and with fine-tuning,
While the correlation positive and significantly different from zero, the correlation is only around +0.5 for ResNet- 18. We therefore investigate if the correlation is higher when only considering specific decomposition choices next.
Correlation across Layers vs. Methods In the previous experiments, we compared how de- composition choices on both different layers and methods correlated with performance. Here we investigate if the correlation is stronger if only one of these choices would be considered. For in- stance, previous works often only include layers as decomposition choice, and have not compared across decomposition methods. We compare correlation on all choices for both sets (IL x M x C) as before, to layers only (LL x {m} x {c} with reported results averaged for all m MI and E C), and to methods only ((I) x M x {c} with reported results averaged for all E LL and E C).
Figure 3 shows that before fine-tuning the approximation error has lower correlation with the per- formance of the model when considering layers only compared to all decomposition choices. Not all layers of neural network have the same efficienny- perroormcy- trade-off (Lebedev et al., 2015; Hawkins et al., 2021). Therefore, the correlation is lower when we fix the decomposition method and compression level. It is better combine layers with compression levels (and decomposition
8

---

Published as a conference paper at ICLR 2023
Without fine-tuning
fun-iignng
1.0
Correlation
0.6
Correlation
all methods
0.4
0.2
Re & CIRAR-20
GaripovNet & CIFAR.10
GärppovNet & F-MMNSST
Restret 18 CIFAR-10
Garpoviket& CIFAR-10
GaripovNet F-MIOST
Figure 3: Correlation (T) calculated over different decomposition choices (colors) for layers and decomposition methods using the Relative Weights error (all is identical to its results in Fig. 2). The reported correlations are averaged over all not-compared choices and runs. The standard deviations only calculated over runs make the three groups comparable. Before fine-tuning, across layers has less correlation than across methods, though interestingly this pattern revered by fine-tuning.
methods). However, fine-tuning recovers some of the correlation for layers. Across decomposition methods, the correlation before fine-tuning comparable to the correlation calculated across all decomposition choices. These results suggest decomposition methods can be compared better than just the layers before fine-tuning, although the former is not an optimization choice considered in previous works. Interestingly, for GaripovNet the correlation across decomposition methods drops significantly after fine-tuning. We find that this due difficulties optimizing the CP decom- posed layers, since the gradient flow through CP convolutions is known problem (Silva & Lim, 2008; Lebedev et al., 2015), whereas ResNet does not suffer from this due to its skip connections. We conclude that (unlike current practice) network compression could consider multiple decompo- sition methods as their approximation errors can be compared, though most reliably when aiming for compression without fine-tuning.
5 CONCLUSION
We have tested Assumption 1, and find that there is positive correlation between the relative ap- proximation error on the weights and the resulting performance error the model for a wide range of TD choices, including layers, methods, and compression levels. We further find that using data to compute the approximation error on the features, rather than simply on the model weights directly, does not improve the correlation. Scaling the approximation error with the sor provides the highest and most stable correlation across all compared models and datasets. Our findings suggest that the Relative Weights approximation error is best suited select among TD decomposition choices.
of the original ten-
While these choices can be made across layers, TD methods, and compression levels, we observe that the correlation before fine-tuning is smaller when comparing between layers for fixed method, than when comparing across methods (here: CP, Tucker, and Tensor Train) for fixed layer. In- tegrating multiple types of decompositions within network compression technique is therefore potential direction for future work, although care has to be taken when the use case includes later fine-tuning, the correlation for selecting across decomposition methods degrade since back- propagation through certain factorized structures remains challenging.
Our experiments are limited to set of decomposition choices and network layers commonly found in the TD literature. Future work can extend to other decompositions and other types of neural network layers, e.g. fully connected layers. While the weights are matrices, tensor decomposition has been applied to fully connected layers by reshaping the weight matrix into a higher-order tensor. The choice of reshaping then becomes additional decomposition choice.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
The authors find it important that this work is reproducible. To this extent the following efforts have been made: The datasets and test-validation splits are described in Section 4.1. The datasets are collected from PyTorch Vision. The models and hyperparameters used for training covered Section 4.1. The implementations of the baseline models are from the PyTorch Model 2oo. The models are factorized with Tensorly-Toroe (Kossaffi et al., 2019b). using CP initialization described in Section 4.1 and ranks provided in Appendix A.1. The experimental setup is explained in Section 4. The calculation of metrics is formulated in Section 3. Finally, the code reproduce these experiments available at: https: thub. com wwsSchuuraans//e.m
ACKNOWLEDGMENTS
Described results are made possible in part by TU Delft Cohesion subsidy, TERP Cohesion project 2020.
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger Generalization Bounds for Deep Nets via Compression Approach. ICML, pp. 10, 2018.
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data- dependent coresets for compressing neural networks with applications to generalization bounds. In ICLR, pp. 35, 2019.
J. D. Carroll and J.J. Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart- Eckart-Young decomposition. Psychometrika, pp. 35, 1970.
Song Cheng, Lei Wang, and Pan Zhang. Supervised learning with projected entangled pair states. Physical Review B, 103(12);12511 March 2021. Publisher: American Physical Society.
Bo-Shiuan Chu and Che-Rung Lee. Low-rank Tensor Decomposition for Compression of Convolu- tional Neural Networks Using Funnel Regularization. arXiv:2112.0300 [cs], December 2021.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A Multilinear Singular Value Decom- position. SIAM Journal Matrix Analysis and Applications, 21(4):1253-1278, January 2000. Publisher: Society for Industrial and Applied Mathematics.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting Parameters in Deep Learning. NIPS, 16:2148-21555 2014.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. In NIPS, pp. 9, 2014.
Moonjung Eo, Suhyun Kang, and Wonjong Rhee. A Highly Effective Low-Rank Com- pression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank. arXiv:2111.15179 [cs], November 2021.
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensoriza- tion: compressing convolutional and FC layers alike. NIPS, November 2016.
Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschen- sky, Andrzej Cichocki, and Ivan Oseledets. Automated Multi-Stage Compression of Neural Net- works. In ICCV Workshop, pp. 2501-2508, Seoul, Korea (South), October 2019. IEEE.
Richard A Harshman. Foundations of the parafac procedure: models and conditions for an "ex- planatory" multimodal factor analysis. UCLA Working Papers in Phonetics, 22:111-117, 1972.
Cole Hawkins, Haichuan Yang, Meng Li, Liangzhen Lai, and Vikas Chandra. Tensor Compression for Neural Networks. arXiv:2111.01697 [es], November 2021.
10

---

Published as a conference paper at ICLR 2023
Cole Hawkins, Xing Liu, and Zheng Zhang. Towards Compact Neural Networks via End-to-Enu Training: A Bayesian Tensor Approach with Automatic Rank Determination. SIAM Journal on Mathematicc of Data Science, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, Las Vegas, NV, USA, June 2016. IEEE.
Frank L. Hitchcock. The Expression of Tensor or Polyadic Sum of Products. Journal of Mathematics and Physics, 6(1-4):164-1)) 189, 1927.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up Convolutional Neural Net- works with Low Rank Expansions. arXiv: 1405.3866 [cs], May 2014.
M. G. Kendall, A New Measure of Rank Correlation. Biometrika, 30(1/2):81-93, 1938. Publisher: [Oxford University Press, Biometrika Trust].
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo. Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. In ICLR, February 2016.
T Kolda and B Bader. Tensor Decompositions and Applications. SIAM, 51(3)4555) 2009.
Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. T-Net: Parametrizing Fully Convolutional Nets With a Single High-Order Tensor. In CVPR, pp. 7822-7831, 2019a.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. TensorLy: Tensor Learning in Python. JMLR, 20(26):6, 2019b.
Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M. Hospedales, and Maja Pantic. Factorized Higher-Order CNNs With Application to Apatio-Temporaa Emotion Esti- mation. In CVPR, pp. 6060-6069, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up up Convolutional Neural Networks Using Fine-tuned De-Decomposition. In ICLR, April 2015.
Lucas Liebenwein, Dan Feldman, Alaa Maalouf, and Daniela Rus. Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition. NeurIPS, pp. 17, 2021.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing Neural Networks. In NIPS, pp. 9, 2015.
I. V. Oseledets. Tensor Traion-Train Decompositinn. SIAM Journal on Scientific Computing, 33(5):2295- 2317, January 2011.
Vin De Silva and Lek-Heng Lim. Tensor rank and the ill-posedness of the best low-rank approxi- mation problem. SIAM Journal Matrix Analysis and Applications, 3013):108--127, 200., 2008.
Ledyard R Tucker. Some mathematical notes three-mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
N. Vannieuwenhoven. J. Nicaise, R. Vandebril, and K. Meerbergen. On Generic Nonexistence of the Sahmidt-Eckaat Young Decomposition for Complex Tensors. SIAM Journal on Matrix Analysis and Applications, (5(3):886-900) January 2014.
Bijiao Wu, Dingheng Wang. Guangshe Zhao, Lei Deng, and Guoqi Li. Hybrid Tensor Decomposi- tion in Neural Network Compression. Neural Networks, 132:309-320, December 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms. 1787:7770. [cs, stat], September 2017.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 RANK AND COMPRESSION LEVEL
Tables and present the ranks that are used for GaripovNet (Garipov et al., 2016) and ResNet- (He et al., 2016) respectively. Note that in these tables, the Tucker rank includes the kernel ranks corresponding to the width and height, and the TT rank includes Rg R4 which are left out of Equation 4 for conciseness.
Table 2: pression level.
Ranks of CP, Tucker, and TT corresponding to layers of GaripovNet with a given com-
Layer Compression (%)
CP
Tucker
TT
2
10
28
(14, 14, 3, 3)
(1, 40, 12, I)
2
25
69
(26, 26, 3)
(1, 64, 11, I)
2
50
138
(39, 3,
61,
2
75
206
(49, 49,
51,,
2
248
(54, 3,333)
(1,64, 65, 1)
4
37
(26, 13,3,3)
63, 18,
4
25
93
(49, 3,
(1, 14,
4
50
186
(7,,37,3,3)
(1, 108, 1)
4
75
279
(94, 4,
(1, (1, 60, 128, 1)
4
90
335
(105, 3,33
(1, 80, 128, 1)
6
10
56
(29,
(1, 4, 12, 1)
141
5,,
(1, 128, 21, 63, 1)
6
50
281
(77,77,3,3)
(28 53, 128, I)
6
422
(98, 98, 3.3)
(28, 101, 128, 1)
6
90
507
(108, 108, 3,
(1, 128, 130, 128, 1)
8
10
56
(29, 2,
(1, 12, 128, 21, 63, I) (18 53, 128,
8
25
141
51,,51,3,3)
8
50
281
(77,77,3,3)
8
75
422
(98, 98, 3)
(1, 128, 101, 128, 1)
8
507
(108, 108, 3,3)
(1, 128, 130, 128, 1)
10
56
(29, 29,
(1, 94, 4,,
10
25
141
51,
128, 21, 63, 1) (1, 128, 53, 128, 1)
10
281
(77, 3,3)
10
75
422
(98, 98, 3)
(1, 101, 128, 1)
10
90
507
(108, 108, 3,3)
(2, 130, 128, 1)
A.2 N-MODE PRODUCT
The definition of n-Mode Product Xn given by Kolda & Bader (2009) is used in this paper. The contraction of tensor G Ra, Rxx with matrix Y IR Y. along the nihh mode of the tensor is defined elementwise as:
R
(X X Y) Po-t-Wieee
X Pno JN Y
(5)
12

---

Published as a conference paper at ICLR 2023
Table 3:
Ranks of CP. Tucker, and TT corresponding to layers of ResNet-18 18 He et al. (2016) with
a given compression level.
Layer Compression (%)
CP
Tucker
TT
15
10
28
(14, 14, 3,
(1, 40, 4, 12,
15
25
69
(26, 3.
(1. 64, 11, 1)
15
50
138
(39, 39, 3,3)
(I, 26, 64, 1)
15
75
206
(49, 49, 3,3)
(1, (1, 64, 1)
15
248
(54, 54,3,3)
(1, 64, 1)
19
10
37
(26, 13,3,3)
(3, 18, I)
19
25
93
2,,333
(1, 61, 19, 1)
19
50
186
37,
(1, 108, 1)
19
75
279
(94, 4,333)
(I, 6,, 128, 1)
19
90
335
32,
(1, 64, 80, 128, 1)
28
10
56
(29, 2,,
(1. 94, 4, 12, 1)
28
141
(5,,
(1, 128, 21, 63, 1)
28
50
281
77,
(1, 128, 53, 128.
28
422
(98, 3,3)
(I, 128, 101, 128, I)
28
90
507
(108, 108, 3,
(1, 128, 130, 128,
38
10
114
(57,
(I, 207, 15,
38
25
285
(103, 1,3, (155, 3,3,
(I, 256, 43, 129, I)
38
50
569
(1, 256, 107, 256, 1)
38
75
854
256, 203, 256, 1) (195, 195,3,3)
38
90
1025
(216, 216,
(1, 256, 262, 256, 1)
41
10
8
1,
(5, 11, I)
41
25
21
(25, 12, 1,
(1,256, 25, 75, I)
41
50
42
(48, 1,11
(1,256, 43, 129, 1) (1, 213, 256, 1) (1,256, 280, 256, 1)
41
75
64
(69,
41
90
76
(82,41,
44
10
114
57,
207, 15, 1)
44
25
285
(103, 1,3,
(1, 256, 43, 129,
44
50
569
(155, 1,5,
256, 107, 256, 1) (1, 256, 203, 256, 1) (1, 262, 256, 1)
44
854
(195, 195,
44
90
1025
216,
60
10
229
(115, 115,
(1,425,5, 15, 1)
60
25
573
(205, 203,3,3)
(1, (1,512 85, 255, 1)
60
50
1145
(310, 3.3)
(1, 213, 512, I)
60
75
1718
(390, 390, 3,3)
(1, 401, 401, 512, I) (1, 512, 526, 512, 1)
60
2062
(432, 432,
63
10
229
(115, 3,3,
(I, 42,4 15, 1)
63
25
573
(205, 205, 3,3)
(1, 85, 255, 1)
63
50
1145
(310, 310, 3,3)
(I. 512, 213, 512, 1) (1, 401, 512, I)
63
75
1718
(390, 390, 3.3)
63
90
2062
(432, 432, 3,
(1,5 512, 1)
13

---

Published as a conference paper at ICLR 2023
A.3 SELECTION OF RESNET-18 18 LAYERS
The following layers are considered for decomposition in ResNet-18: The last layer of the first two blocks. The first layer with stride two. The first layer after 1xl convolution. A 1xl convolution with a similar number of parameters as the first choice layer. The layer before and after the 1xl convolution. The final two convolutional layers, as they are the largest convolutional layers.
Table The layers considered in ResNet-18, The Layer nr. reflects the layer number in the official PyTorch implementation, starting with the first input layer and counting non parameterized layers as well.
Layer nr.
Type of convolution
Order in ResNet block
15
regular 2D Conv regular 2D Conv regular 2D Conv regular 2D Cony 1x1 2D conv
last of the first two blocks first with stride 2 first after a Ixl convolution
19
28
38
last layer of two blocks, before 1x1 Conv 1xl conv
44
regular 2D Cony regular 2D Conv regular 2D Conv
first of block, after 1x1 second to last Conv layer
60
63
last Conv layer before avg.pool and classification head
14

---

Published as a conference paper at ICLR 2023
A.4 DIFFERENCE BETWEEN LAYERS ABSOLUTE WEIGHTS FOR RESNET-18
Let us recall from Figure 2 that when basing the approximation enror on Absolute Weights, the correlation with the performance the difference between layers.
close zero for ResNet-18. The correlation is zero due
Figure plots the approximation errors of Relative and Absolute Weights versus the performance error before and after fine-tuning, The points resulting from CP. Tucker, and Tensor Train with compression of 50%, 75%, and 90% are grouped per layer.
Lag classification error with fine-tuning Log classification error without fine- te-tunnigg
-0.75
-1.00
19
28
-1.25
-1.50
-1.75
63
-2.00
-2.50
-2.1
-2.3
-2.4
-2.5
0.2
0.4
0.6
0.8
1.0
0
25
50
75
100
125
Relative Weights
Absolute Weights
Figure Approximation error calculated with Relative Weights (left) and Absolute Weights (right). Layers 15, 19, and 28 have smaller absolute weight error and large performance етооr relative to layers 38, 60, 63. Compared to the Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to layers 15, 19, and 28.
Layers 15, 19, and 28 have smaller absolute weight eиror and large performance error relative to layers 38, 60, and 63. Compare this to the data for Relative Weights where layers 60 and 63 have a small relative weight error and small performance error and layer 28 has comparable errors to 15, 19, and 28. This leads to the correlation between Absolute Weights and performance errors being close to zero, while is positive for Relative Weights. Therefore, the difference in correlations can be explained by the difference in approximation error between layers.
15


Reviews:
[{"review_id": "k9d7daH9FAL", "reviewer": "Reviewer_kJo3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this study the authors investigate post-training compression of CNN-based image classification neural networks. One method to compress a CNN-based NN is tensor decomposition. Their goal is to learn how much the approximation error of tensor decomposition is predictive of the performance of the compressed CNN-based NN classifier. For example, they ask if the choice of tensor decomposition method can be guided by this or the level of compression. This would be more efficient than using the final task performance of each candidate compressed model. Another question, for example, they investigate is if finetuning after compression does disturb or improve the correlation between approximation error and final task performance of the NN. ", "summary_of_the_review": "The paper is well written, focused and has enough experiments to educate the reader about their findings. ", "strength_and_weaknesses": "S1. The paper investigates a clear objective that is relevant for potential follow-up work and has useful practical implications.\n\nS2. The paper is well written and well structured. \n\nS3. The paper does give a clear picture and considers many practically relevant questions. \n\n\nW1. It is unclear how the increased complexity due to the proposed methods and the level of control on the final task performance do compare against a distillation approach.  \n\nW2. How much does the randomness of many of the TD methods due to their initialization influence to robustness of the approach.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Tr1WQX7V_", "reviewer": "Reviewer_YZcC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically evaluates the correlation between different hyperparameters of tensor compressive layers and the classification performance. Their results show that the error is indicative of model performance, even when comparing multiple TD methods, though useful correlation only occurs at the higher compression levels. ", "summary_of_the_review": "Due to the evaluation of \"Clarity (4/10), Quality (5/10), Novelty (4/10) And Reproducibility (4/10)\", I suggest \"Reject\".\n\n-----------After rebuttal------------\nAfter reading the authors' feedback and other reviewers' comments, I think this is a nice paper which has empirical significance. However, there is still much room for improvement in making this empirical evaluation paper much clearer. So, I decided only to change my suggestion from \"3 reject\" to \"5 borderline below\".", "strength_and_weaknesses": "[Strengths]\n\nS1: This is the first paper that systematic studies on how well the approximation error relates to a compressed neural network’s performance across multiple choices of network layers, TD methods, and compression levels. \n\nS2: The experimental results support the conclusion that there is a positive correlation between the relative approximation error on the weights and the resulting performance error of the model for a wide range of TD choices, including layers, methods, and compression levels.\n\n[Weaknesses]\n\nW1: To test Assumption 1, this work reports some experimental phenomena/results based on extensive experiments. However, it doesn't go far enough in the sense that possible reasons behind these phenomena (e.g., the impact of compression levels on the rank correlation) are not sufficiently investigated. \n\nW2: Although showing the correlation between weight (tensor) approximation error and the classification performance is interesting, it is more significant if some practical suggestions are given for designing compressed deep models based on tensor decompositions. However, the empirical findings of this work seem of limited guiding value for a deep learning practitioner who needs to decide which layer to compress, which tensor decomposition to use, and which compression level to choose for both efficiency and effectiveness.  \n\nW3: The writing is not always satisfactory due to typos like \"no works investigate if the make TD decomposition choices using specific approximation errors are well suited\", \"previous works often only includes\", and \"Shown are averages and standard deviation over runs We observe that a higher compression\". There are also confusing notions used, e.g., \"rank\" in \"the correlation is calculated based on measurements of all combinations of layer, decomposition method, and rank once\". \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z516qLhhYmt", "reviewer": "Reviewer_ZT8V", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the use of tensor decomposition (TD) for compression of the weight tensors in CNNs. It considers the problem of choosing the best compression hyperparameters (which layer to compress, which type of TD to use) for a given level of compression. In particular, it investigates if the decomposition error for a given choice can be used as an indicator of how well that particular decomposition choice will perform both without and with fine-tuning when the CNN performs its intended task. If the decomposition error does provide such an indication, it would simplify the task of choosing compression hyperparameters since it would be enough to consider the weight tensor in isolation, rather than how it interacts with the rest of the network. \n\n", "summary_of_the_review": "I like this paper. Although it doesn't introduce any new methods, it provides a nice empirical investigation into the problem of how to choose an appropriate decomposition when using tensor decomposition to compress CNN layers. It is also overall well-written. There are some details that are hard to understand. I listed these four points under weaknesses (a-d under weakness W1). It is particularly important that the reviewers address these four points.\n\n### Update after rebuttal ###\n\nThe reviewers addressed the concerns I had. I would have preferred to raise the score from 6 to 7. However, since there is no option for 7, I'm raising it to 8.\n\n", "strength_and_weaknesses": "--- Strengths ---\n\nS1. The paper provides a good overview of previous works that use TD for compression of NNs.\n\nS2. The problem the paper tries to solve is relevant in practice if someone wants to use TD to compress a CNN, but doesn't have any prior insight into what type of TD would work well for that particular network.\n\nS3. Although some details are hard to understand, the paper is overall well-written and the different experiments are interesting.\n\n--- Weaknesses ---\n\nW1. Some details are hard to understand. I list these below.\n\na. In Fig 2: What is the difference between Relative Weights and Scaled Weights? Similarly, what is the difference between Relative Features and Scaled Features? Based on Table 1, isn't the relative measures the same as the scaled measures but with the particular choice $n_W = ||W||$ and $n_F = ||F||$? If this is correct, then how is $n_W$ and $n_F$ different for the scaled measures? Also, Table 1 is confusing since $n_W$ and $n_F$ aren't introduced anywhere.\n\nb. In the paragraph \"Comparison of approximation error measures\" on page 7, in the sentence \"In Figure 2, the correlation is calculated ... and rank once.\": By \"rank\", do you mean compression level (i.e., one of the three choices {50%, 75%, 90%})? Calling this rank is a bit confusing, since there's many choices of rank that potentially could yield a certain compression ratio, but only three compression ratios under consideration. \n\nc. Related to the point b above: With 3 types of decomposition and 3 levels of compression and 5 convolutional layers for ResNet-18, shouldn't there be 5 * 3 * 3 = 45 different measurements rather than 40? Similarly, since GaripovNet has 8 layers to choose from, shouldn't there be 8 * 3 * 3 = 72 different measurements for that network rather than 50? Please clarify.\n\nd. In Sec 3, the iterative approach is a bit hard to understand. For example, if my network has 10 layers and I want to compress 5 of them, do I run 5 iterations of the method, compressing one layer each iteration out of the remaining uncompressed layers? Does the iterative method have a target compression, and how does that impact the number of layers to compress? Perhaps you can add a more detailed algorithm for how this works in the supplement?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P_oi9qjVnC", "reviewer": "Reviewer_33s4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper performs an empirical study on the relationship between the approximation error of a model’s weight tensor and the model performance after compression. Multiple decomposition methods (CP, Tucker, and Tensor Train), and level of compression are tested on several models and datasets.", "summary_of_the_review": "As discussed above, I am close to a rejection due to its limited novelty.", "strength_and_weaknesses": "Strength:\n- The paper does a great amount of experiments to validate the assumption, namely there is a positive correlation between the relative approximation error on the weights and the model performance for a wide range of TD choices, including layers, methods, and compression levels.\n\nWeakness:\n- The contribution is not significant since the results are not surprised (e.g., relative weight approximation error is better than using absolute weight and rescaled weight), though I appreciate the engineering efforts the author spent on this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster