Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
Authors: Alper KALLE
Théo Rudkiewicz
Mohamed Ouerfelli
mohamed Tamaazousti
Venue: NeurIPS 2025 poster ()
Keywords: Compression of neural networks
tensor decomposition
low-rank approximation
distribution-aware norm
Primary area: optimization
TL;DR: Distribution-aware ALS algorithms for Tucker-2 and CP decompositions compress CNN weights by directly minimizing output-distribution shift, delivering competitive, fine-tuning-free accuracy that even transfers across datasets.

Abstract: Neural networks are widely used for image–related tasks but typically demand considerable computing power. Once a network has been trained, however, its memory‑ and compute‑footprint can be reduced by compression. In this work, we focus on compression through tensorization and low‑rank representations. Whereas classical approaches search for a low‑rank approximation by minimizing an isotropic norm such as the Frobenius norm in weight‑space, we use data‑informed norms that measure the error in function space. Concretely, we minimize the change in the layer’s output distribution, which can be expressed as $\lVert (W - \widetilde{W}) \Sigma^{1/2}\rVert_F$ where $\Sigma^{1/2}$ is the square root of the covariance matrix of the layer’s input and $W$, $\widetilde{W}$ are the original and compressed weights. We propose new alternating least square algorithms for the two most common tensor decompositions (Tucker‑2 and CPD) that directly optimize the new norm. Unlike conventional compression pipelines, which almost always require post‑compression fine‑tuning, our data‑informed approach often achieves competitive accuracy without any fine‑tuning. We further show that the same covariance‑based norm can be transferred from one dataset to another with only a minor accuracy drop, enabling compression even when the original training dataset is unavailable.
Experiments on several CNN architectures (ResNet‑18/50, and GoogLeNet) and datasets (ImageNet, FGVC‑Aircraft, Cifar10, and Cifar100) confirm the advantages of the proposed method.

Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
Alper Kalle*t alper kalle@cea. fr
Theo Rudkiewicz" t www.kieeicziinnii....
Mohamed-Oume Ouerfelli mohamed oumar. www.ouueeeeeeeee Teelliicea..t. fr www.o.ouereellleeeeeeo
Mohamed Tamaazousti mohamed, tamaazousti iccea.fr fr
Abstract
Neural networks are widely used for image-related tasks but typically demand con- siderable computing power. Once a network has been trained, however, memory- and compute-footprint can be reduced by compression. In this work, we focus on compression through lensorization and low-rank representations. Whereas classical approaches search for low-rank approximation by minimizing isotropic norm such as the Frobenius norm in weight-space, we use data-informed norms that measure the error in function space. Concretely, we minimize the change in the layer's output distribution, which can be expressed II(W W/ [| where / is the square root the covariance matrix of the layer's 's input and W, W are the original and compressed weights. We propose new alternating least square algorithms for the two most common tensor decompositions (Tucker-2 and CPD) that directly optimize the new norm. Unlike conventional compression pipelines, which almost always require post-compression fine-tuning, our data-informed ap- proach often achieves competitive accuracy without any fine-tuning. We further show that the same covariance-base another with only a minor accuracy drop, enabling compression even when the original training dataset is unavailable. Experiments on several CNN architectures (ResNet- 18/50, and GoogLeNet) and datasets (ImageNet, FGVC-Aircraft, Cifar10, and Cifar 100) confirm the advantages of the proposed method.
norm can be transferred from one dataset to
Introduction
For the past decade, neural networks have consistently outperformed other algorithms across a wide array of tasks particularly in computer vision, including character recognition, image classification, object detection, image segmentation, and depth estimation. Notably. even with the rise of other architectures, Convolutional Neural Networks (CNNs) continue to push the boundaries of performance in computer vision, with recent ad vanncements demonstrating their sustained competitivenes: for the offte-of-the-arr results [Liu et al.) 2022 Woo al. 2023]. A defining characteristic of these networks, in contrast to traditional algorithms (e.g., nearest neighbors, decision trees), is their substantial number of parameters. For instance, prominent architectures such as AlexNet (Krizhevsky 2017) VGG-16 (Simonyan and Zisserman 2013), ResNet (He et al. 2016) and GoogleNet [Szegedy et al.
2015 possess tens millions of trainable parameters.
"These authors contributed equally
UUiiversiéé Paris-Saclay, CEA, List, 1-0, Palaiscau, France *TAU team, LISN, Université Paris-Saclay, CNRS, Inria, 91405, Orsay, France
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite the performance benefits often associated with a large parameter count during training, oppor- tunities exist for creating more compact representations of trained models. This particularly critical for applications requiring efficient execution on resource-constrainet embedded devices, such as on-device image classification for smartphones and systems for autonomous vehicles. Consequently, significant body of research has emerged, focusing on techniques to reduce the parameter count in neural networks [Cheng et al., 2018), including methods like quantization, pruning, knowledge distillation, and low-rank factorization.
Classical approaches to low-rank tensor approximation for network compression typically involve finding a compressed representation that minimizes an isotropic norm, such as the Frobenius norm, of the difference between the original and compressed weights. This minimization is performed directly in the weight space of the network. While mathematically convenient, such an approach does not explicitly consider the impact of weight changes on the network's 's functional behavior or its output given the data it processes. Consequently, these methods often lead to a notable drop in accuracy, necessitating subsequent, computationally intensive fine-tuning stage to recover performance. This fine-tuning step typically requires access to the original training dataset, which may not always be available.
In this paper, we formalize a paradigm shift from classical weight-space error minimization for tensorized layers. Traditional tensorization approaches replace a computational block, typically a single layer fe parameterized by 6, with a sequence of more compact layers evers-for&re instance, a composition fABB.C := 9Λ hB ic, where A, B. C are the new, smaller parameter tensors. Classical methods then seek to find A, B, C such that their effective combined parameters are close to the original 0 i.e. [|r B,C) C) 0llp small (where is the operator that combines the low rank tensors). In contrast, the data-informed approach aims to ensure that the function implemented by the sequence of compressed layers, fA,,CC. B,C. closely approximates the original layer's 's function, fe. for the actual data the network encounters. This means we measure the error directly in the function space the layer's input-output behavior rather than just its
|lAA,.CC - Jo a. prioritizing the preservation parameter values. This make functional norm and distribution-awar norm. This leads to properly justify the "data norm" that was used by Denton et al. 2014) in the case of SVD decomposition.
To operationalize this concept, we develop novel Alternating Least Squares (ALS) algorithms specifically designed to optimize this new data-informed norm for two of the most widely used tensor decomposition formats: Tucker-2 and Canonical Polyadic Decomposition (CPD). A significant advantage of our approach is its ability to achieve competitive accuracy levels often without any post- compression fine-tuning, thereby streamlining the compression pipeline and reducing its dependency on extensive retraining.
Furthermore, we demonstrate a crucial property of our covariance-baset norm: its transferability. We show that the input covariance statistics learned from one dataset can be effectively applied to compress models for different, related datasets with only a minor degradation in accuracy. This finding is particularly valuable as it opens up possibilities for compressing models even when the original training data is inaccessible due to privacy, proprietary restrictions, other constraints.
Our contributions
1. The use of distribution-informeed norm for the neural network tensor decomposition.
2. Development of new ALS algorithms for Tucker-2 and CPD tensor decompositions that directly optimize this distribution-inforree norm.
3. In depth comparison of tensor decomposition with Tucker and CP decompositions with and without the knowledge of data distribution, and also comparing different approach: tensor deflation and CP-ALS algorithms with respect to distribution-inforred norm.
4. Empirical evidence showing that our approach often achieves competitive accuracy without fine-tuning, and that the learned covariance- based norm can be transferred across datasets.
We validate our proposed methods through comprehensive experiments on several prominent CNN architectures, including ResNet-18, ResNet-50, and GoogLe LeNet, across diverse benchmark datasets such as ImageNet, FGVC-Aireraft, Cifar10, 10, and Cifar100. The results consistently underscore the advantages of our distribution-informeed compression strategy.

---

2 Related work
Our work intersects three active threads in neural-network compression: tensor-based methods, functional norms, and data-free compression techniques.
Tensorization Tensor decomposition reduces both parameter count and computational cost by approximating weight tensors with low-rank formats. Early efforts include Denton et al. (2014);
subsequent milestones span CP decomposition for CNNs [Lebedev et al.) 2015], Tensor- Train
for fully connected layers [Novikov et al., 2015), Tucker decomposition [Kim et al. 2016) and
Tensor-Ring networks [Zhao et al, 2019]. Most of these methods follow a two-step pipeline: (i) compute low-rank approximation, then (ii) fine-tune to restore accuracy. Several studies instead alternate tensorization with lie-tuning-eee compression [Astrid and Lee.
the tensor-powel method for successive layer 2017] and the MUSCO pipeline, which repeatedly compresses and
fine-tunes [Gusak et al. 2019] Rank selection has also been explored: variational Bayesian matrix factorization (VBMF) offers data-free criterion, applied Tucker layers [Kim et al. (0066) and later to CP layers
Astrid et al. 2018), building on the analytic VBMF solution of Nakajima et al. 2010).
Most of the above techniques minimise error in weight space, which is not directly linked to a network's functional behaviour. Functional norms instead measure distances between functions. They underpin natural -gruurlegg optimisation (Ollivier 2017 Schwencke and Furtlehner, [024] and can guide network growth |Verbockhaven et al. 2024]. In compression, alterna- tive norms are less common; one example Lohit and Jones 2022], who employ an optimal-transpor loss during distillation. Another important example is Denton et al. (2014) who propose the use of two non-isotropic norms: the Mahalanobis distance and the data distance. The latest was not presented as functional norm but, we will show later, their data norm can in fact be seen as the L2 functional norm. For this data distance, they proposed replacing the approximation criterion from the Frobenius norm with a data covariance distance metric, which accounts for the empirical covariance of the input layer. They recommended using this criterion alongside either the SVD method or a greedy CP algorithm, which iteratively computes rank-one tensors for approximate CP decomposition and conducted experiments for this distance using the SVD method. Furthermore, to the best of our knowledge, different datasets to assess the benefits of this new norm. Finally, we also expand the analysis to evaluate the transferability of this norm between different datasets.
Functional norms
the first to conduct multiple experiments several classical CNNs across
Data-free compression Several mainstream compression techniques now operate without access to the original training data. Data-free knowledge distillation was introduced by Lopes al. 2017]. Zero-shot quantization approaches such as ZeroQ [Cai et al.] 2020) and DFQ Nagel et al. 2019
remove the need for calibration data. For pruning. Srinivas and Babu [2015] merge redundant neurons.
Tang et al. [2021] prune using synthetically generated data, and SynFlow prunes networks even before training Tanaka al. 2020). To the best of our knowledge, no existing work tackles the data-free high order tensor approximation problem. We address this by evaluating the use of generic distinct dataset to the one used during model training.
Alternative Norms Defining an importance score, norm, is central to structured network com- pression. Prevailing methods rely on simple weight magnitude or more complex, loss-based metrics derived from the Hessian Matrix [LeCun et al. 1989] or Fisher Information Matrix [Tu et al., 2016). Our work explores an alternative based on activation statistics. Specifically, our distribution-awarr norm uses the activation covariance matrix to model the complete second-order statistical structure of feature maps. This provides a richer measure of importance than common activation- based heuristics, which are often limited to first-order statistics like mean magnitude [Rhu et al. 2018). By capturing chter-channel dependencies, our norm offers a more holistic criterion for guiding the compression.
3 Background on compression by tensor decomposition
The general principle of tensor decomposition is to replace a part of the neural network by lighter one. For example if we have neural network ʃ gol op where g and are two neural networks and 1 is one layer of the neural network, we can replace 7 by a new squence of layers I' == 1 o ..... ln. The new neural network is then := gol' op. The goal of the tensor decomposition is to find the best
3

---

I' such that f and f' are as close possible. In most of the case the layer is either fully connected layer or convolutional layer. In the case of fully connected layer, the tensor decomposition done through the singular value decomposition (SVD) and lead to replace the layer by couple of fully connected layers. In the case convolutional layer, the tensor decomposition can be done through the CP decomposition or the Tucker decomposition as described in the next section.
Convolution A convolution Convx: parameterized by a tensor K. size (T, S. H, i is function from the space of images with S channels like X' RSSH, xW, where Hy. W,, represent the height and width of the image to the space of images with T channels like y "R''"x"f The convolution layer has SHWT parameters and the cost of the convolution operation given above is (HH W") additions-mullillicaionns
By decomposing the kernel tensor we can replace the convolution by a series of smaller layers in terms parameters and computational cost. In the next sections, we present two possible decompositions that can be seen as generalizations of singular value decomposition (SVD) to the tensor casc.
3.1 CP Decomposition for Convolutional Layer Compression
The CANDECOMPPARAFAC (CP) decomposition [Hitchcock 1927] represents tensor sum of rank tensors, where the rank tensors are them self decomposed as the outer product of vectors. For rank R. CP decomposition of kernel tensor K. XXXXXxxW is given by the following formula:
K
U @ UPS)
U.H @ U!!)
where U7!m) E R" for n E [T, H. W] are the equivalent of the singular vectors and & denotes the outer product.
This decomposition leads suggested by Lebedev al. 2015 to replace the convolution by series of four layers: a x convolution parameterized by (U, Dr.. x convolution parameterized by ), and 1 x 1 convolution parameterized by (U!T)) )r. The number of parameters of the CP decomposition is x (T + S H + W) which for small rank R is smaller than the number of parameters of the original convolutional layer; the reduction factor is the same for the computational cost.
(U!")) 1 x W convolution parameterized by (U,W),
To choose a rank for the decompostion without reliying trial and errors, we use the VBMF (Variational Bayesian Matrix Factorization) algorithm as suggested by Astrid et al. 2018). We note RVAME the maximum of ranks computed through application of VBMF to the each possible unfolding of the kernel tensor and Rmax an upper bound of the rank with Rmax := Msx(T.S.H.WT
TS HS
We use
rank Ra that a linear combination of those ranks determined by a parameter a that we call VBMF ratio:
R₂ = RyBMP + (1 - or) (Rmax - RRBME).
(2)
3.2 Tucker Decomposition for Convolutional Layer Compression
The Tucker decomposition [Tucker 1966] countrary to the CP decomposition use different number of eigenvectors for each mode. In the case of convolutional layers, as the two mode corresponding to the kernel size (H and W) small, do not decompose them which leads the Tucker2 decomposition. For a kernel tensor IC E xxxxxxx the Tucker2 decomposition with rank Ry and Rs is K = G X (T) X2 (S
As suggested by Kim et al. [016). this leads to replace the convolution by series of two layers: 1 x convolution parameterized by (T),, full convolution parameterized by G and 1 x 1 convolution parameterized by (S)).
To select the ranks Ry and Rs, ranks RT-VVMM BBMF and Rs FBMP by applying the VBMF algorithm to the unfolding matrices of sizes T x SHW and S x THW. Then we compute the ranks Ri and Rşs using the formula given
the method for CP decomposition. We compute the
Xi indicates product along the ith axis of the tensor G

---

in equation (2 on the VBMF ranks RT-VBMF and RM-V respectvvely (with the Rmax being either T or S).
The Distribution-Aware Tensor Decomposition
4.1 Functional Metric for Network Compression
In general when replacing the layer lo from f q lp o to get f = to retrain the network so we aim to have f f'. Mathematically for data distribution D, for classification task we want to have (d (x))) = 0 where d is way to measure the distance between two distributions like the Wasserstein distance or the KI. divergence.
we don't want
Most of the works use as proxy the Frobenius norm |00 0'1 where 0 are the parameters of the layer and 0' the combined parameters of the new layer (G2 (this combination is different depending on the method (see previous section)). Here we propose instead to minimize the functional norm ||le o p - lgg. o Pll12 where p is the network before the layer la where we take as measure the data distribution D. Hence, we have:
||le op lo plll= M
BEND (Ilo p(w) I/ v()))
To compute the I| in the case where lo is convolution we use the following result: Proposition 1. Consider a distribution D. partial neural network p. and two convolution Convi and Conv & parametrized by the kernel tensor K. E RTxSxxXX and K. Under reasonable assumptions
where is the unfolding operator that transforms the image p(z) into a matrix that can be used to compute the convolution matrix product. We can also define 11/2 the square root of E such that / Σ. Then, we have:
define ; (.)((u(((z)))((((())
Maavgoplle Conve op- Conv & o =| (K-
11/2 (1)
where Ocy is the reshaping of the convolution kernel into (T, S H x W).
See appendix A for the proof.
Frobenius norm Here we can notice that in the case where Σ I,, for example if we consider that D = (o, I), we recover the Frobenius norm over the parameters of the layer. Hence the functional norm is generalization of the Frobenius norm over the parameters of the layer, suited for the case of a non isotropic data distributions. In the following, we call the distribution-awer norm given the Eq. as Sigma norm.
4.2 Decomposition Computation
Although taking the input data distribution into account lead to better compression results, we cannot directly apply the highly efficient algorithms used for kernel approximation based on the Frobenius norm. In this paper, we introduce new efficient algorithms, equivalent to the widely used CP-ALS and Tucker-ALS methods for the Frobenius norm, that approximate the kernel using this metric.
4.2.1 Alternating Least Square Algorithm for CP with Distribution-Aware Norm
The CP decomposition presented in Eq. invollvs 4 terms. The main idea of the ALS algorithm is to minimize successively each of the terms and then to iterate this process. The minimization cach term is convex and can be done with a close formula using the pseudo-inverse. We details below how the minimization can be done with Sigma norm.
The reasonable assumptions that the function is in L for the distribution D. This is likely the case in our range of applications since p is neural network that most cases continuous and D can be considered as restricted to compact set.
https: /ocss httpo/r...r...ooeeett//////////e
nn. Unfoll....nnnnn.nooooooomoo.ld.. html

---

Firstly, we note the properties of vectorization of matrices which we will use to adopt the Sigma norm for ALS algorithm. Let A Rm and B E Rn*k be matrices. Then, we can state the followings, for all i E [ E [I.
Vec(A)(mli - + A[i.j,, and Vec(AB) (B' @ 3d(m)) Vec(.A) - (ld(k) @ A) Vec(B).
(5)
(6)
Given that the first unfolding of CP decomposition is Ee1) - U (US o UH o ) where O denotes the Khatri-Rao product, we have:
Vec( 11))) = (I @ Id(T)) (e)(
(7)
(5 @ Id(()) Vec(U (U8 @ U o )"))
(8)
(E @ Id(T)) (1U1S) o U o U()) s Id(T)) Vec(U)())
Thus, we can iterate over each factor matrices to minimize the error between the kernel tensor I. and its approximation under Sigma norm. In other words, for the factor matrix UT) we have the following minimization problem:
min Vec(KCE P(T) (
(10)
where P(T) = ((E1/3)T Id(T)) ((U(S) O U o ((W) & Id(T)), and similarly for the other matrices corresponding to the components S, H, W, which are detailed in the appendix B.2
Since the least square minimization problems have a closed form solution, we can deduce the factor matrix (TT) (and similarly the other factors) with the following formula:
Vec(U(T) + (P(T))
(11)
where At denotes the Phoore-Penrose inverse of the matrix A. As a result, we present the full algorithm called CP-ALS- in algorithm and we refer the appendix algorithm
B.4nddixB.4ffor for the practical use of
Algorithm 1 CP-ALS Sigma
1: function ( (1$), (WW), (HH), m] CP-ALS-SIGMA
2: Give initializations for matrices ( (S),, U1H), U(W)
3: for n mnnn=,,... -l...,, m do
4: Vec(U((T) + (p(TiJ 1
update ( with least square solution
5:
Vec(U((3)) (p(s))t Vec(ACI
update (S) with least square solution
6:
Vec(U((1)) (p(U)) Vec(AC2
update U (27) with least square solution
7:
Vec(U(())) + (p 1ec(XC2
b update (W) with least square solution
8:
return factor matrices UCT) U18) w ((()
4.2.2 Alternating Least Square Algorithm for Tucker2 with Distribution-Aware Norm
The principle of the ALS algorithm for Tucker2 is very similar the CP case. We derive a quadratic minimization problem for each factor when both others are fixed. Those minimization problems can be solved in closed form using the matrix pseudo-invers inverse. In the end, this leads the following algorithm 2 Full details can be found in appendix B.3 and the practical use of the algorithm is provided in the appendix B.4
Experimental Validation
In our experiments, we investigate the convolutional neural network models Resnet18, 88, Resnet50, and Googl (pretrained on ImageNet 1K) to assess the performance of the proposed algorithms CP- ALS-Sigma (Algorithm and Tucker2-ALS. ALS-Sigma (Algorithm{). We do not compress the first

---

Algorithm 2 Tucker2-ALS-Sigma
1: function ( ( m] = TUCKER2-ALS-SINE
2: Give initializations matrices U .U(())
3: for =1,,,,.. 1,..., do
4: Vec(U(()) + (p(T)) Vec(KC2)/2)
update U with least square solution & update U1(5 with least square solution
5:
Vec(U((5) - (p(s)t Vec(XC21/3)
6: Vec(g) + (p())) /2
P update & with least square solution
7:
return core tensor and factor matrices U(T), Uls)
convolutional layer for each model, as doing so degrades performance. Additionally, we used the Tensorly package [Kossaffi al., (2199] to compute the standard CP and Tucker decompositions, referred to as CP-ALS and Tucker2-ALS, respectively. Some parts of the code were also adapted from the MUSCO library [Gusak et 2019 perform the compression on neural networks.
5.1 Evaluation of the CP-ALS and Tensor Deflation with Distribution-Aware Norm
An alternative to the full ALS method is the tensor deflation method. In details, tensor deflation method suggests to update iteratively ((++1) <- W Ory & 3h & & where or & 8u @ 8 8, the rank-one approximation of ((k) such that the low rank approximation is given by W Ek'n Og & B& 8 @ 6g. We compare this greedy approach with our ALS algorithm that optimize the ranks at the same time, in the case where we optimize the distribution-aww norm. In Table [ we
( (k-
show the reconstruction error for the distribution-awanr norm
1//
where K. is the original tensor and K is the decomposed onc) when decomposing kernels of ResNet 18. We observe clear superiority of the full ALS algorithm in reconstruction error (Table 1 and in accuracy after compression (Table
Table I: Relative reconstruction errors of Greedy Tensor Deflation(TD) and P--AL--Signm algo- rithms applied convolutional layers of ResNet18 where the ranks estimated through VBMF with the ratio a = 0.8.
Conv Layer
Rank
Recons. Err. (Greedy TD)
Recons. Err. (CP-AL.-Sigmaa)
Improvement (Greedy ALS)
Layerl 00.oo22
134
0.195
0.053
3.7
Layer2.0.convl
140
0.277
0.118
2.3
Layer3.0.com
291
0.217
0.094
2.3
Layer3. Lconvl
520
0.207
0.070
3.0
Loner
1023
0.159
0.063
2.5
1.yer4
1167
0.292
0.051
5.7
Table 2: Top-1 accuracies of the decomposed models obtained via Greedy Tensor Deflation (TD) and CP-ALS-Sigma algorithms with respect distribution-aware norm applied to ResNet18 ImageNet dataset with different compression rates.
VBMF Ratio
Compression Rate
Acc. (Greedy TD) Acc.((P-ASS-Sima)
0.8
2.07
35.2
67.9
0.85
2.53
20.9
66.5
0.9
3.25
5.1
63.1
5.2
Model Compression and Fine-Tuning with Limited Data Access
In this section, we consider the scenario where only a limited amount of data is available, common situation in many applications. Specifically, we consider the case where only 50,000 images from the ImageNet training set are available to estimate the matrix Σ and to fine-tune the model.
7

---

We compare the classification accuracy of the compressed model with Tucker2-ALS-Sine -Sggaa, Tucker2- ALS algorithms for Frobenius norm and its fine-tuned version. For the network compressed using the standard ALS algorithm under the Frobenius norm, we fine-tune it with the Adam optimizer, selecting the optimal learning rate from the range 10-5 to 10-10, We test the performance of our method at different compression rates by varying the parameter in the rank formula (2). For this, we selected the following values of a:: [0.4, 0.45, 0.5, 0.55] for Resnet18, [0.8, 0.9, 0.95, I for Resnet50, [0.6, 0.7, 0.8] for GoogLeNet.
As shown in Figure our results indicate that the Tucker2-ALS -igma algorithm consistently outper- forms both the standard Tucker2-ALa algorithm and the fine-tuned compressed models obtained with Tucker2-ALS across all the neural networks evaluated. For additional results on the CP-ALS-Siggh Sigiaa algorithm, we refer to Appendix C.T
ResNet18
ResNet50
GoogLeNet
70
70
% Top accuracy
70-
50
50
40
60
30
92
1.4
1.6
17
1.375
1.400
1425
1450
1.475
1.500
1.35
1:40
1,45
1.50
Compression Ratio
Compression Ratio
Compression Ratio
Original Model
Tucker2-ALS
Tucker2 ALS Sigma of Imagenet Subset
Fine Tuned Tucker2 ALS with Imagenet Subset
Tucker2-ALS-Sigma of Aircraft
Figure Accuracy comparison of decomposed models obtained with Tucker2-ALS- ALS-Sigma and Tucker2-ALS algorithms, also including fine-tuned decomposed model (with Tucker2-ALs algo- rithm) results where the fine-tuning done on the subset of ImageNet train dataset. (r.X denotes the compression ratio, calculated by dividing the number of parameters of the original model by that of the compressed model.)
5.3 Impact of Dataset Changes Proposed Algorithms
This section aims to assess how variations in the dataset affect the performance of the proposed CP- ALS-Sigma and Tucker2-ALS-Sigma algorithms. In particular, we demonstrate that the distribution- aware norm can be computed from a dataset different from the one used for pretraining, highlighting its transferability. To achieve this, we performed compression on ResNet18. ResNet50, and GoogLeNet models, all trained on the CIFAR-10 -- dataset, using our proposed ALS-Sigma algorithms (with the matrix obtained from different datasets) and the standard ALS algorithm, applied to both CP and Tucker decompositions. In addition, we fine-tuned the compressed models that are obtained using the standard ALS algorithms to compare their performance with the Sigma-based methods (without fine-tuning). Fine-tuning was conducted on the CIFAR- training dataset, employing the Adam optimizer and testing various learning rates from 10-3 10-7 77, selecting the berf-pero learning rate. Results for the CP-ALS- algorithm can be found in Appendix C.2
We compare Sarker2-AL-SSigneee with the standard Tucker2-ALSS algorithm and the fine-tuned compressed model obtained from Tucker2-ALSS. We chose a from (0.5, 0.6, 0.8, 0.9, 1] for Resnet18, [0.9, 1.1, 1.2, 1.3, 1.4] for Resnet50, [0.6, 0.8, 0.9, 0.95, for GoogLeNet. Our results, shown in Figure 2 indicate that Tucker2-ALS-Sipee ALS-Sigma algorithm has better performance than the standard Tucker2-A4 AnSS algorithm across all models, even when the Σ matrix computed using different datasets, such as subset of the ImageNet training set or the CIFAR- training set. Additionally, while Tucker2-ALS experiences rapid performance degradation as the compression rate increases, our method remains much more consistent. Furthermore, the Tucker2-ALS-Sinn -Sigma algorithm produces results that are close to those of the fine-tuned compressed model obtained using the Tucker2-Al A- algorithm. Notably, the performance of the Tucker2-Al A-L.S---A algorithm on the CIFAR- 100 dataset comparable that on the CIFAR-10 10 dataset, suggesting that our approach is not limited to using the original training dataset for compression with distribution- norm. In

---

fact, as illustrated in Figure the results for the Jucker2-ALS-Shie Sigmaa algorithm with the 110 dataset even outperform those with CIFAR-10 for the ResNet50 model. However, when the Sigma matrix is derived from a subset of the ImageNet training set, the performance of the Tucker2-AIs. Sigma algorithm is somewhat less effective compared to when it is computed on CIFAR-10 10 or CIFAR-100, likely due to differences in image resolution. We refer to the Appendix C for additional experiments which investigate the effects of image resolution, dataset diversity, and the number of samples chosen for the M matrix the performance of proposed algorithms.
Figure I demonstrates that the Micker2-ALS-Sigma algorithm, when the Sigma matrix is derived from the FGVC-Aircres training dataset, yields higher classification accuracy than the standard Tucker2-ALS method for all tested ImageNet models. Notably, in the case of ResNet18, the Tucker2- ALS-Sigma variant surpasses even the fine-tuned model obtained with the standard algorithm.
ResNet18
ResNet50
GoogLeNet
155
90
111.82
% Top 1 accuracy
72:25
1170
70
60
30
40
1:83
30
2.0
2.5
3.0
3.5
0.0
1.55
1.50
1,s5
1.70
1.75
1.00
1.55
1.40
1.50
1.60
1.70
1.90
Compression Ratio
Compression Ratio
Compression Ratio
Original Model
Tucker2-ALS 11:22--LS-Sgnng Sigima Cifar100 Tuckcr2-ALS Fine-Tured Tucker2-AL with Cifar10
Tucker2-AL5-Sigma of Cifar10
Tucker2-ALS A-s-r----Ls--mm of Imagenet Subset
Figure 2: Accuracy comparison of Tucker2-ALS, Saakke--AL--Sigma and Tucker2-ALS algorithms including the
fine-tuned model results after compression with Tucker2-ALS using Cifar10 10 dataset.
5.4 Evaluation Against Alternative Data- Info-Informed Baselines
To rigorously evaluate our approach, we designed and implemented two strong, data-aware baseline methods inspired by metrics from pruning and activation analysis: Fisher information [Tu et 2016) and activation sparsity (Rhu et af. 2018). Each baseline guides the Tucker2 decomposition by solving specific Weighted Alternating Least Squares (WALS) problem, enabling direct comparison against our proposed method.
Fisher- Weiih-e Low-Rank Approximation (FW-LRA) This baseline adapts Fisher information, a well-established criterion from pruning literature, to guide decomposition. The method solves the weighted objective min Hasher @ (K- - I13, where the weighting tensor Hisher is derived from the diagonal of the Fisher Information Matrix. The intuition is prioritize the preservation of weights that are most sensitive to the network's 's loss function.
Activation-Guided Low-Rank Approximation (AG-LRA) Inspired by sparsity-based compres- sion techniques, this baseline prioritizes filters based on their activation magnitude. It solves the objective min Hget @ (KC - I| where the weighting tensor Hac is calculated from the mean absolute activation of each output channel. This heuristic assumes that filters with higher average activation are more critical for the network's predictions.
Comparison with Baselines Tables 3 and 4 summarize the accuracy of our Tucker2-ALS-SA --ggma method compared to the baseline Tucker2-ALS and the two data-aware variants on GoogLeNet and ResNet18 8, respectively. Across all compression rates, our method consistently outperforms both FW-LRA and AG-LRA, as well as the vanilla baseline Tucker2-ALSS demonstrating the effectiveness of our distribution- norm.

---

Table 3: Accuracy comparison of different data-aware baseline methods and Tucker2-ALS-Sige --L--S
with Tucker2-ALS various compression rates on GoogLeNet.
Compr. Rate
Tucker2-ALS
AG-LRA
FW-LRA
Our Method
1.24
60.2
60.5
53.0
66.3
1.33
52.5
54.9
35.8
64.2
1.42
39.9
43.0
6.0
60.1
1.51
23.6
25.1
1.1
52.2
Table 4: Accuracy comparison of different data-aware baseline methods and Tucker2-ALS S-Simma with Tucker2-ALS various compression rates Resnet18.
Compr. Rate
Tucker2-ALS
AG-LRA
FW-LRA
Our Method
1.4
58.9
54.9
36.7
66.8
1.5
52.2
45.0
18.9
66.1
1.63
36.1
37.5
12.2
64.9
1.77
30.2
31.2
4.2
63.3
Limitations and Future Work
Complete functional norm A first limitation of our work is that when minimizing the reconstruc- tion error in the layer / we only take into account the first part of the network in the functional norm. Indeed, we want to minimize Ilf - f'll where f = la p and f' = l' p. To do so we minimize [g o - l/ PllL2 as proxy. This an important improvement compared to the standard proxy ||0 - 01 but this could be improved by taking the part q of the network into account in the optimization process.
To choose the rank of the decomposed tensors we use the Variational Bayesian Matrix Factorization (VBMF) algorithm. This algorithm is based on the assumption that the matrix we try factorize are perturbed by a Gaussian isotropic noise. Similarly to replacing the Frobenius norm by the functional norm, could use more general assumption the noise. Doing so would require to design new algorithm to compute the VBMF rank but could lead to better rank choice.
Anisotropic VBMF
Performance guarantee In data-free compression, we show that our algorithm improves perfor- mance for all targeted compression ratios. However, we are not able to give performance guarantee of the compressed network without testing it on a dataset. Hence to select the rank we rely on a priori heuristic like the VBMF rank. We believe that possible future work would be to the reconstruction error in the functional norm to select the rank. Indced, preliminary experiments shown in the appendix K demonstrate that the reconstruction error in the functional norm is way better indicator of the performance of the compressed network than the reconstruction error in the Frobenius norm.
7 Conclusion
In summary, we have shown that incorporating distribution-aware (functional) norms into tensor- based network compression leads substantial performance gains. By deriving ALS procedures that directly optimize CP and Tucker decompositions under the Sigma norm, we achieve markedly lower reconstruction error and higher accuracy than with traditional Frobenius-based methods, In addition, ALS--igma consistently surpasses greedy tensor deflation optimized with the same norm. The advantage of our approach becomes even more pronounced higher compression rates, where standard methods degrade sharply. Remarkably, the distribution-awe aware decompositions recover almost of the accuracy otherwise obtained by fine-tuning models compressed with conventional ALS, yet they require no additional training when the Σ matrix is estimated from the data distribution. Even when that distribution learned from a smaller, dataset such as FGVC-Aireraff persist. Finally, when the original training data are unavailable, the Σ matrix can be transferred from related dataset, still delivering significant of our distribution -ware aware compression framework.
benefits
teteeement---ihhigghiin the practicality and robustness
10

---

8 Acknowledgements
We thank Clément Laroudie and Charles Villard for helpful discussions that contributed to this work. This publication was made possible by the use of the FactoryIA supercomputer, financially supported by the Ile-de-France Regional Council. This work is also supported by the PEPR-IA ANR-23-PEIA-0010 and DeepGreen: ANR-23-DEGR-0001
References
Marcella Astrid and Seung-lk Lee. Cp-decomposition with tensor power method for convolutional neural networks compression. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). pages 115-118. IEEE, 2017.
Marcella Astrid, Seung-lk Lee, and Beom-Su Seo. Rank selection of CP-decomposed convolutional layers with variational Bayesian matrix factorization. In 2018 20th International Conference on Advanced Communication Technology (ICACT), pages 347-350, February 2018. doi: 10.23919/ SAACT22018.832375555 URL
ieee. htt///ocument/82233300
Yaohui Cai, Zhewei Yao. Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. ZeroQ: A Novel Zero Shot Quantization Framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.
Jian Cheng, Pei-song Wang. Gang Li, Qing-hao Hu, and Han-qing Lu. Recent advances in efficient computation of deep convolutional neural networks. Frontiers of Information Technology & Electronic Engineering, 19(1):64-77) January 2018. ISSN 2095-9230. doi: 10.1631/FITEE. 1700789.
Daria Cherniuk, Stanislav Abukhovich, Anh-Huy Phan, Ivan Oseledets, Andrzej Cichocki, and Julia Gusak. Quantization Aware Factorization for Deep Neural Network Compression. Journal of Artificial Intelligence Research, 81:973-988 988, December 2024. ISSN 1076-9757. doi: 10.1613/jair. 1.16167.
Remi Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems 27, pages 1269-1277. Curran Associates, Inc., 2014.
Shupeng Gui, Haotao Wang. Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: unified optimization framework. Advances Neural Information Processing Systems, 32, 2019.
Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschensky, Andrzej Cichocki, and Ivan Oseledets. Automated multi-stage compression of neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016.
Frank L. Hitchcock. The Expression of Tensor or a Polyadic as a Sum of Products. Journal of Mathematics and Physics, 6(1-4):164-189, 1927. ISSN 1467-9590. doi: 10. 10.1002//amm112776 164.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications, February 2016.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research (JMIR). 20(26), 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6))84-99) May 2017. ISSN 0001-0782. doi: 10. 10.33453066388

---

Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding- up convolutional neural networks using fine-tuned cp-decomposition. ICLR, 2015.
Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. In D. Touret- zky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-
Kaufmann, 1989. URL http:////pppceeedeaanneeaaaa.ppa nwwrine..ccccccc fttaaperfilleepppeerreeeeaeeeeee
pdf
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11976-11986, 2022.
Suhas Lohit and Michael Jones. Model Compression Using Optimal Transport. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2764-27627770 2773,,273,22 2022.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-Free Knowledge Distillation for Deep Neural Networks, November 2017.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-Free Quantization Through Weight Equalization and Bias Correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
Shinichi Nakajima, Masashi Sugiyama, and Ryota Tomioka. Global analytic solution for variational Bayesian matrix factorization. Advances in Neural Information Processing Systems, 23, 2010.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing Neural Networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Yann Ollivier. True Asymptotic Natural Gradient Optimization, December 2017.
Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W. Keckler. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks. In 2018 IÉEE International Symposium High Performance Computer Architecture 8.00017.
(HPCA), pages 78-91, February 2018. doi: 10.
Nilo Schwencke and Cyril Furtlchner. ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning. The Thirteenth International Conference on Learning Representations. October 2024.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition, April 2015.
Suraj Srinivas and R. Venkatesh Babu. Data-free Parameter Pruning for Deep Neural Networks. In Procedings of the British Machine Vision Conference 2015, pages 31.1-31.122 Swansea, 2015. British Machine Vision Association. ISBN 978-1-901725-53-7. doi: 10.5244/C.29.31. URL
http://wwwwwaa.oroooee. http:://ww bmva. http://wwww.....oo//////////apap
html
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper With Convolutions. In CVPR, pages 1-9, 2015.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, volume 33, pages 6377-6389. Curran Associates, Inc., 2020.
Jilliang Tang, Mingjin Liu, Ning Jiang, Huan Cai. Wenxin Yu, and Jinjia Zhou. Data-Free Network Pruning for Model Compression. In 202/ IEEE International Symposium Circuits and Systems (ISCAS), pages 1-5, May 2021. doi: 10. 109/1CCAS51556 9401
Ming Tu, Visar Berisha, Yu Cao, and Jae-Sun Seo. Reducing the Model Order of Deep Neural Networks Using Information Theory. In 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 93-98, July 2016. doi: 10.1109/SVVLS..21...11
12

---

Ledyard R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279-311, September 1966. ISSN 1860-0980. doi: 10.1007/BFO2228444
Manon Verbockhaven, Théo Rudkiewicz, Guillaume Charpiat, and Sylvain Chevallier. Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally, Transactions Machine Learning Research, July 2024.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, So Kweon, and Saining Xie. ConvNeXt V2: Co-Designing and Scaling ConwNets With Masked Autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133-16142, 2023.
Qibin Zhao, Masashi Sugiyama, Longhao Yuan, and Andrzej Cichocki. Learning Efficient Tensor Representations with Ring-structured Networks. In ICASSP 2019 2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8608 8612, May 2019. doi: 10. 00.11999CAASP2220988822222
13

---

The abstract and introduction clearly state the scope and the contributions NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: There is a limitation section stating what we consider as constraints for the use of the methods we introduce.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide proof' for our proposition in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We make clear how reproduce our algorithms.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide in the supplementary material the code for the algorithm 2 Tuccer22- ALS-Sigma well as the code for the computation of the sigma matrix. These are the two essential blocks that correspond to our contribution, as a complete compression module would be too complex to include. These blocks will be made publicly available on GitHub after paper acceptance: https:///githu: com/ hatpa/aaapaaaaaa..aa.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc cc/
gublic/guides lewggge//ooeeuubiiseeeeeeeeeeeee CodeeSumisssioooolooo....o
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
y //nips ct/ wwwppppppcccaaeeeoccceccceecettt heaiicc/giii////ee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we tried to put all important settings that help appreciate our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See the section for the experiments with multiple random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we put these details on the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: Yes, all the authors confirm that the research conforms to the Neurips code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Since our work focus on the development of theoretical and algorithmic methods the neural network compression with tensor decomposition, don't target specific application which can inherent the societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We don't release data models, propose method to compress generic convolutional neural network.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, in the references.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There is no such research in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There is no such research in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We didn't use LLMs for any important or original part of our paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurre /0tp:///eccccooeeeeee////22///// for what should or should be described.
20

---

A Proof on functional norm
Proposition 1. Consider a distribution D, partial neural network p. and two convolution Convi and Conv k parametrized by the kernel tensor K. E RTTxS=HxW and K. Under reasonable assumptions we can define == (..p(u(((=))((((()) that transforms the image p(z) into a matrix that can be used compute the convolution as matrix
where u the unfolding operator T
product. We can also define /1/2 the square root of E such that E1/2 (E)12) E. Then, we have:
Conve op- Conv & oplle
-x 1///
(12)
(1)
Proof. Convolution as a matrix product
We can write the convolution as matrix product. We first need to unfold the image p(z) of size S x Hy x W₂ into a matrix u(p(x)) of size (S x H x W, Hy x W2). In this matrix each line is patch of the image p(x) of size (S x H x W). We can also reshape the kernel KC of size (T, S, H, W) into a matrix (()) of size (T, H W). The convolution can then be written as:
Convx K. o p(x) K()yu(((a)) E (T, H, W.).
(13)
Technical compuations Let x an input data, we have:
Convic p(z) - Convæ p(xx)] ((n)u((())) - Ro)u(o(t))
(14)
(K-- - x) -K) ubb(=)) F
(15)
(K-- R)) (1) u(((x)), (K (1)
a(p(z))
(16)
Using the fact that (u, vw) = =uw'" .)) we have:
= -( u(p(=))u((<))) (k- K) k)))) (17)
=
(1)
Taking the expectation Now we take the expectation over the data distribution D:
Convy o p- Conv pll E (I Convx o p(z) Conv x o p(=))l
(18)
By linearity of the expectation and of the matrix multiplication and using 17 we have:
IC K) (1) Ez-D Ex==(((((()))(p(t))) (K-K)) k) (1)
R)
(19)
K) (1)
Σ, (K - x) )
(20)
(1))
(K- k) (1)
I1/2 (I4/2) T
(Ic -k- x)))))
(21)
Using the fact that (uwT,)) = {w, vw) we have:
k (K-R) K) (1)
1/2
X (x- -k)
1/1/2
(22)
(1)
(x-
kc
1/2
(23)
(1)
'The reasonable assumptions that the function is in L for the distribution D. This is likely the case in our range of applications since p is neural network that in most cases continuous and D can be considered as restricted to compact set.
https:: /ocs. httpe//c/w..eeeooe//a//////////e
nn. Unfold.
21

---

Hence, we have:
Convx op- Conv o op|1= = (x-x) 1:/2
(24)
(1)
B Details on ALS Algorithm with Distribution-Aware Norm
B.1
Computation of the Square Root of Σ
B.1.1
Definition and Existence
As we stated the Proposition 1 we have = Eg=D (u(p(=))(p(())) In this section, we show that the square root of the matrix E can be computed via its Cholesky decomposition. In this end, we need show that the matrix E Ex-D (u(p(=))u(((())
is symmetric positive definite matrix.
First of all, note that Σ is symmetric:
= EzND ((xp(((()))))) = E (ub=)(((t()))u(((((t) =.
(25)
(26)
Moreover, E is positive semi- mefineeinite matrix, since for any vector E R", we have
T Σz EB-D (xbo(((((())(((((()))
(27)
-EED (T(TTTTTppp u(p(=))u(((()) ELED (1=TT(p(<)))11
(28)
To guarantee that Σ admits a Cholesky decomposition. it must be positive definite. This requires that the columns of the matrix u(p(z)) linearly independent and span the space in which they lie.
Thus, the matrix Σ admits a Cholesky decomposition and only if it is positive definite, which holds when the covariance of up(()) is full rank. However, in practice, the columns of u(p(z)) can be linearly dependent, making E only positive semi-definite. To ensure numerical stability and enable the Cholesky decomposition, we add small regularization term eld, where identity matrix, to the diagonal Σ in cases where it only positive semi-definite. Alternatively, the square root of Σ can be computed using the singular value decomposition (SVD), which remains applicable even when Σ is not full rank.
and Td is the
B.1.2 Complexity Analysis
Computation of Σ Let recall the notations, for the where want to compute Σ at layer of network:
N the number of samples used estimate
p the function corresponding to the first part of the network computing the layer 1
C' the complexity of one forward pass of the full network
Hyl + 1], Wz[ + 1] the height and width of layer / + 1
S the number of channels of layer
H, W the height and width of the kernel of the connection from layer to layer
To compute E for each we compute:
p(z) which has a complexity bounded by C
For each of the Hy 1]W=[( + 1] patches Ps, of size H W in p(x) (each patch corresponds to the receptive field of a neuron of layer + 1) we compute (pij) Pad which has a complexity of (SHW)² per patch. Then we sum all those matrices to get u(p(z)) (u(p(z))) The complexity of this step is Hy 1]W, [1 1](SH))?
22

---

In total, the complexity of the computation of E is O(NC + NH 1| + ())))))) In comparison, the complexity of forwarding trought the connection from layer / layer + is Of, + 1jW,
1JSTHW) where 7 is the number of channels of layer + 1. Thus the computation of Σ is at most SHW times the complexity of classical forward (whith typical values T
T = and H. W small, the computation of comparable to the complexity few (9 for 3 x kernels and 49 for the biggest 7 x 7 kernels) forward passes).
Computation of the Square Root Σ Computing the square root of Σ can be done with Cholesky or SVD which require O(SHW))) operations in both cases. This cost is independant of N thus as long as N SHW the cost of computing Σ is higher than the cost of computing its square root.
Conclusion In general, the computation of 2 and its square root is comparable to at most a few (9 for 3 3 kernels and 49 for the biggest 7 x 7 kemels)) forward passes through the network times the number samples used to estimate E.
B.2 Details of the CP-ALS-Sigma Algorithm
Remember that for given rank R, CP decomposition of a kernel tensor K. E xxxxSxx is given by the following formula:
R u
K Σ U.m &
& l & U.W)
(29)
By iterating over the components of the CP decomposition as given in the Section 4.2. we obtain the following sequence of minimization problems:
Vec(KS)1/2) - ((D1/2) (U(() c U(") @ U(W) & T((T)) Vec(Ur(T))
min 0
(30)
Vec(KCI1/2) -(E112)) V Id(T) ( w( c u(") @ Id((s) Vec(U((S))
min
(73)
=:P(S)
(31)
min tr0
-(E')/) & Id()))
( o (8)) @ U(w) @ Id(H)) Vec(Ur(())
P(
F
(32)
V
- ((2 @ Id(T)) (wi o (S)) c (()) @ Id(W) Vec(Ur()))
min ((W)
=:P(W)
F
(33)
where we used the property (A 19 (B = (AB&C) for 30). In consequence, the associated algorithm called CP-ALS- Sigma described in the algorithm]
23

---

B.3 Details of the Tucker2-ALS-Sigma Algorithm
Remember that Tucker2 decomposition of K E R is given by core tensor g R xHxW and two factor matrices U E RR and USS) E RSSRRs such that
K X2 U()) X1 U
Similar to CP decomposition above, using the properties of tensor unfolding and vectorization of matrices, we end up with the following minimization problems:
min wm
Vec(KC
[111/2) X2 (2)))) (xxx @ T((T))
Vec(U((T))
(34)
=:P(T)
min (
Vec(K2I1/3
(57/2)T & IA(T)) [0 x, U @ Id(s) Vec(U
(35)
=:P(s)
and
(
(1 II((T)) ( s U(()) s Id(H) & Id(w) Vec(g)
min
(36)
P(9)
Thus, we associate the corresponding factor matrix (T) (and similarly for the factor matrix (S) and core tensor G) with
(e (P(T))t V2c(
(37)
Consequently, the full algorithm Tucker2-ALS-Signe is presented in Algorithm 2
B.4 Pseudo inverse computation for CP-ALS-Sigma and Tucker2-A1S-Sigma Algorithms
We refer the section 4.2.1 implementing Sigma the CP-ALS algorithm, obtain the closed form solution for the factor matrix (T) (and similarly for the other factors) with the following formula:
Vec(U((T) (P(T)) (ecT))))
(38)
Since (TT) is matrix of size TSHW x RT, we apply the identity At = A)t to avoid explicitly forming and storing large intermediate matrices of size TSHH x RT, TSHW RS, TSHW x RH, and TSHW x RW, which would otherwise be required for computing the factor matrices ( U and (W)), respectively.
Then, rewrite the above equation 38
Vec(U(T))= + (PITT)TP!T) (P(T)) Vec(KE)
(39)
such that (P(T)) Vec(K() is a vector size RT and (P(T)) (TT)) is a matrix of size RT x RT. However, in practice, we observed that the matrix (P(T)) P(T) can be ill-conditioned, making the computation of its pseudoinverse numerically unstable. To mitigate this issue, reformulate the problem as linear system and solve it using the MINRES (Minimum Residual) method, which is particularly effective for solving symmetric but large-scale systems. Specifically, we define
b (P(T)) Veo(KE1 A = T P(T)
and solve the system
AU = b,
xi indicates product along the ith axis of the tensor G

---

where the solution vector U corresponds to the desired factor matrix. This system is solved efficiently on the GPU using the minres function provided by the CuPy library.
Likewise to CP decomposition, solve the factor matrices U and U1$), and the core tensor & of the Tucker2 decomposition using the the minres function provided by the CuPy library.
C Experiments with CP-ALS-Sigma Algorithm
C.1 Model Compression and Fine-Tunie Fine-Tuning with Limited Data Access
consider the case when only a limited amount of data is available, common situation in many applications. Specifically, we consider the case where only 50,000 images from the ImageNet training set are available to estimate the matrix E and fine-tune the model.
Refering the Section
5.2
Here, we conducted the compression algorithms on ResNet18 and GoogLeNet, and we compare the classification accuracy of the compressed model with CP-ALS -igmmm,, CP-ALS algorithms for Frobenius norm and its fine-tuned version. For the network compressed using the standard AL.S algorithm under the Frobenius norm, we fine-tune it with the Adam optimizer, selecting the optimal learning rate from the range 10-5 to 10-10, In addition, we selected the following values of az: [0.8, 0.85, 0.9] for Resnet18, and [0.85, 0.9, 0.95] for GoogLeNet.
According the results on ResNet18, presented in Figure 3 the CP-ALS-Sigma algorithm con- sistently outperforms both the standard CP-ALS algorithm and the fine-tuned compressed models obtained with CP-ALS, In addition, our experiments on GoogLeNet (see Figure B) show that the CP-ALS-SEA L---migma algorithm achieves higher accuracy than the standard CP-ALS while yielding close results to the fine-tuned compressed model obtained via CP-ALS.
ResNet18
GoogLeNet
70
70
Top-1 accuracy
2453
60
50
45
50
1:1.33
3x25
2.2
2.4
2.6
2.8
3.0
3.2
1.550
1.575
1.60c
1.625
1.650
1.675 1.700
1.725
Compression Ratio
Compression Ratio
Original Model
CP-ALS
CP-ALS-Sigma of imagenet Subset
Fine-Tuned CP-ALS with Imagenet Subset
CP-ALS-Sigma of Aircraft
Figure 3: Accuracy comparison of decomposed models obtained with CP-ALS CP-ALS-Sigma and CP-ALS algorithms, also including fine-tuned decomposed model (with CP-ALS algorithm) results where fine-tuning done on the subset of ImageNet train dataset.
C.2 Impact Dataset Changes on Proposed Algorithms
Here, provide the detailed results for the CP-ALS- Sigma algorithm in the dataset transferability experiments described in Section 5.3 Specifically, we evaluate the compression performance of CP-ALS-Sigmi Sigma ResNet18, and GoogLeNet models trained CIFAR-10, including V matrices derived from datasets different than the original training set. Our goal is to assess how the choice of dataset for computing the distribution-awarn models.
affects the final accuracy of the compressed
We compare CP-ALS-SSI -SiggSSgaaaaa against the standard CP-ALS algorithm, and the fine-tuned compressed model obtained with CP-ALS. Fine-tuning was conducted on the CIFAR-10 10 training set using the Adam optimizer and range of learning rates from 10- to 10-7, selecting the best performing
25

---

rate model. In addition, we selected parameters a from [0.85, 0.9, 0.95, 1] for Resnet18, and 0.95, 1] for GoogLeNet.
The results shown in Figure inndicate that the CP-ALS-Sigma algorithm consistently outperforms the standard CP-ALS algorithm across all models, even when the Σ matrix is computed from different datasets, such as subset of the ImageNet training set or the CIFAR-100 training set. Additionally, as for Tucker2-ALS experiences, while CP-ALS experiences rapid performance degradation as the compression rate increases, our method remains much more consistent. Notably, the CP-ALSSS algorithm achieves similar performance on the CIFAR- 100 dataset as it does on CIFAR- 10, indicating that the use of a distribution-awann dataset. However, when the Sigma matrix derived from subset of the ImageNet training set, the performance of the CP-ALS- -Sgmaa algorithm is somewhat less effective compared to when it computed on CIFAR-10 or CIFAR-100, due to differences in image resolution as observed in Tucker2-ALS-Sii experiments, see the appendix G for further investigation of dataset features that affect algorithm performance. Moreover, as illustrated in Figure 4 using the AP--ALS-Sigma algorithm on ResNet18 results in only 0.8% accuracy drop compression rate of 3.53, and 1.7% drop at a compression rate of 5.22, relative to the original model accuracy 94.3% where Σ is computed using the CIFAR-10 train dataset.
norm for compression is not restricted to the original training
Furthermore, as illustrated in Figure 3 the compressed model obtained using the PP.A-SSiigm algorithm, with the Sigma matrix computed from the FGVC-Aireraft training dataset, achieves close classification accuracy to the fine-tuned compressed model obtained through the standard CP-ALS algorithm applied to the original ResNet18.
ResNet18
GoogLeNet
50
% Top-1 Eccuracy
80
80
70
70
60
40-
r9:97
2216
5
E
1.05
1.90
1.95
2.00
2.05
2.10
2.15
Compression Ratio
Compression Ratio
Original Model
CP-ALS-Sigma of Cifar100
CP-ALS-Sigma of Cifar10
CP-ALS
CP-ALS-Sigma of imagenet Subset
Fine-Tuned CP-ALS with Cifar10
Figure 4: Comparison of CP-ALS-Sigma and CP-ALS algorithms including with fine-tuned model after compression with CP-ALS using CIFAR-10 dataset.
D Additional Experiments on the Impact of Dataset Changes Using CIFAR-100 Trained Models
Building on our previous experiments with CIFAR-10 trained neural network architectures (see the Section 5.3)). we continue assess how variations the dataset influence compression outcomes and further demonstrate that the distribution-awarc norm retains its transfcrability- from dataset different from the one used for pretraining. Specifically, we compress ResNet18, ResNet50, and GoogLeNet models trained on CIFAR- 100 using our ALS-Sigma methods withh with the matrix derived from datasets different from the original train dataset- and compare the results against those of the standard ALS approach under both CP and Tucker decomposition. In addition, we fine-tuned the compressed models that are obtained using standard AL. algorithms to compare their performance with the Sigma-based methods (without fine-tuning). Fine-tuning was performed on the CIFAR-100 training dataset, employing the Adam optimizer and testing various learning rates from 10-3 to 10-7, selecting the perforrrno learning rate.
when computed
26

---

CP-ALS-Sigma Here, we present accuracy comparison of decomposed models obtained using CP-ALSS-igma,, and the standard CP-ALS algorithm, and the fine-tuned compressed model obtained with CP-ALS. We selected a from [0.85, 0.9, 0.95, both for Resnet18, and GoogLeNet. The results shown in Figure indicate that the CP-ALS ALS-Simm algorithm consistently outperforms the standard CP-ALS algorithm across all models, even when the D matrix is computed from different datasets, such as a subset of the ImageNet training set or the CIFAR-10 10 training set. Additionally, as for the Tucker2-ALS experiences, while CP-ALS experiences rapid performance degradation as the compression rate increases, our method remains much more consistent. Notably, we verified again that since the CP-ALS -Sgma algorithm achieves similar performance on the CIFAR- 10 dataset as norm for compression is not restricted to the Tucker2-ALS-Sigma: Firstly, we compare the results of Tucker2-ALS-Siee with those of the standard Tucker2-ALe ALS algorithm and fine-tuned compressed model obtained with Tucker2-ALS algorithm. We selected a from 00.4, 0.5, 0.6, 0.7. 0.8, 0.9, 1] for Resnet 88, [0.9. 1.1, 1.2, 1.3] for Resnet50, [0.6, 0.7, 0.8, 0.9, 0.95, 1] for Google The results, presented in Figure 5 confirm the earlier observations: the Tucker2-ALS-Siimm algorithm consistently outperforms the standard Tucker2-ALS method across all evaluated models. The observed performance difference remains evident even when the Σ matrix is derived from datasets other than the one used for training, such as a subset of ImageNet or the CIFAR-10 dataset.
Moreover, we verified again that while Tucker2-ALS causes to significant accuracy degradation as compression rate increases, the Tucker2-A -atre22-AL--SS Sigma algorithm maintains significantly more consistent performance. Importantly. Tucker2-ALe ALS-SSima fine-tuned Tucker2-ALSS models not involving any additional fine-tuning. These results reaffirm the transferability and effectiveness of our distribution-awar norm implementation under varying dataset conditions.
Sigma remains comparable in performance to
ResNet18
GoogLeNet
ResNet50
DC
1150
accuracy
70
60
60
50
50
90
1975
1977
3755
19B4
20
20
1.5
2.0
255
3.0
3.5
4.0
4.5
3.0
1.4
1.5
1.6
1.7
1.0
1.9
1:60
1:05
1.70
1.75
1.80
Comoression Ratio
Compression Ratio
Compression Ratio
Original Model
Tucker2-AL5-Sigma of Cifar10 ...... Tucker2-ALS Fine-Tuned Tucker2-ALS with Cifar100
Tucker2-ALS-S/ -LL-s-gaa--ss of Cifar100 Tucker2-ALS-Sigma of Imagenet Subset
Figure 5: Accuracy comparison of decomposed models obtained using the Tucker2-Al A------------imm S-Sigma and Tucker2-AI.S algorithms across ResNer18, Google and ResNet50 architectures. The results also include fine-tuned decomposed models obtained via Tucker2-ALS, where fine-tuning was performed on the CIFAR-100 training dataset.
it does on C00, the use of a distribution-aa original training dataset.
In addition, when the Sigma matrix is derived from a subset of the ImageNet training set, the performance of the ALS-Sigma algorithms are somewhat less effective compared when it is computed on CIFAR-10 or CIFAR-100, likely due to differences in image resolution as observed in experiments performed on the CIFAR-10 trained models (see the Sections 5.3 and C.2
E
Additional Experiments on AlexNet
In this section, we evaluate our proposed algorithms on the AlexNet architecture, extending the compression to the fully-connected layers using Singular Value Decomposition (SVD).
First, we introduce SVD-Sigma, a method that adapts SVD to minimize error with respect to the Sigma norm. The goal is to find a low-rank approximation W for given weight matrix W by solving
27

---

ResNet18
GoogLeNet
1.88
70-
Y1850
Top-1 accuracy
70
60
ri:95
-0-
40
09:44
4
6
9
16
1.7
1.8
1.9
2.0
21
Compression Ratio
Compression Ratio
Original Model
CP-AL5-Sigma of Cifar10
(P-ALL-Slgma of Cifar100
CP-ALS
CP-ALS Sigma of Imagenet
Fine-Tuned CP-ALS with Cifar100
Figure 6: Accuracy comparison of decomposed models obtained using the CP-ALS- and CP- ALS algorithms across ResNet1 8 and GoogleNe architectures. The results also include fine-tuned decomposed models obtained via CP-ALS, where fine-tuning performed the CIFAR-100 training dataset.
the following minimization problem:
min ||W- W/
(40)
W
We then compare the performance of two hybrid compression schemes in Table
1. Baseline Hybrid:
The convolutional layers are compressed with the standard Tucker2-ALS algorithm, while the linear layers are compressed with conventional SVD.
2. Proposed Hybrid:
The convolutional layers are compressed with our Tucker2-ALS-Sipe algorithm, while the linear layers are compressed with the proposed SVD-Sigma method.
Table 5: Top-1 accuracies of the compressed AlexNet architecture using Tucker2-ALS followed by SVD, and Tucker2-ALS-Sigme. followed by SVD-Sigma, where the Sigma matrix computed using 5,000 images from the ImageNet training dataset. AlexNet original Top-1 accuracy is % 56.
VBMF
Compression
Tucker2-ALS-Sigma + SVD-Sigma
Tucker2-ALS + SVD
Ratio
Rate
0.6
1.46
49.8
42.4
0.65
1.63
48.1
40.1
0.7
1.85
46.1
34.4
0.8
2.54
40.9
25.8
Stability of the Tensor Decomposition Methods
We examine the stability of our proposed CP-ALS-Sigma -iiga algorithm relative to the standard CP-ALS baseline. Experiments are performed on GoogLeNet trained on ImageNet, CIFAR-1O, and CIFAR- 100, each evaluated across multiple random seeds to measure the sensitivity of the decompositions to initialization. As shown in Tables 7 and [ CP-ALS is more sensitive to random initialization compared to CP-ALS-Sigma, particularly under high compression rates.
For Tucker2-AL.S and Jucker2-ALS-Sigma. we adopt SVD-based initialization, which yields deter- ministic and thus fully reproducible results.
28

---

Table 6: Performance of CP-ALS and PF.ALS-Sigma on GoogLeNet (ImageNet) with the Σ matrix computed with a subset ImageNet training dataset. Top-I accuracy results are reported as mean + standard deviation, obtained using 5 different random seeds.
VBMF Ratio
Compr. Rate
Σ of 50k-ImageNet
CP-ALS
0.85
1.54
65.85 + 0.45
66.4 + 0.07
0.9
1.63
59.5 + 0.58
63.25 + 0.05
0.95
1.73
37.43 +
52.45 0.12
Table 7: Performance of CP-ALS and CP-ALS-Sigma GoogLeNet (trained on CIFAR-10) with the matrix computed with CIFAR-10, CIFAR-IO6 100, and a subset ImageNet training datasets. Top-1 accuracy results reported as mean + standard deviation, obtained using different random seeds.
VBMF Ratio
Compr. Rate
Σ of
Σ of
Σ of
CP-ALS
5k-ImageNet
CIFAR-100
CIFAR-10
0.9
1.54
93.06 + 0.15
92.58 + 0.12
93.44 + 0.09
93.53
0.08
0.95
1.63
89.14 + 0.62
88.31 + 0.24
91.47 1: 0.06
91.88 = 0.04
1
1.73
37.71 +: 3.0
69.86 0.31
79.94 1: 0.24
82.22 1 0.5
Table 8: Performance of CP-ALS and CP-ALS- the Σ matrix computed with 100 CIFAR-100, and a subset of ImageNet training datasets. Top-1 accuracy results are reported as mean + standard deviation, obtained using different random seeds.
GoogLeNet (trained CIFAR-100) with
VBMF Ratio
Compr. Rate
Σ of
Σ of
Σ> of
CP-ALS
5k-ImageNet
CIFAR-10
CIFAR-
0.85
1.68
76.95 + 0.32
77.1 + 0.08
77.7 + 0.19
77.64
0.19
0.9
1.80
73.46 1 0.48
74.97 0.11
76.41 0.15
76.36 = 0.10
0.95
1.95
61.4 1.22
67.3 0.38
1 72.54
0.17
1
2.12
21.25 + 1.84
38.59 - 0.45
51.28 0.11
50.74 0.17
G Impact of Dataset Properties on Algorithm Performance
G.1 Image Resolution
This section investigates the hypothesis that differences in image resolution are primary factor in cross-dataset performance degradation. Our experiments confirm that algorithm performance is sensitive resolution matching and the quality of the image scaling process.
First, we found that matching the proxy dataset's resolution (ImageNet) to the target dataset's resolution (CIFAR-100 10) via downsampling leads to significant performance recovery. The results in Tables and show that using a matrix computed on downsampled ImageNet images consistently outperforms one computed on original-resolution images.
Second, our method is sensitive the choice of interpolation used during preprocessing. As shown in Table II] replacing the default bilinear interpolation (used in other experiments) with bicubic interpolation during the image resizing operation yields better results, underscoring the importance of preserving feature quality during upsampling.
G.2 Dataset Diversity
This section evaluates the impact of dataset diversity on the performance of our proposed algo- rithms. Our findings indicate that the method remarkably flexible regarding the specific semantic composition of the data used compute the E matrix.
First, we found that broad dataset diversity is beneficial but not essential. The results Table 12 confirm that a large-scale dataset like ImageNet yields the best performance. Nevertheless, performance remains strong even with smaller, more specialized datasets like FGVC-Aircratt or Oxford F00wers-102.
29

---

Table 9: Accuracy comparison of the Tucker2-ALS- algorithm using Sigma matrices computed from different datasets on a ResNet-18 model trained with CIFAR-10. 10. The Sigma matrices were derived from the full CIFAR-10 10 and CIFAR- 100 training sets, well as 50,000 ImageNet training images. For the ImageNet (Down) and Aircraft (Down) variants, images were first downsampled to CIFAR- 10 resolution (3 x 32 32) and then upsampled to ImageNet resolution (3 x 224 224)
before computing the Sigma matrix, whereas for the other variants, images were only upsampled to ImageNet resolution.
E of ImageNet (Orig.)
Σ of ImageNet (Down)
Σ of Aireraft (Orig.)
Σ of Aireraft (Down)
VBMF Ratio
Compr: Rate
Σ of CIFAR-10
Σ of
CIFAR R-100
0.5
1.6
93.9
93.9
93.7
94
93.1
93.6
0.6
1.89
93.7
93.7
93.4
93.7
92.1
93.3
0.7
2.25
93.5
93.5
92.7
93.2
89.8
92.4
0.8
2.73
93
92.8
91
92.5
86.9
91.1
0.9
3.37
91.9
91.7
88.6
91
81.7
88.6
1
4.22
90.2
89.5
83.5
87.9
72
82.7
Table 10: Accuracy comparison of the Tucker2-ALS-Sigee Sigma algorithm using Sigma matrices computed from different datasets on a GoogleNet model trained with CIFAR-10. The Sigma matrices were derived from the full CIFAR-10 10 and CIFAR- 100 training sets, as well as 5000 ImageNet training images.
Σ of
E of 5k-ImageNet (Down)
VBMF Ratio
Compr. Rate
M of
Σ of
5k-ImageNet (Orig.)
CIFAR-10
CIFAR-10W
0.6
1.39
94
93.8
92.5
93.4
0.7
1.51
93.4
93.1
90.7
92.3
0.8
1.64
92.6
92
88.1
90.4
0.9
1.76
90.4
89.3
81.7
87
0.95
1.83
88.5
87.1
76.9
83.3
1
1.89
86.3
84.2
70
78.6
Table 11: Accuracy comparison of the Tucker2-ALS -AiS- algorithm showing the impact upsam- pling quality when using low-resolution datasets for the computation of Sigma matrix ResNet18 trained with ImageNet. The Sigma matrices were derived from the full CIFAR-10 and CIFAR-100 training sets.
Σ of
of
Σ of
Σ of
VBMF
Compr. Rate
CIFAR-10
CIFAR-10 (Bicubic)
CIFAR-100 (Bilinear)
CIFAR-100
Ratio
(Bilinear)
(Bicubic)
0.4
1.4
55.4
56.3
55.6
56.7
0.45
1.5
52.3
53.8
52.4
53.9
0.5
1.63
47.2
50.5
48.3
50.8
0.55
1.77
41.7
46.5
42
47
30

---

Furthermore, we investigated whether performance is sensitive to the specific theme of the proxy dataset. As detailed in Table [3] computing Σ matrix from single broad category (e.g.. 'dog''. 'reptiles', or 'aircraft') or from mix of diverse categories has minor impact on the performance, highlighting the method's robustness to the semantic composition of the proxy data.
Table 12: Accuracy comparison of the Bucker2-ALS-Sieen algorithm using Sigma matrices computed from different datasets on the original ResNet18 architecture. The Sigma matrices were derived from the FGVC-Aircrat and Oxford Flowers- 102 training sets, and the 50.000 images of ImageNet.
VBMF
Compr. Rate
Σ of 50k-ImageNet
Σ of
E of
Ratio
Aireraft
Flowers-
0.4
1.4
66.3
64.2
64.6
0.45
1.5
65.2
62.4
63.2
0.5
1.63
63.9
59.9
61.6
0.55
1.77
61.9
56.5
59.1
Table 13: Accuracy comparison of the Tucker2-ALS Sigma algorithm using Sigma matrices computed depending on the categories of ImageNet dataset on the original ResNet18 architecture. We selected the categories Aircraft, Dog, and Reptiles from ImageNet to compute the Sigma matrices and called by ImageNet-Airerafi, ImageNet-Dog, and BaageNet-Reptilee For AmageNet-MixCateg, we have chosen images from 10 different categories of ImageNet.
VBMF
Compr.
Σ of ImageNet-Aireran
Σ of ImageNet-Doog 66.1
Σ of CraggNeetRRetiiie
of Image Netggeeee-MiCatt
Ratio
Rate
0.4
1.4
65.6
65.3
66.6
0.45
1.5
64.3
64.2
65.7
0.5
1.63
62.6
63.7
62.3
64.4
0.55
1.77
60
61.7
60.2
62.6
G.3
Sample Size
We next investigate the sensitivity of our method to the number of samples used to compute the matrix. The results, presented in Tables 14 and 13 indicate that performance is remarkably stable across different sample sizes. This robustness a key practical advantage, as it shows our algorithm does not require excessively large dataset for effective covariance estimation.
Table 14: Performance of Tucker2-ALS-Sigge on ResNet- 18 with the Σ matrix computed using varying sample sizes. Top-1 accuracy results reported as mean + standard deviation, with each sample subset chosen randomly over different seeds.
VBMF
Compr. Rate
Σ of
Σ of 10k-ImageNet
of 20k-ImageNet
Σ of
Ratio
5k-ImageNet
50k-ImageNet
0.4
1.4
66.81 + 0.02
66.80
0.03
66.80 + 0.02
66.80 0.02
0.45
1.5
66.09 + 0.03
66.11
0.06
66.10 + 0.02
66.08
0.02
0.5
1.63
64.88 + 0.03
64.90 + 0.05
64.91 + 0.02
64.89 + 0.01
0.55
1.77
63.25 + 0.05
63.29 + 0.07
63.31 + 0.03
63.30 + 0.01
Table 15: Performance of Tuckkr2-ALS-Sigmm on GoogLeNet with the V matrix computed using varying sample sizes. Top-1 accuracy results are reported as mean + standard deviation, with each sample subset chosen randomly over different seeds.
VBMF
Compr. Rate
of 5k-ImageNet
of 20k-ImageNet 10k-ImageNet
S
Ratio
50k-ImageNet 64.17 0.03
0.6
1.33
64.17 + 0.04
64.13 0.05
64.16 + 0.07
0.7
1.42
60.04 + 0.03
60.11 0.06
60.13 + 0.04
60.11 0.04
0.8
1.51
52.21 + 0.09 0.09
52.18 + 0.08
52.20 + 0.08
0.08
52.17 +: 0.05
31

---

H
Synergy of Tensor Decomposition and Quantization
Tensor decomposition and quantization are two principal and complementary model compression techniques. While our factorization approach reduces model size by exploiting structural redundancy, quantization enhances efficiency by reducing numerical precision (e.g., to INT8) of model weights. Since these methods target different aspects of compression, combining them is expected provide additive benefits, as shown in previous work (Gui et al. 2019].
To validate this idea further, we applied post-training FP16 and INT8 quantization to our compressed GoogLe Leet (trained on CIFAR-10) models. The results in Table T6 confirm this strong synergy, particularly for our Jucker2-AL.-Sign quantization to the factorized model results negligible change in accuracy. Even with more aggressive INT8 quantization, the performance remains remarkably high, demonstrating that our method is robust to a subsequent reduction precision.
algorithm. Across all compression ratios, applying FP16
Table Top-1 accuracies of (raaaee on CIFAR-1O) compressed with Tucker2-ALSS Sigma and Tucker2-ALS. The table compares the performance without quantization (FP32) against post-training quantization (FP16 and INT8) at various compression rates.
Tucker2-ALS-Sigge
Tucker2-ALS
Compr: Rate
WO Quant.
W Quant.
WO Quant.
W Quant.
FP32
FP16
INT8
FP32
FP16
INT8
1.39
93.97
93.94
93.65
92.81
92.82
92.47
1.51
93.43
93.39
93.19
91.16
91.18
91.24
1.64
92.69
92.68
92.29
87.66
87.68
88.38
1.76
90.65
90.68
90.02
76.43
76.34
77.03
1.83
88.77
88.79
88.19
43.57
43.56
44.31
1.89
86.73
86.73
85.91
33.73
33.71
34.33
While Tucker decomposition is generally robust to post-factorization quantization, the high sensitivity of Canonical Polyadic (CP) decomposition can lead to significant accuracy degradation. To address this challenge, Cherniuk et al. (2244) recently proposed quantization-awwal framework called ADMM-EPC. Their method uses novel CP-EPC initialization to produce low-rank factors that are inherently robust to quantization, enabling the successful combination of CP decomposition and numerical precision reduction.
I Compute Resources
To ensure reproducibility, we detail the computational resources used for all experiments:
Hardware: All CP-ALS and Tucker2-ALS experiments were executed exclusively on CPU, using a machine with x86 64 architecture and 1.5TB of RAM. In contrast, all CP-ALS-Sigma and Tucker2-ALS -as--------Siigm experiments were performed on systems equipped with either an NVIDIA A100 GPU (80GB) an NVIDIA H100 GPU (100GB), alongside the same CPU and memory configuration.
Software Environment: We used Py PyTorch 2.6.0 and CUDA 12.4. Experiments were run on Linux system with Python 3.10..4.
The runtime of Tucker2 and CP decompositions varied depending on model size and compression ratio. Fine-tuning phases were run for 30 epochs, taking approximately 1-6 hours per model.
Execution Time:
J Extended Background on Compression by Tensor Decomposition
We detail the implementation of CP and Tucker decompositions for convolutional layers, as introduced in Section We recall the convolution Convi parameterized by tensor K. of size (T, H, W).
32

---

which defines a mapping from the space of images X R to the space of images y E x Assuming stride 1 and no padding, the output of the convolution at given spatial location can be written as
5
hd
Wid Σ
[t, y, 20
Klt, s, h, Wss + h x + w]
(41)
==
by to ol
where 2h + H and 2wd + W, and the output dimensions are reduced accordingly: Hy = Hg - H + 1 and W W, W + 1. For a detailed introduction to the CP and Tucker decomposition methods, we refer to [Lebedev et al. 2013]. We reproduce some of the key equations from that paper below.
J.1 CP Decomposition for Convolutional Layer Compression
Recall that for rank R. CP decomposition of a kernel tensor K. E R x is given by the following formula:
R
K
(T) @ u @ u. @ u. x
(42)
such that it can be written as:
R
Kst, 8, h, w
U (1)
U [ [s))ss
(43)
As proposed by Lebedev et al. 2015). we replace the original kemnel tensor I. with its CP decomposi- tion iC, thereby expressing the convolutional layer sequence of 4 successive layers obtain the followings:
Vlt v,
(1 [w (M) hy + w]
(44)
ha
Σ
(t)
Σ
(W)) [w]]
((H)
Σ U(S) xs, a h +
Wid
h=-ha
conv
depthwise conv
depthwise
convolation
(45)
J.2 Tucker Decomposition for Convolutional Layer Compression
Remember that given the ranks Ry and Rs corresponding to the mode-1 and mode-2 unfoldings (i.c., the first and second axes) of the kernel tensor K. IR. 7XXXXXX the Tucker2 decomposition of K. is
ビ= g x (T) (S))
(46)
Accordingly, the elementwise representation of the decomposed kernel at given spatial location is given by:
Rs
Eltt h, w|
grr h (T)) (18) (r..s.
(47)
33

---

Similarly, as suggested by Kim et al. [2016]. this leads to replace the convolution by series of the three layers: a 1 x convolution parameterized by U full convolution parameterized by 9 and a a x 1 convolution parameterized by U Specifically, this can be expressed as:
Jl, yj
Grr re h w (r. (r, (r. s)
h, w]
hat
Rl Σ [l]
hd
Wid
Rs
Σ
9 G Tay w
U (S) (s, 11 h
1x1 conv
HxW
1x
K Correlation Between the Reconstruction Error and Accuracy
ResNettB-LLayer...comm// Accuracy vs. Reconstruction Error
1.0
90
0.8
Reconstruction Error
Accuracy (%)
70
0.6
60
0.4
40
30
0.2
20
10
0.0
50
100
150
200
250
Rank
Original Top-1
Top-1
Recons. error Fro. norm
Original Top-5
Top-5
Recons, error Sigma norm
Original Top-10 - Top-10
Figure 7: Relative reconstruction error and accuracy with respect to rank of the CP-decomposition of the second convolution of R.sNet18.
In this section we explore the link between reconstruction error and accuracy. To do this we took a neural network trained on ImageNet and we decomposed one layer with the CP decomposition different ranks. For each compression rank computed the relative reconstruction error in Frobenius norm and distribution decomposed layer. The results are shown in Figure 7 We see that the accuracy of the model is not changing linearly with the rank, instead it first increases rapidly and then saturates. The Frobenius reconstruction error more smooth and does not show a clear drop reconstruction error when the accuracy increases. The distribution-aware reconstruction error is bit chaotic for low-ranks. This due to the fact that the optimisation not done for this norm but for the Frobenius norm, hence we don't t have the guarantee that the distribution- reconstruction error will decrease when the
norm. We also computed the accuracy of the model with the
34

---

rank increases. We observe that the distribution-awant reconstruction error is inversely correlated with the accuracy of the model. Indeed, for the low ranks, the distribution-wware reconstruction error rapidly decreases until the accuracy saturates. After that, the distribution-awar reconstruction error still decreasing slowly while the accuracy is not changing. This shows that the distribution-awen reconstruction error is good metric to measure the quality of the decomposition and that we could use to choose the rank the decomposition without having to compute the accuracy of the model.
The behavior observed in Figure 7 is representative of what we consistently observed for the first layers across all the networks tested. In addition, for the last layers we still have similar structure for the accuracy and the reconstruction error in sigma-nom norm but the Frobenius norm tends to be way more aligned with the sigma-norm.
35


Reviews:
[{"review_id": "99FA4lahzY", "reviewer": "Reviewer_ZcfM", "summary": "This paper introduces distribution-aware tensor decomposition for neural network compression, replacing traditional weight-space optimization with function-space optimization using a data-informed norm that incorporates input covariance statistics. The proposed CP-ALS-Sigma and Tucker2-ALS-Sigma algorithms achieve competitive accuracy without fine-tuning and demonstrate transferability across datasets, representing a meaningful theoretical advancement in compression techniques that addresses practical deployment scenarios where original training data is unavailable.", "questions": "### Regarding Limited Experimental Scope:\n1. Have you tested your method on modern architectures like MobileNet or EfficientNet that are commonly used in resource-constrained environments?\n\n### Regarding Insufficient Baseline Comparisons:\n2. How does your method compare against popular quantization techniques like INT8 or mixed-precision compression?\n\n### Regarding Missing Statistical Rigor:\n3. Could you provide error bars or confidence intervals by running experiments multiple times with different random seeds?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper introduces a principled shift from weight-space to function-space optimization by incorporating data distribution into tensor decomposition norms.\n\n2. The proposed CP-ALS-Sigma and Tucker2-ALS-Sigma algorithms provide concrete implementations that directly optimize the distribution-aware norm with closed-form solutions.\n\n3. The demonstrated ability to transfer covariance statistics across different datasets addresses real-world scenarios where original training data is unavailable.\n\n## Weaknesses\n\n1. Testing only three CNN architectures (ResNet-18/50, GoogLeNet) on primarily image classification tasks fails to demonstrate broad applicability of the method.\n\n2. The paper lacks comparison with modern compression techniques like quantization, knowledge distillation, and other state-of-the-art methods beyond traditional tensor decomposition.\n\n3. The absence of error bars, confidence intervals, or multiple-run statistics undermines the reliability and significance of the reported results.", "was_revised": false, "final_justification": "", "rebuttal": "1. Have you tested your method on modern architectures like MobileNet or EfficientNet that are commonly used in resource-constrained environments?\n\nWe appreciate the reviewer’s suggestion regarding MobileNet and EfficientNet. These architectures are indeed optimized for efficiency at design time, using techniques such as depthwise separable convolutions and compound scaling. As a result, they are inherently low in redundancy, offering limited headroom for further compression via low-rank methods. Our method is designed for overparameterized models, where structural redundancy remains exploitable and compression yields more substantial gains.\n\nThat said, we recognize the importance of evaluating generality. To this end, we have added results on AlexNet, which differs significantly from the ResNet and GoogLeNet families in both structure and parameter distribution. Notably, we applied our method to both convolutional and fully connected layers (which can be easily adapted from our original approach to convolutional layers), demonstrating its effectiveness across different types of layers. The results are summarized in Table and further support the applicability of our approach beyond the models initially reported.\n\nWe also emphasize that in practice, there are two complementary paths to efficiency: (1) designing compact models from scratch, and (2) compressing high-performing models post hoc. Our work addresses the latter, a crucial use case in real-world deployments where retraining or architectural redesign is impractical. Continued advancement in post hoc compression techniques, like ours, enables broader applicability across legacy models, diverse architectures, and varying deployment constraints.\n\n**Table:** Top-1 accuracies of the compressed AlexNet architecture using Tucker2-ALS followed by SVD, and Tucker2-ALS-Sigma followed by SVD-Sigma, where the Sigma matrix is computed using 5,000 images from the ImageNet training dataset. AlexNet original Top-1 accuracy is 56%.\n\n\n$$\n\\\\begin{array}{|c|c|c|c|}\n\\\\hline\n\\\\textbf{VBMF Ratio} & \\\\textbf{Compr. Rate} & \\\\textbf{Tucker2 Sigma + SVD Sigma} & \\\\textbf{Tucker2 + SVD} \\\\\\\\\n\\\\hline\n0.6 & 1.46 & 49.8 & 42.4 \\\\\\\\\n0.65 & 1.63 & 48.1 & 40.1 \\\\\\\\\n0.7 & 1.85 & 46.1 & 34.4 \\\\\\\\\n0.8 & 2.54 & 40.9 & 25.8 \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n\n\n2. How does your method compare against popular quantization techniques like INT8 or mixed-precision compression?\n\nQuantization and low-rank factorization target different dimensions of model compression and are complementary rather than competing methods. Our work focuses on exploiting the linear structure in weight tensors through low-rank factorization, which reduces model size and computation without imposing hardware constraints or altering the underlying data distribution. This makes low-rank factorization an ideal candidate to be combined with quantization to further enhance model efficiency.\n\nIn response to the reviewer's concern, we have added experiments where we quantized the compressed models obtained through low-rank factorization. As shown in Table, the performance of the compressed and quantized models remains preserved, demonstrating that the combination of these two methods provides a synergistic effect, enhancing efficiency without sacrificing accuracy.\n\nWhile quantization methods like INT8 reduce model size by focusing on numerical precision, low-rank factorization operates at a structural level, targeting redundancy in weight tensors. Thus, the two methods address orthogonal aspects of compression. Existing research and preliminary findings support the notion that combining these approaches can lead to even greater compression gains than applying either method alone [1].\n\nThough we have not yet explored integration with more advanced, distribution-aware quantization techniques, we believe this is a promising direction for future work. For now, we hope our findings provide a solid foundation for the complementary use of low-rank factorization and quantization, highlighting how these methods can work together to achieve broader deployment flexibility across different hardware and architectures.\n\n[1] Gui, S., Wang, H., Yang, H., Yu, C., Wang, Z., & Liu, J. (2019). Model compression with adversarial robustness: A unified optimization framework. Advances in neural information processing systems, 32.\n\n[2] Cherniuk, D., Abukhovich, S., Phan, A. H., Oseledets, I., Cichocki, A., & Gusak, J. (2024). Quantization aware factorization for deep neural network compression. Journal of Artificial Intelligence Research, 81, 973-988.\n\n**Table:** We present the accuracy results of the GoogleNet model compressed using Tucker2-ALS and Tucker2-ALS-Sigma algorithms, followed by post-compression quantization with FP16 and INT8 precisions including FP32 compressed models accuracies varying VBMF ratios.\n$$\n\\\\begin{array}{|c|c|c|c|c|c|c|}\n\\\\hline\n\\\\textbf{VBMF} & \\\\textbf{Tucker2 Sigma}&\\\\textbf{Tucker2 Sigma} &\\\\textbf{Tucker2 Sigma} & {\\\\textbf{Tucker2}}&{\\\\textbf{Tucker2}}& {\\\\textbf{Tucker2}}\\\\\\\\\n\\\\textbf{Ratio} & \\\\textbf{FP32} & \\\\textbf{FP16} & \\\\textbf{INT8} & \\\\textbf{FP32} & \\\\textbf{FP16} & \\\\textbf{INT8} \\\\\\\\\n\\\\hline\n0.6  & 93.97 & 93.94 & 93.65 & 92.81 & 92.82 & 92.47 \\\\\\\\\n0.7  & 93.43 & 93.39 & 93.19 & 91.16 & 91.18 & 91.24 \\\\\\\\\n0.8  & 92.69 & 92.68 & 92.29 & 87.66 & 87.68 & 88.38 \\\\\\\\\n0.9  & 90.65 & 90.68 & 90.02 & 76.43 & 76.34 & 77.03 \\\\\\\\\n0.95 & 88.77 & 88.79 & 88.19 & 43.57 & 43.56 & 44.31 \\\\\\\\\n1.0  & 86.73 & 86.73 & 85.91 & 33.73 & 33.71 & 34.33 \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n\n3. Could you provide error bars or confidence intervals by running experiments multiple times with different random seeds?\n\nOur Tucker2 experiments are deterministic, as we use a fixed initialization strategy (specifically, SVD of different matrices obtained from the weight tensor) and apply the generalized minimal residual method (GMRES), which is also deterministic. As a result, these experiments do not involve any random variability, and error bars or confidence intervals are not applicable here.\n\nHowever, for the CP method in the supplementary material, we used a random initialization, and the results do vary slightly across runs. To address this, we conducted 10 experiments with different random seeds using the GoogLeNet model trained on the CIFAR-10 dataset, applying the CP-ALS algorithm with varying VBMF ratios. For a VBMF ratio of 0.9, the mean Top-1 accuracy across 10 runs with different random seeds was 93.062\\%, with a standard deviation of 0.145\\% and a 95\\% confidence interval of (92.958\\%, 93.166\\%). With a VBMF ratio of 0.95, the mean accuracy was 89.136\\%, standard deviation 0.615\\%, and confidence interval (88.696\\%, 89.576\\%). At a VBMF ratio of 1.0, the mean accuracy was 37.705\\%, standard deviation 3.001\\%, and confidence interval (35.558\\%, 39.852\\%).\n\nFor the most relevant compression ratios, we observe that the standard deviation is minimal, indicating consistent performance across runs.\n\nMoreover, for the CP-ALS-Sigma method, we conducted 6  experiments with different random seeds using the GoogLeNet model trained on the CIFAR-10 dataset, applying the CP-ALS-Sigma algorithm where Sigma matrix is computed with CIFAR-10 train dataset with varying VBMF ratios. For a VBMF ratio of 0.9, the mean Top-1 accuracy with different random seeds was 93.533\\%, with a standard deviation of 0.082\\% and a 95\\% confidence interval of (93.448\\%, 93.619\\%). With a VBMF ratio of 0.95, the mean accuracy was 91.883\\%, standard deviation 0.041\\%, and confidence interval (91.840\\%, 91.926\\%). At a VBMF ratio of 1.0, the mean accuracy was 82.217\\%, standard deviation 0.496\\%, and confidence interval (81.697\\%, 82.737\\%)."}, {"review_id": "utfItWX5Et", "reviewer": "Reviewer_qrJc", "summary": "The paper introduced a novel tensor decomposition compression method for CNNs. The key contribution is to replace the Frobenius norm with the so-called Sigma norm, which is the root mean squared residual over the dataset of the compressed and uncompressed network. This puts emphasis on network weights that affect the output more. The author also showed that, for convolutional layers, this norm is also equivalent to the Frobenius norm rescaled with the data covariance. The author then derived the CP and Tucker-2 ALS under the Sigma norm. The author then demonstrated empirically that using the data-aware Sigma norm greatly improves the reconstruction accuracy.", "questions": "- Can your method be generalized beyond convolutional networks?\n- How did you compute the covariance in equation 4, and what's the computational complexity?\n- Your methods are kind of similar to compressed sensing. Could CS theory provide more theoretical guarantees or better covariance estimation strategies?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The method is mathematically well-founded, and the motivation of making the tensor compression data-aware makes a lot of sense\n- The writing was very clear.\n- Empirical performance was good. At the same rank, the proposed method is better than the baseline in terms of accuracy.\n\nWeaknesses:\n- The computational complexity of the method was not discussed. For example, the computation of the covariance matrix could scale badly, which could prevent this method from being applied to large networks.\n- Some notations were not introduced, for example, the Khatri-Rao product\n- Using data distribution as functional norm is not new", "was_revised": true, "final_justification": "I will maintain my initial score for the paper. My initial concerns and questions are all well addressed, and the authors have conducted new experiments and obtained new findings as suggested by other reviewers.", "rebuttal": "1. Can your method be generalized beyond convolutional networks?\n\nThank you for raising this important point. Our method is designed to target convolutional layers, leveraging their unique structural properties. This method is not directly applicable to non-convolutional architectures like Transformers without some adaptation.\n\nThe core idea of low-rank factorization itself is not constrained to CNNs. In fact, we’ve successfully applied our method to fully connected layers (e.g., in AlexNet to address a similar concern from the reviewer ZcfM), which have similar structural properties to those found in Transformer models, such as dense layers. This required only minor adjustments to our approach, suggesting that our method can be adapted to other linear layers with minimal changes.\n\nHowever, applying our method to Transformer architectures presents additional challenges due to the attention mechanisms and the non-local dependencies within the model, which do not exhibit the same kind of spatial locality that convolutions do. These structural differences require careful consideration in adapting our low-rank factorization technique to handle attention layers and other components of Transformers.\n\nWhile we have not yet explored these adaptations experimentally, we see this as a promising next step. One of the motivations for focusing on CNNs in our current work was to gain rigorous insight into the behavior of low-rank decompositions in practical, performance-critical scenarios. These insights will be critical in addressing the specific challenges posed by Transformer models in future work.\n\nWe also note that CNNs remain widely used in real-world applications, especially in edge devices and memory-constrained environments, where this method already has high practical impact. Additionally, our approach is hardware-agnostic and can be stacked with other compression methods, including those currently applied to Transformer models.\n\nFinally, as Transformer models continue to grow in size and energy consumption, efficient compression remains a critical research priority—not just for performance, but also for sustainability. Our future work will build on this foundation, with a focus on bringing lightweight, interpretable compression to Transformer architectures in a similarly orthogonal and composable way.\n\nWe’ll make this trajectory clearer in the paper and explicitly mention the applicability of our method to linear layers in Transformers, as well as our plans to extend this work in that direction.\n\n2. How did you compute the covariance in equation 4, and what's the computational complexity?\n\nThe cost (total number of operations) is quadratic in the number of features $(S \\times H \\times W)$ of a layer:\nfor an intermediate representation of size $S \\times H_i \\times W_i$ and $N$ samples, the cost will be $N \\times H_i \\times W_i \\times (S \\times H \\times W)^2$. \n\nThis computational cost is efficiently parallelizable, which makes the process suitable for modern hardware, especially GPUs. The memory bandwidth required for these operations is  $(S \\times H \\times W)^2$, which is manageable given the memory architecture of contemporary systems.\n\nImportantly, this factorization step is performed **once per model** and **dataset**, meaning that it is a **one-time pre-processing step**. As a result, the cost of this operation is negligible compared to the long-term benefits of compression, which include reductions in inference time, computational cost, memory usage, and energy consumption during deployment\n\n3. Your methods are kind of similar to compressed sensing. Could CS theory provide more theoretical guarantees or better covariance estimation strategies?\n\nThank you for the insightful question. While our method shares similarities with compressed sensing (CS) in reducing redundancy, our approach focuses on exploiting low-rank structures in weight matrices, whereas CS primarily deals with sparse recovery of signals. The mathematical foundations are different, as CS relies on sparsity and can recover signals from fewer measurements, while we optimize the Frobenius norm in the output space to capture redundancies in neural networks.\n\nThat said, compressed sensing theory could potentially provide insights into covariance estimation if the covariance structure of layer outputs is sparse. In such cases, CS could help reduce dimensionality and accelerate covariance computation by focusing on the most important components, using techniques like random projections or sparse recovery. We plan to explore whether CS-inspired techniques could help improve our method, especially in the context of combining our approach with pruning to enhance model efficiency.\n\nWe plan to briefly mention compressed sensing in the related work section, specifically in relation to its use in sparse recovery and covariance estimation, while noting that it offers an interesting avenue for future research, particularly for sparsity-aware compression."}, {"review_id": "jldkjSy4iC", "reviewer": "Reviewer_GT7A", "summary": "This paper proposed a tensor decomposition method based on data-aware norms that does not require access to the original training data, aiming to improve the compression efficiency and practical applicability of convolutional neural networks in resource-constrained environments.", "questions": "1. The method proposed in the paper, which utilizes the distribution-aware norm based on the activation covariance matrix and its tensor decomposition, has achieved remarkable results, demonstrating the advantages of the data-driven approach. Are there any other forms of data-aware indicators to guide the compression process?\n\n2. The text states that when the Sigma matrix is derived from a subset of the ImageNet training set, the performance of the Tucker2-ALS-Sigma algorithm is lower compared to when it is calculated on CIFAR-10 or CIFAR-100. The authors speculate that this might be related to the difference in image resolution. Based on this, apart from image resolution, are there any other factors that might affect the performance across different datasets? For example, differences in the category distribution of the datasets, the number of samples, or the diversity of image content?\n\n3. Please read weakness 1\n\n4. Please read weakness 2", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength：\n1. Introduces a covariance matrix derived from data distribution as a weighting norm, thereby converting traditional error measurements in weight space into function space metrics that better reflect perceptual error.\n\n2. Achieves high accuracy without requiring fine-tuning on the original training dataset.\n\n3. Maintains favorable performance even when applied across different datasets.\n\nWeakness:\n1. As shown in Figure 1, the compressed model using this method outperforms even the fine-tuned model in terms of accuracy, despite not undergoing any fine-tuning itself. This result is somewhat counterintuitive, as fine-tuning is generally expected to improve model performance by better adapting it to the data distribution. The paper provides limited explanation for this phenomenon.\n\n2. The paper notes that when the covariance matrix Σ is computed from a subset of the ImageNet training set, the algorithm’s performance is slightly inferior to that obtained when Σ is calculated using the CIFAR-10 or CIFAR-100 datasets. The authors hypothesize that this discrepancy may be attributed to differences in image resolution. Nevertheless, this assertion is supported only by preliminary validation experiments and lacks comprehensive analysis and robust experimental evidence.", "was_revised": true, "final_justification": "For questions:\n\nWhile the authors acknowledge the existence of other data-aware indicators and provide theoretical justification for their approach, a quantitative experimental comparison is still necessary. Even though methods like gradient-based importance (Fisher Information) and activation sparsity may not be directly designed for tensor decomposition, they can potentially be adapted or serve as baseline comparisons to demonstrate the superiority of the proposed method.\n\nCould the authors please provide additional experimental results in a new comment that compare the proposed distribution-aware norm against these adapted baseline methods in terms of both accuracy and compression ratio?\n\nThe author provided the supplementary experimental results. These detailed experimental results and analyses have addressed my concerns.", "rebuttal": "1. \n\nThank you for your insightful question. There are indeed several data-aware indicators that have been explored in other compression settings, such as pruning and quantization. These indicators include gradient-based importance (e.g., Fisher Information) [1], activation sparsity [2], mutual information with outputs, and per-sample sensitivity [3]. While these techniques are valuable for guiding compression, they are typically focused on element-wise operations such as component removal or bit reduction, which are not directly applicable to tensor decomposition.\n\nOur method introduces a novel, functionally grounded indicator tailored specifically for low-rank factorization, shifting the focus from traditional reconstruction error to preserving task-relevant structure based on how the data actually flows through the model.\n\nThat said, we recognize the broader landscape of data-aware compression methods, and we will mention techniques like pruning and quantization in the related work section. While these methods are not directly applicable to tensor decomposition, they provide useful context for how data-driven indicators are used in other compression strategies. This will help to position our method within the broader research field, and we see potential for exploring how insights from these techniques could inform future combinations or extensions of our approach.\n\n[1] Tu, M., Berisha, V., Cao, Y., & Seo, J. S.. Reducing the model order of deep neural networks using information theory. In 2016 I (ISVLSI) (pp. 93-98). IEEE.\n\n[2] Rhu, M., O'Connor, M., Chatterjee, N., Pool, J., Kwon, Y., & Keckler, S. W. . Compressing DMA engine: Leveraging activation sparsity for training deep neural networks. In 2018  (HPCA) . IEEE.\n\n[3] Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I., Mahoney, M., & Gonzalez, J.. Actnn: Reducing training memory footprint via 2-bit activation compressed training. 2021 In ICML. PMLR.\n\n2.\nThank you for your insightful question. This is indeed a very relevant and practical inquiry, especially for users looking to apply our method in real-world settings where dataset choice for Sigma matrix computation can impact performance. While this aspect isn't the central focus of our work, we do recognize the value of providing practical guidelines to help users make informed decisions about which datasets to use.\n\nIn response to your question, we conducted several supplementary experiments to explore how factors like dataset size, category distribution, and image diversity affect performance. These experiments are intended to offer valuable insights into the real-world applicability of our approach, though they are not central to the core algorithmic development and theoretical contributions of the paper.\n\nTo investigate the effect of category distribution and image content diversity on our method, we experimented with subsets of ImageNet consisting of similar categories (e.g., Aircrafts, Dogs, or Reptiles) as well as mixes of images drawn from more diverse categories. We observed that using images from similar categories resulted in slightly reduced performance (at most -2%) compared to broader (mixed) category diversity.\n\nWe also computed the Sigma matrix using datasets with limited category diversity, such as Oxford Flowers 102 and FGVC-Aircraft. Despite their smaller diversity, the results were comparable to those obtained using ImageNet, demonstrating that category distribution and image content diversity have only a minor impact on performance. Importantly, our approach consistently outperforms baseline methods that do not take data distribution into account.\n\nFinally, we evaluated the impact of the number of samples (5,000, 10,000, 20,000) used to compute the Sigma matrices by testing with varying sample sizes from the ImageNet training dataset. The performance of the Sigma algorithms was not significantly affected by the number of samples chosen (differences of at most +-0.1%), supporting the robustness of our method across different dataset sizes.\n\nWe will include these additional results in the supplementary material, as we do not have enough space to present them in the answer.\n\n3.\n\nThank you for pointing this out. The result may indeed appear counter-intuitive at first glance, and we realize that the paper may not have been sufficiently explicit about the setup. To clarify: the fine-tuning was applied to the baseline (standard) compression method—not to our method. Despite this, our approach sometimes achieves better accuracy without any fine-tuning, which we agree warrants further explanation.\n\nThe underlying reason lies in how each method approaches compression. Standard low-rank factorization treats all parts of the tensor equally, aiming to preserve global structure, often at the expense of compressing away information that is critical for generalization. Fine-tuning can partially recover performance, but only within the local basin of the degraded solution, it cannot bring back lost structure.\n\nIn contrast, our method explicitly prioritizes the preservation of tensor components that are most impactful for the actual data distribution. By focusing the compression on the information that directly influences the model’s output on real inputs, it preserves more task-relevant capacity, even with the same tensor size. This likely explains why it can outperform the standard method, even post-fine-tuning.\n\nWe are not claiming that our method always outperforms fine-tuned baselines (for example this is not the case in Figure 2), but we do believe this result highlights the strength of data-aware compression strategies, and we will add a clearer discussion of this effect in the revised manuscript.\n\n4. \n\nThank you for your insightful comment. We agree that the initial hypothesis regarding the effect of image resolution on performance was based on preliminary experiments. To address your concerns, we conducted a more comprehensive set of experiments to explore this further.\n\nImpact of Resolution on Performance:\nTo investigate the potential role of resolution in the discrepancy between ImageNet and CIFAR-10 performance, for models trained on CIFAR-10, we conducted experiments where we reduced the resolution of both ImageNet and Aircraft images using bilinear downsampling. This simulates the effect of compression (e.g., JPEG) commonly seen with CIFAR-10 images. As shown in our supplementary results, we observed a significant improvement in performance on both ResNet18 and GoogLeNet, as we had intuitively expected. Upon analyzing the covariance matrices of the output layers, we found that the entropy of the eigenvalues was much closer to that observed when using CIFAR-10. Additionally, we computed the distance between the Sigma matrices for the different datasets and found them to be much more similar after downsampling. This suggests that the resolution difference does indeed impact the features extracted by the neural network, particularly in the first layers where low-level features such as edges are extracted.\n\nUpsampling Experiments:\nTo further investigate the effect of resolution on models trained on ImageNet and tested on CIFAR-10 or CIFAR-100, we experimented with different upsampling methods. Specifically, we compared bilinear interpolation with bicubic interpolation for rescaling the images to match the ImageNet resolution. We observed that the use of bicubic interpolation resulted in better performance than bilinear interpolation, reinforcing our hypothesis that features extracted in the first layers are particularly sensitive to resolution differences. Despite this improvement, however, the performance was still lower compared to using images of similar resolution. This strongly suggests that resolution similarity is a critical factor, as it directly influences the representation learned by the model in the initial layers.\n\nThese experiments, we believe, provide more robust evidence that resolution differences, particularly in the first layers of the network, have a significant impact on the performance of the proposed method. The results indicate that aligning resolution between the datasets is likely the most important factor when computing the covariance matrix, as it directly affects the features extracted by the network.\n\n**Table:** Accuracy comparison of the Tucker2-ALS-Sigma algorithm using Sigma matrices computed from different datasets on a ResNet-18 model trained with CIFAR-10. The Sigma matrices were derived from the full CIFAR-10 and CIFAR-100 training sets, as well as 50,000 ImageNet training images. For the ImageNet (Down) and  Aircraft (Down) variants, images were first downsampled to CIFAR-10 resolution ($3 \\times 32 \\times 32$) and then upsampled to ImageNet resolution ($3 \\times 224 \\times 224$) before computing the Sigma matrix, whereas for the other variants, images were only upsampled to ImageNet resolution.\n\n$$\n\\\\begin{array}{|c|c|c|c|c|c|c|c|}\n\\\\hline\n\\\\textbf{Ratio} & \\\\textbf{Compr Rate} & \\\\textbf{Sigma of Cifar 10 (Up)} & \\\\textbf{Sigma of Cifar 100 (Up)} & \\\\textbf{Sigma of ImageNet (Orig.)} & \\\\textbf{Sigma of ImageNet (Down)} & \\\\textbf{Sigma of Aircraft (Orig.)} & \\\\textbf{Sigma of Aircraft (Down)} \\\\\\\\\n\\\\hline\n0.5 & 1.6 & 93.9 & 93.9 & 93.7 & 94.0 & 93.1 & 93.6 \\\\\\\\\n0.6 & 1.89 & 93.7 & 93.7 & 93.4 & 93.7 & 92.1 & 93.3 \\\\\\\\\n0.7 & 2.25 & 93.5 & 93.5 & 92.7 & 93.2 & 89.8 & 92.4 \\\\\\\\\n0.8 & 2.73 & 93.0 & 92.8 & 91.0 & 92.5 & 86.9 & 91.1 \\\\\\\\\n0.9 & 3.37 & 91.9 & 91.7 & 88.6 & 91.0 & 81.7 & 88.6 \\\\\\\\\n1.0 & 4.22 & 90.2 & 89.5 & 83.5 & 87.9 & 72.0 & 82.7 \\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n\n5.\n\nThank you for pointing this out. We will correct the table numbering and ensure that titles appear properly before Tables 1 and 2 in the revised version.\n\n\nPlease note that due to space limitations, we were unable to include all the new results in this response, but we will be adding them to the paper. Feel free to ask us to put them here in the second phase."}, {"review_id": "QKuIMjVkkV", "reviewer": "Reviewer_2AV8", "summary": "This paper proposes a data-informed norm to measure the compression error in the function space, which leads to a lower accuracy drop compared to the traditional Frobenius norm. To optimize the new norm, the authors develop new alternating least square algorithms for Tucker-2 and CPD. Experiments show that this approach can outperform the Frobenius norm in terms of accuracy while eliminating the need of post-compression fine-tuning. Additionally, the method maintains its effectiveness even when optimized with unseen datasets, offering advantages for scenarios requiring data privacy protection.", "questions": "see weaknesses", "limitations": "The authors clearly describe the limitations.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The authors propose a new functional norm, which is straightforward to implement.\n3. The authors provide a clear and detailed derivation of the optimization process under the new norm, and the comparative analysis between the ALS-based optimization algorithm and the tensor deflation strategy is well-justified.\n\nWeaknesses:\n1. The insight behind the proposed norm should be clarified. Since the difference between the proposed norm and the Frobenius norm appears to be the multiplication by the covariance matrix, the underlying reason why this modification improves performance should be explained.\n2. The comparative experiments could be more comprehensive. In addition to comparing with Tucker2 and CP using the Frobenius norm in the weight space, it would be valuable to include comparisons with other data-driven compression methods. A basic additional comparison could be Tucker2 and CP with the Frobenius norm in the layer's output space.\n3. According to Equation 3, the functional norm measures the layer's output approximation error. How does this differ from using the Frobenius norm in the layer's output space, which is a common approach in this field?\n4. A minor point: On page 7, 'where p is the netork' contains a typo ('netork' should be 'network').", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We believe there may have been some misunderstanding regarding our core contribution, and we’d like to take this opportunity to clarify the key points of our work.\n\nOur central technical contribution is the formal connection between the functional approximation norm and the Frobenius norm in the layer's output space. While using the output-space Frobenius norm as a proxy for functional accuracy might seem intuitive, it is often used heuristically in the literature without justification or scalability. To our knowledge, no prior work has explicitly established the equivalence we prove, and this result is what allows us to adapt efficient, widely-used algorithms such as Tucker2-ALS and CP-ALS to a data-aware objective.\n\nThis is not a simple reinterpretation. Previous attempts to incorporate output-space norms into low-rank compression have either remained theoretical or faced significant scalability issues. The closest related work [1] we are aware of, which does explore a related goal, is restricted to small datasets (CIFAR10) and concludes that “...the results are encouraging but limited due to the computational bottleneck of solving large scale SDPs.” It does not address convolutional layers or provide a tractable implementation pathway for larger models. We will add this reference to our related work and discuss this in more details.\n\nIn contrast, our contribution enables the practical use of data-aware compression through low rang factorization in real architectures, without requiring new, unproven solvers. We see this as a significant step forward: by identifying a theoretical equivalence and exploiting it computationally, we go from a conceptually appealing but impractical direction to a usable, scalable method with measurable performance gains.\n\n1. About Clarifying the Insight Behind the Proposed Norm:\nThe key insight behind this modification is that the covariance matrix captures the dependencies and variations in the data. By incorporating this matrix into the norm, we effectively \"weight\" the approximation by the importance of different features, allowing the decomposition to focus on the most relevant aspects of the data. In other words, the covariance matrix provides a data-aware bias, ensuring that the model captures the structures that matter most for the task at hand.\n\n2. We don't have direct methods for comparison, as previous approaches were specialized to pruning and quantization, which are complementary to our method rather than competing with it. Additionally, some methods, like the one in [1], are not scalable and are limited to smaller datasets.\n\n3. Please see the explanation above.\n\n4. Thank you for pointing this out, we will correct it in the revised version."}]

Author rebuttal:

Meta-review: This paper introduces a distribution-aware tensor decomposition method for CNN compression, replacing the traditional Frobenius norm in weight space with a covariance-weighted functional norm. The authors derive new ALS algorithms (CP-ALS-Sigma and Tucker2-ALS-Sigma) to optimize this objective and show that the resulting compressed networks achieve competitive accuracy without fine-tuning, with robustness to dataset transfer when the original training data is unavailable.

The reviewers generally agreed that the work is mathematically well-founded, clear, and represents a principled shift from weight-space to function-space optimization. Strengths include the novelty of the proposed norm, tractable algorithms for its optimization, and strong empirical evidence that the method preserves accuracy better than classical low-rank decompositions. Importantly, the authors extended their experiments during rebuttal to address concerns: they added results on AlexNet to test generality, demonstrated synergy with quantization, conducted multiple-seed runs to provide confidence intervals, analyzed computational complexity, and thoroughly investigated dataset transferability (with careful analysis of resolution effects). They also conducted new comparative experiments against Fisher Information and activation-based baselines, showing their method’s superiority.

The main weaknesses are the limited scope of architectures (CNNs only, no modern lightweight models such as MobileNet/EfficientNet, nor transformers), and somewhat narrow baselines relative to the broader model compression literature (quantization, pruning, distillation). While the authors argue that their method is complementary rather than competitive with these approaches, this limits the breadth of impact. One reviewer remained unconvinced that the new comparisons fully addressed their concerns.

Overall, the paper makes a solid theoretical and algorithmic contribution to tensor decomposition–based compression, with carefully executed rebuttal experiments that significantly strengthened the submission. While its scope is somewhat narrow, the work is technically sound, clear, and provides useful insights for model compression in scenarios where retraining is infeasible.
Decision: Accept (poster)