Title: Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Compression of neural networks

tensor decomposition

low-rank approximation

distribution-aware norm
Abstract: Neural networks are widely used for image–related tasks but typically demand considerable computing power. Once a network has been trained, however, its memory‑ and compute‑footprint can be reduced by compression. In this work, we focus on compression through tensorization and low‑rank representations. Whereas classical approaches search for a low‑rank approximation by minimizing an isotropic norm such as the Frobenius norm in weight‑space, we use data‑informed norms that measure the error in function space. Concretely, we minimize the change in the layer’s output distribution, which can be expressed as $\lVert (W - \widetilde{W}) \Sigma^{1/2}\rVert_F$ where $\Sigma^{1/2}$ is the square root of the covariance matrix of the layer’s input and $W$, $\widetilde{W}$ are the original and compressed weights. We propose new alternating least square algorithms for the two most common tensor decompositions (Tucker‑2 and CPD) that directly optimize the new norm. Unlike conventional compression pipelines, which almost always require post‑compression fine‑tuning, our data‑informed approach often achieves competitive accuracy without any fine‑tuning. We further show that the same covariance‑based norm can be transferred from one dataset to another with only a minor accuracy drop, enabling compression even when the original training dataset is unavailable.
Experiments on several CNN architectures (ResNet‑18/50, and GoogLeNet) and datasets (ImageNet, FGVC‑Aircraft, Cifar10, and Cifar100) confirm the advantages of the proposed method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a data-informed norm to measure the compression error in the function space, which leads to a lower accuracy drop compared to the traditional Frobenius norm. To optimize the new norm, the authors develop new alternating least square algorithms for Tucker-2 and CPD. Experiments show that this approach can outperform the Frobenius norm in terms of accuracy while eliminating the need of post-compression fine-tuning. Additionally, the method maintains its effectiveness even when optimized with unseen datasets, offering advantages for scenarios requiring data privacy protection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and easy to follow.\n2. The authors propose a new functional norm, which is straightforward to implement.\n3. The authors provide a clear and detailed derivation of the optimization process under the new norm, and the comparative analysis between the ALS-based optimization algorithm and the tensor deflation strategy is well-justified.\n\nWeaknesses:\n1. The insight behind the proposed norm should be clarified. Since the difference between the proposed norm and the Frobenius norm appears to be the multiplication by the covariance matrix, the underlying reason why this modification improves performance should be explained.\n2. The comparative experiments could be more comprehensive. In addition to comparing with Tucker2 and CP using the Frobenius norm in the weight space, it would be valuable to include comparisons with other data-driven compression methods. A basic additional comparison could be Tucker2 and CP with the Frobenius norm in the layer's output space.\n3. According to Equation 3, the functional norm measures the layer's output approximation error. How does this differ from using the Frobenius norm in the layer's output space, which is a common approach in this field?\n4. A minor point: On page 7, 'where p is the netork' contains a typo ('netork' should be 'network').\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nThe authors clearly describe the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a tensor decomposition method based on data-aware norms that does not require access to the original training data, aiming to improve the compression efficiency and practical applicability of convolutional neural networks in resource-constrained environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n1. Introduces a covariance matrix derived from data distribution as a weighting norm, thereby converting traditional error measurements in weight space into function space metrics that better reflect perceptual error.\n\n2. Achieves high accuracy without requiring fine-tuning on the original training dataset.\n\n3. Maintains favorable performance even when applied across different datasets.\n\nWeakness:\n1. As shown in Figure 1, the compressed model using this method outperforms even the fine-tuned model in terms of accuracy, despite not undergoing any fine-tuning itself. This result is somewhat counterintuitive, as fine-tuning is generally expected to improve model performance by better adapting it to the data distribution. The paper provides limited explanation for this phenomenon.\n\n2. The paper notes that when the covariance matrix Σ is computed from a subset of the ImageNet training set, the algorithm’s performance is slightly inferior to that obtained when Σ is calculated using the CIFAR-10 or CIFAR-100 datasets. The authors hypothesize that this discrepancy may be attributed to differences in image resolution. Nevertheless, this assertion is supported only by preliminary validation experiments and lacks comprehensive analysis and robust experimental evidence.\n\nQUESTIONS:\n1. The method proposed in the paper, which utilizes the distribution-aware norm based on the activation covariance matrix and its tensor decomposition, has achieved remarkable results, demonstrating the advantages of the data-driven approach. Are there any other forms of data-aware indicators to guide the compression process?\n\n2. The text states that when the Sigma matrix is derived from a subset of the ImageNet training set, the performance of the Tucker2-ALS-Sigma algorithm is lower compared to when it is calculated on CIFAR-10 or CIFAR-100. The authors speculate that this might be related to the difference in image resolution. Based on this, apart from image resolution, are there any other factors that might affect the performance across different datasets? For example, differences in the category distribution of the datasets, the number of samples, or the diversity of image content?\n\n3. Please read weakness 1\n\n4. Please read weakness 2\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

comments: SUMMARY:
The paper introduced a novel tensor decomposition compression method for CNNs. The key contribution is to replace the Frobenius norm with the so-called Sigma norm, which is the root mean squared residual over the dataset of the compressed and uncompressed network. This puts emphasis on network weights that affect the output more. The author also showed that, for convolutional layers, this norm is also equivalent to the Frobenius norm rescaled with the data covariance. The author then derived the CP and Tucker-2 ALS under the Sigma norm. The author then demonstrated empirically that using the data-aware Sigma norm greatly improves the reconstruction accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:
- The method is mathematically well-founded, and the motivation of making the tensor compression data-aware makes a lot of sense
- The writing was very clear.
- Empirical performance was good. At the same rank, the proposed method is better than the baseline in terms of accuracy.

Weaknesses:
- The computational complexity of the method was not discussed. For example, the computation of the covariance matrix could scale badly, which could prevent this method from being applied to large networks.
- Some notations were not introduced, for example, the Khatri-Rao product
- Using data distribution as functional norm is not new

QUESTIONS:
- Can your method be generalized beyond convolutional networks?
- How did you compute the covariance in equation 4, and what's the computational complexity?
- Your methods are kind of similar to compressed sensing. Could CS theory provide more theoretical guarantees or better covariance estimation strategies?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces distribution-aware tensor decomposition for neural network compression, replacing traditional weight-space optimization with function-space optimization using a data-informed norm that incorporates input covariance statistics. The proposed CP-ALS-Sigma and Tucker2-ALS-Sigma algorithms achieve competitive accuracy without fine-tuning and demonstrate transferability across datasets, representing a meaningful theoretical advancement in compression techniques that addresses practical deployment scenarios where original training data is unavailable.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper introduces a principled shift from weight-space to function-space optimization by incorporating data distribution into tensor decomposition norms.\n\n2. The proposed CP-ALS-Sigma and Tucker2-ALS-Sigma algorithms provide concrete implementations that directly optimize the distribution-aware norm with closed-form solutions.\n\n3. The demonstrated ability to transfer covariance statistics across different datasets addresses real-world scenarios where original training data is unavailable.\n\n## Weaknesses\n\n1. Testing only three CNN architectures (ResNet-18/50, GoogLeNet) on primarily image classification tasks fails to demonstrate broad applicability of the method.\n\n2. The paper lacks comparison with modern compression techniques like quantization, knowledge distillation, and other state-of-the-art methods beyond traditional tensor decomposition.\n\n3. The absence of error bars, confidence intervals, or multiple-run statistics undermines the reliability and significance of the reported results.\n\nQUESTIONS:\n### Regarding Limited Experimental Scope:\n1. Have you tested your method on modern architectures like MobileNet or EfficientNet that are commonly used in resource-constrained environments?\n\n### Regarding Insufficient Baseline Comparisons:\n2. How does your method compare against popular quantization techniques like INT8 or mixed-precision compression?\n\n### Regarding Missing Statistical Rigor:\n3. Could you provide error bars or confidence intervals by running experiments multiple times with different random seeds?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}