Forget the Data and Fine-Tuning! Just Fold the Network to Compress

Authors: Dong Wang, Haris Šikić, Lothar Thiele, Olga Saukh
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Model compression, model folding, model merging

Abstract:
We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730319574543

thread_type:
official_review

<thread_object>
creator: reviewer_ajd2

<content>
summary:
This paper introduces a method to merge the columns in the weight matrix in a data-free manner. The method is composed of two parts: first, using the clustering method to find and merge the weight matrix. Then, based on the observation that the variance ratio after compressing the model would collapse or overshooting, the authors propose the data-free Fold-AR and Fold-DIR to inverse the model and repair the model. Experiments show that the proposed method can achieve better results than previous data-free methods, especially at high sparsity ratio.

strengths:
1. Rigorous theoretical analysis of the proposed method
2. A comprehensive framework for merging and repairing the model under the data-free setting, also without fine-tuning.
3. The writing of the paper is good

weaknesses:
1. There are plenty of works [1,2,3,4] about how to compress the model under the data-free setting, including the DeepInversion, which is mentioned as part of the method in this paper. These papers are all not mentioned and compared in the paper.
2. The experimental results show that the authors’ proposed methods experience significant performance drops compared to finetuning-based methods. Given that the cost of finetuning remains acceptable but the performance drop is not neglectable, it’s unclear why a tuning-free setting is necessary (since we can get the synthetic by methods like DeepInversion).
3. For methods on LLaMA, [5] also propose a model merging method to compress the model without fine-tuning. Please compare to this paper.

[1] Zero-shot knowledge transfer via adversarial belief matching.  
[2] Data-free learning of student networks.   
[3] Data-free adversarial distillation.  
[4] Data-free knowledge distillation via feature exchange and activation region constraint.   
[5] Shortgpt: Layers in large language models are more redundant than you expect.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551484682

thread_type:
official_review

<thread_object>
creator: reviewer_ao8l

<content>
summary:
This paper presents a data-free and fine-tuning-free method for compressing deep neural networks. This is achieved by identifying and merging redundant parameters through a similarity-based approach, followed by data generation with DeepInversion for further repair. The method is flexible and can be applied to both vision models like ResNet by merging channels and LLMs by merging linear weights, demonstrating effectiveness across different architectures. Extensive experiments on ImageNet, CIFAR-10, and CIFAR-100 highlight the method's robustness, with results clearly illustrated through figures.

strengths:
1. Previous methods typically rely on data-driven fine-tuning to restore the performance of compressed models. This paper addresses this limitation by introducing a data-free method that leverages generated data and an efficient activation repair process to recover model accuracy, making it highly practical for real-world applications.
2. The paper includes extensive experiments on both vision and language models, with clear visualizations and figures that offer detailed insights into the method's performance. Notably, at high sparsity levels (>40%), the proposed method achieves significantly better accuracy compared to prior baselines, such as IFM and INN.
3. The presentation is good, with a clear illustration of the entire pipeline in Figure 1. Additionally, the method is straightforward to implement and highly reproducible.

weaknesses:
1. My main concern with this submission is the limited technical novelty. The work largely combines existing methods, including similarity-based model merging for compression [1], model inversion for data generation [2], and REPAIR [3] for statistics alignment. While these components are integrated effectively, the key contributions could be further emphasized to distinguish this work.
2. Although the paper claims the method is data-free, this feature is primarily enabled by DeepInversion rather than a new development within this work. Clarifying the unique aspects of this approach compared to existing data-free methods would strengthen the contribution.
3. The folding results on LLaMA-7B are not particularly competitive, with a Wiki PPL of 13.33, compared to other pruning methods like LLM-Pruner (PPL=10.53) and Wanda (PPL=8.22). The claim that "model folding achieves comparable performance to data-driven methods" may be overstated due to this significant gap in PPL.  
4. SliceGPT [4], which similarly merges parameter matrices via a learnable transformation matrix, is another "merging"-based method and should be considered as a relevant baseline for comparison.  

Minor Typos: L173 - activation matching matching

[1] Chen, Yiting, Zhanpeng Zhou, and Junchi Yan. "Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory." arXiv preprint arXiv:2310.06756 (2023).  
[2] Yin, Hongxu, et al. "Dreaming to distill: Data-free knowledge transfer via deepinversion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.  
[3] Jordan, Keller, et al. "Repair: Renormalizing permuted activations for interpolation repair." arXiv preprint arXiv:2211.08403 (2022).  
[4] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620909853

thread_type:
official_review

<thread_object>
creator: reviewer_tqyn

<content>
summary:
The paper presents Model Folding, a data-free and fine-tuning-free model compression technique that merges structurally similar neurons within a neural network. This approach contrasts with traditional methods that often require access to training data or fine-tuning after compression.  Experiments on standard benchmarks, including ResNet and large models like LLaMA-7B, show that Model Folding achieves high sparsity while maintaining competitive performance with data-driven methods and outperforming recent data-free compression techniques.

strengths:
1. The method has both theoretical justification and empirical support, demonstrating that k-means clustering is an optimal method for weight fusion in a data-free manner. Results from benchmarks like ResNet18 and LLaMA-7B show that model folding achieves performance on par with or surpasses existing data-driven and data-free compression methods, particularly at high sparsity levels.

2. Model folding is designed to be completely data-free, which differentiates it from other compression methods that typically require fine-tuning with original data. The authors propose two data-free alternatives for adjusting internal data statistics—Fold-AR and Fold-DIR—based on approximate and deep inversion-based techniques. It presents significant improvements over existing state-of-the-art data-free methods, demonstrating its applicability in compressing large-scale models effectively

weaknesses:
1. More data-free and training-free references are needed, such as [1]

2. Lack of sufficient experimental results on compression ratio and speedup ratio.

[1] Haroush, Matan, et al. "The knowledge within: Methods for data-free model compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

questions:
1. Is there a speed comparison of inference on different devices after compression of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667811717

thread_type:
official_review

<thread_object>
creator: reviewer_mvsb

<content>
summary:
The paper proposes a new training- and fine-tuning-free method for neural network compression. Inspired by recent research on neuron alignment, in particular the weight matching algorithm [Ainsworth et al., 2023], the method essentially works by applying k-means vector quantization to cluster similar neurons (i.e., rows or channels of a weight matrix/tensor) together, and applying an additional data-free REPAIR (Jordan et al. (2022)) procedure to preserve the activation statistics of the original network. Experiments on convnets (e.g., ResNets, VGGs) demonstrate improved accuracy / sparsity tradeoff against other SOTA data-free pruning methods (e.g., FM (Chen et al., 2023), and INN (Solodskikh et al., 2023)). Experiment on an LLM (LLaMA-7B) shows comparable / somewhat worse performance than other pruning methods that require data/finetuning.

strengths:
I think conceptually connecting the research and recent methods in neuron alignment / neural network symmetry (Yamada et al., 2023; Ainsworth et al., 2023) to the problem of model compression is somewhat novel and deserves more attention, although it certainly has been done many times, see e.g., [Zhou et al., 2018](https://arxiv.org/abs/1804.05862) and the paper [Chen et al., 2023](https://arxiv.org/pdf/2310.06756) cited in the current work. Methodologically the contribution seems fairly incremental (i.e., weight quantization, extension of the existing REPAIR (Jordan et al. (2022)) method).

The writing quality and clarity is mostly good; the experiments/figures are informative. 

I rate the significance as moderate. In terms of practical impact, the method is relatively easy to understand and implement, and obtains competitive performance among other data- and tuning-free approaches. However, I was hoping to see more in terms of bridging the conceptual understanding of neural network symmetry and compression, e.g., whether the proposed *model folding* method (essentially VQ) compares/relates to the weight matching of (Ainsworth et al., 2023), and whether it also enables some degree of alignment or mode connectivity b/w two randomly pretrained neural networks.

weaknesses:
1. The paper is missing related literature and baselines on model quantization. A simple Google search suggests quite a few related papers, e.g., using vector quantization for model compression [[Martinez et al., 2021]](https://openaccess.thecvf.com/content/CVPR2021/papers/Martinez_Permute_Quantize_and_Fine-Tune_Efficient_Compression_of_Neural_Networks_CVPR_2021_paper.pdf) and post-training quantization [[Nagel et al., 2021](https://arxiv.org/pdf/2106.08295)] which is also training-free.

2. Some of the writing can be improved:
  - The section on Fold-AR is a bit hard to understand; it's unclear what exactly is being proposed. Perhaps an algorithm box can help.
  - The paper repeatedly talks about "merging similar channels" as the basis of the model folding algorithm, without defining it for various network architectures considered (especially outside the context of CNNs). This should be clarified, as how the scalar weights are grouped into vectors can have a big performance impact on VQ.
  - Some typographical errors: Line 278: "The variance ratio of the l-the", "l-the" -> "l-th"; unfinished sentence on Line 414: "Fold-DIR ourperforms Fold-AR as the cost of generating a batch of synthetic images and a forward pass through the network."

questions:
As mentioned earlier, I find the connection between weight matching (Ainsworth et al., 2023) and the proposed VQ method to be somewhat lacking, and the former seems more like an inspiration on which the method is loosely based. Do the authors suspect a similar effect of alignment to be achieved through quantization, such that distant modes can be connected with low loss barrier in between them (as in (Ainsworth et al., 2023))?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a data-free method for merging columns in the weight matrix, consisting of two key steps. First, a clustering-based approach is used to identify and merge similar columns in the weight matrix. Second, observing that compressing the model can lead to variance ratio collapse or overshooting, the authors introduce data-free Fold-AR and Fold-DIR techniques to invert and repair the model. Experimental results demonstrate that the proposed method outperforms previous data-free approaches, particularly at high sparsity ratios.

Most reviewers find the paper well-written, the evaluation is comprehensive with theoretical justifications.