Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Authors: Dong Wang
Haris Šikić
Lothar Thiele
Olga Saukh
Venue: ICLR 2025 Poster (2025)
Keywords: Model compression
model folding
model merging
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Model Folding: Compressing Deep Networks Without Data or Fine-Tuning

Abstract: We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.

Published as a conference paper at ICLR 2025
FORGET THE DATA AND FINE-TUNING! JUST FOLD THE NETWORK TO COMPRESS
Dong Wang** Haris Sikic Lothar Thiele', Olga S
o Graz University Technology. Austria
Complexity Science Hub Vienna, Austria
*ETH Zurich, Switzerland (dong. wange, haris. sikicēstudent., saukhe)tugraz..a thiele@tik. ee. ethz.ch
ABSTRACT
We introduce model folding, novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.
INTRODUCTION
Deep neural networks (DNNs) have emerged as a fundamental technology, driving progress across a multitude of applications from natural language processing computer vision. However, the deploy- ment of these models in real-world settings often constrained by the computational and memory resources available, partieularly
edge devices like smartphones and embedded systems (Wan et al., 2020 Kumar et al. 2017] Chen et al., 2020). This limitation poses a significant challenge, as the growing complexity and size of SOTA models demand increasingly substantial resources. (Bommasani et al.
2021 Chang et al. 2024 Rombach et al., 2022).
Conventional model compression techniques, such pruning (Han et al. 2015 LeCun al. 1989
Li al. 2016b Hassibi et al., 1993) and quantization (Gupta et al. 2015 Zhou et al. 2017 Li et al. 2016a), have been developed mitigate this issue by reducing the model size and computational requirements. These methods usually remove redundant or less critical parameters from the model, thereby reducing the overall size and computational load. For example, pruning eliminates weights that contribute minimally to the model's 's output (Han et al., 2015 Li et al.) 2016b). Quantization reduces the precision of the weights and activations (Gupta al. 2015), which decreases memory usage and speeds up inference (Zhoou et al.) (017). Despite their effectiveness, these approaches often introduce a degradation in model performance, necessitating a phase of fine-tuning to maintain the internal data statistics within the model (Jordan et al.) 2022 and restore the original accuracy levels (Frankle & Carbin 2018 Hassibi et al. 1993] Frantar Alistarh 2022). This requirement can be significant drawback in scenarios where access to the original training data is limited.
Recent methods have sought to circumvent the need for extensive retraining or finetuniigg by exploring alternatives to traditional approaches. Instead, several recent strategies build model merging techniques (Entezari et al. 2022 Ainsworth et al. 2023 Jordan et al. (022) and achieve (multi-)modl compression by fusing similar computational units. For example, Ziplc! (Stoica et al. 2024) merges two models of the same architecture by combining similar features both within and
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Veghtese
LELISTER
kym
1.0
Toptinal
target
IFM
0.8
Fold-Naive Fold-AR
Test Accuracy
0.8
Fold DIR
Speraity
Fold-R
0.4
0.2
JAPPUR
0.0
Variance Collapse
Variance Explesior
10
10a
201 Variance Ratio
10
10*
Figure I: Model compression and repair of data statistics. Left: Model folding pipeline is applied layer-wise, consisting of three phases: weight tensor clustering and merging, and data statistics repair. Right: To maintain accuracy, the data variances of compressed and uncompressed models must align (i.e., the variance ratio must be close 1), variance collapse explosion leads suboptimal performance. Our data-free and tne-tunnng--fre model folding methods (Fold-AR and Fold-DIR) achieve performance comparable data-driven statistics repair (Fold-R), while outperforming naive statistics repair (Fold-naive) and the recently proposed IFM (Chen et al., (023). All methods were evaluated on a public ResNet18 checkpoint trained on CIFARTO. Lines connect the performance of different methods at the same weight sparsity level, applied uniformly across all layers. Variance ratio refers to the activation outputs in the last layer. precise definition and analysis are in Sec. 3
across models. They provide both theoretical and empirical evidence suggesting that features within the same model are more similar than those between models trained on different tasks. This method avoids the need for retraining the compressed model but requires training data to match features based on the similarity of their activations. Similarly. Yamada et al. (223)) examine various model merging techniques and conclude that merged models require dataset such as a coreset- for effective merging and to achieve high accuracy. This data is essential for adjusting internal data statistics that are disrupted by weight fusion, such as updating the running mean and variance in BatchNorm layers (loffe & Szegedy] 2015). The process involves a simple forward pass through the model and is a well-established method to adapt models in low-resource environments (Leitner al.. 2023).
In contrast, IFM (Chen et al. 2023) offers a fully data-free and fine-tuning-fret approach, utilizing weight matching (Ainsworth al. 2023) iteratively merge similar hidden units, similar to Stoica et al. (2024). However, despite a heuristic for preserving data statistics, we demonstrate that IFM fails to maintain performance across standard architectures and for high sparsity. Other data-free approaches, such as (Yin et al., 2020), generate synthetic images directly from the uncompressed model for fine-tuning to restore pruned model accuracy. More related work is covered in Appendix N
This paper presents model compression technique, model folding, that exploits weight similarity through three phases: neuron clustering, merging, and data statisties repair, summarized in Fig. (left). We demonstrate that k-means clustering provides theoretically optimal and data-free method for merging weights. Building on Jordan et al. (2022), which addresses variance collapse using REPAIR with training data, we introduce two data- free alternatives: Fold-AR (folding with approximate REPAIR) and Fold-DIR (folding with Deep Inversion- REPAIR). Fold-AR estimates mean correlations within clusters assuming independent inputs, while Fold-DIR uses Deep Inversion (Yin et al. 2020) to synthesize a single batch of images for updating BatchNorm statistics via a forward pass. Both methods maintain data statisties and prevent variance collapse explosion to avoid suboptimal compression performance, with Fold-AR standing out as a more resource-efficient option while still significantly surpassing existing methods. Fig. [ (right) shows that the highest accuracy at any target sparsity is achieved when the mean variance ratio over the last layer between the compressed and uncompressed models stays close to one. Our contributions are:
We introduce model folding, a novel model compression technique that merges structurally similar neurons within the same network to achieve compression. We provide both theoreti- cal justification and empirical evidence demonstrating that k-means clustering is an optimal and effective method for fusing model weights in a data-free manner.
To enable data- free model compression, we adapt the REPAIR framework proposed by Jordan et al. (2022) to address variance collapse of data statistics within model after
2

---

Published as a conference paper at ICLR 2025
layer compression. We introduce data-free and fine-tuning- versions of REPAIR, that effectively maintain model statistics and achieve high performance.
We demonstrate that model folding surpasses the performance of SOTA model compression methods which do not data or fine-tune the pruned model, including recently proposed IFM (Chen et al. 2023), and INN (Solodskikh et al., (023). in particularly at higher levels of sparsity and when applied to more complex datasets.
We model folding on LLaMA-7B without utilizing data post-tuning and achieve comparable results to methods that require data and fine-tuning.
2 PRELIMINARIES
Our work is inspired by recent advances in two key areas: neuron alignment algorithms for fusing model pairs in weight space, and data-driven methods for recovering from variance collapse fused models. Below, we summarize the relevant results from the literature.
Neuron alignment algorithms. Model merging involves combining the parameters of multiple trained models into single model, with key challenge being the alignment of neurons across these models, particularly when they are trained on different datasets or tasks. Neuron alignment methods can be classified based on their dependency
the input data. Methods like the Straight
Through Estimator (STE) (Ainsworth et al.) 2023). Optimal Transport (OT) (Singh & Jaggi (0200) and correlation- based activation matching (Li et al. 2015) require data for effective merging. In contrast, weight matching (Yamada et al. 2023] Ainsworth et al. 2023) is data-free method, making it efficient in scenarios when training data is not available. In weight matching, neurons are aligned
by minimizing the L2 distance between the weight vectors of neurons across models. Given two models with weight matrices WA A and Wa B, the goal is find permutation P of the weights in W that minimizes the distance:
min
I PW Bl P
where PWn a denotes the weight matrix Wn after applying the permutation P to align it with W Once the optimal permutation is found, the models are merged by averaging the aligned weights:
1
W merged
where P is the permutation that minimizes the L.2 distance. Weight matching solves an instance of the linear sum assignment problem (LSAP), usually solved by Hungarian algorithm (Kuhn) 1955) as done in (Jordan et al., 2022 Ainsworth et al. 2023). to layer-wise align weight vectors. Unlike merging different models, aligning neurons within single model requires acyclic matching graph, a challenge not addressed by LSAP, which assumes disjoint task and worker sets. To overcome the challenge Chen et al. (2023) and He et al. (2018) apply iterative approach greedily merging a pair of the most similar neurons in each iteration. This work extends weight matching to align clusters of similar neurons within the same model, remaining data-free. Appendix C provides more details on the relationship between weight matching and model folding. We show that IFM is inferior to clustering utilized by model folding as described in the next section.
Variance collapse and REPAIR. When interpolating between independently trained, neuron-aligned networks, (Jordan al. 2022) observed phenomenon they termed variance collapse. This occurs when the variance of hidden unit activations in the interpolated network significantly diminishes compared to the original networks, leading to a steep drop in performance. To solve this issue, Jordan et al. (2022) introduced the REPAIR method (Renormalizing Permuted Activations for Interpolation Repair) which uses input data to recompute the internal data statisties.
REPAIR works by rescaling the preactivations of the interpolated network to restore the statistical properties of the original networks. Specifically, it adjusts the mean and variance of the activations in each layer of the interpolated network to match those of the corresponding layers in the original networks. This is done by computing affine transformation parameters parameters-rrecclinng and shifting coefficients- for each neuron, ensuring that the mean and standard deviation of activations in the interpolated network are consistent with those in the original models. REPAIR effectively mitigates the variance collapse, enabling the interpolated network to maintain performance closer to that of
3

---

Published as a conference paper at ICLR 2025
k
1.0
10
10
1.0
1.0
0.8
0.8
0.8
Comelatio
0.6
0.5
0.6
0.6
3.5
0.4
0.4
0.4
0.4
0.4
0.4
0.4
0.2
02
0.2
0.2
0.2
0.2
02
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Fraquency
Figure 2: Layer-wise correlation between matched channels in ResNet18 trained CIFAR10. For each layer, we use activation matching with Lg distance measure to greedily pair similar neurons. Each subplot shows the correlation within all matched pairs.
the original models. This technique has become essential in recent work to preserve model accuracy after merging (Ainsworth et al., 2023 Yamada et al. 2023 Lolicoeur-Martinee al. (022). While REPAIR relies on input data to preserve the network statistical properties, this paper proposes data-free alternative.
MODEL FOLDING
In this section, we introduce model folding. a novel compression technique that reduces the compu- tational complexity and size of neural networks by merging similar neurons in each layer without requiring training data. As illustrated in Fig. [ (left), model folding processes the network layer by layer, involving filter clustering, merging, and correcting data statistics. Below, we present a theoretical analysis of our approach, supported by empirical results ResNet18 using CIFARIO.
3.1 CHANNEL CLUSTERING
Channel similarity. Neural networks trained with stochastic gradient descent (SGD) tend to have many correlated hidden units, illustrated in Fig. 2 Model folding exploits this observation, which is related to the implicit bias of SGD. As discussed in (Gunasekar et al.) 2017). SGD exhibits minimum norm bias, which can be viewed as a form of regularization when no explicit regularization is used. In contrast to L2 regularization, which promotes sparsity, the minimum Euclidean norm solution (I-2 norm) penalizes large weights, encouraging smaller, more regular weights. This not only prevents overfitting but also results in smoother decision boundaries (Bishop) 2006). While the minimum norm solution does not directly enforce weight similarity, we empirically demonstrate in Appendix D that it leads to effective model compression when applying similarity-based methods. Recently published methods (Stoica et al. 2024 Chen et al. 2023) leverage the same observation.
Folding clustering problem. This work extends weight matching (Ainsworth et al. 2023), which minimizes the L2 distance between weight vectors and operates without requiring training data. Instead of finding pairs of similar neurons by solving the linear
assignment problem (LSAP)
with Hungarian algorithm (Kuhn) 1955) as done in (Jordan et al. 2022; Ainsworth et al., (203), we achieve channel matching using k-means clustering. In the following, we justify this approach it provides an optimal weight matrix approximation.
Given neural network layer / with weight matrix Wi Ruxm define the output of this layer as yt = o Wix)), where X, E R is the input vector to this layer, E IR" is the output vector, and (.)) non-linear activation function applied element-wise.
To reduce the number of outputs of layer / we cluster (fold) rows of Wi, i.c., & cluster centroids are determined which serve as a prototype of the respective cluster of rows. All rows of a cluster are replaced by their cluster centroid. This can be formulated as
W, R UM,
where M kxm contains the & cluster centroids and the cluster matrix U {0, 1 determines the membership of a row: j(i.j) = 1 if the i-th row of W, belongs to the j-th cluster, and u(i. otherwise.
4

---

Published as a conference paper at ICLR 2025
As measure of the approximation error when replacing the rows of W, by k < prototypes, |p of the difference between W, and the low-rank factorization UM:
use the Frobenius norm
J |W - UM = w(( + (UUMM"UT)) - 2r(UMW77.
We determine the optimal matrix of cluster centroids by setting the derivative of J with respect to M
to zero:
M =
As result, we can write
W = UM CW, with C C=U(O'U)-'''
As mentioned above, we use k-means clustering for folding as this minimizes J by determining the optimal clustering matrix U and the corresponding cluster centroids M, also see (Bauckhage) 2015).
Interdependence between layers. will expand the above result successive layers and 1. For simplicity of notation, we neglect the bias and get
y1+1 a W (W(x)))
Following the above notation, we describe the folding of activations by some clustering matrix U and C U)U 'U". It
shown in Appendix B that the corresponding approximation satisfies
Vi+1 oo WWWHo ((VC))
o (Wi4)C")
Adding up the individual folding costs J1+1 = || CW P and Jr ||, - cwili yields the combined approximation error Jt,+1 J1+1 + J4 for folding layer which can be rewritten as
J1,+11 |ii+1 CW with W W, w
If we perform k-means clustering WI,+1 and use the resulting clustering matrix i C U(U"U)-' UTT then the combined approximation error Jr,411 is minimized. This approach accounts the impact of compressing one layer on next, leading to more efficient compression that balances the process and preserves learned representations while reducing model size. Our folding methods outperforms other methods experimentally, see Fig. for comparison to other clustering methods and Iterative Greedy (greedy) adopted in SOTA.
Batch Normalization. Now, let us consider batch normalization layer / represented by two diagonal matrices Σ, (scaling) and Σ, (normalization), again neglecting the bias to reduce notation. In this case, we get
y a (W 11 Σ, E. Wix:))
The folding of layer can be distributed to the matrices Eg, En, and W; in various ways, depending on the chosen correction the variance, see Sec. 3.2 For example, one can cluster each matrix separately, leading to
y = F C* (CE (CW.))))))))
Adding up the individual folding costs J141, Js. Jn. and Ji for each of the matrices W1-1 E,, Σ, and Wt, respectively, yields the total approximation error Jut = J1+1 + Ja + Jn + Jr for folding layer I
Jos
W cW with W = [W]] W, diag E.) diag Σ..]
If we perform k-means clustering W then the total approximation error Jus is minimized. This approach is used in the Deep Inversion (DI) REPAIR, see next section.
Instead, if we decompose the folding of layer / according to
Vi+1 = ooWW41(WW Cd (o((CE.)(CC, Wi)x))).
then the individual folding costs of W1+1. E, and the normalized weight matrix E,, W, add up to W wa clall with W = [Σ, W. (E, WTnl.
Jun
Again, if perform k-means clustering on this combined matrix W then the corresponding total approximation error Jmn is minimized. This approach is used in the approximate REPAIR, see Sec. 3.2 For completeness, present in Appendix F how we handle residual connections.
5

---

Published as a conference paper at ICLR 2025
Merging similar channels in each cluster. To fuse similar channels, various approaches have been proposed in the literature, such as fusing weights for multitasking, which involves Hessian calculations (He et al.) 2018). or by combining the matched weights into single channel (Chen et al. 2023). (Matena & Raffel 2022) introduces Fisher-weighted averaging based on the Laplace approximation for merging weights, while (Jin et al. 2023) suggests computing regression mean, which is both computationally efficient and scalable for merging multiple models. In our approach, we use above formulation of the optimization problem as k-means clustering and use simple mean to compute the cluster centroids.
3.2 MAINTAINING DATA STATISTICS IN A COMPRESSED MODEL
Variance collapse and variance overshooting. We use the conceptual framework in (Jordan et al., 2022 to analyze the performance of model compression methods. We use the following definition.
Definition (Variance ratio). Consider neural network f(x, θ) with layer activations {xi}i and its compressed version f(x, ②) with activa- tions [x The variance ratio of the 1-th layer is:
0.8
Test Accuracy
0.6
AC ward
AC single
complete
AC average
Var(*i) Var(xi)
xx Var($t,s)
1
greedy
X
Var(xi,))
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Sparsity
We observe not only variance collapse but also variance overshooting phenomena. Specifically, when data statistics are not accurately corrected after channel merging, as in IFM, variance over- shooting can occur, leading to network perfor- mance decline. Fig. 4 shows layerwise variance ratio between the compressed and uncompressed networks. Staying close to I is essential to miti- gate both phenomena. This highlights the critical need for precise statistical corrections during model merging.
Figure 3: k-means (KM) outperforms other clus- tering methods: Spectral Clustering (SC), Ag- glomerative Clustering (AC) with different linkage criteria and Iterative Greedy (greedy) used to com- press ResNet18 trained on CIFAR10 10. Data-based REPAIR was used to restore data statistics after clustering for all methods.
the context of model compression, particu- larly when using folding as a clustering method, it is crucial to ensure that the compressed model maintains accurate data statistics. This is especially important for layers involving operations like BatchNorm, where maintaining the correct statistical properties of activations is vital for model performance (Jordan et al.) 2022 Yamada et al. 2023).
Fold-AR: Folding with approximate REPAIR.
In the following explanation of the data-free approximate REPAIR, neglect biases for ease of notation. Following the previous section, we consider folding of the normalized weight matrix with
using the post-activation output X of the previous layer and the input 2 to the scaling matrix Σ, A cluster is defined by the column of the clustering matrix U, i.e., all values 21( with u(i,c) 1 belong to cluster c. Moreover, by definition of C. all values (7) belonging to a single cluster c equal the centroid (c) of the cluster, i.e., the average of all values Σ,, Wixi belonging this cluster. More formally,
vu(i, c) 1 :1)(()) = (c)
1 Σ Ei()).
A1 2(c)= N JET.
where I. : c) = 1} denotes the indices of all values belonging to cluster N. [a] denotes the number of values in the cluster, and ã = Σ, Wix, The batch normalization using Σ,, ensures that the variances of all āi( equal 1. The averaging over all ăz(i) belonging to single cluster destroys this property and leads to the observed variance collapse. We will describe various methods compensate this loss in variance, first the data-free approximate REPAIR (Fold-AR).

---

Published as a conference paper at ICLR 2025
10
0.8
Teat Accuracy
20"
0.6
Varience
0.4
Fold Naive
target
Fold-AR
Fold-AR
10
Fold-DIR
0.2
Fold-DIR
Fold Naive
Fold-R
Fold R
0
10
12
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Layer Index
Sparsity
Figure 4: Variance collapse and overshooting on ResNet18 with CIFAR10. The goal to align the layer-wise variance in the compressed network to that of the uncompressed model. Naive aver- aging of statistics (Fold-Naive) leads variance collapse (Jordan et al. 2022), while IFM over- shoots. Fold-AR and Fold-DIR closely match the performance of the data-driven REPAIR (Fold-R). Layer- wise sparsity 0.5.
Figure 5: Data-free folding methods with ap- proximate REPAIR (Fold-AR) and Deep In- version (Yin et al. 2020) (Fold-DIR) and ResNet18 with CIFARIO at various weight spar- sity levels, uniformly distributed across layers. Fold-DIR performs similarly to the data- based REPAIR (Fold-R). Both Fold-AR and Fold-DIR surpass IFM (Chen et al. 2023) by significant margin.
The variance of the cluster centroid (c) of cluster is given by
Var (c))
Var(ār(1))
Σ Cov(Ez(0,,
N2
iEL.
WEELSS
which further simplifies to Var(&r(c)) = No [No+ ( - N.) where E[c] is the mean corre- lation within the cluster. To prevent variance collapse, aim for Var(a((c)) = which would occur Elc] = 1, meaning all channels in the cluster are fully correlated. However, as E(c] < typically, multiply each cluster centroid scaling parameter assuming average cluster correlation Elc]
No
(c) + (c)) V No +NZ Flc
Suppose now that the covariance matrix Σx, of the output x; of the previous layer is available and that we define the normalized weight matrix W, Σn W, with rows (f). Then the correlation E(c] can be computed
W W (wi (T)) (1))
Elc
Σ
N2 - No ijELit
( wT())
In the absence of data, E[c] can be estimated by assuming that the output values of the previous layer are uncorrelated. As the individual variances (t) equal we obtain
Elc]
1
Σ
N2
N
(JELAN) V (w) (1) ())
We term this approach maintain the data statistics within the model folding with approximate REPAIR (Fold-AR). This approach helps to ensure that the statistical properties of the data are preserved even after model compression, maintaining the performance of the network while reducing its size. Fig. shows how the performance of Fold-AR compares to the data-driven REPAIR (Fold-R) and surpasses the SOTA data-free methods.
Correcting data statistics with deep inversion. Deep Inversion (DI) (Yi et al. 2020 a technique that synthesizes realistic images directly from a pre-trained neural network without requiring access to the original data. The process involves inverting the model by optimizing random noise to produce class-conditional images that match the statistics of the data the model was trained on (Mordvintsev et al. 2015). DI leverages the NarchNorm layers within the network, which store
Fold-DIR:
7

---

Published as a conference paper at ICLR 2025
0.9
0.7
Pals-
0.6
Fois-AR
20.7
Fuls-taive
20.5
FolsR
Foid DIR
0.6
FRK-AR
G4
0.3
1..4
0.2
SP
0.3
SP L1
0.1
SP
6.0
0.0
0.1
0.
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Sparstly
Sparshy
0.9
0.7
Fold-ttive
0.0
20.7
20.5
Pois4t
Fois-DIR
0.6
Fols AR
3.4
IFM
0.5
Fols-DIR
6.3
SP
Fale-R
IFM
6.1
0.0
0.3
0.2
0.3
0.4 0.5 0.5
0.0
0.3
0.2
0.3
0.4
0.5
0.5
Figure 6: Comparison with IFM (Chen et al. 2023 and structured magnitude pruning Cai at. 2020; Yin et al.) (002)). Model folding, when tested on ResNet18 (top row) and VGGII-BN (bottom row) trained on CIFARIO 10 (left column) and ImageNet (right column), outperforms IFM with higher sparsity and increasing dataset difficulty.
the running mean and variance of activations during training. By using these stored statistics regularization term in
R(x) = Cotass (x,t)+ + Σ |Ix(t))) (xx))11 +
Σ \arr(*))) - Var(x:)11 + ||x 2 ||x]|
DI ensures that the generated images have similar statistical properties to the original training data, thus producing high-fidelity images. Here, vu({ and Var(*;; are the mean and variance of the feature map & in the synthesized data, and ((xx) and Var(x:) xt) are the expected mean and variance of the feature map in the original data. The term Litass (x, L)) denotes classification loss of the synthetic sample, while | and | denote the L2 and Total Variation regularization terms over the synthetic sample x. Finally t denotes the desired class of the synthetic sample x. Sample images extracted from a pre-trained ResNet18 model on CIFARION 100 with DI are shown in Appendix M
We leverage a single butch of DI-synthesized data within model folding to preserve data statistics after channel merging, eliminating the need for training data. By generating synthetic images aligned with the network's 's internal statistics, DI recalibrates the folded model's parameters, ensuring that activation variance and mean are maintained. This helps the model retain its performance post- folding, mitigating issues such as variance collapse explosion without requiring the original dataset. Notably, updating BatchNorm statistics requires only a forward pass, with no backpropagation needed. Thus, Fold-DIR offers data-free and fine-tuning-free solution for maintaining data statistics. Fig. shows that Fold-DIR closely follows the performance of the data-driven REPAIR (Fold-R), effectively maintaining the data statisties within the model. Fold-DIR outperforms Fold-AR the cost of generating a batch of synthetic images and forward pass through the network.
4 EXPERIMENTS
Following related works model merging (Ainsworth et al. 2023 Chen et al. 2023 Jordan et al.
2022). we evaluate folding on convolutional architectures, including ResNets (He et al.) (016) and VGGs Simonyan & Zisserman 2014) of varying sizes CIFAR10, 10, CIFAR100 (Krizhevsky et al.
2009b and ImageNet (Deng et al. 2009). For models trained on the CIFARIO and CIFARI00 datasets, we used the hyperparameters available from online benchmark For models trained on ImageNet, the pre-trained weights were taken from torchvision For large language models (LLMss, we evaluate model folding on LLaMA-7B (Touvron al. 2023a) with pre-trained weights
8

---

Published as a conference paper at ICLR 2025
C.0
0.7
67
G.8
0.5
0.5
0.4
FoloR
Fole
ITM
INN
0.0
0.1
0.2
0.3
0.4 Sparsby
0.5
0.6 0.7 0.0 0.1
0.2
0.3
0.4
0.5
Sparshy
Figure 7: Comparison of model folding with IFM (Chen et al. 2023), and INN (Solodskikh et al. 2023) using ResNer18 on CIFARIO. the original experiment defined in the IFM and INN papers, where only the last two blocks of a ResNet1 8 are pruned, folding is significantly better than INN while matches the performance of IFM for lower sparsities and becomes significantly better for higher sparsities. Note, the maximum sparsity achievable by INN is 54% (Solodskikh et al. 2023).
Fursen 16
12
Recuerty
Figure 8: Layer-wits wise correlation among matched channels in VGG11 and its wider variants on CIFAR10. This figure shows correlation matrices for each layer of VGG11 and its Ix and 3x wider variants, derived from activation matching. Opaque black represents the Ix wider model, while vibrant colors indicate the 3x wider model, highlighting differences in correlation strength.
from Hugging Face Hub. In all experiments, model sparsity denotes the proportion of weights that have been removed as a result of model compression. Experimental setup is detailed in Appendix A Further evaluation results are in Appendix K and
Model folding mitigates variance collapse. Fig. 6 compares model folding with IFM (Chen et al. 2023), recently introduced data-free, fine-tuning-fies free method that combines aspects of folding and pruning. Unlike model folding, which accurately corrects the data statistics in the compressed model, IFM merges matched input channels by summing one and zeroing the other, followed by weighted average of output channels. In contrast to the original paper, Fig. applies the same sparsity ratio across all layers for every method. We find that model folding significantly outperforms IFM, particularly at higher sparsity levels and for larger networks. Additionally, Fig. (left) replicates the experiment from (Chen et al. 2023) on ResNet18 with CIFARIO, using the same per-layer sparsity pattern where only the last two blocks are sparsified. In this scenario, IFM offers slight performance edge over our method for low sparsity, but struggles with higher sparsity.
Comparison to structured pruning. We compare model folding with the structured magnitude pruning (SP) method used in (Cai et al.) 2020: Yin et al.) 2022), based on I, and L2 norms, without fine-tuning. Fig. 6 demonstraaes that model folding significantly outperforms magnitude pruning, with the performance gap widening as sparsity increases. At 70% sparsity, the folded ResNet18 8 on CIFARIO maintains over 80% accuracy, while pruned networks barely surpass random chance. On ImageNet, the performance collapse is even more pronounced across all methods due to the dataset's s higher complexity, yet model folding consistently performs well across both datasets. Following (Chen et al. 2023). Fig. 7 (right) compares model folding with the SOTA data-free pruning method INN (Solodskikh al. 2023). which struggles to manage even moderate sparsity.
Folding wider models. Do wider networks present more opportunities for model folding? We first examine the layer-wise correlation among matched channels in VGGII and its wider variants on CIFARIO, as shown in Fig. S This ablation study reveals that increasing the layer width strengthens the matched correlations, suggesting greater potential for folding. Building on this, Fig. demonstrates the application of model folding also to 1x/2x/3x wider MLP and ResNet50 architectures, trained on CIFARIO and CIFAR100, showing consistent performance gains as width increases.

---

Published as a conference paper at ICLR 2025
0.25
0.20
F36--
Faida
Foid
30
0.1
0.2
0.5
0.4 0.5
0.5 0.7
0.3
04
05
0.6 07
Figure 9: Model folding performance improves with increasing model width. The MLP model consists of three stacked mlp blocks (including fully connected layer, BN layer, and ReLU layer), followed by a final classifier. Upscaled versions of MLP (left) and ResNet50 (right) architectures, trained on CIFARIO and CIFAR100, demonstrate the consistent advantages of model folding.
Prune ratio
Method
Data usage
WikiTex12)
BaolQ
WinoGrande
ARC-e
ARC-c
Average?
0%
LLaMA-7B (Touvron et al. 2023a
5.68
75.05
69.93
75.34
41.89
65,55
Magnitude Pruning
36136
43.21
35.36
20%:
L..M.Pruner (Ma et al.
2023
Gradients
10.53
59.39
61.33
59.18
37.18
54,27
FLAP (A et
Calibration
20%
Wanda (ptsuu er a 2023
Calibration
8.22
71.25
67.09
71.09
42.58
63.00
20%
SliceGPT (2shiitsws
2024
Calibration
7.00
57.80
67.96
62.67
36,01
56.11
20%
ShorGGP [Mon ot al 2024
Calibration
15.48
62.17
67.40
58.88
31.91
$5.09
20%
Model Folding
13:33
62.29
62.19
26.37
50.17
20%
Model Folding Fine tune norm
8.95
70.09
63.14
59.85
28.24
$5.33
Table I: Structured pruning performance on LLaMA-7B without post-tuning, showing perplexity on WikiText2 and zero-shot task results. "Wanda_sp denotes a structured version of Wanda. Model folding, despite no data or fine-tuning, matches data-driven methods. Fine-tuning only layernorms on wikipedia en further boosts performance.
Folding LLMs. LLMs built with large number of parameters, achieving strong performance across various tasks. However, structurally compressing these deep and large models remains a challenge. LLM-Pruner (Ma et al., 2023) performs structured pruning using gradient calculations, while Wanda (Sun el al. (2023) leverages an importance score by multiplying weights with their corresponding imput activations. FLAP (An al. 2023) dynamically computes fluctuation pruning metric using calibration data. In Tab. I we compare model folding with these methods on LLaMA- 7B (Touvron et al. 2023a), focusing on perplexity on the WikiText2 (Merity al. 2016) validation set and zero-shot performance across four tasks using the EleutherAl LM Harness (Gao et al. (2024). The folded model performs only very slightly worse than models compressed with data-driven methods. Following SOTA, the clustering phase of model folding was applied to LLaMA-7B. introducing 20% and 50% sparsity in the attention and feed-forward layers of decoder blocks 22-29, and 10% and 40% sparsity in the attention and feed-forward layers of decoder blocks 1 11-21, respectively. As there is no batchnorm layer in LLaMA-like LLMs, we just applied clustering in LLMs without REPAIR. Tab, [ shows the generated examples of dense and folded LLaMA-7B processed by model folding without REPAIR in Appendix E Results of folding I.L.MA-7B (Touvron et al.) 2023b) are also provided in Appendix E
When folding with 20% sparsity, the pruned model continues perform well.
CONCLUSION
In this paper, we introduce model folding, a novel compression technique that reduces model size by merging similar channels across layers, without requiring fine-tuning or training data. Model folding achieves high sparsity while preserving data statistics, outperforming traditional pruning and data-free compression methods. Our experiments demonstrate that wider networks, such as VGGI and ResNet50, offer greater opportunities for folding due to increased redundancy, further improving compression efficiency. In L.Mss, model folding can prune models while maintaining performance comparable to data-driven methods, but without the need for data access or fine-tuning, which are typically required by most structured pruning techniques.
Limitations and future work. Model folding offers significant compression without data or fine- tuning, but its effectiveness may be limited in networks with low redundancy. Additionally, it does not optimize sparsity levels per layer, leaving this for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Franz Papst and Francesco Corti for their insightful comments on the early draft of the manuscript. This work was partly funded by the Austrian Research Promotion Agency (FFG) and Pro2Future (STRATP II 4.1.14 E-MINDS strategic project). The results presented in this paper were computed using the computational resources of Zentralen Informatikdienstes of Graz University of Technology and Pro2Future GmbH.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries, 2023.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation based adaptive structured pruning for 11983
large language models, 2023.
Arduino. Arduino nano 33 ble documentation. https:://ocs wwwp:/iooooo...uioooooooooo http:///w....ccrrraaaeeaaanaaaa/ 33--1/// 2024. Accessed: 2024-11-19.
Salch Ashkhoos, Maximilian I.. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. /tRp:htts:://axii.orrr abs/2401. 15024
Christian Bauckhage. k-means clustering is matrix factorization, 2015. URL.intps///arvvv.or///s//// 07548
Christopher M. Bishop. Puttern Recognition and Machine Learning (Information Science and Statistics). Springer- Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bemsstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint w0xiv:2108.07220 2021.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Onco-for-all: Train one network and specialize for efficient deployment, 2020.
Yupeng Chang, Wang. Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3): 45, 2024.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu. Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks, 2019. URL/https: //tttpps///rxiv...o http:://aaxxxvoooaa///////s wwtpab///1990000 001886
Yanjiao Chen, Baolin Zheng, Zihan Zhang, Qian Wang, Chao Shen, and Qian Zhang. Deep learning mobile and embedded devices: State-of of--the-art, art, challenges, and future directions. ACM Computing Surveys (CSUR). 53(44)1-37, 2020.
Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. arXiv preprint arXiv:2310.0675 067566, 2023.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning-taxonomy. comparison, analysis, and recommendations, 2023. URI httphhtss////iiiii orgg/bs//333 abs/aa/a....../ 06767
Francesco Corti, Balz Maag. Joachim Schauer, UIrich Pferschy. and Olga Saukh. HADS: Hardware-aware
deep subnetworks. In 5th Workshop on practical ML for limited/low resource settings, 2024a. URL
Francesco Corti, Balz Maag. Joachim Schauer, Ulrich Pferschy, and Olga Saukh, Reds: Resource-efficien deep subnetworks for dynamic resource constraints, 2024b. URL htttp:///axxiv httpaaa/wii..or/b/////////////// 133493
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 448-255. Icce, 2009.
Rahim Entezari and Olga Saukh. Class-dependent compression of deep neural networks, 2020. URI https. //arxiv. wwt.:/orgg/bbbb//ggge/ 10364
11

---

Published as a conference paper at ICLR 2025
Rahim Entezari. Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in
linear mode connectivity of neural networks, 2022.
Espressif Systems. Esp-eye development board espressif systems. proouutt//eekttsseee-e-e//rrrrer 2024. Accessed: 2024-11-19.
http:///wwwwesrsssi.ffom
Gongfan Fang, Jie Song. Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation, 2020. URL
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 888880033333 2018.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems. 35:4475-440 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URt/htpss:/ htttttts///enooooo
httpa////eeoooooo/rrrrrrrrreeeee wttp://orrreer/rreeeeeeeeeeeee/e 12608662
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference, 2021. URL httpp:://ardvv.org/aab/////
13630
Jiamping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: survey, International Journal of Computer Vision, 179 March 2021. ISSN 1573-1405. doi: 10.1007/ $11263-021-01453-2 URL doip ortp:///dddoiooo//o.ooooo 0000////223002-01110010
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization, 2017. URL hhtps: htttt::://rxiv.... http://wwwwggg/b///////a//// 09280
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. International conference on machine learning, pp- 1737-1746. PMI., 2015.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free http:///wwiv..or//////////.
model compression, 2020. URL
Bahak Hassibi, David G Stork, and Gregory Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 299.299. IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770 778, 2016.
Xiaoxi He, Zimu Zhou, and Lothar Thiele. Multi-task zipping via layer-wise neuron sharing. Advances in Neurul Information Processing Systems, 31, 2018.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv httpx/wwwwrg/b//////////o 150///bbb1111..... 0253T
Samuel Horvath, Stefanos Laskaridis, Shashank Rajput, and Hongyi Wang. Maestro: Uncovering low-rank
structures via trainable decomposition, 2024. URL
14929
Hengyuan Hu, Rui Peng. Yu- Wing Tai, and Chi- Chi-Keung Tang. Network trimming: datannriven neuron pruning approach towards efficient deep architectures, 2016. /tt./nttss http://arriivooorr//////0// 03200
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. /RLL/ittsss//arai.orr//////////
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8)) 87, 1991.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pictro, and Pengxiang Cheng. Dataless knowledge fusion by merging http://aawwww.orrr/////2222.////
weights of language models, 2023.
12

---

Published as a conference paper at ICLR 2025
Alexia Salicoeur-Marinnaa.. Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julien. Population parameter averaging (papa), 2024. htttps://arxxi..rr/abb///// 03094
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair, arXiv preprint arXiv:221 2022.
Leonid V Kantorovich, On the translocation of masses. Journal of mathematical sciences, 133(4):1381-138) 2006.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang. and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications, 2016. URL https
//arxiw.org/abs//55 /0650
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009a.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- and cifar- 10 (canadian institute for advanced research), 2009b. URL http::/wwwww.... toronto. wrrrontt.edd//kii/iiiiie html MIT License.
Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics (NRL). 52, 1955.
Ashish Kumar, Saurabh Goyal, and Manik Varma. Resource-efficient machine learning in kb ram for the internet things. International conference on machine learning, pp. 1935-1944, PMLRR 2017.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up up convolutional neural networks using fine-tuned cp-decomposition, 2015. URL httpss///arxi..org/bbbs
1412. 6553
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky (ed.), Advances in Neural Information Processinng Systems, volume Bonrgan-Kaffmann 1989. URL http:p:://rooccedngg...ttt Paperr
Stefan Leitner, M. Jehanzeb Mirza, Wei Lin, Jakub Micorek, Marc Masana, Mateusz Kozinski, Horst Possegger, and Horst Bischof. Sit back and relax: Learning to drive incrementally in all weather conditions, 2023. URL https://arxiv.
wwt.://g/abb.....oa/ https///arxkv.orr//ab//255.....
18953
Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Temary weight networks. arXiv preprint arXiv: 1605. 140.00111 2016a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 1801,08700 2016b.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2017. URL http:::/t///:tti.....t wwtp:/wwwwww./////b/o///// 08710
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint 1rXiv:1511.0754 2015.
Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, and Wen-Huang Cheng. Lightweight deep learning for resource-conse conuree-constrained environments: A survey, 2024. URL https 7/arxiv. mrrxiw.org/abs/24444 07236
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinct: filter level pruning method for deep neural network compression. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 201
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinct: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017b.
Xinyin Ma. Gongfan Fang. and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances neural information processing systems, 36:21702-21720 2023.
Julieta Martinez, Jashan Shewakramani, Ting Wei Liu, Ioan Andrei Bårsan, Wenyuan Zeng, and Raquel Urtasun. Permute, quantize, and fine-tune: Efficient compression of neural networks, 2021. URL https
Taaxxiw.org/abs///10... 15703
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2022. URL. https //axxiv.org/abs/21111
13

---

Published as a conference paper at ICLR 2025
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL
httpss:/aaxxvv.or//abb/// 03888
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, arXiv preprint arX70:1609.07843, 2016.
Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching, 2019. URL https://arxiv.. wtt.:/wwww.rr///////000000999 09766
Gaspard Monge. Mémoire sur la théoric des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pp. 666-704, 1781.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks,
2015. URL hntpp::/reesaacchggoogl/// http:///tw...cooooo//oo//////pta htwprppppiioiiinnooooooeeeeeeeee html
NVIDIA. Jetson
nvidia developer.
https://www http://www.wiii...a. www/coo http://nnuu/atutoooouuuuuuuuuuu machinnss
seseedddd-sstttms/jet....o nawaaaannoooooouaanooe deve Lopment/ 2024. Accessed: 2024-11-19.
Franz Papst, Daniel Kraus, Martin Rechberger, and Olga Saukh. Sensor-guided adaptive machine learning on
resource-constrrineed devices. Proceedings of the International Conference on the Internet Things, 2024.
Siyu Ren and Kenny Q Zhu. Low-rank wrune-and-factorit for language model compression, 2023, URL https://arxiv. www. attp///../22....... 14152
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High resoluttolt image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated oixture-of-experts layer. arXiv preprint arXiv:1701.066 065388 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint (ee:::409.555550 2014.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Letkimmiatis. Integral neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16112-1612, June 2023.
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, 2024. UtL/httss://axxv. www.:/or///aaeeeeaoe hwrgg/abs/230555 .03053
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1 11695, 2023.
Alexander Theus, Olin Geimer, Friedrich Wicke, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Towards meta-pruning via optimal transport. arXiv preprint arXiv:2402.07889 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. URL
httpps://arxiv. https://arkiv.or///ab///000 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylow, Pushkar Mishra, Igor Molybog. Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor. Adina Williams, Jian Xiang Kuan. Puxin Xu, Zheng Yan. lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kamhadur, Sharan Narang, Aurelien Redriguez, Robert Stojnic, Sergey Edunow, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023h. UR. /tattpi:///aii...... wwwparwwwwwwww.///////ee/// 09288
14

---

Published as a conference paper at ICLR 2025
Shaohua Wan, Lianyong Qi. Xiaolong Xu, Chao Tong, and Zonghua Gu. Deep learning models for real-time human activity recognition with smartphones. Mobile Newworks and Applications, 25(2):743 755, 2020.
Dong Wang, Olga Saukh, Xiaoxi He, and Lothar Thiele. cubfsaae networks, 2024. URL https://arxiv.. abw//305. 13536
Zixiao Wang, Ke Xu, Shuaixiao Wu, Li Liu, Lingzhi Liu, and Dong Wang- Sparse-yolo: Hardwarre/sftwaare co-design of an fpga accelerator for yolov2. IEEE Access, 8:1 8:116569 116585, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rehecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022.
Masanori Yamada, Tomoya Yamashita, Shin's ya Yamaguchi, and Daiki Chijiwa, Revisiting permutation symmetry for merging models between different datasets, 2023. URL httpp:://ax//iii http://.ggg//bb/////e/oo/e 05641
Hongxu Yin. Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha. and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion, 2020. URL https
hwt.v/orgg/bbb//111 08795
Shanzhi Yin, Chao Li, Wen Tan. Youneng Bao, Yongsheng Liang, and Wei Liu. Exploring structural sparsity in neural image compression, 2022. httttpp://aaxtw.orrr/aabsseee
04595
Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24266 24275, 2023.
Aojun Zhou, Anbang Yao. Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights, arXiv preprint arXiv:1702.03044, 2017.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
The following sections provide supplementary information omitted from the main text:
Section
Implementation Details.
Section
Further Theoretical Results to Support Model Folding.
Section
Relationship Between Weight Matching and Model Folding.
Section
Channel Similarity.
Section
Model Folding on LLMs.
Section
Handling Residual Blocks.
Section
Handling Batch Normalization Layers,
Section
Folding Similar Channels in MLPs.
Section
Folding Similar Channels in Convolutional Layers.
Section
Folding Similar Channels in LlamaMLP and LlamaAttention.
Section
Comparison with Knowledge Distillation.
Section
Inference Speed of Folded Models on Edge Devices.
Section
M
Deep Inversion Sample Images.
Section
Further Related Work.
A IMPLEMENTATION DETAILS
We trained over 100 models NVIDIA DGX Station A100 featuring eight NVIDIA A100 GPUs (each equipped with 80GB memory) to evaluate the performance of model folding presented in this work. For folding experiment, we apply the same compression ratio to all layers. Pytorch Hut and Huggingface Hut") are used to load pre-trained checkpoints for complex model-dataset combinations, including BeNNNNeSGRR
on ImageNet and LLaMA-7B (Touvron et al.) 2023a). WandB is used to log training history, folding result, and evaluation metrics. The source code of all experiments available here: hhtpps: //gi thub. wtt. hwtp///ngooyumooee-olllooe universal
B FURTHER THEORETICAL RESULTS TO SUPPORT MODEL FOLDING
Lemma 1. Let x E & and lei U E 10,1)xxk be binary clustering matrix with E, Uij 1. Then with any element-wis nonlinear function a-)) we have
(Ux)) -x)
Proof of Lemma Define y Ux a (Ux) and (x),, Ux(x). Note that in any row of U just one element satisfies Wij = 1. We define such an element by a function with uy = + p(i) jj.
Therefore, y; Xp(1) and z; = oyy:) = (xp(t)) for all 1 S iis S n. Moreover, v; = oxx.) and W; = Vn(i) = σ (xp(t)) Therefore, z; = w, and z = w.
Lemma 2. L les U 1)xk a binary clustering matrix with E, uij = 1, let l() be an element-wise nonlinear function, and define C U' Then
a (Cx) = C. (Cx)
'ntts://pytorchhorg//ubb hhtpss//uugingffcccooeeeeee co/docs/hub///ddee h/tps//wandb.ai
16

---

Published as a conference paper at ICLR 2025
Proof of Lemma
2 We can write
o (Cx)
a(U(("
Ua((U'U))-U'x)
(Lemma
"|""0))100000(("""""""""""
- -'
(Lemma
= =Cx)(Cx))
Lemma 3. Let U be a clustering matrix and let D be a diagonal matrix, then the following is true
(U" (Ou diag(D)
Proof of Theorem
3
The clustering matrix U can be expressed as:
u
M12
Uin
U2
U21
1/2
U2n
U
2k11
dg2
Wkn
where uf represents the rows of the clustering matrix. Each row corresponds to cluster i, and the entries Wi satisfy the binary clustering property: úij and Wi = 0 otherwise.
1 if the j-th data point belongs cluster i,
The product DU is given by:
d
0
U11
U12
Wik
d2
0
U21
U22
U2k
DU
dn
Lun1
Un2
Unk
This simplifies to:
[ di Mnun
diu12
d wik
d2w21
d2
d2 u2k
DU
un
un
d, unk_
Using the clustering property of U, follows that:
11. ifi="'
10, otherwise.
From this, the product DU simplifies to:
U Diag (U" diag(D)).
This result holds because only the diagonal entries remain due the clustering matrix's 's orthogonality and binary properties.
Finally, using the above result, we compute:
(D"')~-''' = (U' U) ( diag(D))
By the property diag(Diag(x))) for any x E R.,, we obtain:
-UTU DU = U)
The lemma demonstrates that projecting the diagonal matrix D through the clustering matrix U preserves its diagonal structure. The diagonal entries are determined by the clustering matrix's mapping of the original diagonal values dDag(D), ensuring efficient computation and alignment with clustering properties.
17

---

Published as a conference paper at ICLR 2025
Lemma 4. Let U be clustering matrix and let E R" and R", then the following is true UDiag( w)x Diag( Uw
Proof of Lemma
The clustering matrix U can be expressed as:
U
where each row v is defined by mapping function f (1,1,...... + .1, For each row v the entries defined as:
1 if f(m):
Um
0, otherwise.
This representation indicates that the clustering matrix U assigns each element m specific cluster [(m).. Each row v has single non-zero element corresponding to the cluster index f(m).
Calculation of the Left-Hand Side (LHS).. The left-hand side of the equality is:
UDiag(w)x.
First, compute Diag(w)x, which scales each element of x by the corresponding element of w:
W1
W222
Diag(w)x =
Then, multiplying by U aggregates these scaled values according to the clusters defined by f. Specifically, the j-th element of UDiag(w)x given by:
(UDiag(w)x),
Σ
Wm
Calculation of the Right-Hand Side (RHS). The right-hand side of the equality is:
Diag(Uw)Ux.
First, compute Uw. The j-th element of Uw is:
(Uw), =
Σ Wm.
which sums the w., values for all elements assigned cluster j.
Next, construct Diag(Uw/ diagonal matrix with entries (Uw), along the diagonal:
(Uw)1
0
(Uw)2 2
Diag(Uw)= =
(Uw)
Finally, compute Ux. The j-th element of Ux is:
(Ux), =
Fmm
mmf(m=))
which sums the om values for all elements assigned cluster j.
Multiplying Diag(Uw) by Ux gives:
(Diag(Uw)JU) = (Uw),(Ux),
Σ m=)(m==
Wm
Σ
18

---

Published as a conference paper at ICLR 2025
Verification of Equality. Both the LHS and RHS compute the same aggregated sums 2m:f(m)=) Wm for each cluster j. The LHS directly performs the aggregation of um within clusters, while the RHS separates the computation into two steps: summing wm and Im for each cluster, followed by multiplying these sums. Since multiplication distributes over addition, the two expressions are equivalent:
UDiag(w) w (iag(Uw) Ux.
The lemma is proven, both sides of the equation compute the same weighted aggregation of Wm over the clusters defined by the clustering matrix U.
Lemma 5. Ler C. be clustering matrix and ler D be diagonal matrix, then the following is true
I Diag( (Cdiag W) [ddaa(W) Cdiag( (W)
Proof of Lemma 5 Let W = Diag( Cdiag( W)), where W represents the diagonal matrix obtained
by clustering the diagonal entries of W using the clustering matrix C. Both W and W are diagonal matrices, their difference W W is also diagonal. The entries of this difference are:
Wi,s 13ss if = J.
W..j
Was
10,
otherwise.
The Frobenius norm of the difference W W is:
||W W Σ
(wi. wrs))
Since W and W are diagonal matrices, this simplifies to:
I
W
Σ I(wrs w2
The diagonal entries of W can be represented as vector diag( (W), and the diagonal entries of W are given by Cdiag( W). Substituting these representations, we have:
||W- - WI Σ
(diag(W): - (Cdiag( (W)))
This is equivalent to the squared fz-norm of the difference between the vectors diag( (W) and Cdiag( (), giving:
W
w | |/diag( (ddiag()- (ddiag(W)]3
Substituting back W = Diag Cdiag( W)). we conclude that:
IW Diag( (Cdiag W)) 1 |diag( (W) Cdiag( W)
Lemma 6. Let A Raax and B Ruxn be diagonal matrices, then:
AB Diag (AdiaAgaaB))
Proof of Lemma 6 Since both A and B are diagonal matrices, their product AB is also diagonal matrix. The entries of the product AB are given by:
(AB),, = aizbi.j.
For diagonal matrices, all off-diagonal entries are zero, S0:
(ai.br.i if j.
(AB),,
0,
otherwise.
19

---

Published as a conference paper at ICLR 2025
Thus, the diagonal entries of AB are as, and the matrix AB
is:
Taibi
0
a262
0
AB
ag bn
where a, = a,,i and b; b..: represent the diagonal entries of A and B, respectively.
Now, let diag( [) denote the vector of diagonal entries of B, i.c.,
bj
b2
diag(B) ()
b
The operation Adiag(B) represents the element wise multiplication of the diagonal entries of A and B:
a bj 0zb2
ABia
an bn
Next, using the function Diag(-), construct diagonal matrix from this vector:
Ta
0
a2b2
Diag( Adiag(B)))
=
Ory b
Clearly, AB and (Bd(ag)))(BB)) are identical, as they both produce the same diagonal matrix with entries a;b; along the diagonal. Therefore:
AB Diag( Ad)ag(B)).
C RELATIONSHIP BETWEEN WEIGHT MATCHING AND MODEL FOLDING
Weight Matching (insswrth et al. 2023 fuses two models into one, whereas Model Folding compresses the weight tensors/matrices of single network. While inspired by Weight Matching, Model Folding addresses a distinct use case, leading to different optimization problems (K-Means vs. LAP). Notably, the Linear Sum Assignment Problem (LAP) can be framed as constrained K-Means variant, where each cluster contains exactly two vectors: one from network A and one from network B.
As an example for this discussion, consider simple feedforward network. The steps of our proposed compression algorithm involve iteratively solving the following:
C = arg min
| - C.W +
- - - 1
C
such that
C = U,(UT
where UT is clustering matrix.
Weight Matching merges two feedforward networks by iteratively optimizing:
P = arg min W P.WBall + || P, P.W.f.il
where P is permutation matrix. To connect Weight Matching with method, frame approach within the model merging domain. This begins by establishing a relationship between K-Means and the Linear Sum Assignment (LAP) problem.
20

---

Published as a conference paper at ICLR 2025
K-Means and LAP Connection. In the standard K K-Means formulation, given dataset represented as rows of a matrix E RTxd the objective to cluster these rows into & groups. This can be represented as:
=
arg min |X CX|
where Rnxn is a clustering matrix satisfying:
Each row of C corresponds single cluster assignment.
C has block diicknal structure that assigns each row of X to single cluster centroid.
The clustering matrix C be explicitly written terms of matrix U Rn* as:
C U((UU))
where U encodes the cluster assignments and centroids.
To connect this with LAP, let X the concatenation of rows from two matrices WA and W W (e.g.. weights from neural networks):
WA A
x
WB B
Now, constrain C such that:
C = P I
where:
P is permutation matrix representing a one-to-one mapping between rows of W and W B-
I is the identity matrix, allowing for exact cluster assignments during merging.
Under this constraint, C enforces a specific structure, aligning rows of WA and Wa 8 pairwise.
Substituting C into Equation we get:
P = arg min
W
W [BB]
A
WB
This is an instance of the Linear Sum Assignment Problem. Minimizing the cost:
W
A
P
W
1
W
W
equivalent to maximizing:
W A WBB
W A WB
J+ = tr
P
Model Folding. Building these results, define Model Folding for merging networks as follows:
W 1.4 W
C
W 1.4 WLB
(W41.44 + [ W44.8] - W41..]
C
Constraining C₁ to C
[P II. where P is a permutation matrix, yields the Weight Match- coordinate descent cost:
ing Ainsworth et (2023)
1 W W -WWTAAPPWW... 11.
J/ = 2 | P.W.
MODEL FOLDING FOR CONNECTING MODELS
We provide small experimental setup comparing WM WM Ainsworth al. (2023), ZipIt! Stoica et al. (2024). and our proposed method for merging networks trained on the same task and networks trained separate tasks.
21

---

Published as a conference paper at ICLR 2025
For the experiments involving the merging of networks trained on disjoint tasks, we used instances of VGG11 and ResNet18 trained on CIFARIO with 5+5 label split. All experiments were performed with REPAIR.
Merging Networks Trained on Separate Tasks.
Model
WM
Ziplt!
Model Folding (Ours)
VGG11
0.57
0.69
0.71
ResNet18
0.48
0.74
0.75
Table 2: Performance comparison for merging networks trained on separate tasks.
Merging Networks Trained on the Same Task. For the experiments involving the merging of networks trained on the same task, we used instances of VGG11 and ResNet18, 88, both trained on CIFARIO. All experiments were performed with REPAIR.
Model
WM
Ziplt!
Model Folding (Ours)
VGG11
0.89
0.87
0.92
ResNer18
0.92
0.91
0.93
Table 3: Performance comparison for merging networks trained on the same task.
D CHANNEL SIMILARITY
Models learned by SGD trend have correlated patterns or similar parameters in the weight space. Fig. I0 shows 3 x 3 filter weights in convl of pre-trained ResNet18. These filters across the first input channels and first 16 output channels ordered by the entropy of filter weight. From the plot, most filters of a channel can find at least one another similar filter in other channels, which means filter similarity may lead to structured redundancy.
Figure 10: Similar patterns in weight map of conv/ layer in ResNet18 pre-trained on Ima- geNet (Deng al., 2209)). Each small square represents the weights of a single filter in cool-warm color map, where each color of grid corresponds to weight value.
To investigate the filter redundancy within a layer, we apply weight matching activation matching from the literature (Jordan et al., 2022) to each layer of ResNet18 pretrained on CIFARI0 (Krizhevsky et al. 2009a) in Fig. 2 and on ImageNet (Deng et al., 2009) in Fig. [1 We observe two findings:
(1) The correlation score distribution varies across layers. The earlier and narrower the lay are,
the more scattered the correlation coefficients are, and only a few have high correlation coefficients. The wider and later the layers are, the more compact the correlation coefficients are, and the matching channels have high correlation coefficients. (2) In the same layer, the distribution of correlation coefficients among matched channels differs across various pre- pre-training datasets. This observation does not fully align with the claim by Chen et al. (2023) regarding the downward trend of similarity before reversal. appears that this characterization might not consistently hold across different models and pre-trained dataset.
of
D. THE IMPACT OF REGULARIZATION
In Fig. 6 the models on CIFARIO were trained without regularization, while the pre-trained ImageNet extend the comparison of folding and pruning
models were sourced from torchvision. In Fig.
22

---

Published as a conference paper at ICLR 2025
kye
Vee3 cuses
1.0
10
10
1.0
1.0
Bores
10
1.0
10
10
1.0
6.8
0.8
0.8
0.8
0.8
0.5
0.8
0.8
0.8
0.5
0.5
0.8
Comssston volue
0.6
0.6
0.6
0.6
0.6
0.
0.6
0.6
0,6
0,6
0.5
0.6
0.4
0.4
0.4
0.4
0.4
0.4
0.4
0.2
Figure 11: Layer-wise correlation between matched channels in ResNet18 trained ImageNet. We compute a layer-wise correlation matrix by matching activations between channels, then assign each channel its best match in the same layer using a greedy pairing based on the correlation matrix.
methods CIFAR10, including ResNet18 (left column) and VGGI1 (right column) models trained with explicit L, and L2 regularization. L1 regularization, in particular, promotes neuron sparsity, leading structured magnitude pruning methods to outperform model folding under these conditions. However, a comparison between Fig. and Fig. [2]sss that model folding with L2 regularization maintains the highest accuracy at higher sparsity levels. surpassing 80% accuracy. In contrast, the accuracy of the pruned network trained with L drops significantly, reaching just 33% at 0.75 sparsity.
0.9
Fold-R
0.9
Fold-R
0.8
Fold-Naive
0.8
Fole-Naive
Accuraxy
0.7
SPL1-R
SPLI-R
0.6
SPL1
0.0
SPL1
0.5
SPL2
SPL2
0.4
Test
0.4
0.2
0.2
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.5
0.6
0.7
Sparsity
Sparsity
0.8
Test Accuracy
Test AccuroC
0.7
0.6
Fold-R
0.6
Fold-R
Foid-Naive
0.5
Fold-Naive
0.4
SPLI-R
0.4
SPL1
SPL1
0.2
SPL2.R
R R
0.3
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.5
0.6
0.7
Sparsilyy
Sparsity
Figure 12: ResNet18 (left column) and VGG11 (right column) models trained with L, (top row) and L2 (bottom row) regularization. Structured magnitude pruning outperforms model folding only if training explicitly regularizes for model sparsity (L, norm). REPAIR is hardly beneficial for all structural pruning methods.
E MODEL. FOLDING ON LLMs
Table presents example outputs from both the original and the pruned LLaMA-7B models, as processed by model folding, From the responses presented in Table it is evident that when folding 20% of the parameters, the pruned model continues to perform well. In Tab. we also compare model folding with these methods on LLaMa2-7B (Touvron et al. 2023b), focusing on perplexity the WikiText2 (Merity et al. 2016) validation set and zero-shot performance across four tasks using the AleutherAl LM Harness
(Gao et al. 2024). We take the same folding sparsity as shown in Tab.
23

---

Published as a conference paper at ICLR 2025
Prume ralio
Method
Data usage
WikiTex22
BoolQ
WinoGrunde
ARC-
ARC-c
Average1
0%
LLaMA2-7B Touvron et al.
2023b
5.12
77.7
68.98
76.34
43.26
66.57
20%
Magnitude Pruning
Inf
44.8
49.8
26.22
21.93
44.52
20%
LLM-Prunnr (Ma al. 2025
Gradicuts
10.58
64.62
63.54
68.39
36.52
51.78
20%
FLAP An et hi. 2015
Calibration
6,87
71.5
08,74
70.20
36.95
61.85
Wanda, -9p
(Son (pr
ar
Calibration
8.78
63.93
70.50
40.01
61.66
20% Model Folding
17.46
62.72
61.80
49.00
26.62
50.03
Table 4: Performance of structured pruning methods on LLaMA2-7B without post-tuning, showing perplexity on WikiText2 and zero-shot performance across tasks. "Inf" represents an extremely great value. The "Average" is computed over four tasks. "Wanda_sy represents adapted Wanda method for structured pruning. Despite not using data or fine-tuning, model folding achieves comparable performance data-driven methods.
Model
Generated Sentences
Instruction: are prime numbers? Response: Prime numbers numbers are only divisible by itself. For 67. 73
example, 2, 3,5, 7, 11, 13, 17, 23, 29, 31, 37, 41, 47, 53, 59, 67,
HLMMA-7B
(ratio=0%)
Instruction: Please introduce Newion's 's second law Response: Newton's 's second law of motion states that the net force acting on an object is equal to the rate change The object's momentum. The net force acting on object equal to the rate of change of the object's momentum. The net force acting on an object equal to the rate of change of the object's momentum. Newton's 's second law of motion states that the net ferce acting on an object is equal to the rate
Instruction: What
difference hetween monkeys and gorillas? Response: Monkeys are primates that are smaller than gorillas. They are also more agile and have longer tails, Gorillas are larger than monkeys and have shorter tails. What is the difference between gorilla and chimpanzee? What the difference between gorilla and a chimpanzee? What is the difference between gorilla and chimpanzee? What is the difference hetween gorilla
Instruction: How many planets are there in the solar system? Response: There are eight planets in the solar system. The planets are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune. What is the name of the planet that is closest the sun? What the name of the planet that the closest to the sun? What the name of the planet that the closest to the sun What is the name of the planet that is the closest to the sun
Instruction: What are prime numbery? Response: A prime number number that is divisible by only two numbers. For example, 2, 3, 5, 11, 13, 15, 23, 23, 27, 31, 33, 35, 37, 41, 43, 45, 47, 51
Instruction: Please introduce Newton's 's second law Response: The second law motion Newton's 's law gravity. Newton's 's law of gravity is that every object is attracted by every other object The more the two objects are attracted, the more strongly they are attracted. Newton's law of gravity the law of gravity. What the second law of motion? What the second law of motion? What the second law of motion? What the second law of motion
LLaMA-7B (ratio=20%)
Instruction: What is the difference between monkeys and gorillas? Response: The answer is simple: monkeys are smaller animals while gorillas are larger animals. What the difference between a monkey and a gorilla? The answer is simple: monkeys are smaller animals while gorillas are larger animals. The size of the body is not the only difference between monkeys and g What is the difference between a monkey and a gri What is the difference between a monkey and a griWhat is difference between
How many planets are there in the solar system? Response: There eight planets the solar system. The planets are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Aster and Nept. The planets are arranged in order of size from smallest largest. The plancts are also arranged order distance from the from closest to farest. What difference between planets and stars? What is the difference between planets and stars? What is the difference between planets
Instruction:
Table 5: Generated examples from the original LLaMA-7B and pruned by model folding. The maximal number of output tokens is set 100 in both models.
F HANDLING RESIDUAL BLOCKS
In this subsection we discuss the behavior of Residual Blocks after compression. In a similar manner to the analysis of Normalized Blocks, we investigate the possible dependencies between the clustering matrices for different parts of the residual block and the incoming layers.
F.1 SIMPLE RESIDUAL BLOCKS
Consider a Simple Residual Block, consisting of a shortcut represented by an identity transform Wr, = I, and preceding layer decomposed using a clustering matrix U, The projection matrix is defined as:
C--1-U-1 (I-,(U,L_,) U--1) "U
24

---

Published as a conference paper at ICLR 2025
This decomposition allows for approximating the residual block while reducing redundancy in the weights. The residual block approximation satisfies:
(w(2) (w." CL1X--1) + CI-
where X;-1 is the input the block, yı is the output, and of-) represents the activation function.
The shortcut Wt, = ensures that the input x;-1 1 is directly added to the output of the main path, preserving information and facilitating gradient flow.
Decomposing w Let the weights w be decomposed using a clustering matrix U and its corresponding projection:
C = u (U)*"
U.er U
Substituting this decomposition into the residual block yields:
Y1 (c w/ (w/ -[ + cf-1x--)).
This approximation captures the effect of clustering and compressing the weights while maintaining the structure of the residual block.
Aligning Clustering Matrices. To simplify the folding process, we assert that U:-1 u!? This ensures consistency in the clustering across the residual block, reducing the need for additional transformations between layers. As result, the folding costs for the preceding layer and the current layer can be summed directly:
J 11 J1-1-
Total Approximation Error: as:
The total approximation error for folding the residual block is defined
Jo = Wiot c Wial?
where:
w
for
W
Here, Wirr combines the weights of both layers in the residual block into a single representation. This unified view allows the clustering process be applied holistically, ensuring that redundancies across the entire block are captured and reduced.
By asserting U,-1 U and summing the individual folding costs n/² and Ji-1, achieve a compact representation of the residual block with minimal approximation error. This approach ensures that the compressed residual block remains effective while reducing redundancy in the weights.
F.2 RESIDUAL BLOCKS WITH NON-IDENTITY SHORTCUTS
Consider a Residual Block with a shortcut represented by a weight matrix Wi.,, and preceding layer decomposed using a clustering matrix U,-1. The projection matrix is defined as:
C -
UU
This decomposition allows for approximating and clustering the preceding layer's weights while maintaining their representational capacity. The corresponding approximation for the residual block
satisfies:
yI (w (w/ (w("CL Wi., (I
where:
W (2) is the weight matrix of the second layer in the residual block,
w(1) is the weight matrix of the first layer in the residual block,
Wie V is the shortcut connection weight matrix,
( represents the activation function.
25

---

Published as a conference paper at ICLR 2025
Decomposition of Weight Matrices. The weights w/2) and W 1,s are decomposed using their respective clustering matrices. For _ the decomposition is:
c U (U,OTU!?)) U.!!T "
For Wt.s, the decomposition is:
Ct = U1. (Uf,, UI..) Uf,--
Substituting these decompositions into the approximation yields:
= a (cf (2) (2)T
(c,2) w ( (w/'cl_ + Ci. Wi, CL (x-I))
To simplify the folding process and ensure consistency across the layers, introduce the constraint:
Consistency Constraint and Total Approximation Error.
Ut U
This ensures that the same clustering matrix is used for both the shortcut weights Wt, i,s and the second
layer's weights w By adding the individual folding costs j(() and Jl,a, ensure that Lemma | holds, leading to the total approximation error for the residual block:
Ju
5(2) + J1.s-
Unified Approximation for Residual Blocks. compactly as:
The total approximation error can be expressed
Jos
W not
C Will
where:
W lot Wi,
w (2)
Here, W wt combines the shorteut weights Wt, and the second-layer weights w into single matrix. This unified representation allows the folding process to be applied holistically, reducing redundancies across the entire residual block.
The decomposition of weights residual blocks with non-identity shortcuts introduces a consistent clustering mechanism for both the shortcut and the second layer. By ensuring that U,,, ufa), we maintain alignment the clustering process, leading to compact and efficient representation with minimal approximation error.
G HANDLING BATCH NORMALIZATION LAYERS
Batch Normalization layers, when combined with linear layers, introduce additional scaling and normalization operations. One special case layer consisting of linear block followed by Batch Normalization block, formally defined as:
21+1 Wino W(x--)),,
where:
Wi: weight matrix of the linear block,
Σ, Batch Normalization scaling matrix,
Σn: Batch Normalization normalization matrix,
Wit weight matrix of the subsequent layer,
a (-): activation function applied element-wise.
26

---

Published as a conference paper at ICLR 2025
A design choice in handling such layers is to decompose ,, En, and W separately while preserving the original structure of the layer. This ensures that the scaling, normalization, and linear blocks treated distinct functional units. The decomposed approximation for the layer then be expressed as:
71+1 2/+1 W441 CCC, C.
where the projection matrices C., Cn, and C₁ are defined as:
C, U.(U U,M,,
C, U, U,,
U,M.,
C = U,(U7U,) U7 U,M.-
Here, U,, Un. and U, are clustering matrices, and M,, M,, and M, are normalization terms.
Clustering Assumptions. To simplify the decomposition and ensure alignment across the layer components, we impose the following consistency constraint:
U, U, U².
This assumption ensures that the same clustering structure is applied to the scaling, normalization, and linear blocks, leading to unified decomposition. Under this assumption, the approximation becomes:
11+ W Cf (UU U.M
where W represents the intermediate scaling factors.
Applying Diagonal Properties. Using Lemma 3 we observe that the normalization and scaling matrices can be represented as diagonal matrices:
ž++ = W Cf (U,Do(UDUD
(M,diaag (E.) M.x-))))))))))))
Furthermore, by applying Lemma
we rewrite this expression as:
2 = W C, o (Diag(C,,iagg W W..)) Diag( Cidiag( (,.iiagE,,))Cr (
This shows that the diagonal structure of the scaling and alignment matrices is preserved through the decomposition, maintaining the original behavior of the Batch Normalization block.
Compression Cost. According to the definition of the Model Folding problem and using the properties stated in Lemma 5 the compression cost for the layer can be expressed as:
Jest | Croell
where:
W toz
W [+1
W diag E.) diag (o,))
This cost quantifies the approximation error introduced by clustering the weights, scaling, and normalization matrices while preserving the layer's functional structure.
By decomposing the Batch Normalization and linear blocks separately and aligning their clustering structures (U, U, Ui), we ensure that the original diagonal properties of the scaling and normalization matrices are preserved. The resulting compression cost captures the overall error of folding the entire layer into a compact representation.
G. ALGORITHMIC DESCRIPTION FOLD-AR
The Fold-AR algorithm for single layer combines the Batch Normalization components and layer weights into a compact representation, followed by clustering to reduce redundancy. The steps are described in Algorithm [
27

---

Published as a conference paper at ICLR 2025
Algorithm 1 Fold-AR for Single Layer
Require: g. En, W. W4+1
> Input components of the layer
1: Compute the normalized weight matrix: Wi En Wi
2: Construct the combined weight matrix: Wio f [WT] [+1
W diag [.,]
3:
Solve the clustering problem:
+ arg min W Iot U(U U W
subject to U" E {0, 1j and m m<n
4: Update the scaling matrix: E. f (U"
5: Update the second-layer weights: W U /+1
6: Update the current-layer weights: W, + (U'U)-'U'
7: for .,..., do
Adjust scaling factors for each cluster
8: Compute cluster size: No t -E,10U (U,. =1)
> [I(.) is the indicator function
9: Compute intra-cluster correlation:
WIS:: Wi
Elc
I(U U,e = 1)I((3 # 1)
N2 -
No 1.
W1,z
10: Update the scaling factor for cluster
N:
(Eo)ee + (Eolac VN₂ + (N? E(c
11: end for
EXPLANATION OF KEY STEPS
1. Combining Normalization and Weights. The normalization matrix Σn is diagonal, and multiplying it with the weight matrix W; produces the normalized weight matrix:
W W.
This step integrates the normalization operation into the weights of the current layer, reducing the complexity of subsequent computations.
Construction of Combined Weight Matrix.
The combined matrix W uo defined as:
W
[W 7+1
Wi
diag(E.)]-
This matrix aggregates the second-layer weights (WT+1) ). the normalized current-layer weights (W)) and the scaling factors (diag(E.)) into single representation, preparing them for joint clustering.
3. Clustering. The projection matrix is computed by solving the clustering problem:
U arg min [ux. U)U' - 1|
subject to U E {0, 1p and m < The clustering minimizes the reconstruction error by projecting the combined weights into a lower-dimensional space defined by m clusters.
4. Scaling Adjustments. To ensure proper scaling within each cluster, the diagonal elements of E, are updated. For each cluster e, the adjustment considers the size of the cluster (N.) and the intra-cluster correlation (E(c)):
N EE)c. (Ex), VN. + (N? N.)E.c.
The intra-cluster correlation E(c] is computed normalized dot product, capturing the redundancy among the weights within the same cluster. This adjustment preserves the scaling properties of the original layer.
28

---

Published as a conference paper at ICLR 2025
5. Final Updates. The current-laye weights W, and second weights W are updated to align with the clustered representation:
W - (U'U))''', W - U
These updates ensure consistency between the clustered weights and the projection matrix U.
This algorithm combines clustering, scaling adjustments, and weight updates to compress the layer while preserving its functional properties. The clustering step minimizes redundancy, and the final updates align all components of the layer with the clustered structure.
H FOLDING SIMILAR CHANNELS IN MLPs
For fully connected networks, where two successive layers are defined as:
X = o (W and X1+1 = al (Wi+11)),
where X; represents the activations of layer W and W are the weight matrices, and the activation function. The channels of the layer are defined as the coordinates X1,; of the vector x Each channel corresponds to a specific dimension the activations.
The folding cost Jr for the 1-th layer is defined as:
J1 = W. C.will++ W
C.
where C₁ is a clustering matrix. This cost function represents the optimization objective minimize the approximation eтror introduced by folding (clustering) the weights of the I-th layer. The first term measures the reconstruction error for the weights Wt, while the second term measures the reconstruction етrоr for the weights W under the transformation C, Together, these terms ensure that clustering transformation preserves the structure and relationships of the weights across layers.
From the perspective of K-Means as a matrix decomposition problem, the grouping of scalar weights into vectors is defined as follows:
p
P2
W
and W4+1
(q1
q2
qnl
P
where PI are the rows of W, and qu are the columns of Wi-1 These groupings reflect the natural structure of the weight matrices in fully connected layers:
Each row of W, represents the weights associated with specific output channel of layer (.
Each column of Wi+1 represents the weights associated with specific input channel of layer I 1+1.
In this formulation, the rows P and columns qi are treated vectors to clustered by the matrix C₁ which aligns with the K-Means decomposition perspective. The clustering matrix C₁ maps these weights into representative clusters, preserving the relationships between input and output channels across layers while enabling efficient compression.
I FOLDING SIMILAR CHANNELS IN CONVOLUTIONAL LAYERS
For convolutional layers, two successive layers can be defined as:
X = (Wi x111) and X1+1 = oWW++ * Xi).
where X is 3-dimensional feature tensor with values x²() x- The first dimension, co, corresponds to the output channels, while and j represent spatial pixel locations. The 4-dimensional weight tensor W; has values ()())
G.j. where:
29

---

Published as a conference paper at ICLR 2025
Co corresponds to the output channels of Xi,
Cj corresponds to the input channels of 2/-1-
To simplify and compress the network, decompose the weight tensor Wi such that output channels of X (i.e., the values x1() folding problem is defined as:
corif for Co = 1,... Cour), which are similar in some sense, are merged. This
J1 I ml IM++1 W44 o Cilli
where C corresponds to 1 x convolution parameterized by the clustering matrix C₁, with
c()) 1,1,1 Ci,c.e"
From this definition, it follows that:
Jr = I| - C, W W W C/ IT:
where the weight tensors Wi and W++1 are mapped to matrices W, and W Wit1 follows:
vec(MW 1.1.3 )
vec(W )"
vec(M (e() L.nie
1.2.11
veo(WL))
vec (W (1) T
vee
2.2.1
W,
(wo 1. )
vec(W 2.:.!
vec(W Car
This means that each convolutional filter contributing to output channel is flattened and stacked into a vector, forming c,-th row of the matrix W,. Similarly, for W1+1, each filter associated with the e--th input channel is flattened and stacked into vector, forming column of the matrix W
vec(W (1-))) vec (t:-1 2,1.
vec (ec(M)((t)) (114))) vec(M 2,2
rec(W(t+1 (++1)
I. vec(W ((+
W41
vec(W
vec (ec(l(++1) 2,.)
vee vec(Witt))) Caasts
Comss bay
From the perspective of K-Means as a matrix decomposition problem, the grouping of scalar weights into vectors defined as follows:
p
W,-
and
W441 [q
qm),
where:
p
vec( (. (ec
vec(W!l) )
and:
q (eo(()((t)))) 1839
vec(WWit (++1))
vec(W
Rusdin yr)
In this formulation, the rows P. of W, and columns q of W are grouped into clusters for the folding process, aligning with the K-Means decomposition perspective.
J FOLDING SIMILAR CHANNELS IN LLAMAMLP AND LLAMAATTENTION
J.1 FOLDING SIMILAR CHANNELS IN LLAMAMLP
The LlamaMLP module is composed of three sub-layers: gate_pra sete__roj, up_proj. and down_proj. These sub-layers define the structure and functionality of the MIP, with the main computation pipeline expressed as:
Comn_projfac__fnsaate_ppoo
up_proj(z)).
We cluster similar channels in both the output channel and input channel of each sub-layer.
30

---

Published as a conference paper at ICLR 2025
Input Channel Folding. To fold the input channels of LlamaMLP, we simultancously consider the input dimensions of both gate_pros and up_proj layers, as they collectively define the effective input to the gate_up sub-layer. The input channels of gate_proj and up_proj are clustered respectively using methods similar to those applied in standard MIP layers.
Output Channel Folding. To fold the output channels of LlamaMLP, we first consider the output channels of both gate_proj and up_proj by clustering and adjusting the input channel of the down_proj. Subsequently, we adjust the output channel of down_proj according to the residual connection used outside of LlamaMLP.
J.2 FOLDING SIMILAR CHANNELS IN LLAMAÅTTENTION
The LlamaAttention module consists of four primary sub-layers: q_proj, k_proj, v_proj, and o_proj. These sub-layers define the query. key, value, and output projections, respectively. For clarity and simplicity, we conceptualize q_proj, k_proj, and v_proj as a unified sub-layer referred to as q_k_v, which computes the intermediate representations required for attention calculations. The o_proj sub-layer processes the final output of the attention mechanism. We treat the attention head as the structure to be folded in LlamaAttention. By reshaping the weights of each sub-layer into MLP-like tensor, we can cluster similar heads, similar to how it is done for a standard MLP layer.
For all configurations of LlamaAttention, including Multi-Head Attention (MHA) and Grouped Query Attention (GQA), the weight shapes of the k_k_ sub-layer differ:
In MHA, the weights for q. k, and projections share the same shape: [num heads X head dim, hidden_ _szee].
In GQA, the weights for and v projections have the shape: [num heads x head dim, hidden_sizzeee size].
Output Channel Folding. When performing output channel folding for the LlamaAttention layer, the clustering of the o_proj sub-layer's output channels is dictated by the residual connection outside of LlamaAttention, ensuring alignment with the clustering results from previous modules. Specifically:
The o_proj weights, originally shaped as [num heads x head dim, hidden hidden_size], are reshaped into [num_heads head_ hidden_size) clustered along the first dimension (num_heads)) and then reshaped back to their original form.
For clustering within the q_k_v sub-layer, the weights for q, k, and v are reshaped into muum_heade head_dim, didd_dimmat hidden_size (or mmum_vy_heade head_dimg hidden_size) for and in GQA) and clustered along the first dimension (num_heads or num_kv_heads). After clustering, the weights are reshaped back to their original dimensions.
Input Channel Folding. To perform input channel folding, the focus is on the input channels of each of their weights
k, and v weights. Since these weights share the same input (ondde_ssizee of their respective matrices. This ensures that
is clustered along the first dimension the clustering process respects the shared input representation across the q_k_v sub-layer while maintaining the integrity of the attention mechanism.
K COMPARISON WITH KNOWLEDGE DISTILLATION
We evaluated some data- data-free knowledge distillation (KD) methods (Micaelli & Storkey 2019 Chen al. 2019 Fang al. 2020 Yu et al., (023), on an NVIDIA A A100 GPU, for all methods using the same pre-trained teacher model, data loader, and student model setup for consistency. The full model a ResNet18 pre-defined by torchvision and trained on CIFAR1O, while the student models for each KD method share the same architecture but differ in the number of channels across all layers to achieve the desired sparsity levels. Specifically, in ResNet18, the number of output channels for all blocks is multiple of 64, which is also the number of output channels in the first convolutional layer. To reduce the model's channel dimensions, we scale this base hyperparameter by a reduction factor, effectively reducing the width of all layers proportionally. The following table presents the test.
31

---

Published as a conference paper at ICLR 2025
accuracy of compressed by KD methods and model folding CIFARIO test dataset. The time taken to achieve each accuracy is provided in parentheses next to the corresponding accuracy value. From the table, it is evident that the proposed model folding achieves model compression within seconds, even at high sparsity levels, compared to other KD methods that require tens of hours to complete.
Sparsity
Full model
10%
25%
50%
70%
ABM (Micuelli & Storkey 2019
94.72
93.30 (17h19m)
91.99 (16h8m)
89.42 (15h30m)
85.43 (13h23m)
DRAD
Chen al.
2019
94.72
93.79 (2h31m)
93.52 (2h3m)
92.04 m)
89.67 (1h54m)
DAFL
Fang
94.72
71.73 (16h488m
77.80 (15h39m)
68.06 (15h1 9m)
53.86(76h34m)
SpacestipNNet
Yu 2013
94.72
94,50 (42h333)
93,99 (40h33)
92.96 (37h57m)
91.53 (27h10m)
Model Folding (ours)
94.72
94 (56.35s)
92 (53.556)
88 (55.758)
82 (54,95s)
Table 6: Performance comparison of knowledge distillation and model folding, showing accuracy (%) and runtime (in parentheses). The sparsity levels indicate the percentage of weights pruned
L INFERENCE SPEED OF FOLDED MODELS ON EDGE DEVICES
We apply model folding on LeNet5 model pre-trained on FashionMNIST with different sparsity, and then evaluate the folded models on NVIDIA Jetson Nano, ESP-EYE, and Arduino Nano 33 BLE. All models are converted and executed as a float32 Tensorflow Lite model all devices.
Sparsity
10%
50%
Plash
Rustine
Fash
Rustime
Flaii
Ims
ESP-EYE
(Spressi Systeus
2591ms
1868ms
55.7K
1532ms
1156ms
36.5K
1.2M
Arduino
Nam STRE Saae Amiiine 2074
50 5K
3.4M
37266m
55.7K
2.8M
48
2900ma
36,5K
Table Performance and resource usage at various sparsity levels across devices, with detailed breakdowns for runtime (ms), RAM usage (K), and Flash storage usage (M).
M DEEP INVERSION SAMPLE IMAGES
Deep Inversion (DI) Yin et al. 2020 generates synthetic images from the uncompressed network by optimizing noise match the internal statistics stored in BatchNorm layers. These images, exemplified in Fig. [3 which reflect the original data's 'ss statistical properties, are used during model folding restore data statisties in the compressed network, ensuring accuracy without requiring external data.
N FURTHER RELATED WORK
Model folding intersects with several established approaches in model compression, network archi- tecture optimization and model merging. This section outlines key related works that inspired the development of model folding, highlighting both their contributions and limitations.
N.1 MODEL COMPRESSION
Model compression techniques reduce models' size and computational requirements while main- taining or minimally sacrificing performance. Various methods have been developed. Most can be classified as pruning, quantization, knowledge distillation, and low-rank factorization. Traditional pruning techniques (Han et al.) 2015): LeCun et al.) 1989 Li et al., 2016b) Hassibi et al.) 1993] Entezari & Saukh 2020), structured or unstructured, involve removing weights, neurons, or filters that are deemed less important, typically measured by the magnitude of their contributions (e.g... Li or L2 norm) (Entezari & Saukh 2020 Li et al. 2011 |
Cheng et al. 2023). While effective in reducing the size of the model, pruming often leads to degradation of performance that requires fine-tuning complete retraining of the network (Cheng et al.) 2023 Han et al.] 2015 Frankle Carbin 2018
Frantar & Alistarh
2022
He al. 2018). Quantization (Gupta et al. 2015 Zhou al. 2017
32

---

Published as a conference paper at ICLR 2025
Figure 13: Sample images generated by Deep Inversion (Yn et al. 2020) using ResNet18 trained on CIFARIO0. These images generated from the uncompressed network and used in model folding to restore data statistics in the compressed network.
et al. 2016a) reduces the precision of the numerical values in model, from floating point lower- bit representations (e.g.., 8-bit integers). This approach significantly reduces the model's 's memory footprint and speeds up computation, especially when combined with hardware accelerators designed for low-precision arithmetic (Gholami et al., 202T). Like pruning, post-training quantization may also require fine-tuning to restore model performance. Knowledge distillation (Hinton al. 2015) trains smaller model, called the student, to replicate well-trained larger model, called the teacher, by mim- icking the output of the teacher model, which transfers knowledge between the teacher model and the student model. While effective in transferring knowledge and reducing model size, even approaches that eliminate data dependency using synthetic samples or adversarial distillation (Micaelli & Storkey 2019; Chen et al. 2019 Fang et al. 2020; Yu et al., 2023 Haroush et al. 2020), the training process for knowledge distillation can be computationally expensive and time-consuming (Hinton et al. 2015) Gou et al.; 2021; Martinez al. 2021). Moreover, knowledge distillation often assumes substantial differences between student and teacher model architectures (Gou et al., 2021). Low-rank factoriza- tion decomposes weight matrices into lower-rank matrices reduce parameter size through such as singular value decomposition (Ren & Zhu 2023] Horvath et al. (024)) or tensor decomposition
(Lebedev al. 2015 Kim al.) (0016). Approaches such mixture of experts (Jacobs al. [991]
Shazeer et al. 2017). bubspace-configurabe networks (Wang et al. 2024, Papst et al.) 20224) and
resource-efficient deep subnetworks (Corti et al. 2024bba a). explore dynamic model reconfi acconiiguration to minimize the number of active weights during inference.
Structured pruning. Structured pruning of particular interest because it removes entire structures (such as neurons, channels, or layers) (Entezari & Saukh 2020 Li et al.) 2016b; Luo et al., 2017a Hu af., 2016 Wen al., (016) rather than individual parameters, reducing model complexity while maintaining even improving performance. This method is especially valuable for enhancing efficiency with easily implemented acceleration in resource-constrainee
environments (Wang et al.,
2020: Liu et al. 2024). However, structured pruning typically requires additional retraining or
fine-tuning (He et al.] 2017: Liu et al. 2024 Luo et al. 2017 Recent work by Theus et al.) (2024) combines model pruning and fusion using Optimal Transport theory, demonstrating that significant portion of pruning accuracy can be recovered without access to training data. However, the impact of pruning the model's data statistics and how to recover them is not addressed.
33

---

Published as a conference paper at ICLR 2025
N.2 MODEL MERGING
Model merging combines multiple models to generate a single, unified model which leverages the strengths and diversity of each individual model. It particularly benefits ensemble learning and distributed training scenarios, where models are trained independently on different subsets of data or across different devices. Merging can be achieved by averaging the parameters of model trained independently. Recently, multiple methods have been developed to enhance model performance and robustness. MTZ (He et al. 2018) and Ziplt! (Stoica et al. 2024) compress multiple models pre-trained for different tasks by merging them through neuron sharing. Model soup (Wortsman ct al. 2022) averages the weights multiple fine-tuned models from same initialization to improve accuracy and robustness without increasing inference time. Taking permutation invariance of neural networks into account, a finding (Entezari et al. 2022) shows the interpolation between models trained with SGD has barrier. Git Re- Basin (Ainsworth al. 2023) utilizes activation matching and weight matching to achieve permutated alignment between models trained from different initialization. REPAIR (Jordan al., 2022) mitigate variance collapse problem while aligning neurons by rescaling the preactivations of fused models. PAPA leverages a population of diverse models trained on different data variations and slowly pushes the weights of the networks towards the population average Jolicoeur-Martinen, et al. 2024). A recent work (Yamada et al. 2023) shows that for model merging on different datasets, using original or condensed datasets during the model merging process can significantly improve accuracy. However, those methods do not consider model efficiency and internal parameter redundancy. Another recent work (Theus al., (024) achieves intra-layer model fusion by integrating optimal transport (Monge) 1781, Kantorovich] 2006) Singh & Jaggi 2020) to fuse computational structures in the model without fine-tuning. We note that this approach is orthogonal to the problem solved in this paper, we do not consider intra-layer dependencies.
Merging multiple computational units. Merging computational units has been extensively explored in ensemble methods. Wortsman al. (2022) demonstrate that combining multiple models fine-tuned from the same pretrained initialization enhances both accuracy and robustness. Ainsworth et al. (2023) extend this approach to models trained the same data with different initializations, albeit with some accuracy loss. Jordan et al. (002) improve upon Git Re-Base Re-Basin by adjusting batch normalization layers where applicable. IFM Chen al. (2023) and ZipIt! [Stoica al. (2024) focus merging multiple computational units within a single model, pioneering this approach.
34


Reviews:
[{"review_id": "CQAFIXI1qZ", "reviewer": "Reviewer_mvsb", "summary": "The paper proposes a new training- and fine-tuning-free method for neural network compression. Inspired by recent research on neuron alignment, in particular the weight matching algorithm [Ainsworth et al., 2023], the method essentially works by applying k-means vector quantization to cluster similar neurons (i.e., rows or channels of a weight matrix/tensor) together, and applying an additional data-free REPAIR (Jordan et al. (2022)) procedure to preserve the activation statistics of the original network. Experiments on convnets (e.g., ResNets, VGGs) demonstrate improved accuracy / sparsity tradeoff against other SOTA data-free pruning methods (e.g., FM (Chen et al., 2023), and INN (Solodskikh et al., 2023)). Experiment on an LLM (LLaMA-7B) shows comparable / somewhat worse performance than other pruning methods that require data/finetuning.", "questions": "As mentioned earlier, I find the connection between weight matching (Ainsworth et al., 2023) and the proposed VQ method to be somewhat lacking, and the former seems more like an inspiration on which the method is loosely based. Do the authors suspect a similar effect of alignment to be achieved through quantization, such that distant modes can be connected with low loss barrier in between them (as in (Ainsworth et al., 2023))?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I think conceptually connecting the research and recent methods in neuron alignment / neural network symmetry (Yamada et al., 2023; Ainsworth et al., 2023) to the problem of model compression is somewhat novel and deserves more attention, although it certainly has been done many times, see e.g., [Zhou et al., 2018](https://arxiv.org/abs/1804.05862) and the paper [Chen et al., 2023](https://arxiv.org/pdf/2310.06756) cited in the current work. Methodologically the contribution seems fairly incremental (i.e., weight quantization, extension of the existing REPAIR (Jordan et al. (2022)) method).\n\nThe writing quality and clarity is mostly good; the experiments/figures are informative. \n\nI rate the significance as moderate. In terms of practical impact, the method is relatively easy to understand and implement, and obtains competitive performance among other data- and tuning-free approaches. However, I was hoping to see more in terms of bridging the conceptual understanding of neural network symmetry and compression, e.g., whether the proposed *model folding* method (essentially VQ) compares/relates to the weight matching of (Ainsworth et al., 2023), and whether it also enables some degree of alignment or mode connectivity b/w two randomly pretrained neural networks.", "weaknesses": "1. The paper is missing related literature and baselines on model quantization. A simple Google search suggests quite a few related papers, e.g., using vector quantization for model compression [[Martinez et al., 2021]](https://openaccess.thecvf.com/content/CVPR2021/papers/Martinez_Permute_Quantize_and_Fine-Tune_Efficient_Compression_of_Neural_Networks_CVPR_2021_paper.pdf) and post-training quantization [[Nagel et al., 2021](https://arxiv.org/pdf/2106.08295)] which is also training-free.\n\n2. Some of the writing can be improved:\n  - The section on Fold-AR is a bit hard to understand; it's unclear what exactly is being proposed. Perhaps an algorithm box can help.\n  - The paper repeatedly talks about \"merging similar channels\" as the basis of the model folding algorithm, without defining it for various network architectures considered (especially outside the context of CNNs). This should be clarified, as how the scalar weights are grouped into vectors can have a big performance impact on VQ.\n  - Some typographical errors: Line 278: \"The variance ratio of the l-the\", \"l-the\" -> \"l-th\"; unfinished sentence on Line 414: \"Fold-DIR ourperforms Fold-AR as the cost of generating a batch of synthetic images and a forward pass through the network.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Os2z337Ax9", "reviewer": "Reviewer_TqyN", "summary": "The paper presents Model Folding, a data-free and fine-tuning-free model compression technique that merges structurally similar neurons within a neural network. This approach contrasts with traditional methods that often require access to training data or fine-tuning after compression.  Experiments on standard benchmarks, including ResNet and large models like LLaMA-7B, show that Model Folding achieves high sparsity while maintaining competitive performance with data-driven methods and outperforming recent data-free compression techniques.", "questions": "1. Is there a speed comparison of inference on different devices after compression of the model?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method has both theoretical justification and empirical support, demonstrating that k-means clustering is an optimal method for weight fusion in a data-free manner. Results from benchmarks like ResNet18 and LLaMA-7B show that model folding achieves performance on par with or surpasses existing data-driven and data-free compression methods, particularly at high sparsity levels.\n\n2. Model folding is designed to be completely data-free, which differentiates it from other compression methods that typically require fine-tuning with original data. The authors propose two data-free alternatives for adjusting internal data statistics—Fold-AR and Fold-DIR—based on approximate and deep inversion-based techniques. It presents significant improvements over existing state-of-the-art data-free methods, demonstrating its applicability in compressing large-scale models effectively", "weaknesses": "1. More data-free and training-free references are needed, such as [1]\n\n2. Lack of sufficient experimental results on compression ratio and speedup ratio.\n\n[1] Haroush, Matan, et al. \"The knowledge within: Methods for data-free model compression.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsZIQrh5uj", "reviewer": "Reviewer_Ao8L", "summary": "This paper presents a data-free and fine-tuning-free method for compressing deep neural networks. This is achieved by identifying and merging redundant parameters through a similarity-based approach, followed by data generation with DeepInversion for further repair. The method is flexible and can be applied to both vision models like ResNet by merging channels and LLMs by merging linear weights, demonstrating effectiveness across different architectures. Extensive experiments on ImageNet, CIFAR-10, and CIFAR-100 highlight the method's robustness, with results clearly illustrated through figures.", "questions": "Please see weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Previous methods typically rely on data-driven fine-tuning to restore the performance of compressed models. This paper addresses this limitation by introducing a data-free method that leverages generated data and an efficient activation repair process to recover model accuracy, making it highly practical for real-world applications.\n2. The paper includes extensive experiments on both vision and language models, with clear visualizations and figures that offer detailed insights into the method's performance. Notably, at high sparsity levels (>40%), the proposed method achieves significantly better accuracy compared to prior baselines, such as IFM and INN.\n3. The presentation is good, with a clear illustration of the entire pipeline in Figure 1. Additionally, the method is straightforward to implement and highly reproducible.", "weaknesses": "1. My main concern with this submission is the limited technical novelty. The work largely combines existing methods, including similarity-based model merging for compression [1], model inversion for data generation [2], and REPAIR [3] for statistics alignment. While these components are integrated effectively, the key contributions could be further emphasized to distinguish this work.\n2. Although the paper claims the method is data-free, this feature is primarily enabled by DeepInversion rather than a new development within this work. Clarifying the unique aspects of this approach compared to existing data-free methods would strengthen the contribution.\n3. The folding results on LLaMA-7B are not particularly competitive, with a Wiki PPL of 13.33, compared to other pruning methods like LLM-Pruner (PPL=10.53) and Wanda (PPL=8.22). The claim that \"model folding achieves comparable performance to data-driven methods\" may be overstated due to this significant gap in PPL.  \n4. SliceGPT [4], which similarly merges parameter matrices via a learnable transformation matrix, is another \"merging\"-based method and should be considered as a relevant baseline for comparison.  \n\nMinor Typos: L173 - activation matching matching\n\n[1] Chen, Yiting, Zhanpeng Zhou, and Junchi Yan. \"Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory.\" arXiv preprint arXiv:2310.06756 (2023).  \n[2] Yin, Hongxu, et al. \"Dreaming to distill: Data-free knowledge transfer via deepinversion.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.  \n[3] Jordan, Keller, et al. \"Repair: Renormalizing permuted activations for interpolation repair.\" arXiv preprint arXiv:2211.08403 (2022).  \n[4] Ashkboos, Saleh, et al. \"Slicegpt: Compress large language models by deleting rows and columns.\" arXiv preprint arXiv:2401.15024 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qimDfymeLU", "reviewer": "Reviewer_AJd2", "summary": "This paper introduces a method to merge the columns in the weight matrix in a data-free manner. The method is composed of two parts: first, using the clustering method to find and merge the weight matrix. Then, based on the observation that the variance ratio after compressing the model would collapse or overshooting, the authors propose the data-free Fold-AR and Fold-DIR to inverse the model and repair the model. Experiments show that the proposed method can achieve better results than previous data-free methods, especially at high sparsity ratio.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Rigorous theoretical analysis of the proposed method\n2. A comprehensive framework for merging and repairing the model under the data-free setting, also without fine-tuning.\n3. The writing of the paper is good", "weaknesses": "1. There are plenty of works [1,2,3,4] about how to compress the model under the data-free setting, including the DeepInversion, which is mentioned as part of the method in this paper. These papers are all not mentioned and compared in the paper.\n2. The experimental results show that the authors’ proposed methods experience significant performance drops compared to finetuning-based methods. Given that the cost of finetuning remains acceptable but the performance drop is not neglectable, it’s unclear why a tuning-free setting is necessary (since we can get the synthetic by methods like DeepInversion).\n3. For methods on LLaMA, [5] also propose a model merging method to compress the model without fine-tuning. Please compare to this paper.\n\n[1] Zero-shot knowledge transfer via adversarial belief matching.  \n[2] Data-free learning of student networks.   \n[3] Data-free adversarial distillation.  \n[4] Data-free knowledge distillation via feature exchange and activation region constraint.   \n[5] Shortgpt: Layers in large language models are more redundant than you expect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)