Cut Less, Fold More: Model Compression through the Lens of Projection Geometry
Authors: Olga Saukh
Dong Wang
Haris Šikić
Yun Cheng
Lothar Thiele
Venue: ICLR 2026 Poster (2026)
Keywords: model compression
structured pruning
model folding
projection geometry
Primary area: optimization
TL;DR: Folding - low-rank, geometry-aware projection - typically outperforms structured pruning for calibration-free compression, supported by a one-rank slack guarantee and results on  >1000 CNN/ViT checkpoints

Abstract: Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1'000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training). We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate–high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Published as a conference paper at ICLR 2026
CUT LESS, FOLD MORE: MODEL COMPRESSION THROUGH THE LENS OF PROJECTION GEOMETRY
Olga Saukh $ Dong Wang", Haris Šikic Yun Cheng", Lothar Thicle*
Graz University of Technology. *Complexity Science Hub, Austria "Swiss Data Science Center, 'ETH Zurich, Switzerland
(saukhē, dong.wange, haris. siiiisssiiisstudntt hitiiiastuueetttturaa...... yun. chengesdss.....ooc etcls....ehcc.co. ch, thiele@tik. ee. ettzz.
ABSTRACT
Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, evaluate >> 11000 check- points spanning ResNer18, PreActResNer11, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimiz- ers, learning rates, augmentations, regularization, sharpness-awan training), as well multiple LLaMA- familly 60M and 130M parameter models trained on C4, We show that folding typically achieves higher post-compression accuracy, with the largest gains moderate -ighh compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry- aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.
INTRODUCTION
Neural network compression critical for deploying models in resource-constrained environments. Common approaches include quantization, which reduces the precision of weights and activations, and knowledge distillation, which transfers information from a large teacher model to a smaller student model. In this work, focus the class of calibration trree post-training structured compression methods that optimize the model architecture itself without access to training data. Among these, the most widely used is magnitude-based pruning, which prunes tensor elements according their magnitudes, using them as proxy for their contribution to model accuracy (Han et al.) 2015 Mishra al. 2021 Lu et al. 2023 Ding et al. 2024 Bambhaniya al. 2024). When combined with line-tuning or lightweight BatchNorm reset (Saikumar & Varghese 2025). this approach achieves significant compression rates with negligible accuracy loss (Kurtic et al.) 2022, Sanh al., 2020).
In contrast, the recently introduced model folding clusters similar weights and ties them together, providing an approximation of the original network (Wang et al.) 2025). Both pruning and folding reduce parameter count but differ fundamentally: pruning removes weights entirely, while folding preserves them in merged representations.
In this work, develop unified theoretical and empirical framework to compare pruning and folding through the lens of orthogonal projections in parameter space. We show that both compression methods can be viewed as projections onto -ivee-dimeniional subspaces, but with crucial differences in geometry: pruning corresponds to axis- axis-aligned coordinate projections, while folding projects onto cluster-structured subspaces that retain directional information.
At a high level, both pruning and folding compress the weights of a model. We show that for any pruned solution there exists folded alternative that is almost snall-using one extra component

---

Published as a conference paper at ICLR 2026
in the compressed representation -yt is strictly closer to the original weights (smaller Frobenius norm), which in turn bounds the change in the network function. Intuitively, folding merges weight vectors with similar directions rather than zeroing coordinates, so the compressed model stays closer in behavior to the initial network.
Empirically, we perform comprehensive calibration-free study over > 1'000 checkpoints spanning CNNs and ViTs on CIFAR-10 10 and ImageNet- trained under diverse hyperparameter choices (optimizers, learning rates, augmentation, regularization, sharpness-aware training). We also train and process 18 LLaMA-family models with 60M and 130M parameters on C4, by varying learning rates, warmup lengths, and weight decay strength. After compression and also followed by lightweight and full fine- tinietuning folding typically attains higher cosp-eompresionn accuracy. with the largest gains moderate to high compression. The margin narrows, and can occasionally reverse, at very low compression or under specific training setups, but the overall trend is consistent with our theoretical analysis. Our projection-basee perspective opens new directions for designing compression methods that explicitly optimize for functional closeness. This paper makes the following contributions:
We introduce unified projection framework that casts pruning and folding as orthogonal projections onto, respectively, axis-aligned and cluster-structured subspaces. We prove that at a compression rank difference of one, folding achieves smaller parameter reconstruction error and tighter function penttioo-pettuuaationn bounds under mild smoothness assumptions.
A large-scale evaluation across I '000 checkpoints and diverse hyperparameters, covering CNNs and ViTs on CIFAR- 10 and ImageNet- K, as well as LLLMA-60M and LLaMA- 130M on C4. In addition, we use post-compression lightweight LayerNorm reset for ViTs, or full-fine- full-fnnetuuning to show that the strong performance of folding is preserved in these settings.
We show that folding is geometry-aware alternative that is often superior in practice, with clearly identified regimes (e.g., moderate- high compression) where its advantage is most pronounced, and corner cases where the gap narrows.
We discuss related work in Appendix C however, the main text already positions pruning and folding within our projection framework and clarifies the novelty of our approach.
UNIFIED FRAMEWORK FOR PRUNING AND FOLDING
2.1 PRELIMINARIES AND DEFINITIONS
We consider a neural network with input æ E Rd. We assume ReLU activations and normalization layers (e.g., BatchNorm LayerNorm) are present.
To develop the theoretical framework, we focus on compressing a single layer at a time. This layer has p inputs and outputs with its parameters collected in matrix W Rmmx, A row wii) of W is denoted as the ith parameter vector with individual weights w(i, j). Since all other network parameters are treated as fixed, the network function can be expressed [a;; W), which is trained to minimize loss function L(W).
We assume that the loss function L is Lipschitz continuous, i.e., there exists constant & >> 0 Ouchh that ([(W,) - [.(22) S k |[ - W2llF
(1)
for all admissible parameter matrices W and W2. The Frobenius norm of a matrix is defined as ||-|||- EE., la:yl? P. that is, the square root of the sum of the squares of its entries, or equivalently, the & f2-norm of the vectorized matrix. This Lipschitz condition controls the change in loss with respect parameter perturbations.
Orthogonal Projection. We formalize structured pruming and model folding as orthogonal projec- tions in parameter space. A matrix C E mR is an orthogonal projection if C C C² i.e., it is symmetric and idempotent. Such projections map any parameter vector to its closest point (in the Euclidean norm) within lower-dimensional subspace.
If the columns of U E m form a basis of a k-dimensional subspace, the corresponding orthogonal projection is
C = -i
2
2

---

Published as a conference paper at ICLR 2026
Equivalently,
Cу =
arg min 1 - - 2|/2
2 RERagee(U)
meaning Cy is the orthogonal projection of y onto the subspace spanned by U.
2.2 COMPRESSION AS ORTHOGONAL. PROJECTION
Structured pruning. Pruning can be viewed as projection onto a coordinate-aligned subspace at the level of neurons, filters, channels. Assume the layer outputs are ordered that the last k pruned. The corresponding basis Up spans the k-dimensional subspace, with projection matrix Cp and transformed weight matrix Wp p-
()) C
( ( :)
U
W, W.
Consequently, the last 1 of W, zero, and the corresponding neurons, filters, channels can be simply removed.
Model folding. Folding groups the parameters into clusters and replaces each cluster with its mean. Depending on the choice of clusters, different folding results. Folding can be represented as an orthogonal projection onto the k-dimensional subspace spanned by U E {0, 1}m=>,, where each row contains exactly one nonzero entry indicating the cluster assignment. A cluster S₃ comprises all indices of parameter vectors belonging it; thus, (i.
if and only if Sj.
The projection Cj defined in Eq. 2 maps each cluster to its mean (Wang et al.
2025) Specifically,
W, = C,W, e S, wy(() = Pjr Mj
w(1),
S
iES,
where nj is the mean of cluster j. After projection, all parameter vectors within cluster are replaced by their mean, making them identical. As a result, the corresponding layer outputs are also identical, leaving total of k distinct neurons, filters, or channels. Practically, the identical layer outputs can be joined while adapting the next layer appropriately, see (Wang et al. 2025).
2.3 FOLDING DOMINATES PRUNING
To compare pruning and folding, first show that for any choice of pruning, there exists folding that yields more accurate approximation of the parameter matrix W.
Theorem 2.1. Given any pruning with basis Up of rank S kp I m 1 (i.e., at least one parameter vector is prumed), there exists a folding with basis U', and rank kj kpp + such that
I|W - W Z IIW - W
where W, p Cp W and W = C. W, with Cp and C'2 denoting the orthogonal projections defined Eq.
In the above theorem, U' denotes the constructive clustering obtained by merging all pruned rows into a single additional cluster. The proof is in Appendix BB By the Lipschitz continuity of the loss function in Eq. [ the superior approximation property of folding implies tighter bound on the loss difference compared to pruning:
[[(W) I. (W))) & W Wylle, [[(W) I(W,)) S WW W,llr;
with
IW w < W W
Furthermore, the rank difference ky = kp + between pruning and folding is practically negligible, since in typical scenarios many parameter vectors are pruned. For instance, under uniform 50% per-layer retention, a ResNet-18 stage with 256 channels keeps k, = 128 (so folding uses ky = 129), and a ViT-B/32 block with width 768 keeps kp 384 (so ky 385); the relative increase is just 1/k, xs 0.78% and 0.26%, respectively. negligible in practice. Moreover, for all layers and architectures observe that loss and accuracy vary smoothly the rank increases from k, to kip +
3

---

Published as a conference paper at ICLR 2026
no jumps in loss accuracy, and the error difference between ranks k and & + is typically much smaller than the difference between pruning and folding at the same rank (see Appendix[F3]
Finally, we show that folding using optimal k-means clustering never yields less accurate approxi- mation of the parameter matrix W than pruning.
Theorem 2.2. Let U' be the basis obtained from an optimal k-meaans clustering with kiş clusters, i.e., the folding clusters are determined by a k-means algorithm minimizing the accumulated within-cluster sum of squares. Then, for any pruning with basis Up of rank kip = k 1, we have
IW 10--Wpll
Wplli N
IW w
where W p C, W and W = C W, with Cp and Cjj denoting the orthogonal projections defined in Eq.
The proof is given in Appendix B
This result demonstrates that k-means folding is not merely a heuristic, but an optimal projection under clustering constraints. Note that the special folding W in Theorem 2.1 is suboptimal. while Theorem 2.2 shows that W achieves the minimum possible reconstruction error over all clusterings, producing a strictly stronger improvement as IW W 2 W 1 > IW- W Unlike pruning, which relies on parameter vector removal, folding generalizes the idea by enabling coordinated parameter merging. Thus, folding incurs less parameter distortion and provably smaller loss perturbation under local parameter- Lipschitz assumption.
In addition, Theorem 2.2 has implications for possible fine-tuning after compression. Matrix W contains the optimized weights and Wp p or W contain the weights after pruning and folding the optimized network. As a result Theorem 2.2] the quadratic distance between the optimized weights and the compressed optimized weights is smaller for folding in comparison to pruning.
Our theoretical results employ a one-rank slack comparing pruning rank kp to folding ky kp + 1, as proof device to obtain a clean monotonicity guarantee on projection error. This slack does not reflect our evaluation protocol. all experiments we enforce matched sparsity budgets and compare methods at the same retained size (parameters and FLOPss. Hence, empirical accuracy gaps cannot be attributed to extra capacity.
EXPERIMENTAL RESULTS
Most pruning studies vary only seeds by training several checkpoints under single hyperparameter recipe, leaving the role of upstream training underexplored. We instead benchmark > 1'000 check- points spanning diverse hyperparameters (optimizers, learning rates, augmentation, regularization, SAM) to quantify how training choices interact with folding and pruning. Concretely, we train 216 R (Adam) and 576 ResNet18 (SGD) models on CIFAR-10, include 50 PreActResNet118 and 200 ViT-B/32 checkpoints from (Andriushchenko et al.] (023), and add 72 CLIP VIT V/3-B/32 models fine-tuned on ImageNet-IK from (Wortsman et al.) 2022b). The two ViT families differ markedly in scale (~~1MM vs. 151M parameters). We also train 36 LI A--fmilyylyy 60M and 130M parameter models on the Colossal Clean Crawled Corpus (C4) (Raffel et al. 2020). Training details are in Appendix D The results for LLaMa- 130M are Appendix F
We empirically compare model folding and structured pruning across CNNs, ViTs and LLaMA-60M models under matched training setups. Unless stated otherwise, we do not apply gradient-based fine-tuning: for CNNs we only re-estimate aatch-normalizationn statistics via single forward pass using REPAIR (Jordan et al., 2023) isolate structural effects, and ViTs / LLaMA-60M, models are left uncalibrated. Note that R REPAR was recently shown to substantially improve post-compression performance for pruned models (Saikumar Varghese 2025), and has also been applied on top of folding (Wang al., 2025). We report results (i) immediately after compression (CNNs after REPAIR, ViTs with no further step), (ii) for ViTs additionally after a LayerNorm reset, and (iii for both families after 1-5 epochs of full fine-tuning.
Folding vs. Structured Pruning on CNNs and ViTs. We compare model folding (FOLD) with structured magnitude pruning (MAG) under L1 and L2 criteria (MAG1, MAG2) across representative CNN and ViT architectures. Fig. summarizees results: scatter plots show accuracy of MAG1 vs. FOLD for each trained model, with compression ratio indicated by color. Results for MAG2 are in Appendix B
4

---

Published as a conference paper at ICLR 2026
Aamuney (44 IratD MALS)
Accunacy (S) (LGLD -MG
10LD west Aocumecy (5)
30
100
TOLD accurscy 15
80
40
1.
Carre Rodd
1.0
OWED Inter 9ob3
10
20
40
60
20
40
60
Test aaayyyycyyyyyy 196
Layalwise Compressior
Accu acy [96]
Layetuiise Comp uasion Ratio
(a) 88 Adam, MAG1 vs FOLD, no Il regularization
(b) PreActResNer18, MAG1 vs FOLD
A.cunncy (44 (re01 Matts
Acuueey [S] DULLL Matnt
1OLD seet Aecuracy 183
80
80
300
60-
90
Comp woliiiiiin kodis
5-
40-
0.e
1.
-10
(c) ViT-B/32, MAG1 vs FOLD, base accuracy 755%
(d) CLIP VIT-B/32, MAG1 vs FOLD
Figure 1: Folding outperforms magnitude pruning across diverse training regimes. Top row: ResNet18 and PreActResNet188 on CIFAR-10. ResNet18 checkpoints were trained from scratch with Adam using different hyperparameter configurations. PreAAttReNNet18 checkpoints are from Andriushchenko et al. (2023). Bottom row: ViT-B/32 on CIFAR- 10 from (Andriushchenko et al. 2023) and CLIP ViT-B/32
ImageNet-1K 1K from (Wortsman al. 2022b). See Appendix D for details. In these plots, we use checkpoints that were trained without L1 regularization. Scatter plots show cospeeompnnsn accuracy for magnitude pruning (L1 criterion) versus folding uniform per-layer compression ratios. Bar plots depict the accuracy gain by folding, computed as A = Acc(FOLD) Acc(MAG1), as function of layer-wise compression ratio. Folding yields the largest improvements at moderate to high compression, confirming its robustness across architectures and datasets. Fig. g shows the results for magnitude pruning with L2 criterion.
90
96 2
Amuurry [S] IPOLD MALL!
1011 lest Accurecy (h))
90
25
50
20
60
Momptros hoto
15
Kg
5.7%
3
2.33
Compoest
i5
30-
60
60
80
20
60
60
MAG1 Test Accuracy (%6
Layer wise Compression Retio
MAGI Test Accuracy [P5]
Layer wize Compression Raão
(a) ViT-B/32, MAG1 vs FOLD, base accuracy >75%
(b) CLIP VIT-B/32, MAG1 vs FOLD
Figure 2: MAG1 versus FOLD on ViTs after LayerNorm-only fine-tuning for ViT-B/32 on CIFAR-10 and CLIP ViT-B/32 ImageNet- IK. In the scatter plots, points checkpoints, color encodes layer-wise compression. Bar plots depict the accuracy gain A Acc(FOLD) Acc(MMG1). which is positive, indicating that under LayerNorm adaptation FOLD retains advantage over pruning.
Box plots depict the distribution of accuracy differences between FOLD and MAG1. Positive differences indicate folding outperforms pruning, with the gap widening at higher sparsity. This trend holds across ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on both CIFAR-10 10 and ImageNet- 1K, demonstrating robustness architecture and dataset scale. These results support our theoretical claim (Sec. p) folding projects onto cluster-structured subspaces, preserving parameter alignment and reducing functional distortion, yielding consistent accuracy gains magnitude pruning.
Performance Comparison after Lightweight and Full Fine-Tuning The previous results isolate structural effects by evaluating models without further optimization. We now test whether folding's
5

---

Published as a conference paper at ICLR 2026
In
FO.D acourscy
Accurran After
Accurrry
20
Accurecy (60)
MAC1 Accunary (4)
Lyyer wise Compresiion Rade
40
60
20
46
20
00
(a) MAG1 vs FOLD, fine-tuning for I (left) and 5 (right) epochs.
(b) MAG1 vs FOLD
(c) Before and after fine-tuning for epochs:
accuracy gap after ft. FOLD (left) and MAG1 (right).
3 srcc prour. MAUC1
10.0 Rccunn
Ausunne After
to-
65
60 d
Laystooon Compenssan flabs
0.3 04
20
20
sefore
(d) MAG1 vs FOLD, fine-tuning for 1 (left) and 5 (right) epochs.
(e) MAG1 vs FOLD accuracy gap after ft. FOLD (left) and MAG1 (right).
Before after fine-tuning epochs:
Figure 3: Folded models retain their accuracy advantage after fine-tuning. Results for ResNet18 trained by Adam on CIFAR-10 (top row) and CLIP-ViT-B/32 trained on ImageNet-IPe IK (bottom row): (a,d) compares post-compression accuracy of magnitude pruning (MAG1) versus folding (FOLD) after I and 5 epochs of fine-tuning. (b,e) show the accuracy gap between folding and pruning as a function of fine-tuning epochs, demonstrating that folding maintains a consistent lead, i.e., the FOLD accuracy delta is positive. (c,f) illustrate accuracy trajectories before and after 5 epochs of fine-tuning for both methods, highlighting that folded models recover accuracy faster. Further results in Appendix B
advantage persists after fine-tuning, Fig. 2 compares MAG1 and FOLD on ViTs under lightweight LayerNorm-only adaptation: across V/3-B/32 (CIFAR-10) and CLIP ViT-B/32 (ImageNet-1K) folding consistently reaches higher post-compression accuracy, with the gap Acc(FOLD) Acc(MAG1) remaining positive and typically growing with compression.
Next, we allow brief fine-tuning (1-5 epochs). Fig. 3 shows that folded models (a.d) start from higher accuracy and retain their lead, (b.e) maintain positive relative gap, and (c,f) recover faster with
weight_decom
warmup_steee
max Jr
PPL. 0% sparsity
PPL. MAG2 (20%)
PPI. FOLD (20%)
MAG2 MAG2 (50%)
PPL. FOLD (50%)
0.01
S80
0,001
32.11
54.51
47.17
398.62
221.32
32.14
220.54
172.57
0.01
2200
0.001
32.20
46.57
47.54
174.58
216.36
S80
0.001
32.17
51.14
48.23
220.33
223.86
1100
0.001
32.21
50.03
47.47
231.41
204.47
2200
0,001
32.40
46.38
46.92
177.48
185.27
302.43
29.77
$64.96
234.56
0.01
2200
0.005
29.60
$4.50
47.04
360.52
208.02
880
0.005
30.47
78.73
62.35
762.05
395.04
1100
0.005
30.17
$9,20
49.58
544.87
184.74
2200
0.005
29.75
56.18
46.55
353.35
165.21
910.48
977.92
367.94
0.01
2200
0.01
29.25
51.46
44.28
323.68
288.83
880
0.01
108.56
129.77
123.85
279.17
198.72
1100
0.01
30.31
97.97
61.19
860.14
533.62
2200
0.01
29.57
54.43
47.77
351.11
209.06
Table 1: Evaluation of FOLD and MAG2 on LLaMA-60M. We train and evaluate 18 L.LMA-family models with 60M parameters on C4 by varying max_h, Ir, warmup steps and weight decay. Columns 4-8 show perplexity of the trained model before compression and after pruning / folding using layer-wise pruning ratio of 20% and 50%. We prune only FFN blocks. Except for low learning rates with long warmup schedules, FOLD outperforms MAG2 (highlighted in bold).

---

Published as a conference paper at ICLR 2026
fewer plateaus. Thus, folding provides a better initialization and requires fewer updates to regain performance, making it advantageous in settings with limited fine-tune mueettunng.
Performance Comparison LLaMA-60M. Tab. I evaluates FOLD and MAG2 LLaMA-60M trained on C4 under 18 hyperparameter settings (varying learning rate, warmup, and weight decay). We prune fold only the FFN blocks and report perplexity at baseline and 20% and 50% layer-wise sparsity. Except for models trained with very low learning rates and long warmup, FOLD consistently outperforms MAG2. Similar finding have been obtained for LLaMA- 130M models in Tab. 6
MODEL COMPRESSION ABLATION STUDIES
The previous sections demonstrated that folding often outperforms structured pruning across archi- tectures and compression ratios. On ResNets and ViT',, we probe which training factors impact this advantage. Specifically, we analyze sensitivity to learning rate, the use of sharpness-awan training (SAM) (Foret et al., 2021), regularization and data augmentation (Prabhu et al. 2019) the hyperpa- rameters known to influence loss landscape geometry and generalization (Fort & Jastrzebski 2019) Li al. 2018 Neyshabur et al. 2017 Chen et al.) (022)) in non-trivial ways (Andriushchenko et al. 2023). To validate these curvature- related hypotheses, Appendix [ incluuees a sharpness analysis. Our measurements quantify how hyperparameters shift the local geometry of the loss landscape and help
explain when FOLD 's advantage widens or narrows.
Role of Optimizer. We repeat the ResNet18 8 analysis under Adam and SGD to gauge optimizer sensitivity. Compared to the Adam-trained sweep in Fig. Fig. the complementary SGD sweep in Fig. 5 shows the same qualitative ordering-FOLD exceeds MAG1 across compression levels- with different baselines and dispersion: SGD checkpoints form a tighter cloud and exhibit a smaller median gap, whereas Adam yields larger variance and at times more pronounced FOLD advantage, especially at higher compression. The FOLD-MAG1 difference remains positive under both optimizers in most cases, but its magnitude is optimizer--eeendent.
Effect of Learning Rate. Fig. 4 reports post compressonss accuracy for FOLD versus MAG1 across learning rates on ResNet18 (Adam, SGD), PreAActReNNet18, and ViT-B/32. With Adam, FOLD 's edge is largest at moderate-low rates, narrows and can reverse at very high rates, and vanishes again extremely small rates (both methods degrade). For SGD, the dependence is weaker and can be inverted (e.g., ViT-B/32). The effect of learning rate is expressed through sharpness (see Appendix F): when training places the model in regions where folding produces smaller sharpness increase than pruning, folding wins. When folding produces a larger sharpness increase (most visible at high learning rates under Adam), pruning can outperform. Adaptive methods like Adam are associated with sharper minima and distinct generalization behavior compared to SGD, amplifying this sensitivity (Wilson et al. 2018 Jastrzębski et al., 2018
Zhou et al. 2021
Effect of SAM. Fig. 6 evaluates training with and without SAM and measures post- compression accuracy. Across models, SAM lifts both methods. With light LI regularization (10-5) during training shown in (b), pruning narrows the gap at low compression (where in- duced sparsity aligns with LI), yet FOLD regains and extends its lead as compression increases. These trends are consistent with the view that SAM steers training to flatter solutions, reducing curvature sensitivity. Within this flatter neigh- borhood both pruning and folding projections operate inside the same robustness ball, so their geometric differences matter less and the gap narrows- an effect stronger for ViT-B/32, where high p homogenizes head/ channel saliencies and reduces the relative advantage of clustering.
Accunccy (N) (FOLD- MAG1
30
FoR Tex Accurecr
20
60
10
CaT Taato
MA311 TeAE Accuracy (S)
Ratin
Figure 5: Optimizer effect evaluated on ResNet18 checkpoints trained on CIFAR- 10 with SGD (no LI regularization). The figure complements Fig. a).
Effect of Data Augmentation. Fig. 7 plots the distribution of AAccuracy (FOLD - MAG1) across checkpoints versus the layer-wise compression ratio, contrasting runs without (gray) and with Augment (green). For R (Adam and SGD) and PreActResNet18,, RAUG reduces or shifts FOLD 's relative benefit. In contrast, for ViT-B/32 RAUG increases FOLD 's advantage: the median A
7

---

Published as a conference paper at ICLR 2026
Accunaty (S) DOLL (AaL))
Uh-10"
scnedule
Schedure
Accornaty (id)
LA-10
LR
Acourny (POLD
80
60
80
90
Lt-20
S1-22"4
Li-1 30 10
20
1n 12"0
Ut 10
(R=10
LAw10
40
60
60
50
Accuracy 186]
Layaroiis Campression
MAGL Accuracy 186]
Laysonise Comp Comprossion
(a) R8sNet18, Adam, no 1.1 reg., no weight decay
(b) ResNet18, SGD, no 1.1 reg., no weight decay
Accuracy (S) (roUD- MKG3)
Accunasry (M) COLL (OAL1))
80
FOLD Accuraccy
FOLD Accuracy
7n
80
80
80
20
0.0
Accurady
Accuracy
(c) PreActResNet18, SGD
(d) ViT-B/32, SGD
Figure 4: Learning rate modulates folding's edge. Post-compression accuracy of FOLD and MAG1 across learning rates: ResNet18 with Adam (a) and SGD (b), PreActResNet18 (c), and ViT-B/32 (d). FOLD leads moderate-lon low rates. With Adam, the gap shrinks reverses very high rates, and closes again at extremely small rates. SGD shows weaker or opposite dependence.
0.999
A.rumecy 1S, (rai (okr))
CLASS B22 OF
Arcuuny [S) tro nous)
0.000
0.033
8o
8D
o23
0.023
1ULD Accurnty
TuLD Accurecy
60
40
-10
20
MAGI Accuraty 1961
Ratio
MAGE Accuracy 1%6]
Ratio
(a) ResNet18, Adam, MAG1 FOLD, without LI reg.
(b) ResNet18, Adam, MAG1 FOLD, with LI reg.
Accunncy [S] (rGLD MMG()
3.75
Racunatry US) traul (oas))
0.00 a 0.005 100
8
(OLD Accurwey (hs)
20
60-
Accuraty
&
0.000 0,33
80.
0.305 3
10
80
20
0,200
No SAM 5AM
SAN
20
50
20
80
MAG1 Accurady 1%6]
Layeriise Comp ession Ratio
MAG1 Accuracy (86]
Layor-n 50
Compresseon sssiion
Ratio
(c) PreActResNet, MAG1 vs FOLD, LI regularization
(d) VIT-B/32, MAG1 vs FOLD, no LI regularization
Figure 6: SAM (Foret et al. 2021 can boost model compression. Post-compression accuracy under training with without SAM. (a) ResNet18 (Adam), no L1. (b) ResNet18 (Adam), LI= 10 (c) PreActResNet18 (SGD), no LI. (d) ViT-B/32, no LI. SAM improves both FOLD and MAG1, but the uplift is consistently larger for FOLD, especially with Adam. Light LI regularization helps MAG1 low compression, yet FOLD retains a clear advantage at moderate-high compression.
rises with compression, suggesting that augmented ViT representations especially amenable to projection- based removal. A plausible mechanism is that augmentation biases training toward flatter, more invariant solutions. This is supported by our sharpness analysis in Appendix and consistent
8

---

Published as a conference paper at ICLR 2026
with recent theory linking augmentation induuedd input perturbations to equivalent parameter-space perturbations and showing that augmentations bias training toward flatter minima (Yoo & Yoon 2023). Standard augmentation (augm=True) shows similar trend and is omitted for brevity.
Accurcy (h (o WM611
No RAUG
SAUG
A.ccomn (NOu M631
30
BAUG
MALO
Acturacy (N6 (FOLD M6G1)
40
NC RAUD
BAUG
A.ccurscyy (POL M631)
20-
16
20
20
%
15
10
10
-0
-10
-20
8 8
Ratio
Layatotis Compression Rabio
Layersonse
Rabio
Compression Rabio
ResNet18, Adam
(b) ResNet18, SGD
(c) PreActResNet
(d) ViT-B/32
Figure 7: Augmentations have generally positive effect on the post-compression accuracy. Post-compression accuracy without with random augmentations for (a) ResNet18 (Adam), (b) ResNet18 (SGD), (c) PreActResNet18 8, and (d) ViT-B/32. Augmentations boost both FOLD and MAG1, On ResNer18 8 they also narrow FOLD 's advantage consistent with added invariances making axis-aligned removals less damaging. In ViT-B/32, augmentations are essential for folding")
These ablations reveal a consistent pattern: conditions that encourage flatter and structured solu- tions- moderately low learning rates and SAM with small- radius-magnite FOLD advantage, whereas extremes reduce it: very high or very low learning rates, stronger augmentations, or large SAM radii narrow the gap; SGD generally dampens all effects relative to Adam. This aligns with our projection view (Sec.2) when weights are well aligned, clustering reduces projection error more than coordinate removal and thus perturbs the function less, while weaker alignment or broad robustness neighborhoods make the two projections behave more similarly.
5 CONCLUSION, LIMITATIONS, AND OUTLOOK
We framed structured pruning and model folding as projection-based compression and showed that folding achieves smaller parameter deviation with one-rank slack, implying tighter func- tional preservation under mild smoothness. calibration ffree evaluation over ~1'00 checkpoints (ResNet18, PreAetResNNt118, ViT-B/32, CLIP VTT- CIAR--00 ImageNet-IKG IK; and LLaMA-60M and ALaMA-130M on C4) found that FOLD typically surpasses MAG1 in post-compression accuracy, with the clearest gains high compression and under training conditions that induce flatter, more structured solutions (e.g., moderate learning rates, SAM). The gap narrows at very low compression and can shrink under strong data augmentation or large SAM radii.
Limitations. Our theoretical guarantee allows a one-component increase in compressed rank but does not establish universal dominance at exactly matched sizes. Empirically, we focus on standard CNN and ViT families on CIFAR- 10 and ImageNet- IK, as well as small LLaMA models on C4. For ViTs and LLaMA, pruning and folding are applied only to the FFN blocks. Extensions to attention layers is left for future work. We evaluate in strictly calibration-free settings, with optional BatchNor- m/l m//LayerNorm resets and short fine-tuning budgets, and compare primarily against magnitude- structured pruning. Interactions with quantization, distillation, and unstructured sparsity are not considered. Larger LLMs are beyond the scope of this study due to the computational cost of training across diverse hyperparameter settings. We note that most SoTA pruning methods for LLMs rely on calibration data (e.g., activation activation-awwre/secomm www.-aawwwwdddaarrreeeeeeeeeeeee -order) and are exclusively pruning-bast based.
Outlook. We plan to extend folding to pruning/ folding attention blocks, calibration- -asedd settings and evaluate on larger LLMs/ / VLMs. We also plan to study interactions with quantization and adaptation methods. More broadly, our projection-based view positions folding geometry-awan primitive for compression: a foundation on which novel calibration-basee compression methods, hybrid pipelines with quantization and distillation can be built, and a step toward principled model compression framework. In this sense, folding is not only a practical tool but a building block for the next generation of compression methods tailored foundation models and deployment at scale.
Note that the base accuracy of ViT-B/32 checkpoints trained without RAUG lower than with RAUG.

---

Published as a conference paper at ICLR 2026
Reproducibility Statement. Our compression operators and evaluation protocol are described in Sec. Sec. 3] with ablation studies in Sec. | A repository with configs and scripts to regenerate all figures/tables is linked in Appendix [A] Complete proofs are in Appendix [ Related literature is summarized in Appendix [ Training setups, datasets, links to the used checkpoints, and hyperparam- eter grids in Appendix [ and extended results in Appendix [E] and Appendix [F] Our limited L.M usage statement is in Appendix G Together, these materials enable re-running the full pipeline and regenerating the results.
This work was in part supported by the FFG COMET K1 Centre "Pro" II" (Cognitive and Sustainable Products and Production Systems of the Future; Contract No. 91 1655). The results presented in this paper were obtained using the computational resources of Pro2Future GmbH, and the Austrian Scientific Computing (ASC) infrastructure.
Acknowledgments.
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re- Re-Basin: Merging models modulo permutation symmetries, 2023. /ttps:////// wttp://ww..o/////////////// 04836
Maksym Andriushchenko, Francesco Croce. Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship hetween sharpness and generalization, 2023. https:ttt:://// httpps://arxiv.or//aaa
2302.0700 07011
Anna Bair, Hongxu Yin, Maying Shen, Pavlo Molchanow, and Jose Alvarez. Adaptive sharpness-aware pruning for robust sparse networks, 2024. URL httpitss//arxxi.or///////////// 14306
Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-Chun Kao, Shivani Agrawal, Utku Evci, and Tushar Krishna. Progressive gradient flow for robust n:m sparsity training in transformers, 2024, URL
Christian Bauckhage. K-means clustering matrix factorization. arXiv:1512.07548, 2015. URL https "/arxiv.org/abs//5112.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks, 2019.
01188
Xiangning Chen, Cho-Jui Hsich, and Boqing Gong. When vision transformers outperform resnets without 0154/
pre-training or strong data augmentations, 2022.
Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. arXiv preprint arXiv:2310.06756, 2023.
Bita Darvish Rouhani, Daniel Lo, Ritchic Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vino- gradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Tacsik Na. Prerak Patel, Shuai Che, Lok Chand Koppaka, XIA SONG, Subhojit Som, Kaustav Das, Saurabh T, Steve Rein- hardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing cloud scale with microsoft floating point. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 10271 10281, Cur- ran Associates, Inc., 2020. URL http::://proceedingg www.rrreeeeinnnnnrrrieiineeeeeee httppp.cc//pppr..ele/e////peeeee www.s//apeeeeee//eeeeeeeeaapeeet /tt.:/pprr...eee///ee///////e... /2000ffile/
840-- pdf]
Shaojin Ding, David Qiu, David Rim, Yanzhang Hc, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang, Tara N. Sainath, Zhonglin Han, Jian Li, Amir Yazdanbakhsh, and Shivani Agrawal. Usm-lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models, 2024. URL
https://arxiv.org// abtp://.aa/223...eee 085533
Rahim Entezari and Olga Saukh. Class-dependent compression of deep neural networks, 2020. URL https //arxiv.org/ab//190 10364
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks, 2022. URL https://aaxxx.v...... org/pdf///11 06296
Gongfan Fang, Jic Song, Chengehao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation, 2020. URL httpss///aani.oorg/ab///////
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2021. URL. https: http:://arxv..or/aa//000..0000
10

---

Published as a conference paper at ICLR 2026
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes, 2019. URL
fttps://arxtv.orr//ab/////// 04724
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
Athanasios Glentis, Jiaxiang Li, Qiulin Shang, Andi Han, Ioannis Tsaknakis, Quan Wei, and Mingyi Hong. Scalable parameter and memory efficient pretraining for Ilm: Recent algorithmic advances and benchmarking. 2025. http:s //axiiv. wwwp:/i...voooooeoee haa///bs//2/000 2505 22922
Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong. Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SItrain: sparse plus lawk approach for parameter and memory efficient pretraining, 2024. URL
https://arxiv. org/ httgg//bb2244666 02214
Song Han, Jeff Pool, John Tran, and William J. Dally, Learning both weights and connections for efficient neural networks, 2015. URL https://arxii wtggabs///gaoooeooooooeee 02626
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression, 2020. URI. httpps://arxi..or//aaaa httpg//////////////////g
J. A. Hartigan and M. A. Wong. A k-means clustering algorithm. JSTOR: Applied Statistics, 28(1):100-108, 1979.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL 02531
Samuel Horvath, Stefanos Laskaridis, Shashank Rajput, and Hongyi Wang. Maestro: Uncovering low-rank structures via trainable decomposition, 2024. URL
14929
Hengyuan Hu, Rui Peng. Yu-Wing Tai, and Chi-Keung Tang. Network trimming: data-driven neuron pruning .32550
approach towards efficient deep architectures, 2016. htl.:itttss//rrxv..orr//aasss
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd, 2018. URL http:tsss//arrv..ogg//////////1/ 04623
Keller Jordan, Hanic Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair, 2023. //RL:/ttss///aaav.orrr/bb////// 08403
Hyeong-Ju Ju Kang. Accelerator pruning for convolutional neural networks. IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-1, 2020. ISSN 1558-2205. doi: 10.1109/csw.2009922111647 URL http:///x.dde doi org/10... 100.111000 1109/TCSVT 2019. 201..2911677
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications, 2016. URL https
//arxiv.org/abs////
.65530
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Blias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models, 2022, URL tttps:/tts::::ii...oo http://wwww..rgg//////e..a/ 07259
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned ep-dccomposition, 2015. URL /ttp///t////// abs/ 1412. 6553
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087: 08710. 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS'18, 18, pp. 6391-6401, Red Hook, NY, USA, 2018. Curran Associates Inc.
Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa. and Amir Yaz- danbakhsh, Step: Learning n:m structured sparsity masks from scratch with precondition, 2023. URL
https://arxiv.. www.:////bbb//......mm 01172
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial helief matching, 2019. URL. https://arxiv. wwwpeeiwi..oooogooeeeeeeeee wtggabs///ra 09766
11

---

Published as a conference paper at ICLR 2026
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius, Accelcrating sparse deep neural networks, 2021. httlhttpss///arii.orr//b//////// 08378
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning, 2017. URL htttps://arxiv.orggg.e..
08947
Vinay Uday Prabhu, Dian Ang Yap, Joyce Xu, and John Whalcy. Understanding adversarial robustness through loss landscape geometries, 2019. URL http://://aaw...or////aaa//////e
.9061
Sai Qian Zhang, Bradley McDanel, and H. T. Kung. Fast: Dan training under variable precision block floating point with stochastic rounding. In 2022 IEEE International Symposium on Higg-PPefformacce Computer Architecture (HPCA). pp. 846 860, 2022. doi: 11A/////HAA5366622222220000
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021. URL /tKL/ttts///prceedinng mml press/v139/ 3ttp////r1///rarod2./.d2 html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 1-7, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Ii, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
/ttpshtts///arxiiv wttpr/iwwwwrg/bb//11111 10683
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. /tt//ttps://arxiv herrrg/ab/2204... 06125
Siyu Ren and Kenny Q Zhu. Low-rank prune-and-factorize for language model compression, 2023. URL
httpp:///axiv org. wwt.:/rg///bb/aaa/ee ab3// 14152
Dhananjay Saikumar and Blesson Varghese. Signal collapse in she-shot pruning: When sparse models fail to distinguish neural representations, 2025. URL http:::///aaiii.oor/a/////////// 15790
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning, 2020. UtRL/htpp://aakiv wwwpaaaawvv.oogbbb//aaaeea wtt.:/org/bbb//00555/// 07688
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, 2024. URL httpps://aaaiv..orrg//////////// orgp.////./......./ 03053
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024, URL /tttps://arxiv.. www.tvooor/r/sssse. htrrg/bbs/3300 11695
Alexander Theus, Olin Geimer, Friedrich Wicke, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Towards meta-pruning via optimal transport. arXiv preprint arXiv:2402.07839 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. URL
httpps://axxi. wttp://wwggbbbbb/./.a///// 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylow, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor. Adina Williams, Jian Xiang Kuan. Puxin Xu, Zheng Yan. lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kamhadur, Sharan Narang, Aurelien Redriguez, Robert Stojnic, Sergey Edunow, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023h. URI/httss:: //appiii///xxv...../ wwwpaaawwwww.ggr//////aaa/////// 09288
12

---

Published as a conference paper at ICLR 2026
Dong Wang, Haris Sikić, Lothar Thiele, and Olga Saukh. Forget the data and fine-tuning! just fold the network to compress. The Thirteenth International Conference on Learning Representations, 2025, URL
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Learning structured sparsity in deep neural networks, Advances in neural information processing systems, 29, 2016.
Ashia C. Wilson, Rebecca Roclofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning, 2018. URL httl://s///arxi..orr//bbbb////// 02822
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-opes, Ari Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs. Raphael Gontijo-Lpes Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022b. URL httttsss///axii wwwrrriiiiioooaaeee wrg./bs/////3 05482
Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. Balanced sparsity for efficient dnn inference on gpu. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):5666-5683, July 2019. ISSN 21.59-5399. doi: 10. 111116699aii. .33015676. htRlitttpp//dx.dol www.oogg//...ooo 16t.g///...111//aa 1609/aaai V33101 33015676
Weebum Yoo and Sung Whan Yoon. A flat minima perspective on understanding augmentations and model robustness, 2025. URL https://arxiv. httpps://arxxi.orr//aa//255 24592
Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. In Proceedings of the IEEE/CVF Conference Compuler Vision and Putlern Recognition, pp. 24266-24275, 2023.
Qiaozhe Zhang, Ruijie Zhang, Jun Sun, and Yingzhuang Liu. How sparse fundamental limit perspective, 2025. https:////// wtgg/bs/////0000o https////arkk...rrraa//2366 05857
prune deep network: A
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Hoi, and Weinan E. Towards theoretically under- standing why sgd generalizes better than adam in deep learning, 2021. httl:httss.//ariivorr////// 05627
13

---

Published as a conference paper at ICLR 2026
APPENDIX
The following sections provide supplementary information and complement the main paper:
Appendix A
Code, Data, and Resources.
Appendix
Proofs of Theoretical Claims.
Appendix
Related Work.
Appendix
Training Details.
Appendix
Extended Empirical Comparison of Folding and Pruning.
Appendix
Additional Analyses: Sharpness, Runtime, and LLMs.
Appendix
Use of Large Language Models.
A
CODE, DATA, AND RESOURCES
Code and logs. repository with all source code, experiment configs, and figure-generation scripts (including the exact logs used to render every plot/table) are released at /tttps://githun httoo/ouuu/ooodinngg.ooe
_soog___asppoottionnn The repo contains: implementations of folding and pruning
operators, training/evaluation pipelines, scripts to plot ablations, and notebooks to reproduce figures directly from logs. We log all training metrics and hyperparameters with Weights & Biases and export logs alongside the code for reproduction. Additionally,
provide another repository for reproducing results of compressing LLaMA-60M and LLaMA-130M with folding and magnitude structured pruning at /hhttpss//githuu com
Our folding implementation is based on the code by Wang et al.
2025
Datasets. We use CIFAR-10" and ImageNet-1 CIFAR-10 is downloaded automatically via torchvision. ImageNet- IK requires the official credentials and follows its license. Pretrained/fine- tuned checkpoints referenced in the paper are either trained by us (configs in the repo) or obtained from the cited works (Andriushchenko et al. 2023 Wortsman et al. 2022b). The download links are also provided in Appendix [
Compute resources. Experiments were run on a cluster featuring 8x NVIDIA A100 (80 GB RAM) GPUs. All random seeds are fixed in the configs and scripts.
Computational complexity and memory cost. At inference and matched retained sizes, folding and structured pruning yield the same compute and memory. The difference lies in the compression step: magnitude pruning is one-pass scoring and selection procedure (0(pm) to score p filters of dimension m, plus O(p logp) selection), whereas folding runs k-means on layer weights with T sweeps. Using Hartigan's algorithm (Hartigan & Wong 1979), one sweep costs (pkm), with max T 10 sweeps the total is Oppmml) (effectively linear in pm when l is small). This cost is paid per layer and is small compared training.
Runtime overview. The most expensive step in our study fine-tuning of CLIP VIT-B/32 on ImageNet- (1-5 epochs), which dominates wall-clock time (order of hours per run). In contrast, compression lightweight. We detail measured runtime overhead of compression in Appendix
B PROOFS OF THEORETICAL CLAIMS
Below we prove that for any choice of pruning, there exists a folding that yields a more accurate approximation of the parameter matrix W.
Weights & Biases: httpps://vee htaps::/wwanb....
"Model folding universal: http:///tttt....ooaaa/////////es and model folding
for CNNs: httpp::// ttw.bbb..tttt..u http:///..tt...c.mmmrree///aaaaa
'CIFAR- 10: https: http://www....orrrrttoeee/////ce
ImageNet- IK https://imageenet..org
14

---

Published as a conference paper at ICLR 2026
Theorem 2.1. Given any pruning with basis Up rank ky I 11 (i.e., least parameter vector is prumed), there exists a folding with basis U' and rank ky = kp + such that
I|
wpll 2 IW -
where W Cp W and W, = C' W. with Cp and C' denoting the orthogonal projections defined Eq.
Proof. The rows of W can be ordered such that the pruned parameter vectors are first: w(1) ... w(m kp). Then find that
w(1) w m - kip)
W W
0
using Eq. For the existence proof, choose folding that clusters all parameter vectors ((1)..... w m - kip) into single cluster, all other parameter vectors have individual clusters, Le.,
w
m ky
w(m
kp)
U =
W W
w
hp
1=1
using Eq. 4
We have WW- wpl1. = EmTT my w(i)Tw(()) and
k-
m-k, kp
W w/1 = Σ
(wli) (w)(( - -w) Σ 1=1
(w(i)T w() - -2w(i) + xTTH)
==1
w(i) (ii (m -kp)p= p
w(i) w(1) IW- w
The latter inequality directly establishes the theorem.
The following theorem shows that folding using optimal k-means clustering never yields less accurate approximation of the parameter matrix W than pruming.
Theorem 2.2. Let U be the basis obtained from an optimal k-means clustering with ky clusters, i.e., the folding clusters are determined by a k-means algorithm minimizing the accumulated within-cluster sum of squares. Then, for any pruning with basis Up of rank kip kg 1, we have
|| Wpll
> W 1;
where W Eq.
Cp W and Wi = C W. with Cp and Cj denoting the orthogonal projections defined
Proof. According to BBucchage (015) and Wang et al. 2025), the problem of k-means clustering can be formulated as the following constrained matrix factorization problem:
min W U(U UW - w|l subject to u(i.)) E (0,1), 1}. Σ Eu(i.)) Vi.
15

---

Published as a conference paper at ICLR 2026
This formulation coincides with the orthogonal projection of model folding, see Eq. and Eq. 4 Theorem 2.1 guarantees the existence of a folding basis U' and the corresponding projection Cy for any pruning W, v of W, such that
JW W P 2
I - WW W
Since optimal k-means clustering achieves the minimal possible error W will S |W W'ilis lli,, the theorem follows.
C RELATED WORK
Model compression encompasses a wide range of approaches designed to reduce inference cost while preserving model utility. We focus on post-training, calibration- structured compression, where the model architecture modified without access data or gradients. this setting, the dominant baselines are structured pruning and, more recently, model folding. Below we discuss these families of methods and clarify how our projection- thheoretcc view relates to and extends prior work.
Post-training compression. Quantization reduces arithmetic precision (Darvish Rouhani et al. 2020
Qian Zhang et al., 2022). but typically requires calibration to maintain activation ranges. Know ledge distillation (Hinton et al. 2015) produces reduced students trained to imitate teacher logits. Even
data-free variants (Micaell & Storkey, 2019; Chen et al. 20199 Fang et al., 2020 Yu et al. 2023
Haroush et al. (220)) require full training dynamics and do not yield structural compression. Low-rank
factorization via matrix or tensor decompositions (Ren & Zhu 2023; Horvath et al. 20244] Lebedey
ct al. 2015 Kim al., (0166 approximates pretrained weights by continuous subspaces but generally requires fine-tuning for restoration. These approaches differ fundamentally from our objective: they modify numerical precision or parameterization, not the discrete structure of the model.
Structured pruning. Structured pruning removes neurons, channels, filters, or blocks (Li et al. 2016 Luo al. [217 [Hu et al. 2016 Wen al. 2016). Magnitude- criteria (Han et al. 2015 Lu et al. 2023] Ding et al., (2224] Entezari & Saukh 2020) dominate due to simplicity and hardware alignment. However, structured pruning typically requires fine-tuning or recalibration (Kurtic et al., 2022 Sanh et al., (020)) to mitigate accuracy degradation, and even calibration bbased methods such as SparseGPT (Frantar & Alistarh 2023) or Wanda (Sun et al.) 2024) operate through axis-aligned
removal of coordinates. One-shot improvements using N:M sparsity Yao et al. 2019] Kang 2020)
or OT-based structural alignment (Theus et al.) 2024) still operate within the same paradigm: pruning corresponds to enforcing that the retained parameter vectors lie in a fixed coordinate- allgned subspace.
Our work shows that such axis-aligned projections are geometrically restrictive. We formalize pruning orthogonal projection onto coordinate subspace and demonstrate that, matched ranks up to one slack, pruning is provably dominated by projections onto cluster-structured subspaces.
Weight clustering and model folding. Model folding, recently introduced by Wang et al. (0225)
ties groups of similar channels by replacing them with their mean, yielding dense low-rank layers that preserve structural couplings. Folding implicitly performs cluster-structured projection determined by discrete assignments, and practical implementations rely on k-means clustering. This operator class strictly richer than axis-aligned pruning: folding enables coordinated merging rather than coordinate removal, while remaining compatible with dense inference. IFM (Chen et al. 2023) is related in that it also merges channels via grouping, but its coolaps correction is ineffective (Wang et al., 2025). leading to substantially weaker performance.
Our work strengthens this line along two axes. First. we provide a unified conjection-geomerre framework showing that both pruning and folding are orthogonal projections, but onto fundamentally different subspaces: coordinate-alignes versus cluster-structured. Second, we prove that for any pruned solution of rank k. there exists folded solution of rank k+11 with strictly smaller parameter reconstruction error, and that optimal k-means folding minimizes this projection error among all cluster-structure projections. This establishes a strict theoretical separation between pruning and folding and explains the empirical superiority of folding in calibration free settings.
Model merging and alignment. Model merging combines independently trained models via pa- rameter averaging or permutation alignment. Model soups (Wortsman et al.) 2022a) exploit shared
initialization. Permutation matching (Entezari et al. 2022 Ainsworth et al. 2023) constructs neuron
16

---

Published as a conference paper at ICLR 2026
correspondences. REPAIR (Jordan et al. 2023) stabilizes fused models by re- re-normalizing preactiva- tions. mntra-model merging approaches such as Ziplt! (Stoica et al. (024) combine computational units but do not target compression under fixed architectural constraints.
These works differ from ours in both objective and mechanism. Merging seeks functional fusion across networks, whereas folding compresses single network by exploiting intra-layer redundancy. Our projection-theore explicit geometric optimality guarantees- not shared by merging methods.
formulation shows that folding operates as a structured projection with
Positioning of this work. Across pruning, folding, and merging, prior efforts lack a unifying math- ematical framework that characterizes the geometry of post-training structural compression. Our contribution is to introduce such a framework: we cast pruning and folding as orthogonal projections and show that cluster-structuredd projections admit strictly smaller distortion than coordinate projec- tions under practically negligible rank slack. This perspective yields nontrivial theoretical guarantees and aligns closely with the empirical phenomena observed across CNNs, ViTs, and LLaMA models.
D TRAINING DETAILS
The following subsections detail the hyperparameters used to train our checkpoints. For checkpoints taken from the literature, we summarize the available training details.
D.1 RESNET18 18 ON 10 TRAINING SETUP WITH ADAM AND SGD
We trained a total of 792 ResNet18 models on CIFAR- 10 by varying hyperparameter configurations. We used two optimizers: Adam and SGD. Tab. 2 summarizes the parameter combinations explored for each optimizer. For Adam, we used learning rates and momentum value. For SGD, we used learning rates and 2 momentum values. The remaining parameters shared across both optimizers: weight decay (3 values), LI regularization (2 values), Rand/ugment (2 values). Sharpness- Aware Minimization (3 values), and learning rate scheduling (2 values). This resulted in 216 models trained with Adam and 576 models trained with SGD. In the ablation studies, we filter checkpoints (as specified in the figure captions) to highlight the observed effects.
Parameter
Values
Optimizer
adam, sgd adam: 0.1, 0.01, 0.001
Learning Rate
sgd: 0.1, 0.05, 0.01, 0.001
Momentum
adam: 0.0
sgd: 0.9, 0.99
Weight Decay
0.0, 0.0005, 0.001
L1 Regularization
0.0, 1 x 10-5 True, False None, 0.05, 0.1
Rand/Augment
SAM (Sharpness-Awan) Aware Minimization)
Learning Rate Schedule
True, False
Table 2: Hyperparameter combinations used for ResNet18 training on CIFAR-10
D.2 PREACTRESNETI ON CIFAR-10
We 50 trained PreActResNet18 models on CIFAR-10 from (Andriushchenko et al. 2023 The models are trained using a fixed set of training parameters and a sweep over few key hyperparameters. Tab. summarizes varied parameters used in this experiment. All checkpoints used the same training protocol: 200 epochs, batch size 128, and no label noise. The model width was fixed at 64 and the learning rate schedule followed a eyclic pattern. Only the maximum learning rate (lr_max), SAM strength (sam_rho), and augmentation settings were varied. For the learning rate ablation studies, we adopt the reported maximum learning rate.
Download link: https:: https://drive.. http://wwww.....ooooee.eeeeeeeer
17

---

Published as a conference paper at ICLR 2026
Aununey (44 IranD Matar)
Accuracy (%) (LOLD -MG-2
10LD lest Aacumecy (5)
80
40
40
10
Layen Ni0I
CurrE Rodd
00000
1.0
OWED SoM 9ob3
20
15
20
40
60
20
40
60
eaccuuccyyyy 196]
LayarwWse Compression
Accu acy [96]
Layetulise uasion
(a) ResNet18, 8, Adam, FOLD vs MAG2, no Ll regularization
(b) PreActResNer18, FOLD vs MAG2
Aeuuney (%4 DE Malut
Socuraty 183
80
60
70
25
40
20:
Compo Complesiiin frsiin kodis
1.5
40
0.e
30
30
(c) ViT-B/32, FOLD vs MAG2
(d) CLIP VIT-B/32, FOLD vs MAG2
Figure 8: Folding outperforms magnitude pruning across diverse training regimes. The same setup as in Fig. [ but compared to the L2 magnitude pruning criterion. Top row: ResNet18 and PreAActResNet18
CIFAR-10. ResNet18 checkpoints different hyperparameter configurations. Bottom row: ViT-B/32 on CIFAR- 10 and CLIP ViT-B/32 on ImageNet-1 Scatter plots show post-compreesion accuracy for folding versus magnitude pruning (L2 criterion) at uniform per-layer compression ratios. Bar plots depict the accuracy gain by folding, computed A Acc(FOLD) Acc(MAG2), as function of layer-wise compression ratio. Folding yields the largest improvements at moderate to high compression, confirming its robustness across architectures and datasets.
trained from scratch with Adam using
22
Accuray [-] 00 30013)
300
rous teet Accurntt (M)
80
suns net Accurecy Po
8
Mapelii Bailig the
25
20
Compres Rott
15
60
5.7%
30
2.50
3
40
3
9
6
40
MAGZ Test Accuracy (75]
Layer wise Compressior Ratio
MAGZ Test Accuracy (75]
Layer wise Compression Rabo
(a) ViT-B/32, MAG2 FOLD, base accuracy >75%
(b) CLIP VIT-B/32, MAG2 FOLD
Figure 9: FOLD versus MAG2 on ViTs after LayerNorm-only fine-tuning for ViT-B/32 on CIFAR-10 and CLIP ViT-B/32 ImageNet-IK: IK. In the scatter plots. points are checkpoints, color encodes layer- wise compression. Bar plots depict the accuracy gain A = Acc(FOLD) - Acc(MAG1). which remains positive and typically grows with compression, indicating that even under lightweight LayerNorm adaptation FOLD retains consistent advantage over pruning. The figure follows the same setup as Fig. in the main paper, but for MAG2.
D.3 VIT-B/32 ON CIFAR-10
The 200 Vision Transformers (ViT) also from (Andriuscchenko et al., 2023), width=256, were trained on CIFAR-10, batch size 128, for 200 epochs with a cosine learning rate schedule and linear warmup. The main hyperparameters use of data augmentation, and the use of Sharpness-Awe Aware Minimization (SAM) in our evaluations. All other settings were fixed.
summarized in Tab. 4 We made of the maximum learning rate, the
18

---

Published as a conference paper at ICLR 2026
Parameter
Values
Optimizer
sgd
Max / Base Learning Rate (?r_max)
from 0.0504 to 4.9759
SAM Strength (sam_rho)
0.0, 0.05, 0.1
Standard Augmentation (augm)
True, False
RandAugment (randaug)
True, False
Table 3: Fixed and varying parameters for PreActResNett18 training on CIFAR-10.
FOLD Accuracy
speck
Accurcy After
22
MAGL Accuracy 760
H4G1 Accuracy (4)
Lesaewise Compreaaios Tabi
25 4)
(a) MAG1 vs FOLD, fine-tuning for I (left) and 5 (right) epochs.
(b) MAG1 vs FOLD (c) Before and after fine-tuning for epochs:
accuracy gap after ft. FOLD (leff) and MAG1 (right).
2 55.5
Accurasty After 2
12
75
Accurecy
Laye Commiression Rotio
50
20
39
(d) MAG1 vs FOLD, fine-tuning for I (left) and 5 (right) epochs.
(c) MAG1 vs FOLD (f) Before and after fine-tuning for epochs:
accuracy gap after ft. FOLD (left) and MAG1 (right).
Figure 10: FOLD outperforms MAG1 after full fine-tuning for 1-5 epochs on PreActResNet188 and VIT-B/32 on CIFAR-10. Results for 8 (top) and ViT-B/32 (bottom). (a,d) accuracy of MAG1 vs. FOLD after and epochs of fine-tuning. (b,e) accuracy gap over epochs, remaining positive. (c.f) accuracy trajectories from post-compression through epochs, showing faster recovery and higher final accuracy for FOLD. The figure extends Fig. Biin in the main paper to PreActResNet18 and ViT-B/32 architectures where FOLD is benchmarked against MAG1.
D.4
CLIP VIT-B/32 ON IMAGENET-IK
CLIP Radford al. 2021) models known for the widespread use of CLIP features (Ramesh et
We use the pool of models introduced by Wortsman et al. (2022b). who fine-tuned the CLIP ViT-B/32 architecture on ImageNet-1e multiple times using different randomly sampled training hyperparameter[] These hyperparameters include learning rate, number of training epochs, weight decay, label smoothing, and augmentation strategies, as stated (Wortsman et al., 20226). The resulting collection of 72 fine-tuned models provides strong basis for evaluating the performance of model folding compared to pruning on CLIP ViT architectures. All checkpoints were evaluated jointly in our study, without parameter-specific ablations.
2022
D.5 LLAMA-60N ON COLOSSAL CLEAN CRAWLED CORPUS (C4)
We train 36 LLaMA-family models (Tovrron et al. 2023a b) with 60M and I 130M parameters on the Colossal Clean Crawled Corpus (C4) (Raffel al. 2020) on NVIDIA DGX Station A100 featuring eight NVIDIA A 1100 GPUs (each equipped with 80GB memory). The training time for LLaMA-60M model is about 45 minutes. Tab, 5 summarizes the fixed hyperparameters used to train LLaMA-60M and LLLMA-130M,..... The learning rate is linearly warmed followed by a cosine
link: ott.a///iiitteiaaiooooooaanoannn
19

---

Published as a conference paper at ICLR 2026
Parameter
Values
Optimizer
sgd
Max / Base Learning Rate (lr_max)
from 0.005087 to 0.492936
SAM Strength (sam_rho)
0.0, 0.05, 0.1
Standard Augmentation (augm)
True, False
RandAugment (randaug)
True, False
Table Fixed and varying parameters for ViT-B/32 Base training on CIFAR-10.
annealing schedule that decays 10% of the initial value. We the base tokenizer (Raffel et
2023 and AdamW optimizer, consistent with prior work (Glentis et al. 2025 Han et al. 2024).
Params
Hidden
Intermediate
Heads
Layers
Steps
Data (Tokens)
60M
512
1376
8
11K
1.3B
130M
768
2048
12
12
22K
2.6B
Table 5: Training hyperparameters LLaMA-60M architecture.
Note that in our work, pruning and folding are applied exclusively to the feed-forward network (FFN) layers of the trained LLaMA-60M and 1LMMA-130M models.
E EXTENDED EMPIRICAL COMPARISON OF FOLDING AND PRUNING
We provide additional experiments complement the main results. Fig. S miirrros the setup of Fig. ! in the main paper, but replaces the 1.1 criterion for magnitude pruning with 1.2 (MAG2). Similarly, Fig. 9 Fig. TO Fig. [T] and Fig. 12 extend the corresponding figures in the main paper to other network architectures and to the 1.2 case. Across all comparisons, the qualitative picture remains the same: FOLD consistently matches or outperforms magnitude pruning, independent the chosen norm.
We further include ablations to study the robustness of these findings with respect to training hyperparameters. Fig. [3 Fig. 14 and Fig. 15 report the effect of varying learning rate, SAM strength, and RandAugment, respectively. Finally, Fig. 16 shows the influence of weight decay. Taken together, these studies confirm that the relative advantage of FOLD is stable across different regularization strategies and training configurations.
F ADDITIONAL. ANALYSES: SHARPNESS, RUNTIME, AND LLMS
Below report additional evaluations. We extend our study by training and compressing 60M- and 350M-parameter LLaMA models on C4, and provide analyses of sharpness, and measure runtime overhead.
F.1 SHARPNESS AND BARRIER ANALYSIS
We compute sharpness following the implementation by Andriushchenko et al. (202)). Sharpness for CLIP is measured only on the final projection layer using 1000 images, while for PreActResNets computed over the full model and dataset.
Sharpness increases with compression ratio for methods and architectures (Fig. reaching peak before stronger compression pushes the model out of its original basin and into flatter, lower-capacity region. This tise-theen-fa pattern appears consistently in both PreActResNet and CLIP models, see Fig.
The correlation analysis in Fig. 20 further supports this interpretation. Across the 200 compressed ResNet18 models, 50 compressed PreActResNet18 8s and 72 compressed CLIP models, FOLD exhibits
20

---

Published as a conference paper at ICLR 2026
accanery alre 1944
F0.0 accurccy
0
10
Accurecy (50)
Jccurary (4)
Lyyer wise Compresiion Rads
MA22 Safora
(a) MAG1 vs FOLD, fine-tuning for I (left) and 5 (right) epochs.
(b) FOLD vs MAG1
(c) Before and after fine-tuning for epochs:
accuracy gap after ft. FOLD (left) and MAG1 (right).
FOLD sccurcy
20
epect
Arnuary Alter FT
30 66
Accurscy
B 6 Accuracy
Layerweoo Compriaaac Tabe
94
30
FOLD Sefore
MA62 before
41
(d) MAG1 vs FOLD, fine-tuning for I (left) and (right) epochs.
Before after fine- fine-tuning epochs: accuracy gap after ft. FOLD (left) and MAG1 (right).
(e) FOLD vs MAG1
Figure 11: Folded models retain their accuracy advantage after fine-tuning. Results for ResNet18 trained by Adam (top row) and PreActResNet18 trained by SGD on CIFAR-10 (bottom row): (a,d) compares post-compression accuracy of magnitude pruning with L2 criterion (MAG2) versus folding (FOLD) after I and 5 epochs of fine-tuning. (b,e) show the accuracy gap between folding and pruning as function of fine-tuning epochs, demonstrating that folding maintains consistent lead, L.e,, the FOLD accuracy delta is positive. (c,f) illustrate accuracy trajectories before and after 5 epochs of fine-tuning for both methods, highlighting that folded models recover accuracy faster and reach higher final performance than pruned models. The figure extends Fig. in the main paper and Fig. T0 in the appendix to MAG2.
weight_decay
warmup_ steps
max Jr
PPL. 0% sparsity
PPL.
MAG2 (20%)
PPL. FOLD (20%)
PPL., MAG2 (50%)
PPL. FOLD (50%)
0.01
1100
0.001
23.90
39.48
236.16
308.77
2200
39.61
0.01
3300
0.001
24.08
38.54
39.10
289.67
451.27
1100
0.001
24.01
42.39
42.27
270.31
477.70
0.0
2200
0.001
21.12
40.01
41.53
239.25
489.48
3300
0,001
24.19
38.72
40.31
277.10
$31.09
1100
0.005
42.11
72.31
62.63
536.53
298.65
2200
333.59
0.01
3300
0.005
22.66
44.35
36.22
589.33
222.21
0.0
1100
0.005
44.92
73.38
63.75
414.17
261.23
0.0
2200
0.005
23.32
$7.62
43.04
1616.77
342.64
3300
0,005
23.00
46.87
39.11
904.07
305.85
2200
252.99
3300
0.01
51.34
97.35
76.15
440.71
229.42
0.0
1100
0.01
282.11
282.48
282.38
345.74
329.58
0.0
2200
0.01
140.20
169.78
149.58
352,14
234.69
3300
0.01
86.18
118.05
100.14
339.37
179.43
Table 6: Evaluation of FOLD and MAG2 on LLaMA-130M (in addition to I.LAMA-0MM evaluations in LaMilffamll models with 130M parameters on C4 while varying Ir, warmup steps, and weight decay. Columns show perplexity of the pretrained model (0% sparsity) and perplexity after structured magnitude pruning and folding with 20% and 50% sparsity in FFN blocks. For higher learning rates, especially for the settings with the best achieved performance in each sparsity category (underlined), FOLD consistently outperforms MAG2 (bold).
Tab. I.. We train and evaluate
negative correlations between A-sharpness and A-accuracy (A (AAcuuracy Acc(FOLD) (MGG)))) As shown in the scatter plots and correlation tables of Fig. 20 larger reductions sharpness under FOLD relative MAG associated with larger accuracy gains. This relationship holds across the
21

---

Published as a conference paper at ICLR 2026
2.
Accurasy After
1.0
2
13
Accurecy
tNd
Layerwide Compress on Ratio
43
6
00
62
(a) MAG2 vs FOLD, fine-tuning for I (left) and 5 (right) epochs.
(b) MAG2 vs FOLD accuracy gap after ft. FOLD (left) and MAG2 (right).
(c) Before and after fine-tuning for epochs:
10L0 Accunnn th
15
Ausunner Alter
A ure (hg
13 32
65
65
60 6
044 Laystroos Compeiss flabs
0.3
20
sefore
(d) MAG2 vs FOLD, fine-tuning (left) and (right) epochs.
(e) MAG2 vs FOLD
Before after fine-tuning epochs:
accuracy gap after ft. FOLD (left) and MAG2 (right).
Figure 12: FOLD outperforms MAG2 after full fine-tuning for 1-5 epochs ViT-B/32 and CLIP VIT-B/32. Results for ViT-B/32 on CIFAR-10 (top) and CLIP ViT-B/32 on ImageNet- 1K (bottom). (a,d) accuracy MAG2 vs. FOLD after and 5 epochs of fine-tuning. (b,e) accuracy gap A over epochs. remaining positive. (c,f) accuracy trajectories from post-compression through 5 epochs, showing faster recovery and higher final accuracy for FOLD. The figure extends Fig. in the main paper and
Fig. 10 in the appendix to MAG2.
With LA Schcoue Ne Schadiln
A.ccureyyy bs (toui MMbc3)
With SCHCOuuE No Scheduii
a.ccurncyy (Sl DLL werc)
Aacuracy (S)
80
80
x
60
20
20
Ratio (a) ResNet18, Adam, Ll reg., no weight decay
MAGZ [96]
ResNet18, SGD, L.1 reg., no weight decay
(S) (rou) Nocc))
Accurccy [S] (ILLD 160021
FOLLS Accuraccy
8
60
8
30
80
Iogiman LiN
20
20
20 40
60
MAGZ Accurscy 176)
Layer vise Compression Rato
MAG2 Accuracy (P6]
Layer Compression Ratio
(c) PreActResNet18
VIT-B/32
Figure 13: Learning rate modulates folding's edge. Post-compression accuracy MAG2 and FOLD across learning rates: ResNet18 with Adam (a) and SGD (b), Net1 (c), and ViT-B/32 (d). FOLD typically leads at moderate-low rates; the gap shrinks or reverses at very high rates, and closes again extremely small rates. The same setup as in Fig. in the main paper, but for MAG2.
evaluated compression ratios, up to the point where one of the models leaves the original basin and sharpness becomes less informative.
22

---

Published as a conference paper at ICLR 2026
0.100
(ennonyyySl)aaa (rouS MAACS
01:00
102
Amouaty DNJ RAALA)
25
RIED R015 BO0S
9.073
B075 Ba22 0 6.000
007 x
uz2
t. To TO
Recurnaty [sg)
000
603
60
I5.
60-
40
10
80
No SAM SAN
No SAM SAN
20
40
60
20 40
60
50
Accurady
196]
Layeneise Compo prossion Ratio
Accuracy
186]
Layorwss Compressior
(a) ResNet18, SGD, MAG2 vs FOLD, without 1.1 reg.
(h) ResNet18, SGD, MAG2 vs FOLD, with L1 reg.
Accurecy (%) (roLD MoGz)
Acccuracyy (nd broaus (0822)
2.075
0.075
80
30
-8 16
000
FOLD Aecurncy
60
So
0.200
60-
0.05
sp
0.525
%0
SAM
30
%
Accuracy
Raboo
MAG2
[86]
(c) PreActResNet, MAG2 vs FOLD, no L1 regularization
(d) ViT-B/32, MAG2 vs FOLD, no L1 regularization
Figure 14: SAM can boost model compression. Post-compression accuracy under training with without SAM. (a) ResNet18 (Adam), no LI. (b) ResNet18 (Adam), LI= 10 (c) Pre-
ActResNet18 (SGD), no L1. (d) ViT-B/32, no L1. The figure extends the results in Fig. 6 to MAG2.
Brcunery (8) jrean (0022)
Aucusatt [4] [rau Matuz)
30
80
25
Recunacy (86)
20-
20
t0
80
80
-10
-20
20
40
60
80
Accuracy 196]
Leyer-Wise Compression
MAGZ Acouracy [96]
Layer Compression
(a) Adam, MAG2 vs FOLD
(b) SGD, MAG2 Vs FOLD
A.renaery (s) prett (oa2))
Alsunity [N] (HOLD Moc.)
MAU
80
FOuD Aicuracy (S)
30
70
35:
60
20
10:
80
10
RING 1021
06
60
MA52 (96]
(c) PreActResNet, MAG2 FOLD
(d) VIT-B/32, MAG2 FOLD
Figure 15: Random augmentations narrow the folding folding-pruning gap. Post-compression accuracy on ResNet18 (CIFAR-10) trained without vs. with random augmentations: (a) Adam, (b) SGD, (c) PreActResNet, (d) V/T-B/32. The figure extends Fig, to MAG2.
In addition to the global sharpness trends discussed above, Fig. 21 and Fig. 22 provide a more fine-grained view of how training hyperparameters influence the relationship between A-sharpness and A-accuracy under compression. For -lam-traiied ResNet models (Fig. (T)). the scatter plots reveal strong and stable negative correlation: whenever FOLD produces lower sharpness than magnitude pruning, it also achieves higher accuracy across almost all pruning ratios. The structure of the point clouds, especially high learning rates, shows that Adam's adaptive scaling induce
23

---

Published as a conference paper at ICLR 2026
Aeronntyy (S) (PoiL (MAE2)
100
Aecunaty (43 DOLL RoaL
Accuracy
90
-10
Wo-- wD 3x
WD-4
WD 10
WD-1a
WD+l0
20
40
60
20
60
100
Accuracy 186]
Layerowse Compression Ratio
Accuracy (96)
Layaseiie Cor mosiion Ratio
(a) Adam, MAG2 vs FOLD
(b) SGD, MAG2 vs FOLD
Figure 16: ResNet18: Weight Decay. Test accuracy ResNet18 checkpoints trained with varying weight decay values. Weight decay does not diminish the ad advantage of FOLD compared to MAG2, especially for SGD-trained models.
Ncturacy [8] (rou -M631
prau nou..))
touD keat Recurecy 19
80
aie hate
Comsifo Kore
20
20
20
40
60
Accuracy 186]
Layerwise Compression
Accuracy [96]
Layer Compression
(a) ResNet18, Adam, FOLD vs MAG1, no LI regularization
(b) PreActResNeu18, FOLD vs MAG1
Accuncy (4g (ran Matcnt
Accuracy (%] (COLD MMGz)
HOLD Rocumery (8)
80
60
60
20
wole Compiiaalern Ruds
10
Compert morter wudss
-20
(c) R, Adam, FOLD MAG2, no regularization
(d) PreActResNet18,, FOLD vs MAG2
Figure 17: Folding magnitude pruning before REPAIR. The same setup as in Fig. 1 and Fig. for CNNs (ResNet18 and PreActResNet18 on CIFAR-10), but the performance is compared for both pruning and folding before REPAIR. Top row: MAG1, bottom row: MAG2. In both cases, folding shows stronger performance already before data-based REPAIR applied.
highly anisotropic sharpness profiles, which in turn amplify the divergence between the compression trajectories of FOLD and MAG.
In contrast, SGD-trained models (Fig. exhibit a weaker and more dispersed relationship between A A-sharpness and A -acccurayy, consistent with the flatter and more isotropic minima typically found by SGD. Under SGD, FOLD often remains slightly flatter than magnitude pruning even when A-sharpness as 0, explaining why FOLD maintains a mild yet more weakly correlated accuracy advantage. The interaction with SAM and augmentation further differs across optimizers: SAM tightens the A- sharpness distribution under SGD, stabilizing the performance gap in favor of FOLD, while RandAug tends primarily to reduce variance without introducing strong directional trends.
These results highlight that the predictive power of sharpness for pruning outcomes is optimizer- and ppperparameter-dependdee
sharpness differences are highly informative for Adam-trained networks but less so for SGD, even though FOLD consistently follows a smoother and less disruptive compression path than magnitude- saaded pruning in both regimes.
These findings align with recent work linking compression and landscape geometry. AdaSAP (Bair et al. 2024) treats pruning sharpness-aware process, and Zhang et al. (2025) show that
24

---

Published as a conference paper at ICLR 2026
(h POEDD 1903
6.0eeem Do BOLD (00)
Aocurncyy Dh POLD (9023
Rocsroo POLDD 990C3
140
POLLD
(h)
(a) ResNer18, Adam
(b) ResNet18, SGD
(c) PreActResNer18
(d) VIT-B/32
(e) CLIP VIT-B/33
A.ccuror INCL MAC3
ddd
M.tunr ( URLD MAD0
1. mary Do uoan Mapo)
Leteney (N) FCLD NM62
S
X15
Base Modd Texk Accurscy (S]
Base Modd Tosk Accurasy 1S]
Joan Nodel Tost Accunssy (P0)
Beac Model Teal Accurecy (Tc
Bose Model Texl Accurecy (S]
(I) ResNet18, Adam (g) ResNet18, SGD (h) PreActResNet188
(i) ViT-B/32
(i CLIP VIT-B/32
Figure 18: Uncompressed model accuracy vs. performance difference AAcuuracy
Acc(FOLD) (MG(M The same setup as in Fig. T and Fig. g Top row: MAG1, Bottom row:
MAG2. Model folding shows strong performance on models of difterent quality, with amplified effect on highhppeforriig models (especially on ResNet18, SGD and ViT-B/32).
feasible pruning ratios depend on intrinsic flatness. Our results support this perspective: compression initially increases sharpness degrees of freedom are removed within the same basin, but stronger compression forces the model into a flatter basin with reduced curvature. FOLD follows this trajectory more smoothly, maintaining basin structure and yielding lower barriers than MAG.
Worst-Case f. Sharpness
Worst-casee case F. Sharpness
Shargress
Worst-Casae f. Sharpnass
30n1
10
Wioret
2-
Compression Ratio
Compression Ratio
Compression Ratio
Compression Radoo
(a) ResNet18, Adam, FOLD (b) ResNet18, SGD, FOLD
(c) PreActResNet18,, FOLD
(d) CLIP ViT-B/32, FOLD
Worst-case sts-s---see-e.. z. Sharpness
Worst- Worst-cese care Starpness
Worst.com cace Sharpness
Woortt-case 2. Sharpmess
107
0-12--
53 31 25 de Compression Ratio
02 63 0.1 el. d6 07 Comgression Ratie
03 03
66 a3 6a
64
53
02
Compression Ratio
Compression Ratio
(e) ResNet18, Adam, MAG1
(f) ResNet18, SGD, MAG1
PreActResNet18, MAG1
(h) CLIP VIT-B/32, MAG1
Figure 19: Worst-case l sharpness as a function of compression ratio across architectures and pruning methods. Each subplot reports the sharpness distribution for independently trained models at three perturbation radii (p- = 10 4 x 10 10 10-3). Panels (a)-(d) show FOLD sharpness for ResNet18 trained with Adam and SGD, PreActResNett1s and CLIP VIT V/T-B/32, respectively. Panels (e)-(h) show the corresponding results for MAG1. Observed trends: (i) Sharpness generally increases with compression ratio up to moderate levels before flattening dropping extreme compression. (ii) FOLD produces on-average lower sharpness than MAG1. (iii) Transformer models (CLIP ViT-B/32) experience substantially sharper solutions under compression compared residual networks. These patterns indicate that FOLD maintains flatter loss landscapes across wide range of settings, whereas MAG more often drives models toward sharper and less stable minima.
25

---

Published as a conference paper at ICLR 2026
&eest Accumncy
Cammp nabio
Toxt Accurscy
Cam losatio laflo
Test Accuncy
Taat 44 accumicc
retto
10
38-18-12-4113
Worsscate J. Sttrppiis
3 3.Vrsicame J, Sharpnins
W.Vorelccee Sharpess
5 &Wersisure Sharpmano
Pearson
Spearman
Ratio
Pearson
Spearman
Ratio
Pearson
Spearman
10%
-0.635
-0.837
10%
40.219
-0.437
10%
-0.932
-0,520
20%
-0.488
-0.399
Spearman
30%
-0.830
-0.896
30%
-0.505
-0.654
30%
-0.612
-0.418
10%
-0.359
-0.400
40%
-0.628
-0.630
-0.481
-0.222
50%
-0.200
-0.943
50%
40.535
-0.655
50%
40.40
00388
30%
-0.520
-0.462
60%
-0,862
-0.910
60%
40.369
-0.403
60%
-0.164
-0,135
40%
0.148
0.058
70%
-0.775
-0.788
70%
-0.399
-0.419
70%
0.005
0.050
50%
-0.139
-0.383
80%
-0.454
-0.493
80%
-0.277
-0,163
80%
0.065
0.115
0.051
0.368
CLIP ViT-B/32,
FOLD vs MAG1
R Adam,
ResNet18, SGD,
PreActResNet18,
FOLD vs MAG1
FOLD vs MAG1
FOLD MAG1
Figure 20: Correlation between A-sharpness and A-accuracy across architectures and pruning baselines. Each column shows the relationship between pruning-induced differences in worst- case fx sharpness (A sharpness- FOLD MAG) and differences in test accuracy (A accuracy FOLD - MAG), for the pair of pruming methods indicated below each plot. Color encodes the layer-wise compression ratio. The tables underneath each subplot report Pearson and Spearman correlations at every compression ratio, quantifying how predictive the sharpness difference of the accuracy difference. Results are shown for FOLD MAG1 for ResNet 88 trained with Adam and SGD, Nee and CLIP ViT-B/32. Statistics are computed over 200 independently trained ResNets18 and 50 PreActResNet18 on CIFAR-IO, and 72 CLIP ViT-B/32 models on ImageNet- IK. Correlations use the sharpness value at = 5 x 10-4 Results at = 10 and = 10-3 are qualitatively very close.
F.2 RUNTIME OVERHEAD AND EQUIVALENCE OF COMPRESSED MODELS
We profile both the compression procedures and the inference behavior of the resulting compressed models on dedicated DGX A100 server equipped with dual-socket AMD EPYC 7742 CPUs (256 hardware threads) and 8x NVIDIA A100 80GB GPUs. All measurements use the THOP profile and report compression time, peak memory during compression, per-batch latency, FLOPs, and peak forward-pass memory before and after compression. For each architecture (PreActResNett1 and CLIP ViT-B/32), all compression methods generate the same compressed network topology (identical channel counts and tensor shapes). Consequently, all methods yield identical FLOPs and nearly identical latencies, demonstrating that inference-time behavior is determined entirely by the resulting architecture, not by the choice of compression algorithm. FOLD introduces a moderate one-off compression overhead, but its inference-time profile matches the other compressed models.
Method
Params
Comp. time [s] Comp, peak mem [MB]
Lat. [ms/Potcc]
Lat. [mssing]
PLOPS (MHLDP/ingg]
Feed peak mem [MB]
Original
11.122.170
3.69
0.0288
557.65
214.30
FOLD
4,008.466
9.45
457.47
J99.05
170.53
MAG
4.008.366
L.77
115.22
3.15
0.0216
199.05
169.82
Table 7: Runtime characteristies of PreActResNet18 before and after compression (64.1% parameter reduction). Latency is measured for a full batch. FLOPs are reported per image. Comp. time and comp. peak mem refer the overhead of running the compression method once.
Table 7 compares the original and compressed NretctResNNet18 at a layer-wise compression ratio of 0.4 (i.e., 64.1% reduction in model parameters). Compression reduces FLOPs from 557.65 MFLOP- s/image to 199.05 MFLOPs/imaee (a 64.3% reduction), improves latency from 3.69 ms/batch to roughly 3.15 ms/batch, and lower peak forward-pass memory (from 214.30 MB about 170 MB).
httpss//ggthbb.coouuttraattsssst
26

---

Published as a conference paper at ICLR 2026
rati=0..1
r0.
rotio-0.3
ratlo=0.4
razlo=0.6
ratio-0.8
learning Rate
p-00001
B 1-1.1
Test Accuracy
-23
Worst Sharpness
ratlowD.1
rotlo=0.2
raflowD.3
reflow..4
rarlowo.5
redooo.dd
ratoow.77
ratooo.8
ratlo99
SGN
A Test Accuracy
Worst-case Sharpness
Augionntation re.Reddug
Sundiog
Text Accur aby
Worst-case Sharpness
Figure 21: Sharpness-accuracy trade-off between FOLD and MAG1 for ResNet18 trained with
Adam. Each column corresponds to layer-wise compression ratio (0. (0.1-..9), and the three group models by learning rate, SAM configuration (including p), and RandAug usage. Points show A worst-case for sharpness (FOLD MAG1) A test accuracy (FOLD MAG1). Observations: (1) The difference in model sharpness strongly predicts the difference performance between FOLD and MAG1
almost all pruning ratios. (2) Learning rate: higher learning rates lead more dispersed sharpness changes. For models trained with Adam using high learning rates, MAG1 moves the model along less sharp path compared FOLD, which struggles to catch up. However, the behavior flips for moderate and low learning rates. (3) SAM/p: p: using SAM reduces the variability in the sharpness shift between methods, especially for larger p. A sharpness gets closer zero. (4) RandAug: augmentation show little specific visible trend.
Method
Params Comp, time [s]
Comp. peak mem [MB)
Lar [ms/batch]
Lat. [ms/ing]
FLOPS (MPLOPs/igg
Pud peak mem [MB)
Original 151,790,313
19.851
0.6203
2946.76
19418
PoLD
140,447,253
92.833
681.23
17.343
0.5420
2379.90
636.97
140,447,253
Table 8: Runtime characteristics of CLIP ViT-B/32 before and after compression (7.47% parameter reduction). Latency is measured for a full batch. FLOPs are reported per image. Comp. time and Comp. peak mem refer to the one-off overhead of running the compression method.
Table shows the same evaluation for CLIP ViT-B/32, where FFN blocks are compressed to with 20% layer-wise compression ratio. Here, FLOPs decrease from 2946.76 76 MFLOPs/inage to 2379.98 MFLOPs/mmaee 19.2% reduction), and latency improves from 19.8 ms/batch roughly 17.35 ms/batch (about 1.14x speed-up). Again, FOLD is the slowest method due to its iterative nature compute k-means clusters, but the compressed models share the same FL.OPs, memory and latency.
27

---

Published as a conference paper at ICLR 2026
13:0000.1
ratlo00.2
retiow0.3
ratlo-0.4
rotio-0.5
ratio=0.6
rato-6.7
ratio=3.8
leeming Rate
BI 1w210
TTsst Accuracy
Worst Sharpness
rarloup.11
ratlo=0.2
ratlowo.4
ratlow0.5
ratoowp.6
ratlow..7
ratlo0.9
SGN
4 Test Accuracy
A Worst-case L. Sharpness
ratiom0.7
ratlo=0.?
Augiiontation Rendlag
Sanding
Test Accuracy
Worst-case Sharpness
Figure 22: Sharpness-accuracy trade-off between FOLD and MAG1 for ResNet18 trained with
SGD. Each column corresponds to layer-wise compression ratio (0.1-0.9), and the three rows group models by learning rate, SAM configuration (including p), and RandAug usage. Points show A worst-case loo sharpness (FOLD MAG1) A test accuracy (FOLD MAG1). Observations: (1) For SGD, the relationship between A sharpness and A accuracy visibly weaker and more scattered than for Adam, reflecting the flatter and more anisotropic minima found by SGD. (2) Learning rate: Unlike Adam, higher SGD learning rates do not systematically increase the sharpness gap between the methods. For most ratios, FOLD tends to remain less sharp than MAG1, producing positive A accuracy even when A sharpness is near zero. (3) SAM/ p: SAM has a strong flattening effect under SGD A sharpness clusters tightly around zero, and the accuracy advantage of FOLD becomes more stable as p increases. (4) RandAug: Augmentation increases robustness pruning under SGD, reducing the spread in A accuracy and further weakening the sharpness sharpness-accuracy link. Overall, SGID-trained models exhibit regime where FOLD consistently follows gentler sharpness trajectory than MAG1, leading to a clearer accuracy advantage at moderate pruning ratios.
F.3 IMPACT OF ONE RANK SLACK AND SINGLETON FOLDING
In this section we separate two effects in our pruning folding comparison: (i) the influence of the one-rank slack in Theorems 2.2 and (ii) the intrinsic difference between pruning and folding projection operators. We therefore contrast the gain from increasing the pruning rank from & to k+1 (blue curves in Fig, 23) with the gain from replacing pruning by folding at the same nominal rank (orange curves). This isolates the contribution of rank from the contribution of the projection geometry.
Fig. 23 shows both effects: for each weight matrix W (in every layer of ViT or ResNet18), it plots the relative Frobenius error change when the retained rank increases by one (blue) versus when the method changes from pruning to folding fixed rank (orange), as function of k.
28

---

Published as a conference paper at ICLR 2026
Lager
Parami
Parani, mag
Ay
FL/PPs
PLOPS, mag
Act. fold
mag
N7. mag
conr
1026
1026
1050624
1050624
0
38912
18912
38912
38912
layer1.0.ooo 1.ccom [
12996
12996
13307904
13307901
0
38912
38912
0
38912
38912
layer! 1.0c0m22 2
12996
12996
133079014
13307904
38912
38912
38912
38912
layer1.
13307904
13907904
38912
38912
38912
38912
layar1.1Loom
1.1oww
13307904
13307904
38912
38912
38912
layer2.0.coov 2.0cowl
25962
25992
6653952
6653952
19456
19456
19456
19456
layor2.0.coav2
51984
51984
13307904
13307904
0
19456
19456
0
19456
19456
2888
2555
734328
734328
19456
19156
19156
19156
layer2.
51984
13307904
13307904
19456
19456
layer2.1 1.omm2
51984
51984
13307904
13907904
19456
19456
19456
19456
layer3.0.com
104652
104652
6697728
6697728
9792
9792
0
9792
9292
layer3.0.com
21068]
21068]
134135K4
13-135584
0
9792
9792
0
9792
9792
0
loyer3. Seeer3..shoriin Si-hhortatt
11628
11628
744192
744192
9792
9792
0
9792
9792
layar3.1
210681
210681
13489584
13483584
9792
layer3.1
21068]
13483594
9792
9792
layerd.0.com
422739
422739
6763822
6763824
0
4912
4912
0
4912
4912
layer 2.caa22
$4824]
$15241
13571856
13571856
0
4912
4912
0
4912
4912
layerd.0 1......sorrii ShorticutO
4697]
16971
751536
751536
4912
1912
4912
4912
layar.1 Loons
848241
13571856
13671856
layes41 Loom2
848241
13571856
13571856
4912
4912
4912
4912
linear
3080
3080
3070
3070
10
10
0
10
10
TOTALS
20035688
4003078
197728902
197725902
0
366370
365370
0
365370
365370
Table 9: Per-layer comparison of PreActResNet18 after FOLD and MAG2 at compression ratio 0.4. For each convolutional and linear layer we report parameters, per-image FLOPs, activation size, and the number of non-zero activations (effective activations). All per-layer differences are zero, confirming that parameters, FLOPs, activations, and effective activations are exactly matched between the two compressed models.
W w!
IW
A (lank())
W/
This quantity measures the improvement obtained when increasing the retained rank from k to k + 1 within magnitude pruning MAG2. It isolates the rank slack effect. Across all examined layers, the improvement from a single additional retained channel is small, especially in deeper layers.
IW W (k) IF W IW- ( ||F
Amethod (k)
|
This measures the gain obtained by switching from structured magnitude pruning to optimal folding at the same rank k. The improvements are one to two orders of magnitude larger than the corresponding Annk values for nearly all layers.
The results empirically support the clarification presented the rebuttal: although Theorems 2. 2.2 compare pruning at rank & to folding at rank ++ the contribution of the rank difference is negligible in practice. The blue curves show that ||W w [| changes only minimally when - 1++,, while the orange curves demonstrate that folding provides a substantially tighter approximation than pruning at comparable compression levels. This confirms that the practical advantage of folding arises primarily from the richer family of cluster-based projections rather than the added rank.
Fig. 24 reports the relative squared error for pruning, special folding W and optimal folding W across all layers of ResNet18 and ViT-B/32 on CIFARIO 10, evaluated at multiple keep ratios. Three consistent phenomena appear:
Pruning always yields the largest error. For every layer and every keep ratio, the pruning curves lie above both folding curves. This confirms empirically that pruning introduces the largest distortion of the original weight matrix.
Special folding W strictly improves over pruning. The construction used in Theorem 2.1 obtained by merging all pruned rows, leads to smaller reconstruction error for all layers and all keep ratios. This empirically validates the first inequality error (W,) 2 crror(W')
Optimal folding W achieves the smallest error. The k-means solution consistently attains the lowest error, verifying the second inequality error(W()) error(W;).
Importantly, the gap between pruming and both folding methods is larger than the very small difference induced by adding single additional cluster (iee, increasing the rank from A to k + 1). This supports
29

---

Published as a conference paper at ICLR 2026
the clarification made in the rebuttal: the practical advantage of folding does not stem from the +1 change rank, but from the richer family cluster-based projections that folding can realize.
The observed ordering, Le., error W error W', error(W;;, holds uniformly across all layers of 8 as well as FFN layers of ViT-B/32. This indicates that the theoretical inequalities not only valid in principle but also manifest strongly and consistently in real trained models.
Across all layers of ResNet18 and ViT-B/32, the gain from increasing the retained rank by one is consistently negligible, while the gain from replacing pruning with folding is one to two orders of magnitude larger (Fig. 23). Thus, the empirical advantage of folding is not a byproduct of the k - ++1 rank slack, but stems from the richer family of cluster-based projections that folding can realize. This is further corroborated by the layer-wise reconstruction errors in Fig. 24, where error (Wp) p) error (W') > error wW) holds uniformly. These results confirm that folding's 's superior approximation properties are structural rather than an artifact of rank.
The theory controls the loss via the Seraameter-Lischhiee bound L(W) L(W.)) S w W.IlF. In regimes where к is moderate, e.g., flat solutions obtained with smaller learning rates SAM- folding's s smaller Frobenius error reliably translates into smaller loss degradation. However, sharp minima such as those produced by Adam at large learning rates (see Fig. 21 and Fig. (2). the effective local becomes extremely large. In this setting, even tiny parameter perturbations cause large loss changes, and the ordering of Frobenius errors longer predicts the ordering of accuracies. Thus, the discovered failure cases of folding are not contradictions of the theory but instances where the Lipschitz assumption required for loss control breaks down due extreme curvature.
G USE LARGE LANGUAGE MODELS
We used ChatGPT for sentence-leve grammar correction and improvement, drafting trivial plotting snippets to produce figures from logs, and code readability edits. All ideas, proofs, experiments, and analyses are ours.
"ChaGGPT GPT-5: https::/p//.. //tp:://///ttttcomm
30

---

Published as a conference paper at ICLR 2026
Luyer
Purun
Av FLOPs_Ri FLOPSRRAR
AF
NZ,RM
NZ
das vileetatiinn head
513000
515000
munteeraeeeellllee peo]
262656
262656
mnanformmrmette restledee if inpic
105033
1051634
uansfoenerasiiiiii it pppp._.pn)
1049085
1040058
0
maneforneeeeehhane c aa oue poo
363896
262656
11
0
0
transforneteaaellede I inppec Sc
1050634
1058644
8
B
0
transheceeesstleess I mnlpic pro)
105906
O
II
0
363656
262656
6
6
candprrneennstiiess c
O
1049065
1040065
wanseeeneeeesilllls.. pood
263656
263656
tnavformermete aeeeladde mlpp:
1050634
1050634
massfrcneeestiiiiss 11 L.mip..
1049085
1049058
trandferneeeccehins... 2 camont pes)
262596
262656
tran transfornettacblleee.. 2 mlpp- fc
1050524
1050624
0
0
mranticaeeissttess mlppc pin
1058595
0
toan 3
proj
362556
262656
1050000
1050654
mransfornee wtw.oeeeeeesiosseeee. tonneformernntie ahallade mlpe fc
png
263586
253656
1080624
transfocner sesiiiesssssssissss.... anppp..ppp.... proj
1049085 9
1049000
trans mannforrnetacsieee aalp-e_ Jc
wannforreeeacchlee Satin peij
1050524
260.85
1059554
262656
1059624
0
transformeeenstttees 5 mlp pin
0
connfeeeeeteillllee it
proj
262556
abblakk mlpp. png
1040700
transfornee sestooliss 7
203586
200634
dernfoemenncceee a mlppe. Jc
1050424
cunsfrcaeeiissiiiieeee. anpp pro)
1049044
0
mrasshenncc ins 3 arm our pen)
2030.06
203356
0
trans maansoorneettsslleee & anlp- St
1050524
0
transformmeeetttee 88 mlp ping
1049
pooj
262556
0
0
0
milp...pg proj
1040065
conet
2359296
315005500
115005500
37652
37683
37632
visual trannfoemerreeablee D Dustiuue
500592
0
visual mmanstoeeee..ee... reshlocks 1) mlp
1859433
122850
122A50
visual iranefemerr reshincks 11 mlp: pry
1857754
9450000
1500
S8000
visual aassocenctteesllell
traedfemer i mly:
1989451
550592
9138800
122830
122890
B
0
122830
D
srasfoeeesseeeeleii 1atlp.c_ppp prooj
1857744
94345900
30400
0
visual traisfsccteeessstooooo 10 our prnj
04546800
943488000
0
1220500
visual transfonnerrcollllu. 11 Lalls Lalts.ol
transfocmer. reshorcsssttsss..cc.. 100mpp poor
300593
D
0
0
0
0
viont mmaedooeeee.eettee resteeeeeeeest 11 mipu to
1
viore inaedfemer mebbaaie 11 mipe peoj
1887766
0
280592
0
0
O
0
D
vostel traedeeme meblicc 2 mly=
0
1228300
122580
0
122810
proj
0
0
yonn trassficnte wtt.rrreeetoccssss.sss.
pedj
1
0
Viuill aasfoeeeer.ess www.eee..esllllla....mm A J out
Viuull wassfooeee.rseeeee.. wew.eeeeeeeellccssseeeeee A alpp.
transformer www.hncttttssooo 3.
1899433
500592
04546500
94345830
0
0
0
122890
0
0
1
0
122830
0
voaee inneeeemeennomm www.nnebbbbbbbbe mlp: pnj
0
viuual wassfocaee..eettle. um out peo
wanseemmrmmsiiiee mlp=
1889497
2
0
0. 1ET
122590
122598
0
0
122820
0
D
visual trassfemerr m.lp... gnn
94544999
94348890
6
38400
0
36-20
petj
1859433
04345300
A4348300
84348930
122850 1
122850
trandfemer meblocks
1867744
18400
visual wrsssooeeereseeeeee resticcssss..ss........ 7 uun
out
590592
0
0
0
0
D
visual www.saaeeeeeeeeeeeo.o.o. restieckss 7 aalp c
1859433
04345330
94345800
123830
L22850
122450
122850
viorel traadiennn.eeeeee.. reblinskk mlp: pnnj
1867744
1887744
94548300
94548800
36400
38400
36400
36420
visual musssorcnteeeeete rewteecksssstssss um out poo
590092
590599
0
0
visual trnsfeenrrrrriiins mly.
1889433
1889453
94346800
94348800
132850
122850
122350
123850
visual trassfeenrr alpp ppmj
1867744
1887744
94346900
94348820
C
38400
38400
0
38400
38420
penj
www.laaaseeeeeessleeeee allp t_.t
1559433
1859453
94345300
94345800
122850
122850
122350
122850
1867744
1887744
94546500
94548800
36400
18400
16400
38420
TOTALS
83633900
83633400
0
2379976704
2175976704
0
1972632
1972603
1
1973632
1972632
Table 10: Per-layer comparison of CLIP ViT-B/32 after FOLD and MAG2 at compression ratio 0.2 (global parameter reduction 7.47%). The table reports all convolutional and linear layers in the vision transformer and classification head, including their parameters. per-image FLOPs, activation sizes, and effective activations. The rransformerrreebblooks * modules belong to CLIP's text encoder. Because the ImageNet- 1k fine-tuned variant evaluates only the vision encoder and classification head, the text encoder is not part of the forward graph. THOP therefore records zero FLOPS and zero activations for these layers, while their parameters remain included in the model. Note that the per-layer totals (%= 8.36 x 107 parameters) are smaller than the full model parameter count (Rs 1.40 x 10*) because this table excludes components without FLOPs, such as token embeddings, positional embeddings, and LayerNorm parameters, which are included in the global counts but not part of the per-layer FLOP/activation analysis. Several projection layers inside the attention blocks show zero FLOPS because CLIP implements attention using fused operations; these operations are profiled at the block level by THOP rather than attributed to the individual Linear submodules. All per-layer differences are zero, showing that FOLD and MAG2 produce structurally identical compressed models on every layer affected by compression.
31

---

Published as a conference paper at ICLR 2026
NAGIIK-AAGUI-1)
MAGZICI-FOLDII!
layeri.o
Syer
Myyer
Mayrr
0.05
Arwaiite watter
6.02
a reative ere:
8.00
6000
reaanvv
6013
000
0.53
0.50
1aylr2 d.convl
1ayer2.0 cors?
layer2. d.dennampllee
layer2...cont
layar2.1.co W2
07707
0.030
Eililttee erter
reattre eme-
0.23
0.435
0.300
DCT0
0:000
GoOD
0000
0.00
0000
100
100
1001
leyer't commml
Layertt cemommmppee
leger's cennet
layert mrow
0.000
60225
nw aree arite
2.300
10775
10392
0.00
0.000
0.5
0000
Leyoed C.carv
Leyerd D mneo?
Leyord deaeneemppe
leyeed Loonel
Loywed rmrw
0.0129
0.33
GITEr
0.33
0.0000
0.22
0,005
01977
0:3
0.00
0.23
0000
356
400
700
160
200
600
303
400
200
400
MMG2)6--AG221-411
m.mehho: MAG2Xd-FOLDRd
tra montsoommrr.co Lytts...Lnntt.
Iaanafomerllyess..o 0.Ieet.3
transformerlay Le..
trans/omms.ayers.11..ee.
Grleetive emn
2.08
Eox
6:00
0.2
0.00
0.00
1000
200
1060
ayers
loyers
layers
Araiaive ertz
0.0
Grelbove
1:2
0.21
-5.0
560
2.2
2te toltformntt. ayan S.A. L.tt.1
Battdommrr la 6.1..
Banssfrmer Baparss L.st.1
frans/ormas www.:/omm..lyys....m 1.MLL
0.31
0.2
treatheec C70P
0.00
0.00
3.22
0.03
5.00
-204
3:00
Figure 23: Effect of increasing the retained rank by one is significantly lower than changing the compression method from MAG to FOLD. Comparison of the effect of increasing the retained rank by one (blue curves) and (ii) the effect of switching from MAG2 to FOLD at the same nominal rank (orange curves). Each panel corresponds a single weight matrix W in ResNet18 convolutional layers (top) and VTT-B/32 FFN layers (bottom) and shows the relative Frobenius error difference A as function of the retained rank k.
32

---

Published as a conference paper at ICLR 2026
Specia
Oyttmaal My
layeri c.com2
6.2
6.0
0.0
044
0.5
0.
24
0.6
38
kunp lollo
keep nall
beep rallo
lesg ralio
kusp robi
ayyr2..0 convt
layar2 D.como
layer2 deemaample.d
1ayer2.1.com/
layar2.1.omm
ra8lo
ratio
ratio
ayyer3 O convt
byyer3 O conw?
layer3 O doansempled O
1syer3 1: conv]
1aye/3 conv2
18.4000 sqpareed 2717
0.0
0.6
0.4
0.
0.5
0.4
24
04
06
58
keep
cocou
layeri doenssmmlle
byyerd:)
conu2
6.-
6.4
0.2
0.0
0.5
0.5
0.0
0.4
0.5
0.0
0.4
2.0
3.0
knpp 10/0
keep rall
koap rallo
locgg valio
kuup robr
Pruna
Special leld It
Optimal fute W/
burra layars.
tranffomme 3...et.....nd..3
bannfoomm Aayans
Inaraformoo layats
0.4
2.6
06
04
2.6
0.
keep reñi
keep ratio
keep ratio
keep ratio
bayens
b.3.rs
layers
layors
0.4
5.6
0.
0.4
08
0.4
2.6
08
knep mido
keep
keep ratio
keep ratio
bayers
byees
0.1
2.6
0.6
0.5
0.
0.3
2.6
0.5
1awp ratic
kewp
Sawp ratio
ratio
Figure 24: Relative reconstruction error of pruning, special folding W (from the proof of Theorem 2.1. and optimal folding Wi for all layers of ResNet18 (top) and ViT-B/32 (bottom). For each layer, we report the normalized squared Frobenius error |W w, w.lli//ll || || at several keep ratios hp/m, where e E (MAG2, singleton, FOLD]. MAG2 (blue) denotes structured magnitude pruning with kp retained rows. The special fold W (orange) merges all pruned rows into single extra cluster (ky = kp + 1). The optimal fold W (green) is the k-means solution with ky clusters. Across all layers, W', consistently outperforms pruning, and W yields the smallest error, empirically validating error W, )) 2 error(W) 2 crror(W?)
33


Reviews:
[{"review_id": "t2TT8UX1q7", "reviewer": "Reviewer_VBA4", "summary": "This paper introduces a unified theoretical framework showing that model folding and structured pruning are orthogonal projections onto different subspaces, with folding provably achieving smaller parameter reconstruction error by projecting onto cluster-structured rather than axis-aligned subspaces. The extensive empirical validation (>1,000 models across ResNet18, PreActResNet18, ViT-B/32, CLIP ViT-B/32 on CIFAR-10/ImageNet-1K) confirms folding consistently outperforms magnitude pruning, especially at moderate-to-high compression ratios, with advantages persisting through fine-tuning. While limited to vision models, the work makes a valuable contribution by establishing folding as a principled, geometry-aware alternative to pruning that better preserves network functionality, supported by both rigorous theory and comprehensive ablations identifying when folding excels (moderate learning rates, SAM training) versus when the gap narrows (extreme hyperparameters).", "questions": "see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "• **Solid theoretical contribution**: Formalization of pruning and folding as orthogonal projections with provable guarantees on parameter reconstruction error, providing principled understanding of compression methods\n\n• **Extensive empirical validation with thorough ablations**: Over 1,000 checkpoints across diverse architectures (CNNs/ViTs) and datasets, with systematic investigation of learning rates, SAM, augmentation, and regularization effects that clearly identify when folding excels\n\n• **Clear and informative figures**: The scatter plots with color-coded compression ratios and bar plots showing accuracy gaps effectively communicate results across all tested architectures\n\n• **Good organization and presentation**: Paper flows logically from theory to experiments to ablations, with comprehensive appendices providing complete experimental details", "weaknesses": "**Major Weaknesses**\n\n- **Limited Scope to Small-Scale Models and Datasets**: Evaluations are confined to relatively small architectures such as ResNet18 (11M parameters) and ViT-B/32 (~86M parameters) on CIFAR-10 and ImageNet-1K. The absence of experiments on large-scale models like LLMs (e.g., GPT series) or diffusion models limits confidence in the method’s scalability and its relevance to modern deployment settings with billions of parameters.\n\n- **Insufficient Comparisons to State-of-the-Art Methods**: Although the paper benchmarks against basic magnitude pruning (L1/L2), it lacks head-to-head comparisons with stronger recent baselines such as SparseGPT [1] and Wanda [2] approaches, despite referencing them in Appendix F. As a result, it remains unclear whether folding offers advantages over cutting-edge compression methods.\n\n- **Lack of Empirical Validation for Some of the Claims**: The paper offers intuitive rationales for hyperparameter effects (e.g., SAM for flatter minima, augmentations for invariant solutions) but lacks direct measurements like Hessian eigenvalues or sharpness metrics to quantify them. It cites literature on optimizers and learning rates influencing curvature but conducts no targeted experiments to verify causal links, leaving claims as unverified hypotheses.\n\n- **Lack of Predictive Metrics**: The paper's ablation studies spot cases where pruning beats folding (e.g., very high learning rates with Adam or large SAM radii), but it fails to create predictive tools like sharpness thresholds, weight sparsity ratios, or loss curvature metrics—to foresee these without tons of tests. This after-the-fact analysis offers little real-world advice on choosing methods.\n\n- **Calibration-Free Claim**: While described as calibration-free, the approach still depends on REPAIR (BatchNorm reset) for CNNs, LayerNorm resets for ViTs, and short fine-tuning (up to 5 epochs) to reach its best results. These requirements imply partial data access and tuning, which conflict with the notion of a truly data-free or zero-shot compression pipeline.\n\n**Minor Weaknesses**\n\n- **Deferred Related Work**: A detailed discussion of prior art is confined to Appendix F due to space constraints, which may force readers to cross-reference for context on novelty, potentially disrupting flow in the main text.\n\n- **Lack of Runtime Analysis**: Folding involves k-means clustering per layer, which is computationally heavier than magnitude-based pruning's simple sorting, but the paper provides no measurements or discussions of overhead, overlooking efficiency trade-offs.\n\n[1] Frantar, E. \"SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot.\" arXiv preprint arXiv:2301.00774, January 2023. \n\n[2] Sun, M. \"A Simple and Effective Pruning Approach for Large Language Models.\" arXiv preprint arXiv:2306.11695, June 2023 (published at ICLR 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ciAESUjkoV", "reviewer": "Reviewer_b6Qf", "summary": "This paper studies calibration-free model compression by framing pruning and model folding as orthogonal projections in parameter space. The authors argue that folding (weight clustering) preserves functional similarity better than pruning (weight removal) and provide both theoretical guarantees and large-scale empirical validation over 1,000 CNN and ViT checkpoints across CIFAR-10 and ImageNet-1K. Folding is shown to outperform pruning, especially at moderate-to-high compression levels.", "questions": "See the weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The projection-theoretic framing of pruning and folding is reasonable. Framing compression as orthogonal projection provides an solid interpretation.\n\n- The paper is writing well with good visualizations.", "weaknesses": "- The main argument is questionable. The paper argues that model folding is better than model pruning with closer distance to the original models. This statement is a little problematic. Since pruning domain has been developed broadly and deeply in the past decade, many pruning methods could produce pruning models even surpassing original models. The logic here is that pruning could eliminate some harmful neurons to make the pruned model to be stronger. Therefore, the distance to original model is not sufficiently strong and convincing enough to support the outperformance.\n\n- Numerical experiment is limited. The paper only compares with basic magnitude pruning schema. Given theoretical support being not sufficient, it should compare and discuss with more recent representative works, e.g., DepGraph, OTO, etc., which contains more pruning protocols.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t0xxC0ieIR", "reviewer": "Reviewer_y5Wu", "summary": "The paper frames structured pruning and model folding as orthogonal projections in parameter space: pruning is an axis-aligned projection, while folding is a low-rank, cluster-structured projection. The authors prove that folding yields smaller parameter reconstruction error and tighter (Lipschitz-based) bounds on functional perturbation than pruning. A large-scale, calibration-free study shows folding usually achieves higher post-compression accuracy, especially at moderate to high compression, and it keeps its edge after lightweight or short fine-tuning.", "questions": "Can you provide per-layer tables verifying that, for every comparison, parameters and FLOPs (and effective activations) are exactly matched between folding and pruning after the next-layer adaptation, not just approximately matched globally? If there are exceptions, quantify their impact.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Casting pruning and folding as orthogonal projections cleanly explains their geometric differences; theorems show folding’s strictly smaller projection error and hence tighter loss perturbation under mild smoothness\n\n- The evaluation spans >1000 checkpoints and multiple architectures/datasets, with consistent wins for folding at moderate to high compression and robustness to training variations.\n\n- Folding’s advantage persists after minimal recalibration (e.g., BN/LayerNorm reset) and 1-5 epochs of fine-tuning, which mirrors realistic deployment where full retraining is expensive.", "weaknesses": "- The core guarantee uses a one-rank slack (pruning rank vs. folding rank)  Although the authors state experiments match retained parameters/FLOPs, a theoretical result at exactly matched rank would remove any residual ambiguity, especially at low compression where a single unit can matter. Consider strengthening theory (e.g., conditions under which folding dominates at equal rank) or adding tighter empirical per-layer equality checks to preclude hidden capacity differences.\n\n- Results focus on accuracy; there’s little on real-world latency, memory bandwidth, or kernel fusion performance after folding - despite identical FLOPs at matched size, wall-clock behavior can differ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9eAnfk2edt", "reviewer": "Reviewer_kdrS", "summary": "The paper builds on model folding introduced in Wang et al. (2025), comparing it to structured magnitude pruning both theoretically and empirically. The authors show that both structured pruning and folding can be viewed as orthogonal projections in parameter space, and based on that show that folding has leads to a smaller reconstruction error with one-rank slack. Experiments on more than 1,000 checkpoints of CNNs and ViTs show that folding leads to a higher post-compression accuracy than structured magnitude pruning in most cases, especially at moderate to high compression; even without the one-rank slack assumed for the theoretical results. In-depth ablation studies show that the improvement of folding over structured pruning are most pronounced in training settings that promote convergence to flatter local minima.", "questions": "- How does the pre-compression model accuracy relate to delta accuracy? This is especially interesting since augmentation is both known to strongly impact final model performance and has a large impact on delta accuracy.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Overall, the paper is written well, easy to follow, and the reader is not left with many questions. The experiments are executed well, showing results for a multitude of models over four architectures, and in-depth ablation studies are provided. The experiments without post-compression reconstruction show that in most cases, FOLD outperforms MAG, with the gaps increasing with compression ratio, validating the findings in Wang et al. (2025). It is interesting to see how the training procedure impacts the performance of the two post-training compression techniques studied in the paper. Experiments comparing FOLD and MAG with post compression fine tuning of only normalization layers or the full models provide novel insights on how folding and magnitude pruning compare in this setting.", "weaknesses": "I have several concerns regarding soundness and contribution of the work, which I explain below. I hope that these remarks are helpful in improving the work and I am happy to discuss my evaluation.\n\n- Theorems 2.1 and 2.2 compare structured magnitude pruning with $k$ retained rows and folding with $k+1$ retained rows and conclude that folding, in theory, outperforms pruning. However, the error inequalities given in the theorems say nothing about the performance of folding compared to pruning as pruning with $k+1$ retained rows also outperforms pruning with $k$ retained rows.\n- The claim of \"folding incurs provably smaller functional deviation\" seems to refer to the output or decision of the model, but is based on the parameter-Lipschitz assumption of the loss. This bound does not control the change in model output directly. \n- The setup is called \"calibration-free\" yet for CNNs, there is no evaluation without data-based post-pruning reconstruction. Even in the no fine-tuning setting, REPAIR is used to re-estimate the batch-normalization. \n- The experiments are conducted only with small scale architectures, the largest being CLIP ViT-B-32 at 151M parameters. The authors point out that calibration-free structured pruning methods collapse in the LLM pruning setting. However, experiments on larger vision transformers like ViT-22B would be interesting.\n- The figures in the paper mostly contain plots comparing model quality after FOLD and MAG compression but not the quality of the uncompressed models. In some cases, this makes it hard to judge if low accuracy is due to compression performing badly or the base model itself having low accuracy - it might be the case that FOLD performs better especially in the cases where the base model itself is weak.\n- Minor typo: In line 146, the loss difference is measured using the Frobenius norm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)