Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging
Authors: Max Zimmer
Christoph Spiegel
Sebastian Pokutta
Venue: ICLR 2024 poster (2024)
Keywords: pruning
retraining
model averaging
neural networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Neural networks can be significantly compressed by pruning, yielding sparse models with reduced storage and computational demands while preserving predictive performance. Model soups (Wortsman et al., 2022) enhance generalization and out-of-distribution (OOD) performance by averaging the parameters of multiple models into a single one, without increasing inference time. However, achieving both sparsity and parameter averaging is challenging as averaging arbitrary sparse models reduces the overall sparsity due to differing sparse connectivities. This work addresses these challenges by demonstrating that exploring a single retraining phase of Iterative Magnitude Pruning (IMP) with varied hyperparameter configurations such as batch ordering or weight decay yields models suitable for averaging, sharing identical sparse connectivity by design. Averaging these models significantly enhances generalization and OOD performance over their individual counterparts. Building on this, we introduce Sparse Model Soups (SMS), a novel method for merging sparse models by initiating each prune-retrain cycle with the averaged model from the previous phase. SMS preserves sparsity, exploits sparse network benefits, is modular and fully parallelizable, and substantially improves IMP's performance. We further demonstrate that SMS can be adapted to enhance state-of-the-art pruning-during-training approaches.

Published as a conference paper at ICLR 2024
SPARSE MODEL SOUPS: A RECIPE FOR IMPROVED PRUNING VIA MODEL AVERAGING
Max Zimmer', Christoph Spiegel' & Sebastian Pokutta'
'Department for Al in Society, Science, and Technology. Zuse Institute Berlin, Germany Institute of Mathematics, Technische Universität Berlin, Germany tz tzimmer, spiegel, pokutta ( @zib.
ABSTRACT
Neural networks can be significantly compressed by pruning, yielding sparse mod- els with reduced storage and computational demands while preserving predictive performance. Model soups (Wortsman al., 2022a) enhance generalization and out-of-distribution (OOD) performance by averaging the parameters of multiple models into single one, without increasing inference time. However, achiev- ing both sparsity and parameter averaging is challenging as averaging arbitrary sparse models reduces the overall sparsity due to differing sparse connectivities. This work addresses these challenges by demonstrating that exploring a single retraining phase of Iterative Magnitude Pruning (IMP) with varied hyperparameter configurations such as batch ordering or weight decay yields models suitable for averaging, sharing identical sparse connectivity by design. Averaging these models significantly enhances generalization and OOD performance over their individual counterparts. Building on this, we introduce SPARSE MODEL SoUPs (SMS), a novel method for merging sparse models by initiating each prune-retrain cycle with the averaged model from the previous phase. SMS preserves sparsity, exploits sparse network benefits, is modular and fully parallelizable, and substantially im- proves IMP's performance. We further demonstrate that SMS can be adapted to enhance of
sirliindduinn--riiiiii approaches.
INTRODUCTION
State-of-the-an Neural Network architectures typically rely on extensive over- paramterrizationn with millions or billions of parameters (Zhang et al., 2016). consequence, these models have significant memory requirements and the training and inference process is computationally demanding. However, recent work (e.g. Han et al., 2015; Lin et al., 2020; Renda et al., 2020; Zimmer et al., 2022) has demonstrated that these resource demands can be significantly reduced by pruning the model, i.e., removing redundant structures such as individual parameters or groups thereof. The resulting sparse models demand considerably less storage and floating-poin operations (FLOPs) during inference, while retaining performance comparable to dense models.
different line of research has shown that the performance of predictor can be significantly enhanced by leveraging multiple models, instead of selecting the best one on a hold-out validation dataset and discarding the rest. Such ensembles combine the predictions of m E N individually trained models by averaging their output predictions (Ganaie et al., 2021; Mehrtash et al., 2020; Chandak al., 2023; Fort et al., 2019). Prediction ensembles have been shown to improve the predictive performance and positively impact predictive uncertainty metrics such calibration, out-of-distribution generalization as well as model fairness (Lakshminnarayaaan et al., 2017: Mehrtash et al., 2020; Allen-Zhu & Li, 2023; Ko et al., 2023). A significant drawback of ensembling is that all models have to be evaluated during deployment: the inference costs are hence increasing by a factor of m, a problem that has partially been addressed by leveraging ensemble of sparsified, more efficient models (Liu et al., 2021; Whitaker & Whitley, 2022; Kobayashi et al., 2022).
Several works propose to instead average the parameters (Izmailov al., 2018; Wortsman et al., 2022a; Rame et al., 2022; Matena & Raffel, 2022), constructing a single model for inference. Unlike prediction ensembles that require sufficiently diverse models for better performance, such

---

Published as a conference paper at ICLR 2024
Model Soups (Wortsman et al., 2022a) need models to lie in linearly connected basin of the loss landscape. However, training models from scratch with differing random seeds but identical initialization often yields models whose parameter average will perform much worse than the individual models (Neyshabur et al., 2020) with recent studies investigating neuron permutation to align them within single basin (Singh & Jaggi, 2020; Ainsworth et al., 2023). Beyond the initial challenge of identifying networks suitable for averaging, another problem emerges when attempting leverage the computational advantages of sparse networks: averaging models with different sparse connectivities reduces overall sparsity (cf. Figure 1) and may require to prune again (Yin et al., 2022a;b), potentially resulting in further performance degradation.
In this work, we tackle the challenge of concurrently lever- aging sparsity as well as the benefits of combining multiple models into single one. We draw our inspiration from recent work in the domain of transfer learning (Neyshabur et al., 2020; Wortsman et al., 2022a; Rame et al., 2022), which has shown that fine-tuning multiple copies of a pretrained model, differing only in random seed, yields models sufficiently similar for averaging and sufficiently diverse for generalization improvements. At the core of our work lies the observation that single prune-retrain phase in standard prune after training strategies, such as ITERATIVE MAGNITUDE PRUNING (IMP, Han et al., 2015), closely resembles the transfer learning paradigm. Starting from pretrained model, the optimization objec- tive shifts abruptly, either due to new target domain or subspace constraints imposed by pruning, followed by training process termed 'fine-tuning', often used interchangeably with 'retraining' to recover from pruning (Hoefler et al., 2021).
(0 + 02)
Creating the average (mid- dle) of two networks with different spar- sity patterns (left, right) may lower over- all sparsity, changing pruned weights (dashed) to non-zero (solid), with reacti- vated weights highlighted in orange.
Figure 1:
We find that, akin to the fine-tuning phase in transfer learning, exploring various hyperparameter configurations during the retraining phase after pruning generates models that are suitable for averaging while sharing the same sparse connectivity by design. Such sparse averages exhibit superior performance compared to both their individual counterparts as well as to models retrained m times as long, effectively reducing IMP's runtime. Additionally, initiate the next prune-retrain cycle from the averaged model just obtained, which remarkably also enhances the performance of the individual retraining runs before averaging again. Our proposed approach, SPARSE MODEL SOUPS (SMS), tackles the aforementioned challenges and enables inference complexity independent of m, utilizes pretrained models without requiring training from scratch, preserves the sparsity pattern while leveraging sparsity benefits, and considerably improves IMP's generalization and OOD performance.
Contributions. To summarize, our contributions can be stated as follows.
1. We demonstrate that pruning a well-trained model and retraining multiple copics with varied hyperparameter like batch ordering, weight decay, retraining duration and length, produces models suitable for constructing an averaged model which exhibits superior generalization and OOD performance compared its individual components. Importantly, these models retain the sparsity pattern of their pruned parent, preserved in their parameter average.
2. We propose Sparse Model Soups (SMS), a novel method for merging sparse models into a single classifier, leveraging the idea of starting each prune-retrain phase of IMP from an averaged model. SMS significantly enhances the performance of IMP in two ways: first, the average improves upon the individual models in terms of generalization and OOD performance and, secondly, the models retrained from an average exhibit better performance compared to those retrained from single model.
3. We extend our findings to the pruning during training domain, demonstrating SMS's s versatility by integrating it with multiple other state-of-the-an approaches. This yields substantial performance improvements and enhances their competitiveness in comparison to other leading methods that sparsify during training.
2

---

Published as a conference paper at ICLR 2024
Algorithm 1 Sparse Model Soups
Retrain
Merge
Input: Pretrained model 0
01
Output: Sparse model soup
Prune 0p
I: for each prune-retrain cycle do Prune
0
02
0 Davg
for i 1 to do
Fully parallelizable
03
0 0
5: Retrain 03 with specific hyperparameters
6: end for
Repeat
0 <- -ereee(()....... .m)
8: end for
return 0
Sketch of the algorithm for single phase and Right: Pseudocode for SMS. Merge(-) takes m models as input and returns a linear combination of the models (cf. Section 2.2).
Figure 2: Left:
Outline. We introduce our framework in Section In Section 3, we experimentally validate our findings across image classification, semantic segmentation, and neural machine translation architectures and datasets. Section reviews relevant literature, followed by a discussion in Section
METHODOLOGY: SPARSE MODEL SoUPS
2.1 PRELIMINARIES
Our focus lies on model pruning which aims at removing individual weights as exemplified by the previously introduced IMP approach. IMP, a prune after training algorithm, follows three-stage pipeline. It starts with pretrained model parameterized by 0, prunes weights with magnitudes below a certain threshold, and then restores predictive power through retraining. This prune-retrain cycle is repeated multiple times, with each pruning step's 's threshold determined by the suitable percentile to achieve the desired target sparsity after a predefined number of such phases. Recent studies (Gale al., 2019; Zimmer et al., 2023) have demonstrated that magnitude pruning results in sparse models with performance competitive to more complex algorithms.
Given sparse models fo, with weights 0 TR" i {1,...,m)) m), prediction ensembles construct model as the functional equivalent of the average of the models' output (Liu et al., 2021; Whitaker & Whitley, 2022; Kobayashi al., 2022). This ensemble requires m forward passes for evaluation, but maintains the overall sparsity level. In contrast, our focus lies on examining the performance and sparsity of single model, specifically linear combination of other models, Given scalars X, R. we consider the prediction function fö. parameterized by the weights given by
Ū= Σ X2O, Isism
(I)
special case occurs when A; 1/m for all i, resulting representing the average of all networks. Averaging the weights of arbitrary sparse models result in reduced overall sparsity, as different networks may possess distinct sparse connectivities, causing the averaging process to eliminate zeros from the tensors (ef. Figure 1). Yin et al. (2022b) and Yin al. (2022a) address this issue by pruning 0 to align with the original networks' sparsity levels. However, this approach has a notable drawback: the sparsity patterns differ significantly, pruning-indee induced performance degradation may occur.
2.2 SPARSE MODEL SOUPS
Inspired by recent advancements in the transfer learning domain (Neyshabur al., 2020; Wortsman et al., 2022a), which demonstrate that models fine-tuned from the same pretrained model end up in the same loss basin and can be combined into soup, hypothesize that similar behavior can be achieved during retraining from the same pruned model. Our motivation stems from the resemblance between the transfer learning paradigm and single phase of IMP. When transitioning from the
3

---

Published as a conference paper at ICLR 2024
source to the target domain, the optimization objective changes abruptly, requiring adaptation (i.e., fine-tuning) to minimize the new objective. Similarly, 'hard' pruning alters the loss abruptly and requires adapting (i.e., retraining) given the newly added sparsity constraints.
A single phase of this idea is illustrated on the left of Figure 2. The pretrained model's weights 0 are pruned, yielding model 8g. which is then replicated times. In this setup, pruned weights remain permanently non-trainable Subsequently, each of the m models is independently retrained with different hyperparameter configurations, such as varying random seeds, weight decay factors, retraining lengths, or learning rate schedules. Finally, the m retrained models are merged into a single model. This process ensures that all m retrained models f1,.. 8r share the same sparsity pattern, as they all originate from the same pruned network with a fixed pruning mask. However, when combining models after multiple prune-retrain cycles, identical sparsity connectivity between all models is not guaranteed. To address this, we average the models after each phase and begin the subsequent phase with the previously averaged model. The resulting method, termed SPARSE MODEL SOUPS (SMS), is presented pseudocode on the right of Figure 2.
SMS offers several benefits and addresses key challenges. First, the inference complexity of the final model remains independent of m. The method is highly modular, allowing for different hyperparameter configurations and different m in each phase. Further, the retraining of the m models can be fully parallelized, enhancing efficiency as detailed in Section 3.2. By initiating each phase with the merged model from the previous one, sparsity patterns are preserved, and the advantages of sparse networks utilized; the number of cycles increases, the networks become sparser, potentially leading to further efficiency gains. Moreover, SMS effectively leverages the benefits of large pretrained models without the need for training from scratch.
Effectively merging models for enhanced generalization can be challenging, as models may end up far apart. We primarily employ two convex combination methods from Wortsman al. (2022a): UniformSoup and GreedySoup. UniformSoup equally weighs each model with A 1/m. On the other hand. GreedySoup orders models by validation accuracy, sequentially including models only if they improve validation accuracy over the prior subset.
3 EXPERIMENTAL RESULTS
We first outline our general experimental approach. For reproducibility, our implementation is We evaluate our approach on well-known datasets for image
available at Sithub.com/ZIB-ION recognition, semantic segmentation, and neural machine translation (NMT), including ImageNet- (Russakovsky et al., 2015), CIFAR-IO/10 (Krizhevsky et al., 2009), Celeb-A (Liu et al., 2015), CitSSaape (Cordts et al., 2016), WMT16 DE-EN (Bojar et al., 2016) and the benchmark OOD- datasets CIFAR-1O0 architectures, such as ResNets (He al., 2015), WideResNets (Zagoruyko & Komodakis, 2016), MaxVIT (Tu et al., 2022), PSPNer (Zhao et al., 2017), and the 75 transformer (Raffel et al., 2020). For validation, we use 10% of the training data. We use magnitude-baxe norm-based structured pruning as suggested by Li et al. (2016). For retraining, we stick to the linear learning rate schedules LLR and ALLR (Zimmer et al., 2023), with further details in Appendix A.2. Appendix A describes exact hyperparameters and settings for pretraining, pruning and retraining.
and ImageNet-C (Hendrycks & Dietterich, 2019). We utilize state-of-the-ar
unstructured pruning and filter
Recomputing Natch-Normalization (BN) (loffe Szegedy, 2015) statistics crucial in both pruning and model averaging, as observed by Li et al. (2020) and Jordan et al. (2022), respectively. When reporting test accuracy for single or averaged models, we reset all BN layers and recompute statistics using a forward pass the entire training dataset.
3.1 EVALUATING SPARSE MODEL SOUPS
We evaluate SMS against key baselines, beginning with a comparison at each prune-retrain phase to the best-performing single model among all averaging candidates (best candidate), the mean accuracy of these candidates (mean candidate) and regular IMP (i.e., averaging candidates are retrained from a previous model soup, distinguishing best candidate from regular IMP without averaging. Given the lower computational demands of regular IMP compared to SMS, which trains m models per phase and hence increases the total number of retraining epochs by a factor of m, we also contrast SMS with extended IMP version retrained times as long (IMP mix ).
1). From the second phase onwards,
4

---

Published as a conference paper at ICLR 2024
Table 1: WideResNett200 on CIFAR-100 and ResNet-50 on ImageNet: Test accuracy comparison for target sparsities 98% (top) and 90% (bottom) given three prune-retrain cycles. We report results using UniformSoup as well as GreedySoup for merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
1-1AR-10 (98%)
Spursity 72.8% (Phase 1)
Sparsity 92.6% (Phase 2)
Sparsity 90.0% (Phase 31
Accuracy of
3
3
10
10
SMS (uniform)
76.50
76.59 #6.13
76.75 a0.28
75.55
76.19
76.21
72.67 s0.29
72.90 s0.64
73,05
best candidate
75.58 +0.19
75.71 +0.08
75.96 +0.13
+0.47
75.01
+0.74
75.00 -0.34
71.77 +0.04
71.77 +0.37
72.21 +0.00
macin condidate
75.37 20.12
75.58 20.03
75.55 :0:26
74.32
74.71 50.48
74.70 =0.42
71.43 20.09
71.61 10:40
71,66 s0.19
SMS (gresdy) mean candidate
76.06 #0:69
76.43 =0.24
76.60
20.4
75.34 40.15
75.39 50.44
75.51
=0.66
72.08
#0.25
71 86 10.64
72.44 +0.20
75.58
75.65 s0.00
75.94
=0.21
71.05
75.37 +0.12
75.54 +0.03
75.54 +0.27
74.52
74.27
74.20 -0.31
70.84 +0.41
70.69 +0.75
70.87 +0.01
IMP IMP- IMP-RePrunc
75.85 +0.26
76.05
+0.00
75.76 +0.34
74.09 +0.34
74.19
+0.44
74.74-0.06
70.92 +0.07
70.51 +0.52
71.85 +0.15
N/A
NIA
68. #0.46
65.53 #0.06
63.62
$0,90
75.54
74.09
70.74
ImageNet (90%)
Sparsity 53.6%
(Phase
Sparsity 78.5%
(Phase 2)
Sparsity 90,0% (Phase
Accuracy of
mi =
mh m=
10
m =
m = 10
SMS (umilormn)
76.74 #0.20
76.89
20.18
77.01 10,05
76.04 10:21
76.30
10.13
76.49 20.13
74.53-20.04
74.82 20.06
74.96 10.16
best candidate
76.07 s0.01
76.07
s0.21
76.14 20.18
75.48
211.16
75.46
s1.11
75.70=0.1
74.00 s0.03
74.19
74.25 211.33
maan candidate
75.99 s0:04
75.95 20.14
75.96 20.08
20.11
20.10
15.55 20.05
73.91
2000
74.11 20.11
#0.12
SMS (greedy)
76.74
76.92 20.15
76.88 #0.11
76.12 20.18
76.35
+9.21
76.11 =0.25
74.58 20.03
74.77 20.03
74.52 #0.11
best candidate
76.08 50.01
76.08 20.21
76. 20.18
75.48
10.18
75.53
#0.24
75.34
=0.19
74.03 10.11
74.21 10:00
73.95 +0.07
candidate
75.98 20.14
75.95
75.95
75,42 211.15
75.45
75.24
20.17
13.94.001
74.09
73.76
Pox IMP-RePrune
76.25 s0.0
76.21 #0.14
76.46 s0.04
75.74
#0.11
75.93 =0:03
74.34 20.99
20.24
all.0
N/AA
NIA
72.97 20.25
72.58 #0.01
72.08 #0.12
IMP
75.97 20.16
75.19 20
73.59 20.04
Unlike SMS, IMP mx cannot be parallelized in each phase as the extended number of epochs are executed sequentially, yet, the overall computational and memory requirements remain identical between both methods. We also compare SMS to another baseline termed IMP-RePrunc where regular IMP is executed m times and model averaging is performed after the final phase. Unlike SMS, which merges after every phase and hence maintains a consistent pruning mask, the individual models in
may develop diverging pruning masks over multiple phases, potentially reducing the overall sparsity when averaged. To ensure comparable sparsity levels, MMP-RePrune incorporates repruning step to address any sparsity reduction after averaging (Yin al., 2022b).
Table 1 presents results for three- preee IMP using WideResNet-20 on CIFAR- 100 and ResNet-50 on ImageNet, employing ALLR and ten retraining epochs per phase. For SMS, we vary the random seeds across each of the m models. The three main columns correspond to phases and sparsities, targeting 98% sparsity for CIFAR-100 and 90% for ImageNet. Each main column has three subcolumns, indicating the number of models to average (3, 5, or 10). We discuss the main observations below.
1. SMS significantly enhances generalization. We find that SMS consistently improves upon the test accuracy of the best candidate, often with a 1% or higher margin. This confirms that the models after retraining are averageable, resulting in better generalization than individual models. SMS notably improves upon both regular IMP and its extended retraining variant, IMP mx with up to 2% enhancements even when using m 3 splits.
2. Starting from model soup enhances generalization. Surprisingly, the best candidates in the second and third phase frequently exceed both IMP and IMP While some improvement is anticipated when picking the best among multiple candidates, it is notable that the mean candidate accuracy (i.e., mean candidate) often surpasses both IMP and IMP mx as well. This suggests that initiating from a soup, as opposed starting from a singular model as in regular IMP, enhances generalization in the subsequent phase.
3. MMP-RePrune faces sparsity reduction and performance degradation. Naively averag- ing IMP's models in the final phase often leads to reduced sparsity due to differing sparse connectivities, requiring repruning which typically degrades performance compared to indi- vidual models. Under certain conditions, MMP-RePrune can remain competitive, suggesting that similar pruning patterns may emerge across multiple pruning rounds (cf. Appendix B.
5

---

Published as a conference paper at ICLR 2024
In summary, we find that averaging after each phase and starting subsequent phases from the soup of the previous phase capitalizes on two dynamics, which often enable significant improvements over IMP: first of all, the model soup consistently improves upon individual soup candidates, demonstrat- ing that pruned and retrained models are indeed averageable and exhibit enhanced generalization. Secondly, models retrained from a pruned soup also outperform those following the classical prune- retrain cycle. Pruning a model with higher generalization performance yields better models after retraining, despite experiencing larger cruning-induces performance drop. For full results on different architectures, datasets, target sparsities, and structured pruning, we refer to Appendix B.
Table also contrasts uniform and greedy soup selections. With just the random seed varied for the m models, none appear to diverge to different basin, rendering greedy subset selection unnecessary. The uniform approach predominantly outperforms the greedy one, notably when comparing the best or mean candidates in scenarios like the last phase of CIFAR-100, indicating that retraining from previous greedy soups yields less performant models.
3.2 EXAMINING SPARSE MODEL MERGING
Having established the merits of SMS, we now investigate its success and limitations more detail.
Exploring parameters beyond random seeds. Previously, we focused on varying the random seed for simplicity. Fig- ure 3 presents a scatter plot for One Shot IMP (70% spar- sity) of ResNet-50 trained on ImageNet, comparing the effects of varying the random seed, weight decay strength, retraining duration, and initial learning rate of linearly decaying schedule. Exact hyperparameters Appendix B.2. Parameter averages are constructed from two-clement pairs of models in the uniform soup setting. The plot displays the test accuracy of the averaged model versus the maximal test accuracy of each pair. In this set- ting, most averaged models show a net improvement over their individual components, demonstrating that varying dif- ferent hyperparameters in the retraining phase of One Shot IMP produces models within the same loss basin. Com- paring different hyperparameters, the most substantial and consistent improvement comes from varying the random seed. Unlike the random seed, which only introduces vari- ability due to inherent randomness, other parameters such as weight decay have a direct, controllable impact on the results; for instance, a poorly chosen weight decay value could significantly degrade performance.
ResNet-50 en ImageNet
76.0%
25.3%
listed in
relmiin aclasdila
setzoin weight decoy
74.3%
randon wvii
74.500
75.00%
75.00%
76.0055
bas: candidate accuracy
Figure 3: Accuracy of average of two
models vs. the maximal individual ac- curacy. All models are pruned to 70% sparsity (One Shot) and retrained, vary- ing the indicated hyperparameters.
DOD-robustness and fairness. We also explored if SMS enhances robustness to fitto---ditribttionn data, akin to regular model soups (Wortsman et al., 2022a), using benchmark robustness datasets CIFAR-100-C and ImageNet-C: (Hendrycks & Dietterich, 2019) for evaluation. SMS consistently outperformed individual models from IMP and other baselines, displaying better resilience to common corruptions, especially with ImageNet-C, which showed up to a 2.5% increase in OOD-aceuracy (see Figure& 8, Table 14 in Appendix B.2.3 for more details). Further, previous research suggests that pruning can exacerbate unfairness across data subgroups (Hooker al., 2019; 2020; Paganini, 2020). In Appendix B.2.4, we examine if SMS could alleviate such pruning induced unfairness, and found SMS to exhibit less severe negative impacts on individual subgroups than IMP.
Instability to randomness and recovering it. Neyshabur et al. (2020) demonstrated that during training from scratch, the inherent randomness in batch selection alone suffices to cause divergence between two models to the extent that they are not averageable, even when starting from identical (random) initialization. Such instability to randomness can be mitigated by ensuring sufficient pretraining: Frankle et al. (2020) specifically analyze the amount of training required before splitting a network into two copies further trained with different random seeds, such that the final models reside within linearly connected basin. In that vein, several works (Frankle et al., 2020; Evci et al., 2022; Paul et al., 2023) study the stability of IMP with weight rewinding (IMP-WE in the context of the Lottery Ticket Hypothesis (Frankle & Carbin, 2018). In contrast, explore retraining networks

---

Published as a conference paper at ICLR 2024
WRN-20 on CIFAR-100
WRN-20 on CIFAR-100
2,0%
2.0%
Ass (soup) -Ace(bes))
1.0%
AIc
0.0%
0.0%
-1.0%
-2.0%
22.0%
Acx (ooup)
-3.0%
-4.0%
Spansity 9055
UniformSoup
Sparsity 96%
-4.0%
GreelySoop
Sparsh Sparlity 9855
-6.0%
70%
75%
80%
05%
100%
30
spansity
nrmbber of reiraining eporhes
(a)
(b)
Figure WideResNet-20 on CIFAR-100: (a) Accuracy difference between the soup (m = 5) and best averaging candidate after One Shot pruning and retraining for varying sparsity levels. (b) Accuracy difference between the soup (m = 3) and IMPxx retrained three times as long as indicated on the x-axis, using One Shot pruning to 90%, 95% and 98% sparsity. Results are averaged over multiple random seeds with min-max bands indicated.
without rewinding. Based on the aforementioned instability analysis, we conjecture that different splits of pruned network converge common basin under low sparsity and moderate learning rates, while high pruning levels may potentially reduce stability to randomness.
Figure 4a shows the difference in test accuracy between soup of m 5 models and the best candidate at different sparsity levels. Each point corresponds to the best configuration across varying retraining lengths (5, 20, and 50 epochs) and schedules (LLR and ALLR). UniformSoup and GreedySoup enhance accuracy by up to 2% over individual models. Yet, as sparsity increases, this benefit declines, with UniformSoup collapsing in performance. Beyond certain sparsity, stability randomness declines, causing model divergence and hindering beneficial model averaging. Unlike UniformSoup. GreedySoup performs at least as well as the best individual model.
Similarly, Figure 4b depicts the difference between the model soup (m = 3) and IMP where the latter is retrained for k epochs and the former trains m models for k epochs, with k denoted on the x-axis. Again, we plot the best configuration varying the retraining schedule (LLR, ALLR) and merging method (UniformSoup. GreedySoup). For moderate sparsity (green), averaging m models is more effective than training single model m-times as long, even with brief retraining. At high sparsity (red), short-term retraining and averaging m models underperforms compared extending single model's 's training. However, a break-even point emerges around 15 epochs, beyond which the benefit of extended single model training diminishes, with the m models sufficiently trained for merging. In iterative pruning, expect SMS to require fewer retraining epochs per phase, benefiting from the gradual sparsity increment. Despite extensive retraining, IMP mx fails to match SMS.
Efficiency of SMS. Each IMP-cycle consists of k epochs, while IMP, mox extends this to k epochs sequentially. Contrarily, SMS executes each phase with m distinct models, independently trained for k epochs, allowing parallelization that can lower wall-time by factor of 1 1/m compared to IMP mox. Nevertheless, the overall compute and memory requirements for both IMP and SMS remain at units. Resource-wise IMP mx and SMS are hence par; however, the parallelization in SMS underscores a practical advantage. Further, our data shows SMS often significantly outperforming IMP mxx suggesting comparable accuracy can be achieved with fewer total retraining epochs.
In Appendix C, we conduct ablation studies to individually assess the impact of the retraining schedule and the number of epochs per phase during the execution of SMS.
3.3 IMPROVING PRUNING DURING TRAINING ALGRITHMM
We extend our findings mngnitude-pruning based methods within the pruning during training domain. These methods, unlike IMP, start with a randomly initialized model and sparsify it during regular training. We focus GMP (Zhu & Gupta, 2017: Gale al., 2019), DPF (Lin et al., 2020),
7

---

Published as a conference paper at ICLR 2024
Table 2: ResNet-50 on ImageNet: Comparison of BIMP, GMP and DPF with their SMS-extended variants for goal sparsity levels of 70%, 80% and 90%. Each subcolumn denotes the top-1 accuracy and the theoretical speedup at a given sparsity. All results are averaged over multiple seeds and include standard deviations. The best, second best, and third best values are highlighted.
ImageNet
Sparsity 70%
Sparsity 80%
Sparsity 90%
Method
Accuracy
Speedup
Accuracy
Speedup
Accuracy
Speedup
BIMP+SMS
76.20 +0.09
2.7 =0.0
75.76 #0.11
3.7 =0.0
74.05 =0.02
6.1 #0.0
BIMP
75.62 #0.02
2.7 #0.0
75.08 #0.16
3.7 #0.0
73.53 #0.05
6.1 +0.0
GMP+SMS
75.10 +0.00 10.00
2.7 #0.0
74.48 #0.00
3.9 #0.0
73.12 #0.02
7.7 #0.0
GMP
74.55 +0.07
2.7 +0.0
73.92 #0.12
4.0 +0.0
72.81 +0.00
7.0 20.0
DPF+SMS
76.26 #0.10
2.7 #0.0
75.85
+0.05
3.6 +0.0
74.31
=0.00
6.0 +0.0
DPF
75.74 40.02
2.6 +0.0
75.27
+0.02
3.6 #0.0
73.88 s0.01
5.9 +0.0
GSM
73.69 +0.70
2.9 #0.1
72.75
+0.62
40.3
70.08
#0.94
9.5 40.8
DNW
75.81 20.05
2.5 =0.0
75.35 20.21
30.0
74.24 a0.12
5.5 +0.1
LC
75.03 40.20
2.4 +0.0
73.87 20.62
3.20
67.57 42.71
5.1 +0.0
DST
10.01
4.1 =0.0
72.32 +0.03
9.7 +0.3
71.35 +0.09
13,2 =0.4
and BIMP (Zimmer al., 2023). GMP applies a pruning schedule to iteratively update pruning mask throughout training. DPF, while following the same schedule, enables error compensation by updating the dense parameters using the pruned model's gradient. BIMP divides the training budget into a pretraining phase and multiple IMP cycles thereafter, rendering it closest to IMP.
These three approaches can be easily adapted using SMS. BIMP can integrate SMS within individual phases. Both GMP and DPF prune at uniformly distributed timesteps during training. We regard the interval between two such steps as a phase, during which we create m copies of the recently pruned model, train them with different random seeds, and merge them before the next pruning step. Unlike IMP or BIMP, GMP and DPF follow the original learning rate schedule throughout a phase.
Table 2 compares accuracy and sparsity- theoretical speedup (Blalock et al., 2020) of the three methods and their SMS-enhanced versions to other state-of-the-art pruning during training methods like GSM (Ding et al., 2019), DNW (Wortsman et al., 2019), LC (Carreir-Peeppnnae & Idelbayev, 2018), and DST (Liu et al., 2020). For a fair comparison, we solely consider the dense-to-span training paradigm,
opposed to pruning initialization (Lee et al., 2019; Tanaka et al., 2020) or dynamic sparse training (DST, Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evei et al., 2020) methods. We applied I.C and GSM both randomly initialized and pretrained models, selecting the best results for each sparsity, noting the original works only applied these to pretrained models. Further, we experimented with STR (Kusupati et al., 2020), but omitted the results we were unable to cover the exact sparsity range, being controllable only indirectly through regularization parameters. Detailed training and hyperparameters can be found in the corresponding subsection of Appendix A.2.
Incorporating SMS into BIMP. GMP, and DPF consistently improves performance, despite their deviation from IMP. BIMP benefits the most, likely due to the decaying learning rate facilitating convergence in each phase. In comparison, SMS stands out as a simple yet effective way to enhance the competitiveness of magnitude- based pruning methods, noting that SMS- -MM-ennhaced methods increase computational costs by branching into models, a cost mitigable through parallelizing.
4 RELATED WORK
We review the related literature, focusing on sparsity-related studies. We refer to Hoefler et al. (2021)
for a comprchensive review of sparsífication approaches.
Model Averaging. Stochastic Weight Averaging (Izmailov et al., 2018) averages parameters across the SGD trajectory for improved generalization. Wortsman et al. (2022a) and Rame et al. (2022) demonstrate model soups' enhanced generalization and perfforfoee by averaging models finetuned with varying hyperparameters. The approach closest to ours in Section 3.3 is Late-phase learning (von Oswald al., 2020), independently training and averaging specific parameters, albeit
8

---

Published as a conference paper at ICLR 2024
without pruning. Gueta al. (2023) explores fine-tuning in language models, revealing clustering- like behavior with regions around close models containing potentially superior models. Croce et al. models, while Choshen et al. (2022) enhance base models by merging multiple finetuned ones. Wortsman et al. (2022b) introduce robust fine-tuning through averaging zero-shot and fine-tuned models. Similar SMS, concurrent work by Dolicoeur-Martinee et al. (2023) regularly averages independently trained models, although without pruning.
(2023) explore soups of adversarialll-robut
We highlight key distinctions between our work and existing studies that combine sparsity with parameter averaging. Yin et al. (2022b) utilize dynamic sparse training, averaging models within a single run with fixed hyperparameters, in contrast to our prune- trua-anntr-training method that averages models across multiple rums with diverse hyperparameter settings. Their prune-and grow approach, exploring different sparsity patterns and requiring re-pruning to maintain sparsity, contrasts with our method which deliberately avoids re-pruning by keeping consistent sparsity patterns. We explicitly demonstrate that this approach significantly improves upon the re-pruning approach (IMP-RePrune)) even when using strategies like CIA CAA that are designed to mitigate the impact of re- re-pruning (Yin et al., 2022b). Similarly, Yin et al. (2022a) employ IMP with weight rewinding, averaging IMP subnetworks of different prune-retrain-cyee -yycles across a single training trajectory, which also requires re- re-pruning, unlike our approach of averaging parallely trained models. Furthermore, their objective to generate lottery tickets, as opposed to creating sparse models for inference. In a similar vein, Stripelis et al. (2022) introduce FedSparsify, a Federated I Learning algorithm that centrally updates a global mask with local client masks, resolving disparities through majority voting. Furthermore, our work is distinct from that of Jaiswal et al. (2023), who focus on averaging early pruning masks for mask generation, whereas we concentrate on averaging parameters of sparse models.
Mode Connectivity. Neyshabur et al. (2020) demonstrate that models trained from scratch are not linearly connected, while models finetuned from a pretrained model tend to be similar and reside within the same loss basin. Entezari et al. (2022) conjecture different-seed trained models are linear mode connected up to neuron permutations. Partially demonstrating this, Ainsworth et al. (2023) propose permutation algorithms for transforming models into shared loss basin and Singh & Jaggi (2020) employ model fusion for neuron soft-alignment, further also enhancing filter pruning by fusing dense into sparse models. Similarly, Benzing et al. (2022) introduce permutation algorithm, demonstrating that models share a loss valley (up to permutation) even at initialization. Jordan et al. (2022) explore 'variance collapse' in interpolations of deep networks, proposing mitigation strategies. Several works (Frankle et al., 2020; Evci et al., 2022; Paul et al., 2023) study IMP's stability to randomness, specifically with weight rewinding (IMP-WR). Evei et al. (2022) demonstrate that trained lottery tickets and IMP-WR solutions converge to identical basins, while Paul et al. (2023) find successive IMP-WR solutions at varied sparsity are linearly mode connected, maintaining loss stability along the linear interpolation between adjacent solutions.
Prediction Ensembling. A range of studies focus on prediction ensembling, where outputs of multiple models are averaged (Lakshminarayanan et al., 2017: Huang et al., 2017; Garipov et al., 2018; Mehrtash al., 2020; Chandak al., 2023). In the sparsity context, Liu et al. (2021) leverage DST for efficient generation of diverse ensemble candidates. Whitaker & Whitley (2022) form ensembles by randomly pruning and retraining model copies, while Kobayashi et al. (2022) finetune subnetworks of a pretrained model. We refer to Ganaie et al. (2021) for a survey of ensembling.
5 DISCUSSION
Efficcint, high-performing sparse networks are crucial in resource-constrained environments. How- ever, sparse models cannot easily leverage the benefits of parameter averaging. We addressed this issue proposing SMS, technique that merges models while preserving sparsity, substantially enhanc- ing IMP and outperforming multiple baselines. By integrating SMS into magnitude- pruming methods during training, elevated their performance and competitiveness. Despite the focus pruning, a single type of network compression, we think that our work serves as an important step towards understanding and improving sparsification algorithms.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was partially supported by the DFG Cluster of Excellence MATH+ (EXC-2046/1, project id 390685689) funded by the Deutsche Forschungseemeinshaat (DFG). We would like to thank Berkant Turan and Christophe Roux for providing useful comments.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023. URL https::/// http:///ppeeeeeeeeeeem net/ forum? d d- CQs.mMmmmmMm.P1
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. In The Eleventh International Conference Learning Representations, 2023. URL wwwptttp///t/ /twp/rppeeeeeeeee//ee/eee ew net/ rum???
Frederik Benzing, Simon Schug, Robert Meier, Johannes von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. September 2022.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In I. Dhillon, D. Papaillopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2. pp. 129-146, 2020. URL https: / http///oooedii/eeooeooo nggs mlays. org/ brg/pape pppppp/p220000 weeffflllfffeeeem d2ddea 8 www.ff0666e868. 6233e 355 Paperer.77pppppp.p......
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http: /ttp://wwwwwwwo www. aclweb. org/ anthology /////////62200
Miguel A Carreira-Perpinall and Yerlan Idelbayev. Leaming-compression algorithms for neural pruning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018
Amit Chandak, Purushottam Kar, and Piyush Rai. Gradient perturbation-based efficient deep ensem- bles. In Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD), pp. 28-36, 2023.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. April 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Francesco Croce, Sylvestre- Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. February 2023.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing July 2019.
performance. arXiv preprint
Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo. Jungong Han, and Ji Liu. Global sparse momentum sgd for pruning very deep neural networks. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' d'Alché-BBi Alché-Buu, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https: httpps:/ppooeeednnge http///ppcccce.ccce/eepppoo neurips. pdd.
cc/paper// / 22009 9 /ile/ 3 B5c4c55d58 (wt.:/5555771144411111
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity neural networks. In International Conference on Learning Representations, 2022. URL https: wpw.reviee..eeeee/e/ee/ee net forum baaprruumidddNiiytteeeee
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference
Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2943-29929999 2952. PMLR, 13-18 Jul 2020. URL https: https://proceedi http//peeciigggeeeeeeeeeeeeeecep mlr. press/vl 9 evei 20a aaa html.
11

---

Published as a conference paper at ICLR 2024
Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pp. 6577-65888 1586, 2022.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. December 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, arXiv preprint arXiv:1902.0955 09574, 2019.
M. A. Ganaie, Minghui Hu, A. K. Malik. M. Tanveer, and P. N. Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 2022, April 2021. doi: (0.10166..nngapai2222..1111...11
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowl- edge region in weight space for fine-tuned language models. February 2023.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran Asso- ciates, Inc., 2015. URL https: http///ooeeedngggeeeeoee meurinn wwwpuuupss.ccpppee//eeeeeee wttpccppp//////////// //t01/ pdf.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. March 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. March 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training neural networks. arXiv preprint arXiv:2102.00554, January 2021.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? November 2019.
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. October 2020.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snap- shot ensembles: Train 1, get for free. In International Conference on Learning Representations, 2017. URL https://// http///ppeeeevieeeeeeeeeeeeeeeee
net www.nnee/oorrrae
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org, 2015. URL http:/ / ptwp/eocceeiicce ngs ml p.r.ssses////em... v
html.
12

---

Published as a conference paper at ICLR 2024
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. March 2018.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, and Zhangyang Wang. Instant soup: Cheap pruning ensembles in single pass can draw lottery tickets from large models. June 2023.
Alexia Jolicoeur- Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julien. Population parameter averaging (papa). April 2023.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renor- malizing permuted activations for interpolation repair. November 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1412 6980, 2014.
Wei-Yin Yin Ko, Daniel D' D'souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. eair-ensembl:: When fairness naturally emerges from deep ensembling. March 2023.
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. Diverse lottery tickets boost ensemble from single pretrained model. May 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5544-5555. PMLR, 13-18 Jul 2020. URL https: /pppceedinggs mlr presss /atess/vlli wwwl9//kusupat///
html.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Duong Hoang Le and Binh-Son Hua. Network pruning that matters: case study on retraining variants. In International Conference on Learning Representations, 2021. URL https:// openreview , net/ forum? diw.ummdddbb44a AMMMHFF....
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer- sparsity for the magnitude-ban based pruning. In International Conference on Learning Representations, October 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Singleshot network pruning based on connection sensitivity, In International Conference on Learning Representations, 2019. URL https: http:s//ppeerevviee net/ / forum? 1d=B1V74AACYX.
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient
neural network pruning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 639-654. Springer, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. August 2016.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In International Confer- ence Learning Representations, 2020. URL https: /oopnrevviii ew. net / / forum? d= SJ1bGGrEED.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. Proceedings of the International Conference Machine Learning (ICLR 2022), June 2021.
13

---

Published as a conference paper at ICLR 2024
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Michael Matena and Colin A Raffel. Merging models with fishhr-weighted averaging. Advances in
Neural Information Processing Systems, 35:17703-17716 2022.
Alireza Mehrtash, Purang Abolmaesumi, Polina Golland, Tina Kapur, Demian Wassermann, and William Wells. Pep: Parameter ensembling by perturbation. Advances in neural information processing systems, 33:8895-8998 8906, 2020.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1), June 2018. doi:
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 13::122-23,, 2020.
Michela Pagannii. Prune responsibly. September 2020.
Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? In The Eleventh International Conference on Learning Representations, 2023. URL https: net hwwpeewweee//ffooroeeeee www.mmmmddddsasmmmmmam
Sebastian Pokutta, Christoph Spiegel, and Max Zimmer. Deep neural network training with frank- wolfe. arXiv preprint arXiv:2010.0724: 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
htttp://jmse ott/papers/pppppoooooooopp.eesst pttppppppss/v2/222 074.html.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, patrick gallinari, and Matthieu Cord. Diverse weight averaging for dutto--distribution generalization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neurul Information Processing Systems, 2022. URL https:/ opwnrevieeeeeeeeeeeeeeeeee net / forrmm/rrmmm//mm// id=tq_ J M_MMMBB 3UB.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):21 1-252, 2015. doi: 1007/s// 1263-015-08 16-y.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
Dimitris Stripelis, Umang Gupta, Greg Ver Steeg, and Jose Luis Ambite. Federated progressive sparsification (purge, merge, tune)+. April 2022.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural net- works without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems 2020, June 2020.
Cuong Tran, Ferdinando Fioretto, Jung- Eun Kim, and Rakshit Naidu. Pruning has disparate impact on model accuracy. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /woporpeeeeeeeeeeeeeeeee ew. net/Eorum? iid InMVZKOWYM.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao
Li. Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 459 479. Springer, 2022.
14

---

Published as a conference paper at ICLR 2024
Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans, Christian Henning, Benjamin F. Grewe, and João Sacramento. Neural networks with late late-phase weights. Published as a conference paper ICLR 2021, July 2020.
Tim Whitaker and Darrell Whitley. Prune and tune ensembles: Low-cost ensemble learning with sparse independent subnetworks. February 2022.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari, Discovering neural wirings. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https::/prroeeedingg neurips /ttpg/gnenniipppss.paae/////// d010396caBate 6ead8 www.cccc2cwccccceee222
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 33965-23999, PMLR. 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 1959-7971. 2022b.
Lu Yin, Shiwei Liu, Fang Meng, Tianjin Huang, Vlado Menkovski, and Mykola Pechenizkiy. Lottery pools: Winning more by interpolating tickets without increasing training or inference cost. August 2022a.
Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Dece- bal Constantin Mocanu, and Shiwei Liu. Superposing many tickets into one: A performance booster for sparse neural network training. May 2022b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 46 May 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 161.00000,,,,0 November 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2881-2890, 2017.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. arXiv preprint 1rXiv:/710.0153 October 2017.
Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Compression-awan training of neural networks using frank-wolfe. arXiv preprint 1.1:22205.11921, 2022.
Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. How Learned To Stop Worrying And
Love Retraining. In International Conference on Learning Representations, 2023. URL https: // /oppnneeiiee net /formmmmmm d=__dd_________-- imFIQI.
15

---

Published as a conference paper at ICLR 2024
APPENDICES
Technical details and training settings
17
A.I Pretraining
17
Training settings and metrics.
17
A.2 Pruning and Retraining
17
Pruning settings.
17
Retraining schedules.
18
Hyperparameters for Retraining.
18
Model Soups and Batch-Normaie Normalization statistics.
18
A.3
Pruning during ................................
19
Hyperparameters for Pruning during Training algorithms.
19
Extended results
20
B.1
Evaluating Sparse Model Soups
20
B.2
Examining Sparse Model Merging
24
B.2.1
Exploring parameters beyond random seeds.
................................
24
B.2.2
Instability to randomness and recovering it.
25
B.2.3
OOD- ROD-Robustness of Sparse Model Soups..
26
B.2.4
Reducing compression-induce -nnducce unfairness.
28
C Ablation studies
29
C.1 Ablation: SMS hyperparameters
29
C.1.1 Ablation: The retraining schedules.
................................
29
C.1.2 Ablation: The retraining length.
29
C.2 Ablation: Suitable baselines for SMS
30
................................
C,3 Ablation:
Differences to Stochastic Weight Averaging
30
C,4 Ablation:
Performance degradation for extreme levels of sparsity
31
16

---

Published as a conference paper at ICLR 2024
A TECHNICAL DETAILS AND TRAINING SETTINGS
A.1 PRETRAINING
Training settings and metrics. Table shows the exact pretraining settings for each dataset- architecture pair, reporting the number of epochs used for pretraining, the batch size, weight decay as well as the learning rate used. We stick SGD as the optimizer, noting that variety of other optimization methods for training deep neural networks exist (see e.g. Kingma & Ba, 2014; Pokutta al., 2020). We keep momentum at the default value of 0.9. The last column reports the performance we achieve when performing regular dense training. For image classification tasks, we report the top-1 I test accuracy being the fraction of correctly classified test samples. For semantic segmentation, we used pretrained backbones and evaluated the mean Intersection-over-Unee dataset, which we use as the test set. For NMT, we report the BLEU score on the test set with sequence length limited to 128. We utilized label smoothing and gradient clipping for Max VIT. If needed, we report the theoretical speedup, a metric indicating the FLOPs ratio for inference between dense and sparse models (Blalock al., 2020). The speedup, defined Fa/ F, where Fa and F, the FLOPs required for dense and pruned models respectively, depends solely on the distribution of pruned weights, not on the values attained by non-zero parameters. FLOPs are computed using single test batch, with code adapted from the ShrinkBench framework (Blalock et al., 2020).
(loU) on the validation
Table 3: Exact pretraining configurations in our experiments.
Dataset
Network Coumther of weiglits)
Epochs
Batch size
Weight docay
Learning rate ( training epoch!
Uspruned test accuracyyooU/BLLL
[1,90]
CIBAR 10
ResNet 18 (11 Mio)
200
128
Sc 4
0.01
F 180).
95.0% 2014
2000
Celeb-A CIERR ImnogeNar
ResNet-18 Mio) WRN 20 Mio)
1e-5
26-4
lincur from
ResNes-50 Mio)
1o-4
lincar from
766.22% 10.0
0.2%
E 20)
220, co).
ImageNett
MasVIT (5) Mio)
200
256
le-5
0.02
E 661, 120)
78.0%
0,002
160.
0.0002
E [161, 200
0.15
[1,20)
0.1
< (201 100)
ChyScapes
PSPNec Mio)
16-5
1.001
0.01
< (201) 277]
[101, 200)
58.3 20.5
0,0001
e [271,29]
0.00001
< (291, 200)
0.1
N. 1.0.
0.1
1 [L.2].
WMT16 (EN DE
T5 small (77 Mio)
5
16 165
0,01
32.3
24.56 BLEU 20007
0.001
E 3/.
0,0001
A.2 PRUNING AND RETRAINING
Pruning settings. Identifying which weights remove is essential for successful magnitude pruning, with multiple methods developed to address this. Zhu & Gupta (2017) presented the UNIFORM allocation that prunes each layer to the same relative sparsity level. Gale et al. (2019) refined this approach to UNIFORM+. leaving the first convolutional layer dense and capping pruning in the final fully-connected layer at 80%. Evci et al. (2020) reformulate the Erdös-Rényi kernel (ERK) (Mocanu et al., 2018) to consider layer and kernel dimensions for layerwise sparsity distribution. Lee al. (2020) suggested Layer-Adaptive Magnitude- based Pruning (LAMP), which minimizes output distortion at pruning time measured by the Iız-distortion on the worst-case input. Throughout this work, we stick to the GI.OBAL. allocation, in which all trainable parameters are treated as a single vector and a global threshold is computed to remove parameters independent of the layer they belong to. In experiments where we prune convolutional filters instead of weights, we adopt the L2 -2-morm criterion from Li et al. (2016), ensuring uniform sparsity distribution across layers.
We follow the recommendations of Evci et al. (2020) and Dettmers& & Zettlemoyer (2019) and refrain from pruning biases and nomiaaiaomaiiiiionnn parameters, as their negligible weight contribution is offset by their significant performance impact. Moreover, for GMP experiments, we opt for the global selection criterion due to its superior performance compared to UNIFORM+ (Zimmer et al., 2023).
17

---

Published as a conference paper at ICLR 2024
Retraining schedules. The choice of learning rate schedule during the retraining phase has recently attracted interest due to its significant influence on the performance of pruned networks. To avoid the undesired necessity of individually tuning the learning rate schedule in each phase, various retraining schedules have been devised to transpose the original learning rate schedule to the retraining phase. We briefly outline these schedules. Let 7 represent the total epochs the original network is trained for with the learning rate schedule (me (SS),,T, and let Ty denote the epochs allocated for retraining per prune-retrain cycle. The following retraining schedules have been proposed.
FINE TUNING (FT, Han et al., 2015); Retrains the pruned network using the constant learning rate, 7r, from the last epoch of the original training.
LEARNING RATE REWINDING (LRW, Renda et al., 2020): Utilizes the last T P--1 T,,, learning rates from the original training.
SCALED LEARNING RATE RESTARTING (SLR, Le Hua, 2021): Compresses the original learning rate schedule into the retraining timeframe with short warm-up.
CYCLIC LEARNING RATE RESTARTING (CLR, Le & Hua, 2021): Employs a cosine based schedule with short warm-up to TT--
LINEAR LEARNING RATE RESTARTING (LLR, Zimmer al., 2023): A linear decay from to zero during each retrain cycle.
ADAPTIVE LINEAR LEARNING RATE RESTARTING (ALLR, Zimmer et al., 2023): LLR but dynamically adapts the initial learning rate based on the impact of the previous pruning step and the retraining time available, addressing both the length of a prune-retrain cycle and the performance drop induced by pruning.
Hyperparameters for Retraining. The choice of hyperparameters during retraining significantly impacts the tradeoff between model performance and achieved sparsity. We briefly discuss the hyperparameters applied during the retraining phase, differentiating them from those leave unchanged compared to the pretraining phase. The exact retraining hyperparameters are specified explicitly in the descriptions of each experiment in the corresponding subsection in Appendix B.
Specifically, we retain the same batch size and weight decay parameters as used in pretraining. For each IMP-dased experiment, we treat the retraining learning rate schedule, the number of retraining epochs, and the number of phases as ppperiment-speciffe hyperparameters.
Learning Rate Schedule: The retraining schedule has dramatic impact on the final performance, as outlined the previous paragraph. Zimmer et al. (2023) demonstrate that LLR and ALLR surpass previously proposed methods across a broad spectrum of architectures, sparsity levels, and retraining durations. Thus, we adhere to these schedules in experiments, specifying our choice explicitly when important.
Number of Retraining Epochs: The number of epochs in retraining influences the extent to which the model can recover the pre-pu pre-pruned accuracy. Further, in the high sparsity regime, sufficient retraining is required to ensure that models averageable, highlighted in Figure 4b.
Number Phases:
The number of prune-retrain phases similarly impacts the performance vs. sparsity tradeoff. In general, high goal sparsity levels require multiple phases.
Model Soups and Batch-Normalizattoo statistics. Throughout our experiments, we also explored variants of LearnedSoup (Wortsman et al., 2022a), which learns the coefficients X; to maximize the validation accuracy. Specifically, we observed improvements when utilizing knowledge distillation techniques (Hinton et al., 2015) with the original pretrained model as the teacher, instead of minimiz- ing the validation loss as suggested by Wortsman et al. (2022b). Nevertheless, these improvements were marginal, so we opted for UniformSoup and Greedy for simplicity and to avoid introducing new hyperparameters.
Moreover, we noticed that in later IMP phases, assuming standard IMP rather than starting from an averaged model as in SMS, sparse connectivities tend to diverge, leading to diminished sparsity upon averaging. Specifically, while in the first phase two model splits converge to the same loss basin, subsequent pruning may project them into different subspaces, motivating the combination of
18

---

Published as a conference paper at ICLR 2024
individual models into single at the end of each phase to ensure that effectively averaging them remains possible. Therefore, we also explored weight alignment, as notably proposed by Ainsworth et al. (2023) and Singh & Jaggi (2020) (see Section 4 for detailed discussion), hoping to permute models in later phases to a common linear subspace. Although we partially mitigated the sparsity unable fully recover the original sparsity. This approach may only work if different IMP runs share the same distribution of sparsity among layers (i.c., a specific layer in model one must have the same sparsity as the same layer in model two), which is generally not the case.
reduction,
When constructing an averaged model or changing the parameters in any other way, the Batch- Normalization statistics have to updated, which can be done by performing a forward pass on (part of) the training data without backpropagation. Since doing so only for model soups could potentially skew the results, we decided to recompute the statistics for single models as well to have comparable setting independent of whether we changed the parameters or not. In particular, we enforce using the entire train data loader and we fixed random batch ordering to ensure reproducibility and avoid the batch ordering having any influence.
A.3 PRUNING DURING TRAINING
Hyperparameters for Pruning during Training algorithms. Unless stated otherwise, we set weight decay to le-4 and momentum 0.9, with all methods following linear learning rate schedule starting from le-1. For GSM and LC, we select the best result either from scratch or using a pretrained model, applying these methods for 10, 20, or 40 epochs. When extending BIMP, GMP and DPF with SMS, we choose the number of copies to create within phase, m, between 2 and 3. Further, we tuned the epoch at which we begin to train multiple copies between and 75. Otherwise, we applied the following hyperparameter grids.
BIMP
-nitial training budget epochs: 60, 75.
Number of pruning phases of equal length: 2,2,3.
GMP
Equally distributed pruning steps: 5,9, 9, 18, 45.
DPF
Equally distributed pruning steps: 9, 18.
GSMM
Momentum: 0.9, 0.95.
Weight decay: le-4, le-5.
LC
-Weight decay: le-4, le-5.
DNW
Weight decay:
le-4, le-5.
DST
Weight decay: le-4, le-5.
a: le-7, 5c-7, le-6, 20-6, 5c-6. 8c-6, le-5, lc-4.
19

---

Published as a conference paper at ICLR 2024
B EXTENDED RESULTS
This section contains additional tables and plots. The subsections follow the same structure as the main experimental section. To maintain transparency, we explicitly mention the retraining schedule and duration in the captions of tables and figures, or in the beginning of the subsection if suitable.
B.1 EVALUATING SPARSE MODEL SOUPS
Table WideResNet-20 on CIFAR- 100 (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 90% (top) and 98% (bottom) given three prune-retrain cycles. We report results using UniformSoup as well as GreedySoup for merging, employing ALLR as the retraining schedule for 10 epochs of retraining per phase. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR-100 (90%)
Spursity 53.6% (Phase
Sparsity (Phase 2)
Sparsity
(Phase
Accuracy of
mi
m -
m.
m
m
10
10 -
20
m. 10
SMS (uniform)
76.30 20.17
76.38 =0.29
76.45 20.34
76.55 s0.37
76.89
+0.42
76.94 c0.25
15.98 20.43
20.76
76.67 20.82
best candidate
76.03 st.0
76.18 20.13
76.13
75.88 21149
75.78 +1.35
75.82 =0.21
75.15
75.23 20:38
75.52 20130
macn candidate
75.88 s0.00
75.86
75.93
75.50
75.48
20.12
20:13
74.99 s0.2
75.06 20/42
SMS (greedy)
76.28 20.10
76.01 20.16
20.32
26.16
76.45 20.51
76.45 =0.25
15.48
75.91 20.08
75.81 #0.13
best candidate
76.06 20.03
16.18 s0.1
76.11 +0.35
75.44 s0.11
75.66 #0.19
75.59 =0.35
75.
20.6
75.08 +0.35
74.97 st1.1
candidate
75.93 s0.12
75.86
75.92 20.26
75,26
75.34
75.24
74.87
74.81
74.72 20121
...
76.30 ab.42
75:97 20.30
76.27 stu02
75.91 =0:25
74.59 s0
40-42
all.5
IMP-RePrune
-NAA
NIA
75.70 20.59
75.68 20:25
75.60 20.23
IMP
75.64 ell.1
75.51 a0.52
74.91 =0.71
CIEAR-100 (98%.)
Sparsity 72.8% (Phase
Sparsity 92.6% (Phase 2)
Sparsity 98.0%
(Phase 3)
Accuracy of
mi -
3
10
m - 10
YI1
112
10
100 -
27
m.
10
SMS (uniform)
76.50 #0.16
76.59 s6.13
76.75
75.55 40.60
76.19
60.37
76.21 20.43
72.67
=0.29
72.90 20.64
73.05 +0.45
best candidate
75.58
75.71 .0.08
75.96
75.00-0.34
71.77
a0.04
mean candidate
75.37 +0.12
75.58
75.55 40.26
74.32 +0.40
74.71
+0.48
74.70
71.41
+0.09
71.61
+0.40
+0.19
SMS (gready)
76.06 +0.69
76.43 -0.24
76,60 +0.47
75.34 +0.15
75.39 +0.44
75.31 -0.65
72.08 -0.23
71 86 +0.64
72.44 +0.20
best candidate
75.58 =0.19
75.65
20,00
75.94 #0.15
74.85 #0004
74,53 #0.42
74.57
=0.21
71.05
20.45
71.01 #0.49
71.47 sl1.33
candidate
75.37
75.54 80.03
74,20
IMP..:
75.85 60.26
76.05 a0.00
75.76 50.24
74.09
10:24
74.19 60.44
74.74 e0.06
70.92
70.31 10.52
71.85 +0.15
MMP-RePrune
N/A
N/A
68. 19 +0.44
65.53 +0.06
63.62 +0.50
IMP
a0).1
74.09 10
70:74 =0.05
Table 5: ResNet-18 on CIFAR-10 (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsity 98% given three prune-retrain cycles. We report results using UniformSoup as well as GreedySoup for merging, employing ALLR as the retraining schedule for 20 epochs of retraining per phase. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR- (98%)
Sparsity 72.8% (Phase 1)
Sparsity 92.6% (Phase 2)
Sparsity 98.0% (Phase 3)
Accuracy
3
10
10
SMS (uniform)
95.41 20,00
95.46 s0.13
20.01
95.45
2011
20,09
20.03
20.16
best candidate
91.84 20.10
91,96
10.13
94.94 40077
91.92
20.07
95.11 20:06
95.12 10,02
94.10
20.02
94.69
20.16
91.81 10.09
mean candidate
94.70 20.15
94.82
a0.00
94.75 s0.8
94.85 20114
94.93 s0300
94.94 s0.01
94.32
20.12
94.57 s0.16
94.66 211.33
SMS (greedy)
95.42 =0,14
95.45
s0.18
95. 50 s0.0
95.46 20.18
95.27 s0.24
95.32
30.20
95.01
=0.08
94,92 20.04
94.94 20
best candidate
94.84 +0.10
944
94 +0.13
94.94 67.07
20 94.96
94. 86 .0:09
94.92 +0.0
94.55
94.48 +0.01
94 +0.06
mean candidate
94.70 =0.15
94.82
+0.00
94.75 s0.8
94,80 s0.15
94.76 #0.11
94.79 +0.07
94.44 =0.06
94, +0.01
94.29 +0.06
IMP
95.18 =0.08
95.16 +0.16
95. 10.18
95.02 #0.11
95.11 +0.18
95.20 +0.02
94.62 =0.25
94.61 +0.00
94 +0.23
IMP-RePrune
94.71 +0.08
94.92 +0.01
94.17 +0
20

---

Published as a conference paper at ICLR 2024
Table 6: ResNet-18 on CIFAR-10 (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 80%, 90%, 95% the One Shot setting, using ALLR for retrain length of 20 epochs per phase. We report results using UniformSoup as well as GreedySoup for merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR-10
Sparsity
Sparsity 90.0% Shol)
Sparsity Shof)
Accuracc of
m.
3
(uniform)
95.03
best candidate
95.04 10.06
94,84 20.03
94.63 10.06
mean candidate
10.8
94.76 #0,03
SMS (greedy)
95.36 10.16
95.41 20.07
95.04 10.19
candidate
95.04 10.05
94.84 20.03
94.63 10.06
mean candidate
94.92 +0.88
94.76 +0.03
94.48 +0.05
IMP IMP
95.17 10.17
95.02 20.01
94.72 10.24
95.02 10.05
94.71 s0.28
94,38 1000
Table 7: Vix on ImageNet (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 75%, 80%, 85% in the One Shot setting, using ALLR for a retrain length of 10 epochs per phase. We report results using UniformSoup well as GreedySoup for merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
ImageNet
Sparsity 75.0% (One Shot)
Sparsity 80.0% Shot)
Sparsity 85.0% Shot)
Accuracy of
m
3
SMS (unifoem)
78.31 10.21
78.12 a0.17
77.59 10.24
best candidate
78.11 +0.77
77.85 =0.10
77.37 +0,2
mean candidate
SMS (greedy)
78.21 10.14
78.06 20.01
77.45 10.13
candidate
4007
77.85
77.37
meam candidate
77.83 +0.01
77.66 #0.06
77.17 +0.04
IMP
78.17 10.15
77.99 s0.12
77.68 10.23
IMP
78.07 +0.09
77.88 +0.06
77.34 +0.12
Table 8: PSPNet on Cityscapes (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsity 90% given two prune-retrain cycles of 50 retraining epochs each. We report results using UniformSoup well as GreedySoup merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CityScapes (90%)
Sparsity 68.3% (Phase 1)
Sparsity 90.0% (Phase 2)
Accuracy of
SMS (uniform)
$8.52 sti.15
58.47 10.10
58.73 s0.20
58.40 +0.30
candidate
+0.37
58.25
mean candidate
57.96
$7.80 20.33
58.38 #0.41
57.62 20.48
(gready)
58.
+0.00
best candidate
58.26 +0.06
$8. e0.14
$8.13 13 #0.27
58.73 +0.06
mean candidate
#0.13
IMP
$7.39 +0.06
58.35 +0.88
58.27 +11.42
58.39 +0.17
IMP
57.92 +0.04
58.89 +0.23
21

---

Published as a conference paper at ICLR 2024
Table 9: PSPNet on Cityscapes (unstructured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 60%, 70%, 80% and 90% in the One Shot setting, using LLR as the retraining schedule for 50 epochs of retraining. We report results using UniformSoup as well as GreedySoup merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CityScapes
Sparsity
Shot)
Spursäty Shol)
Sparsity Shot)
Spursity Shot)
Accuracy of
3
3
SMS (uniform)
best candidate
57.74 #0.11
$7.36 20.43
57.97 10.06
$7.87 20.49
candidate
57.37
57.04
57.88
57.70
SMS (greedy)
58.41 10.13
58.13 a0.31
$7.95 10-40
57.30 s0.21
candidate
58,16
57.78
57.78 +0.36
57.32
mean candidate
58.05 10.50
57.55 s0.07
57.49 10.38
57.18 10.25
IMP
58.02 #0,09
58.09 =1.04
58.26 #0.13
S8.47 s0.22
+0.34
Table 10: T5 WMT16 (unstructured pruning): BLEU score comparison of SMS several baselines for target sparsities 50%, 60%, 70% in the One Shot setting. We report results using UniformSoup as well as GreedySoup merging, employing ALLR the retraining schedule for 2 epochs of retraining per phase. Results are averaged over multiple seeds with standard deviation included. The best value highlighted in bold.
WMT-16
Sparsity 50.0% (One Shot)
Sparsity Shot)
Sparsity (One Shot)
Accuracy of (uniform)
in
3
10.52
25.09 at.I0I
24.51
best candidate
25.39 10.03
24.96 20.26
24.12 10.01
mean candidate
10.08
24.79 20.19
24.03 10.01
SMS (greedy)
25.51 +0.28
24.92 #0.47
24.14 +0.02
best candidate
25.39 10.03
24.96 a0.26
24.12 10.01
mean candidate
25.16 10.08
24.79 s0.19
24.03 1001
IMP
25.36 10.12
25.09 #0.05
24.00 20.04
IMP
25.15 10.20
24.90 40.20
24.04 10.28
22

---

Published as a conference paper at ICLR 2024
Table 11: WideResNeet20 on CIFAR-100 (structured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 60% (top) and 80% (bottom) given three prune-retrain cycles. We report results using UniformSoup as well as GreedySoup for merging, employing ALLR as the retraining schedule for 10 epochs of retraining per phase. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR-100 (60%)
Sparsity 26.2% (Phase 1)
Sparsity 45.6% (Phase 2)
Sparsily 60.0% (Phase 3)
Accuracy of
3
5
10
10
(uniform)
76.54
76.56
76.46
75.34
best candidate
75.12 +0.04
75.19 .0.06
75.23 +0.07
74.73 +0.15
74.90 s0.12
74.85 -0.16
74.31 40.42
74.45 +0.11
74.53 +0.16
macan candidate
74.98 #0.12
20.07
74.86
74.49 20.14
74.53 +0.14
71,53 =0.06
73.81
73.95 10.19
73.95 +0.08
SMS (greedy)
76.55 =0.41
76.15 20.24
20.10
75.85 10:06
76.10 #0.38
76.03 =0.43
15.10
75.24 20:20
74.48 40.42
candidate
75.09
75.19 #006
s0:15 74.04
20:37
mean candidate
74.96 +0.09
74.87 .0.06
74.88 +0.02
74.62 +0.12
74.56 +0.28
74.43 =0.20
73.85
+0.05
73.78
60.12
73.87 +0.14
IMP
75.55 40.04
75:72 a0 18
75.92 #0.16
74.84 $0.02
75.12 40.09
75.68 o0.47
74.28
74.63 40.32
74.73 +0.94
IMP IMP-RePrunc
NIA
NIA
74.65
#0.63
75.54 =0.28
75:19 40.33
73:47
CIFAR-100 (80%)
Spursity 41.5% (Phase 1)
Sparsity 65.8% (Phase 2)
Sparsity 80.0% (Phase 3)
Accuracy
10
(uniform))
75.94
=0.40
71.
best candidate
74.65 20.29
74.65 20.11
74.78 20.17
73.27 20:26
73.22 +0.27
73.71 20.18
70.61 20:11
70.58 20.50
70.96 +l1.3
macan condidate
74.41 20:17
71.37
74.39.00115
22.90
72.91 50.23
73.19 20.09
70.50
20.05
20.08
70.31 20.52
70.40 s0.23
SMS (greedy)
75.87 =0.10
75.07 s0.44
76.14
+0:23
74.21
74.48 -0.35
13.63 20.25
71.70 +0.27
71.06
74.66
s0.11
s0.15
73.59
20.30
#0.04
maan candidate
74.47 20.13
20.16 74.37
74.38 20.14
72.95 20.33
73.03 40,00
73.13 20.10
70.56 20.45
70.37 20.04
69.83 all.3
IMP
75.06 20:34
75.48 20.16
75.39 20009
73.44 atl.2
73.70 a0.06
73.96 =0.54
72,06 20.10
72.15 #0.58
72.32 2t0.5
IMP-ReP RePruee
NIA
NIA
70.52
60 12.9)
69.89 21.34
73.95 at108
20.15
69.88 =0.50
Table 12: ResNet-18 18 on C0 (structured pruning): Test accuracy comparison of SMS to several baselines for target sparsities 40%, 50%, 60% in the One Shot setting, using ALLR as the retrain schedule for retrain length of 20 epochs. We report results using UniformSoup as well as GreedySoup for merging. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR-10
Sparsity 40.0% (One Shol)
Sparsity 50.0% Shot)
Sparsity Shof)
Accuracy of
m.
3
m 3
SMS (uniform)
s0.00
=0.02
94.32 +0.10
best candidate
94.42 10.01
94,28 10.01
93.8 10.06
mean candidate
94.31 10.1
94.16 =0.03
93.77 +0.66
SMS (greedy)
94.87 40.02
94.70 20.08
94.31 10.11
candidate
94.42 10.04
94,28 a0.01
93.5 10.06
mean candidate
94.31 +0.01
94.17 +0.04
93.77 +0.06
IMP m-
94.77 10.01
94.38 20.23
94.34 10.16
IMP
94.45 10.18
94,06 s0.17
93.96 10.34
23

---

Published as a conference paper at ICLR 2024
B.2 EXAMINING SPARSE MODEL MERGING
B.2.1 EXPLORING PARAMETERS BEYOND RANDOM SEEDS.
Similar to Figure 3, Figure 5 visualizes the effects of different hyperparameters given One Shot IMP (90% sparsity) with WRN-20 trained on CIFAR-100. Again, we created eight variations for each parameter (random seed, weight decay strength, retraining duration, and initial learning rate in linear decay schedule) based on equidistant values around the defaults. The scatter plot compares the test accuracy of models in the uniform soup setting (averaged from pairs of models) against the maximum test accuracy within each pair.
To generate the averaging candidates, employed the following hyperparameter configurations. For ImageNet, as showcased Figure 3, our base configuration utilized ALLR for 5 retraining epochs with weight decay as stated in Table 3. While varying the random seed, maintained the base configuration and selected eight distinct random seeds. In adjusting the weight decay, we adhered to the base configuration and experimented with weight decay strengths of 4e-5, 6e-5, 8e-5, le-4, 1.2c-4, I 1.4c-4, 1.6e-4, 1.8e-4, For retraining length variation, we examined all integral values between 2 and 9 epochs. In terms of retraining schedule modification, we adopted linearly decaying learning rate schedule, tuning the initial value among 2e-2, 4c-2, 6e-2, 8c-2, le-1, 1.2c-1, 1.4e-1 and 1.6e-1. For CIFAR-100 100, as Figure 5, we used ALLR for 10 epochs with weight decay as in Table To adjust the weight decay, we experimented with weight decay strengths of le-4, 2e-4, 3e-4, 4e-4, 5e-4, 6e-4, 7e-4, 8e-4. For retraining length variation, we examined all integral values between and 13 epochs. In terms of retraining schedule modification, we adopted a linearly decaying learning rate schedule, tuning the initial value among 6e-2, 7e-2, 8e-2, 9e-2, le-1, 1.le-1, 1.2e-1 and 1.3e-1.
WRN-20 CIFAR-100
76.5%
76.0%
soup accuracy
75.5%
75.0%
random seed weight decay retrain length
74.5%
retrain schedule
74.50%
75.00%
75.50%
76.00%
76.50%
best candidate accuracy
Figure 5: WedeResNet-20 on CIFAR-100: Accuracy of average of models vs. the maximal individual accuracy. All models are pruned to 90% sparsity (One Shot) and retrained, varying the indicated hyperparameters.
24

---

Published as a conference paper at ICLR 2024
B.2.2 INSTABILITY TO RANDOMNESS AND RECOVERING IT.
Figure replicates the plots from Figure 4, albeit for CIFAR-IO, adhering the identical retraining hyperparameter configuration delineated in the main text.
ResNet- 18 on CIFAR-11
ResNet-18 on CIFAR-10
1,0%
0.8%
Ace (coup) - (bersbes))
(snogg (IMP
0.55%
0.0%
0.3%
-1.0%
0.0%
-2.0%
-0.3%
0.5%
-3.0%
UniformSoup GreedySoup
Sparsity 90%
Sparsily 98%
70%6
75%
807
83%
9075
90%
100%
10
spansity
number of retraining eporks
(a)
Figure 6: ResNet-18 CIFAR-IO: (a) Accuracy difference between the soup and best performing model after One Shot pruning and retraining. The lines for UniformSoup and GreedySoup show envelope considering all retraining schedules and durations.
(b) Accuracy difference between the soup (m = 3) and IMP 3x retrained three times as long as indicated on the x-axis, using One Shot pruning to 90% and 98% sparsity. Results are averaged over multiple random seeds with min-max bands indicated.
25

---

Published as a conference paper at ICLR 2024
B.2.3 OOD-ROBUSTENN OF SPARSE MODEL SOUPS.
Figure depicts OOD-robustness effects for One Shot IMP WRN-20 trained on CIFAR-100 90% sparsity, while Figure 8 does the same for ResNet-50 on ImageNet at 70% sparsity. For each parameter (random seed, weight decay strength, retraining duration, and initial learning rate in linear decay schedule), we formulated eight variations centered on default values (see subsubsection B.2.1 for exact hy hyperparametter) Contrary to previous scatter plots, these evaluate models on the robustness benchmarks CIFAR-100 100-C and Imagenet-C. The plots contrast the OOD accuracy in the uniform soup setting (averaged across model pairs) with the peak OOD accuracy of each pair. The OOD accuracy is computed the entire corrupted dataset, i.c., among all corruption types and degrees of severity (ranging from to 5).
Further, Table 13 and Table 14 display the OOD-robustness evaluated CIFAR-100-C and ImageNet-C. respectively, when aiming for higher sparsities and using multiple cycles. SMS con- sistently improves the baselines and improves the out-of-distribution accuracy significantly. For the pretrained models we obtain a base ood accuracy of 49.03% 19.03%(+0.33))) for CIFAR- 1-00------------ and 40.35%(+0.30) for ImageNet-C.
WRN-20 on CIFAR-100-C
49.5%
49.0%
Soup ood accuracy
48.5%
48.0%
47.5%
retrain length
retrain schedule
47.0%
weight decay
random seed
46.5%
46.5%
47.0% 47.5% 48.0% 48.5% 49.0% 49.5% best candidate ood accuracy
Figure 7: WideResNet-20 evaluated CIFAR-100-C: OOD Accuracy of average of two models vs. the maximal individual OOD accuracy. All models are pruned to 90% sparsity (One Shot) and retrained, varying the indicated hyperparameters.
26

---

Published as a conference paper at ICLR 2024
ResNet-50 on ImageNet-C
40.5%
40.0%
soup ood accuracy
39.5%
39.0%
38.5%
retrain schedule
retrain length
38.0%
weight decay
37.5%
random seed
37.5% 38.0% 38.5% 39.0% 39.5% 40.0% 40.5% best candidate ood accuracy
Figure 8: ResNet-50 evaluated on ImageNet-C: OOD Accuracy of average of two models vs. the maximal individual OOD accuracy. All models are pruned to 70% sparsity (One Shot) and retrained, varying the indicated hyperparameters.
Table 13: WideResNNet220 trained on CIFAR-10 100 and evaluated on CIFAR-100-C (unstructured
pruning): OOD accuracy comparison of SMS several baselines for target sparsities 90% (top) and 98% (bottom) given three prune-retrain cycles. We only report results using UniformSoup for merging, employing ALLR as the retraining schedule for 10 epochs of retraining per phase. Results averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
CIFAR-100 (90%)
Spursity (Phase
Sparsity (Phase 2)
Sparsity
(Phase
Accuracy of
m:
in
21.
m
10 -
20
SMS
49.60 s0.14
49.48 20.20
49.57 20.24
49,90 20.11
$0.09 s0.18
50.33 =0.14
04.65-50 s0.31
48 +0.06
49.38
21.34
best candidate
49.31 20.14
49.18 s0.22
49.17 20:25
48.51
48.64
48.82 20.17
47.42
20.15
20.15
47.87
48.97 20.15
48.97
20.12
20.00
47.29 2028
47.51
IMP
49.25 20.10
49.01
20.32 N/A
49.17 20.38
48.06 20.07
48.24 20.52
48.17 20.6
47.10 20.74
46.59 20.05
47.12 2l.52
IMP. RePrane
NIA
48.72 20.25
49.03 a0.15
49.35 s0.11
IMP
49.41 a0:22
48:25 2000
46.88
CIFAR-100 (98%)
Spuesity 72.8% (Phase 1)
Sparsity 92.6% (Phase 2)
Sparsity 98.0% (Phase 3)
Accuracy of
10
10
SMS
50.155
50.01
50.28 40,09
8..00
44.18
44.81 40.26
best candidate
49.36 +0.17
49.01
16
+0.02
47,07
47.27
+0.41
+0.23
mean candidate
48.95 20.07
48.68 20.09
48.67 +0.09
45.87 +0.34
46,80
+1.38
46.87
=0.23
42.94 s0.52
43.31 +0.34
43.17 sl..4
IMP
48.70 10.19
48.81 20.10
48.66 #0.17
46.20
40.27
45,90
80.10
46.17 20.07
41.97 20.17
41.62 a1.55
42.76 +0.30
IMP-RePrune
39.57
48.
68.60-10
+0.14
-0.58
Table 14: ResNet-50 trained on ImageNet and evaluated on ImageNet-C (unstructured pruning): OOD accuracy comparison of SMS to several baselines for target sparsity 90% given three prune-retrain cycles. We only report results using UniformSoup for merging, employing ALLR as the retraining schedule for 10 epochs of retraining per phase. Results are averaged over multiple seeds with standard deviation included. The best value is highlighted in bold.
ImageNet (90%))
Sparsity (Phase 1)
Sparsity (Phase 2)
Sparsity 90.0%
(Phase 3)
Accuracy of
10 =
10 =
m = 10
m
on = 10
m/:3
on
10
SMS
42.19 #0.17
42.65 #0.17
42.94 s0.02
41.17 +0.15
41.62 10.10
42.11 10.06
38.70 40.02
39.23 40.15
39.70 40.82
candidate
mecn candidate
39.87 20.10
39,90 20.22
39.91
20.09
39.19 20.10
39.28 10.05
39.47 10.01
37.21 10.01
37.39 20.14
37.62 4000
IMP
40.05 20.17
=0.26:0029
40.44 =0.1
39.31 s0a2
39.46 #0.0
39.45 +t.288
37.13 +01.2
37.47 20.36
37. 20.34
IMP-RePrune
N/A
N/A
36.48
36.64
27

---

Published as a conference paper at ICLR 2024
B.2.4 REDUCING COMPRESSION UNFAIRNESS.
Classification model performance. usually measured by the top-1 accuracy, can mask the dispropor- tionate effect of compression on individual class performance (Hooker et al., 2019; 2020; Paganini, 2020). Pruning often sacrifices difficult samples. benefiting well-performing classes and worsening the performance of others (Tran et al., 2022). Recent research by Ko et al. (2023) highlights the bene- fits of prediction ensembling enhancing fairness metrics, including minority group performance.
We investigate whether model averaging mitigates pruning's s adverse effects on fairness using ResNet- 18 trained on the Celeb-A facial attribute recognition dataset, fairness benchmark due its strong sub-group and target label correlation. Table 15 compares the dense model, One Shot SMS, and IMP at sparsities of 90%, %, and 97%, where 20 epochs of retraining yields similar top-1 test accuracies for SMS and IMP. We report the recall for the disjoint sub-groups: Male-Blonde (MB), Male-Non- Blonde (MN), Female-Blonde (FB), and Female-Non Non-Blonee (FN), in addition to top-1 test accuracy. Although SMS and IMP reach similar test accuracy, differences in subgroup performance arise for sparsities above 90%. SMS notably increases recall for MB and FB, the most challenging classes in the dense model. This emphasizes the negative effects of regular pruning through IMP, which sacrifices weakly represented subgroups to maintain high overall accuracy. In contrast, SMS has less significant impact on MB and FB but leads to a more nuanced decline in casier-to-llassffy subgroups.
Table 15: ResNet- 18 on Celeb-A: Comparison of the pretrained (i.e. dense) base model against SMS and IMP for different sparsity levels, employing ALLR the retraining schedule for 20 epochs of retraining. We indicate the top-1 test accuracy as well as the recall on the four different sub-groups All results are averaged over multiple random seeds with standard deviation included.
Celeb-A
Sub-group Recall
Setting
Spursity
Top-1
Balanced
MB
MN
FB
FN
Pretrained
0%
98.99 s0.01
98.09 s0.02
94.41 a0.20
99.93 s0.00
98.48 10.00
99.54
s0.00
SMS
90%
99.02 =0.00
98.13 50.0
94.52 +0.20
99.92 10.00
98.57
+0.02
99.50
IMP
90%
98.99 +0.02
98.12 20.0
94.52 +0.20
99.93 +0.01
98.51 +0.04
99.52 +0.03
SMS
95%
98.91
98.08 08 10.05
94.74 10,20
99 88
98.53
99.17
IMP
95%
98.79 20.01
97.89 20.08
94.02 20.31
99.92 +0.01
98.20 20.01
99.41 #0.01
SMS
97%
97.79 +0.08
96.04
.4 1444+1.775 +1.78
99.67 +0.07
97.38 +0.24
97. 78 40.49
IMP
97%
97.74 #0.08
95.73 #0.32
81.42
99.88 +0.01
96.71 +0.07
98,90 +0.04
28

---

Published as a conference paper at ICLR 2024
C ABLATION STUDIES
C.I ABLATION: SMS HYPERPARAMETERS
We conduct several ablation studies to assess the influence of key hyperparameters in IMP: the retraining schedule and retraining length. Given the large number of individual runs in each ablation study, we restrict ourselves to examining WideResNet-20 trained on CIFAR-100 100. Besides the default pretraining parameters shown in Table 3. we employ pretraining learning initiated at le-1, decaying by a factor of 0.2 at epochs 60, 120, and 160.
C.1.1 ABLATION: THE RETRAINING SCHEDULES
We begin by isolating the impact of the retraining schedule, comparing LRW, SLR, CLR, LLR and ALLR. Figure 9 depicts the difference between soup accuracy and best candidate accuracy for wide range of sparsities in the One Shot setting, where we distinguish between = 3 (left) and m = 5 (right). Note that each retraining schedule also influences the accuracy candidate models. Throughout these experiments, the number of retraining epochs is fixed at 10.
First of all, we observe that all schedules except LRW effectively train pruned models to a state suitable for averaging. LRW, solely basing the initial learning rate magnitude retraining duration, potentially falls short in recovering high pruning-induces performance degradation, thus hindering feasible averaging. Contrastingly, for all other schedules see consistent improvements upon their averaging candidates, with strategies performing comparably well, although ALLR also augments performance high sparsity scenarios. The lesser convergence of, for instance, SLR versus LLR, as identified by Zimmer et al. (2023), does not notably affect the accuracy disparity between soup and best model, even though LLR results in superior candidates and a better soup model.
We conclude that SMS requires retraining that is sufficiently accelerated by a proper learning rate schedule.
WRN-20 on CIFAR-100 (m 3)
WRN-20 CIFAR-100 (m
1.5%
Acc (soup) Acc (bees)
1.0%
Acc (boos)
1.0%
0.8%
0.5%
Ane (soup)
0.0%
0.0%
ALLLR
SL. LRW
-0.5%
LLR
CLR
-0.5%
90%
92%
94%
96%
98%
90%
92%
94%
96%
98%
sparsity
spursity
Figure 9: WideResNet-20 CIFAR-100: Test accuracy difference between the soup of a) = 3 or b) m = models compared to the best candidate model for a wide range of sparsity levels. Each line depicts one retrain schedule. Note that only consider the One Shot case and that the candidate models themselves depend on the retraining schedule at hand. Results are averaged over multiple random seeds with min-max bands indicated.
C.1.2 ABLATION: THE RETRAINING LENGTHI
Next, we evaluate the impact of the retraining duration by comparing retraining lengths 1, 2, 5, 10, and 20 epochs. As before, Figure 10 illustrates the accuracy difference between the soup and best candidate models across a spectrum of sparsity levels in the One Shot setting, differentiating between 3 (left) and 5 (right). We emphasize that the number of retraining epochs affects both the soup model accuracy as well as all candidate models for averaging. We stick to ALLR as the retraining schedule.
29

---

Published as a conference paper at ICLR 2024
We encounter diminishing returns scenario: the longer retrain, the smaller the improvement of the soup upon the individual models. More surprisingly however, averaging models yields consistent improvements
single retraining epoch, which is clearly not enough for recovering performance in the high sparsity regime. The learning rate schedule ALLR seems to be of particular importance here, since it also incorporates the retraining length when choosing the learning rate schedule. As visible in Figure 9, such a consistent improvement is not achievable with other schedules, even when using 10 epochs of retraining.
with
We conclude that SMS is able to consistently improve upon the individual models even when using short amounts of retraining, provided that proper care is taken of the learning rate.
WRN-20 90 ou CIFAR- 100 (m = 3)
WRN-20 on CIFAR- 100 (m 5)
2.5%
Anc (soup) ((amup)-Ao -Aoe -Aoe(bes))
2.0%
Ace (soup) Aec (bos)))))
2.0%
1.0%
1.5%
1.0%
1.0%
20 epoclas
epocia
0.5%
10 epochs
epoch
5 epochs
0.0%
90%
92%
94%
96%
98%
90%
92%
94%
96%
98%
sparstt
sparsity
(a)
(b)
Figure 10: WideResNet-200 on CIFAR- 100: Test accuracy difference between the soup of or b) m = models compared to the best candidate model for a wide range of sparsity levels. Each line depicts that the candidate models themselves depend on the length of retraining at hand. Results are averaged over multiple random seeds with min-max bands indicated.
retraining length configuration. Note that only consider the One Shot case and
C.2 ABLATION: SUITABLE BASELINES FOR SMS
We have demonstrated that SMS. which trains m models per phase in parallel for k epochs each, surpasses IMP mxx natural baseline where IMP retraining duration in each phase is extended by factor of m (totaling k m epochs). In Table 16, we compare IMP mox to another relevant baseline in the same setting in Table 1: increasing the number of IMP phases by m, matching the total retraining epochs of SMS and IMP mx but with a reduced pruning rate per phase. The results indicate that IMP mx and consequently SMS, outperform this additional baseline.
Table 16: Comparison test accuracy for different IMP baselines on ResNet-50 trained on ImageNet. Results are averaged over multiple seeds with standard deviation included.
m 3
m 5
m 10
Method
Accuracy
Accuracy
Accuracy
IMP mx
74.34% +0.09%
74.56% +0.24% 74.08% +0.04%
74.50% +0.09% 74.70% +0.02%
IMP with m phases
73.69% +0.10%
C.3 ABLATION: DIFFERENCES TO STOCHASTIC WEIGHT AVERAGING
Stochastic Weight Averaging (SWA, Izmailov et al. (2018)) is a popular procedure to improve the generalization performance of models by averaging their parameters along the training trajectory. In consequence, SWA and SMS are similar approaches, despite SWA being designed for dense models. We highlight some of the main observations and problems when combining SWA and IMP:
30

---

Published as a conference paper at ICLR 2024
1. SWA is only beneficial if models of the same sparsity level and pattern averaged, as differing sparsities will densify the model (see Figure 1). We hence apply SWA separately in each phase, starting each phase with the averaged model from the previous phase and reinitializing SWA accordingly.
2. In general, SWA and SMS are not excluding each other, they can be used in conjunction, potentially further improving the effect of SMS.
3. SWA requires either a eyclic or high constant learning rate to explore multiple optima for beneficial averaging. However, retraining after pruning uses specific translated learning rate schedules (such as FT, LRW, SLR, CLR, LLR or ALLR) to maximize performance.
Despite these issue of differing learning rate schedules, conduct experiments using ResNet-50 on ImageNet, following the setup of Table for a sparsity of 90% three cycles. Precisely, Table 17 compares classical IMP to IMP with SWA, where update the SWA-model after each epoch and set the retrained model to its averaged variant at the end of the phase as discussed above. We observe slightly inferior results when adding SWA, comparing wide range of retraining learning rate schedules.
SWA is not able improve the results of classical IMP (and hence also falls behind SMS by large margin, cf. Table 1), We think that this is mostly due to the specific retraining schedules used for IMP, which stand in conflict with the requirements for SWA.
Table 17: ResNet-50 on ImageNet: Test Accuracy comparison of IMP (first row) vs. IMP with SWA (second row) for different retraining schedules when aiming for a goal sparsity of 90% in three cycles of ten retraining epochs each. Results are averaged over multiple seeds with standard deviation included.
Method
FT
LRW
SLR
CLR
LLR
ALLR
IMP
27.38% 20.51%
73.65%
20.08%
73.29%
20.07%
73.36%
20.02%
73.38%
20.25%
73.80% 10.10%
IMP SWA
22.11% a0.36%
73.34% 20.08%
72.10% 40.02%
72.19% 10.18%
72.25% a0.11%
73.01% +0.0%%
C.4 ABLATION: PERFORMANCE DEGRADATION FOR EXTREME LEVELS OF SPARSITY
We have argued that extremely high sparsity levels lead to model that is not stable to randomness anymore, i.e., two retrained models not lie in the same basin and thus cannot be averaged. For WRN-20 on CIFAR-100, this problem occurs at 99% pruned in One Shot and above, see Figure 4a where the uniform approach unable to average the models with increasing performance.
To investigate this issue, we track the L2-norm distance between the candidates for averaging. Table displays the mean and maximal Le distance between each pair of five candidates for averaging, using One Shot pruning and retraining in the same setting as in Figure 4a. We observe that for sparsities in the range 90%-98% the mean and maximal L2 Li-dittance between the five candidate models relatively stable among sparsities. Increasing the sparsity 99% and 99.5% however leads to a much increased distance between the retrained models. At this sparsity, the models are driven further apart, supporting our hypothesis of instability randomness they do not converge the same basin.
Table 18: Mean (first row) and maximal (second row) Læ-distance when comparing each pair of the five candidates for averaging in Figure 4a for different sparsity levels between 90% and 99.5%. Results averaged over multiple seeds, where omit the standard deviation for the sake of clarity.
Sparsity
90%
92%
94%
96%
98%
99%
99.5%
mean -iddistance
29.17
29.44
29.61
29.66
30.13
32.95
39.41
max Lygddistace
29.22
29.48
29.69
29.70
30.24
33.35
40.18
31


Reviews:
[{"review_id": "m7R3ZgDUvX", "reviewer": "Reviewer_TvoY", "summary": "The paper proposes merging sparse models by initiating each prune-retrain cycle with the averaged model from the previous phase. They show that averaging these models significantly enhances generalization and OOD performance over their individual counterparts. Overall, in summary, it is an extension of model soups for sparse models.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1. The experimental section of the paper + supplementary is rich illustrating noticeable gain.\n2. OOD experiments are new for sparse model soup showing SMS consistently improves over the baselines.", "weaknesses": "I have significant novelty concerns with the draft. \n\n1. The idea of sparse model averaging has been widely explored including the model soups (eg. https://arxiv.org/abs/2205.15322 https://arxiv.org/abs/2208.10842 https://arxiv.org/abs/2306.10460  etc). \n2. The authors have failed to detail how their method contrasts with existing sparse model soup papers in their related work section. I feel it is just an incremental work over the existing literature. The benefits of averaging the sparse masks is already known.\n3. Although I appreciate the extensive experiments by authors, I still feel the experiments are limited to small-scale datasets and models (maybe ViT scale or OPT models-based experiments will add value).\n4. I feel auxiliary benefits of model soups like OOD robustness, fairness, etc are good directions to explore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77qU4jCT2j", "reviewer": "Reviewer_ja8N", "summary": "The idea is to combine model soups from transfer learning with pruning. The proposal is : in prune-retrain paradigm (be it from scratch or pretrained), the training portion is replaced by model-soup. It improves the overall quality of the final pruned model as is validated extensively in experiments.\n\n[Update]\nIn light of the responses by authors, I am increasing my score to 6.", "questions": "1) How is IMP $m \\times$ implemented? Is the pruning rate for each prune step reduced? or is the training portions increased m$\\times$. The latter, I suspect, will not be very useful. \n2) Are there any challenges that are specific to using model soups for training portion of prune-retrain algorithm which differentiate it from applying model soups to finetuning of pretrained models? I felt that there were no new challenges here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This being an empirical paper with simple idea proposal, the authors do an excellent job of evaluating their idea in terms of \n(1) extensive experiments on different domains \n(2) covering baselines that are natural competitors to the proposal.", "weaknesses": "1) I am not sure if this paper contributes new ideas or analysis. The proposal is to replace the training portion of prune-retrain with model soups. I do not have background on transfer learning, but as a general machine learning person, it is not surprising that it improves the accuracy of the model given the backdrop of model soups paper . Since in both the cases it holds that m copies of model start from the same initialization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SPBYAgdxf5", "reviewer": "Reviewer_fESa", "summary": "This paper presents the Sparse Model Soups (SMS) framework, which applies the model soup algorithm to the neural network pruning procedure. Experimental results show that the model soup algorithm, a one of the most well-known weight-averaging methodologies that had demonstrated notable success in training dense neural networks, is also effective in iterative pruning procedures for training sparse neural networks.", "questions": "1. Alongside model soups, another prominent weight-averaging strategy is Stochastic Weight Averaging (SWA). The fact that SMS requires m training runs at each cycle makes SWA, which performs weight averaging within a single SGD trajectory, somewhat appealing. Are there any baseline results using SWA instead of model soups?\n\n2. I understand that the authors have opted to show exclusively the UniformSoup results for CityScapes and WMT16 since the GreedySoup algorithm utilizes validation data, which are the test split here. However, despite the potential fair comparison issue, presenting additional GreedySoup results might offer valuable insights into the benefits of selectively using soup ingredients at high sparsity levels.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality and significance: The proposed SMS framework does not bring a substantial level of novelty, as one can consider it as an application of the existing model soup algorithm to an arbitrary iterative pruning process. However, its originality comes from the actual implementation in the domain of neural network pruning, even though the individual elements may already exist separately. Considering the recent success of weight averaging methods for dense neural networks, it is valuable to explore the extension of these techniques to sparse neural networks.\n\nQuality and clarity: Based on observations in the field of transfer learning, where fine-tuned models from the same pre-trained model tend to stay in the nearby region, the hypothesis that a similar phenomenon will occur when re-training the same pruned model is well-founded. The effectiveness of the proposed SMS framework is confirmed through a range of experiments conducted in the domains of image classification, semantic segmentation, and neural machine translation.", "weaknesses": "While the proposed SMS framework incorporates the model soup algorithm in the context of neural network pruning, it does not provide specific insights into the unique factors that are especially pertinent to sparse network training. The questions remain: What attributes of the model soup algorithm contribute to its effectiveness in the neural network pruning regime? Is it the same reason why weight-averaging methods have succeeded in conventional dense network training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FPU0Rv9uH9", "reviewer": "Reviewer_hicK", "summary": "This paper introduces a new pruning technique named Sparse Model Soups, which combines the weight averaging methods from [1] with the pruning method described in [2]. They provide empirical evidence to support the idea that this straightforward aggregation enhances the performance of pruned models in image classification tasks.\n\nReferences\n\n[1] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.\n\n[2] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2015.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clarity\n- The paper is well written, ensuring it is understandable for readers.\n- The suggested method is straightforward and effectively explained for easy understanding.\n\nOriginality and Significance\n- This paper introduces a novel pruning method that incorporates model averaging techniques based on the Model Soup methods.\n- They provide empirical evidence demonstrating that the proposed method offers improved performance when compared to baseline approaches.", "weaknesses": "Method\n- Although they can parallelize the training process, performing $m\\times k$ training epochs still imposes a substantial computational burden. And if the training cost becomes small, the overall performance gain significantly drops for the extreme sparsity cases.\n\nExperiments\n- It would be valuable to conduct empirical analyses to investigate why performance degradation occurs in regions of extreme sparsity.\n- Similarly, it would be beneficial to empirically analyze why SMS performs well in situations of early sparsity, where batch randomness can lead to divergence between averaged weights [3].\n- It would be advantageous to include an ablation study exploring different combinations of averaging coefficients where the $\\lambda_i$ values differ from each other.\n\nRecommend\n- I suggest that the authors consider adding an Ethics Statement and a Reproducibility Statement immediately following the main paper.\n\nReferences\n\n[3] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)