Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Authors: Mohammed Adnan
Rohan Jain
Ekansh Sharma
Rahul Krishnan
Yani Ioannou
Venue: ICML 2025 poster (2025)
Keywords: Lottery Ticket Hypothesis
sparse training
linear mode connectivity
weight symmetry
deep learning
deep neural networks
random initialization
git re-basin
optimization
Primary area: deep_learning->theory
TL;DR: Lottery Tickets can’t be trained from random init. We show that permuting the mask to align with the new init's optimization basin results in a mask that better trains from random init and approaches LTH generalization performance.

Abstract: The Lottery Ticket Hypothesis (LTH) suggests there exists a sparse LTH mask and weights that
achieve the same generalization performance as the dense model while using significantly fewer parameters. However, finding a LTH solution is computationally expensive, and a LTH sparsity mask does not generalize to other random weight initializations. Recent work has suggested that neural networks trained from random initialization find solutions within the same basin modulo permutation, and proposes a method to align trained models within the same loss basin. We hypothesize that misalignment of basins is the reason why LTH masks do not generalize to new random initializations and propose permuting the LTH mask to align with the new optimization basin when performing sparse training from a different random init. We empirically show a significant increase in generalization when sparse training from random initialization with the permuted mask as compared to using the non-permuted LTH mask, on multiple datasets (CIFAR-10/100 & ImageNet) and models (VGG11 & ResNet20/50).

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Mohammed Adnan 12
Rohan Jain
Ekansh Sharma
32 Rahul G. Krishnan 32 Yani Ioannou'
Abstract
and practitioners alike. To overcome this issue, different model compression methods, such as pruning, quantization and knowledge distillation, have been proposed reduce the model size at different phases of training or inference. Post-training model pruning (Han al., 2016) has been shown to be effective in compressing the model size, and seminal works have demonstrated that large models can be pruned after training with minimal loss in accuracy (Gale et al., 2019; Han et al., 2015). While model pruning makes inference more efficient, it does not reduce the computational cost of training the model.
The Lottery Ticket Hypothesis (LTH) suggests there exists sparse LTH mask and weights that achieve the same generalization performance as the dense model while using significantly fewer parameters. However, finding a L.TH solution is computationally expensive, and a LTH's sparsity mask does not generalize to other random weight initializations. Recent work has suggested that neural networks trained from random initialization find solutions within the same basin modulo per- mutation, and proposes method to align trained models within the same loss basin. We hypothe- size that misalignment of basins is the reason why I.TH masks do not generalize to new random ini- tializations and propose permuting the LTH mask to align with the new optimization basin when per- forming sparse training from a different random init. We empirically show a significant increase in generalization when sparse training from random initialization with the permuted mask as compared to using the non-penn non-permuted LTH mask, on multiple datasets CIFFAR-101000 & ImageNet) and models (VGG11 & ResNer20/50). Our codebase for re- producing the results is publicly available at here.
Motivated by the goal of training sparse model from random initialization, Frankle & Carbin (2019) demonstrated that training with highly sparse mask is possible and proposed the Lottery Ticket Hypothesis (LTH) to identify sparse subnetworks that, when trained, can match the performance of dense model. The key caveat is that a dense model must first be trained to find the sparse mask, which can only be used with the same random initialization that was used to train the dense model. Despite I.TH seeing significant interest in the research community, LTH masks cannot be used to train from new random initialization. Furthermore, it has been observed empirically that the LTH is impractical for finding a diverse set of solutions (Evci al., 2022).
This posits our main research questions: How can we train a LTH mask from a different random initialization while main- taining good generalization? Would doing so find a more diverse set of solutions than observed with the LTH itself?
1. Introduction
In recent years, foundation models have achieved state-of- the-art results for different tasks. However, the exponential increase in the size of sate--of-the-are models requires similarly exponential increase in the memory and compu- tational costs required to train, store and use these models decreasing the accessibility of these models for researchers
In this work, we try to understand why the LTH does not work for different random initializations from a weight-space symmetry perspective. Our hypothesis is that to reuse the LTH winning ticket mask with different random initialization, the winning ticket mask obtained needs to be permuted such that it aligns with the optimization basin associated with the new random initialization. We illustrate our hypothesis in Figure 1.
randomm
'Equal contribution 1 Schulich School of Engineering, University of Calgary "Vector Institute for AI 'Dept. of Com- puter Science, University of Toronto. Correspondence to: Mohammed Adnan <adnan.ahmad cannand@ullgarycca. Yani Ioannou loannou @ ucalgary.ca>.
To empirically validate our hypothesis, we obtain a sparse mask using Iterative Magnitude Pruning (IMP) (Renda et al., 2020; Han et al.,, 2015) on model A (from Figure 1) and show that given a permutation that aligns the optimization basin of model A and a new random initialization, the mask can
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the au)hor(s).

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
W1
W1
WB
intialization
solution dense
sperse
high loss
low loss
W
T-T A
Wo
m/
(a) Dense training and pruning model A.
(b) Sparse training model B with A mask.
Figure J. Weight Symmetry and the Sparse Training Problem. A model with single le layer and only two parameters, w= (wo, ), operating single input za has weight symmetry in the 2D loss landscape as illustrated above. In (a) the original dense model, WA- is trained from random dense initialization, w to dense solution, w 1= which is then pruned using weight magnitude resulting the mask m,, (1,0). In (b), naively using same mask train a model, from different random initialization will likely result in the initialization being far from good solution. Permuting the mask to match the (symmetric) basin in which the new initialization in will enable sparse training. Figure 12 in Appendix
full figure also including Lottery Ticket Hypothesis (LTH).
be reused. The sparse model (with the permuted mask) can be trained to closer match the generalization performance of the LTH solution, and the permuted mask improves the generalization of the trained sparse model compared to the non-permuted mask. Furthermore, increased functional diversity when using our approach compared to LTH solutions. Our contributions are as follows:
Furthermore, our experiments provide novel insights about the LTH and the corresponding dense model: we show that for a fixed initialization, the dense solution and the corresponding LTH solution remain in the same loss basin once we take into account variance collapse. Notably, our conclusion differs from the conclusion drawn by Paul et al. (2023), where they did not consider the variance collapse issue when interpolating between the sparse and dense solutions.
observe drastically
1. We hypothesize that the LTH (Frankle & Carbin, 2019) fails to generalize well with a new random initialization due mismatch between the optimization basin of the winning ticket mask and the new random initialization's solution basin. We propose method based on permuta- tion matching between two dense models, that permutes the winning ticket's ' sparse mask to align with the opti- mization basin of the new random initialization. We em- pirically demonstrate on CIFAR 10/100 and ImageNet datasets using VGG11 1 and ResNet models of varying widths that permuting the LTH sparse mask to align with the new random initialization improves the performance of the trained model (permuted), compared to the model trained without permuting the sparse mask (naive).
2. Background & Related Work
Linear Mode Connectivity. A pair of trained neural networks are said to be linearly connected if the loss along the linear path between the models remains small. The phenomenon of linear (mode) connectivity was first observed in the context of Stochastic Gradient Descent (SGD) by Nagarajan & Kolter (2019), where they showed that two neural networks trained from the same initialization but with different data orders exhibit linear connectivity. The term Linear-Mous Mode Connectivity (LMC) was introduced by Frankle et al. (2020), where they showed that independently trained neural networks can be linearly connected.
2.
We show that models trained from random initialization using the permuted LTH mask much more func- tionally diverse in the solutions they learn than those found from training the LTH winning ticket mask and initialization alone (Evci et al., 2022), across several existing functional diversity metrics and improved ensemble performance.
Linear Mode Connectivity modulo Permutation. En- tezari et al. (2022) further observed that while model and its randomly permuted counterpart are functionally equiva- lent, they are rarely linearly connected in the weight space. This misalignment suggests the presence of loss barriers regions along linear path between models where the loss
2

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
significantly higher than at the endpoints. They conjectured that independently obtained SGD solutions exhibit no loss barrier when accounting for permutation symmetries, sug- gesting that all SGD-trained networks converge to a single basin modulo permutations. Building on this conjecture, sev- eral algorithms have been developed to address permutation invariance by aligning trained networks to the same optimiza- tion basin (Ainsworth et al., 2023; Jordan et al., 2023; Singh & Jaggi, 2020; Tatro et al., 2020). Ainsworth et al. (2023) demonstrated that two models trained from different random initializations find solutions within the same basin modulo permutation symmetry. They proposed permutation match- ing algorithm to permute the units of one model to align it with a reference model, enabling LMC (Frankle et al., 2020). The use of activation matching for model alignment was originally introduced by Li et al. (2015), to ensure models learn similar representations when performing the same task. Jordan et al. (2023) investigated the poor performance of interpolated networks, attributing it to phenomenon they termed "variance collapse". To address this, they proposed a method that rescales the hidden units, leading to significant improvements in the generalization performance of interpo- lated networks. A rigorous study from Sharma et al. (2024) introduced notion of simultaneous weak linear connectivity where permutation, I, aligning two networks also simulta- neously aligns two larger fully trained networks throughout the entire SGD trajectory and the same π. also aligos succes- sive iterations of independently sparsified networks found via weight rewinding. Sharma al. (2024) also showed that for certain neural networks, sparse mask obtained via weight rewinding can be reused modulo permutations without hurt- ing the test performance.
train mask
1- ~
N
N
train
train
Sparee Toitin
mask
wha
W F
W
wh mi
prune
train
W
T
mA
Wh
w
I
w 7 m
activation matching
Prosiam (Nava) Soiddon (Pherozeed)
Figure 2. The overall framework of the training procedure, hegin- ning with two distinct dense random weight initializations, w w t sampled from a normal distribution, .. The sparse training problem attempts train the random initialization, w's using the naive mask m, found by pruning a dense trained model, w'A However, this results in poor generalization performance (Frankle ct al., 2020). We propose to instcad train w at some rewound epoch k equipped with a permuted mask (mm)). We show that this achieves more comparable generalization to the pruned model/trained LTH solution, w @.m..
re-learns the original pruned solution it is derived from (Evci et al., 2022). To make any practical use of sparse training, finding methods of sparse training from random initialization is necessary to realize any efficiency gains in training.
Weight Symmetry. Hecht-Nielsen (1990) demonstrated that neural networks are permutation invariant, where swap- ping any two neurons within a hidden layer does not alter the underlying function being learned. The permuted network remains functionally equivalent to its original configuration, i.e. neural networks are symmetric functions. The existence of permutation symmetries in weight space creates copies of global minima at different points in weight space (Entezari et al., 2022; Goodfellow et al., 2016; Simsek et al., 2021). Weight sparsity, achieved through pruning, can reduce the number of weight symmetries in a neural network. Pruning neurons reduces the number of permutation symmetries in layer. Unstructured pruning, in heterogeneously remov- ing individual weights, can break the weight symmetry of individual neurons in layers.
Lottery Ticket Hypothesis. The LTH proposes to solve the sparse training problem by re-using the same initialization as used to train the pruned models. For very small models, training from such an initialization maintains the generalization performance of the pruned model and demonstrates that training with a highly sparse mask is possible (Frankle & Carbin, 2019). However, subsequent work has shown that obtaining winning tickets for modestly- sized models requires using weight rewinding (Frankle et al., 2020) - requiring significantly more compute than dense training alone, especially considering that LTH also requires IMP, i.e. training of iteratively sparsified models. We include a detailed description of IMP in Appendix A.3. Paul et al. (2023) analyzed the IMP algorithm and showed that sparse network obtained after th IMP iteration is linearly connected to the sparse model obtained after K + 1th IMP iteration. In this work, we show that once we take into account the variance collapse studied in Jordan et al. (2023), we are able to show that the sparse solution obtained after the 1hth iteration is linearly connected to the dense solution. Furthermore, recent work has shown that the L.TH effectively
3. Method
Motivation. In this work, we try to understand why LTH masks fail to transfer to a new random initialization. Our hy- pothesis is that the loss basin corresponding to the LTH mask is not aligned with the new random initialization, as shown in Figure 1. Since the sparse mask is not aligned with the basin of the new random initialization. sparse training does not work well; therefore, aligning the LTH mask with the new random initialization may improve sparse training and enable the transfer of LTH masks to new random initializations.
3

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Ainsworth al. (2023) showed that the permutation symmetries in the weight space can be leveraged to align the basin of two models trained from dif- ferent random initializations. The permutation mapping can be obtained by either matching activations or weights. In this work, we use activation matching to obtain the permutation mapping as has been shown to be more stable in recent works (Sharma et al., 2024). Activation matching tries find permutation mapping, π Sa (where Sa is the permutation group of order d!) such that by permuting the parameters of the second model, the correlation between the activations of the two models is maximized. For a model consisting of L layers, each layer is sequentially matched and permuted starting from the input layer. Let 2,A,, 2 E Rdxx be the activations of layer of model A and B respectively obtained using the training data, where d represents the dimensionality of the activations at layer and n is the number of training data points. Then a permutation mapping for layer 1, Tt, is obtained by solving:
Permutation Matching.
To ensure that the permutation mapping, T, can closely match model A and model B, we evaluate the loss barrier between the permuted model A and model B. However, aligning neurons alone is not sufficient to establish a low loss barrier due to variance collapse (Jordan et al., 2023). To overcome the variance collapse issue, we used REPAIR (Jordan et al., 2023) to correct the variance of the activations in the interpolated/merged model. As shown in Figure 3, the loss barrier after permutation matching and correcting the variance (REPAIR) is lower than the loss at random initialization, showing permutation mapping can match the models to bring them closer/in the loss basin.
Aligning Masks via Weight Symmetry. In contrast to previous works (Ainsworth al., 2023), we are interested in permuting the mask obtained by LTH such that the op- timization basin of the permuted sparse mask and the new random initialization is aligned. To validate our hypothesis, we train two dense models, w and w'z where denotes the epoch, to convergence (trained for T epochs) and then use activation matching (Jordan al., 2023) find the per- mutation mapping R,, such that the activations of IT (w" and wi are aligned. Mask a obtained using IMP, is also permuted with the same permutation map T. The intuition is that the permuted mask aligns with the loss basin of model w which is necessary for sparse training and, therefore, the sparse model can be more easily optimized (see Fig- ure 2). We denote training with the permuted mask, π (m))) as permuted and with the non-permuted mask, M, as naive.
क. argmin|| -
(T, (2
where (..-)= denotes the Frobenius inner product. Equa- tion (1) can be formulated as Linear Assignment Problem (LAP) (Bertsekas, 1998; Ito et al., 2024) solved via the Hun- garian algorithm (Kuhn, 2010); however, the permutation found is not global optima but a greedylapproximate solution as permutation matching is a Na-hard problem. Once the permutation mapping is obtained for all the layers, the model A can be permuted to match model B. To ensure that the per- muted model does not change functionally when permuting the output dimension of layer I, the input dimension of the next layer is also permuted accordingly. Let W/ and by be the weights and bias of layer / respectively, then the permuted weight matrix WP and permuted bias b9 for each layer can be mathematically represented as,
can
Sparse Training. We first show that the dense solution w and the LTH solution obtained by training a model with sparse mask m 4 remain in the same linearly connected mode if one fixes the variance collapse identified by Jordan et al. (2023) by updating the activation statistics via the REPAIR method. We show in Figure 4a the error barrier after applying the REPAIR method remains considerably low as we increase sparsity by iteratively pruning (IMP). These results extend the findings of Paul et al. (2023) to show that when variance collapse is taken into account, the LTH solution remains in the same linearly connected basin as the original dense solution.
WP m. WIT
Up =m-
Evaluating Permutation Matching, Since LAP uses a greedy search to find an approximate solution, to ensure that the permuted model A and model B lie in the same basin, we evaluate the LMC (loss barrier) between the two models. More formally, let 0g be the parameters of two networks, then the loss barrier 23 is defined as:
In sparse training, the model is trained with a mask m, masking some of the weights, during both forward and backward passes. To evaluate the transferability of the per- muted LTH mask we train, a different random initialization w1=0 the LTH sparse mask m,A and permuted LTH mask t (m)), which we denote the naive and permuted solution respectively. We also evaluate the LTH baseline, i.e., training model w/=0 with mask M Since LTH requires weight rewinding to an earlier point in training, we also use rewind checkpoint from epoch t= k T for both the baselines and permuted solution. Figure 4b shows that LTH, dense model A and sparse permuted solutions all lie in the same mode.
B(01.02):= := sup (11-a))) o
-(L-a) 20, (3)
where L is the loss function evaluated on the training dataset.
If B(0,, ,,)) ss O, it is said that 01 and 0g are linearly mode connected.
4

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
3.0
16
16
Loss
0.5
Test Loss
2.5
Test Loss
0.4
2.0
0.34
1.54
0.2
1.0
0.0
0.2
0.4
0.6
0.0
176
0.0
0.2
0.4
0.6
0.8
1/0
0.0
0.2
0.4
0.6
1/0
(a) ReReNett0x (esNee20x(W)/CIARR
(b) ResNer20x (w)/CIAAR-ICCI 100
(c) ResNet50x { (aleeees000u////eeeeeee AmmagNNtt
Figure Larger width exhibits better LMC. Plots showing linear interpolation between (w and WB t= where through activation matching between two dense models for varying widths, w. As the width of the model increases, the permutation matching algorithm gets more accurate, thereby reducing the loss barrier (i.e., better I.MC), which is evaluated the test This shows that the permu- tation matching can find a better mapping, ग,, for wider models, explaining why the permuted mask works better in the case of wider models.
obtained
4.1. Experimental Results.
15.0
22-
12.5
14
ResNet20/CIFAR-I & CIFAR-100. We trained ResNet20 on the CIFAR- datasets. As shown in Fig- ures 5 and 6, the permuted solution outperforms the naive baseline across all model widths and rewind points. Since is more difficult to train models with higher sparsity, the gap between naive and permuted solutions increases sparsity increases, shown in Figure 5d for width multiplier 1,4,8, and 16. It can also be observed that at higher sparsity increas- ing the rewind point improves both the LTH and permuted so- lution but not the naive solution. The improved performance of the permuted solution over naive supports our hypothesis and shows that misalignment of the LTH mask and loss basin corresponding to the new random initialization could explain why LTTH masks do not transffffer to different initializations. We also show accuracy sparsity plots for k k= (10,25,50,100) (details in Appendix A.5); as sparsity increases, the gap between permuted and naive solution increases for all rewind points. As illustrated figure Figure 5, neither the LTT nor the permuted solution performs effectively with random initializaaton (k: 0) but improves on increasing the rewind point up to certain point, beyond which plateaus. Detailed results are presented in Tables 5 to 8 in Appendix A.4.
Barreer
10.0
184
7.5
500 L2VER
X7 A/REPUR
IMP Iterations
24
25
(a) (4NNer20xx4))CTFAA
(b) 0-1 loss landscape
Figure 4. LTH solution remains in the same linearly connected mode as the dense solution. In Figure 4a we plot the crror barrier between the dense solution and the sparse solution (y-axis) the IMP iteration corresponding to the sparse solution (x-axis), for 90% sparsity. We observe that after fixing variance collapse via the REPAIR method, the error barrier hetween the dense and the sparse solutions remains small, thus showing that I.TH solution remains in the same linearly connected mode as the dense solution. In Figure 4b we visualize the Joss landscape of ResNet20x [4)/CIAAR- 10. The figure is generated by evaluating the 0-1 loss spanned by three models the figure. We show that, modulo permutations, reusing the permuted mask leads to convergence in the same mode the original model, i.c. the LTT solution. Hence, there is a small loss harrier between the permuted and I.TH solutions, demonstrating they are within the same linearly connected mode.
We also validated our hypothesis on CIFAR- 100 using ResNet20 with varying widths. As shown in Figure 6, the permuted solution consistently outperforms the naive solution, showing that our hypothesis holds true across different models and datasets. Similar to the CIFAR-10 dataset, we increase the model width multiplier, the gap between the permuted and naive solution increases, showing the efficacy of our method. Detailed results are presented in Tables 11 to 14 in Appendix A.4.
4. Results
To validate our hypothesis. we trained ResNet20 (He
et al., 2016) and VGG11 (Simonyan & Zisserman, 2015)
models on the CIFAR-10/100 datasets (Krizhevsky, 2009)
(details Appendix A.1) across different levels of spar- sity 0$=00.8000.900000 95,9777) We used ResNet20 with vary- ing widths (w 1,4,8, 16) to study the effect of increasing width on the permutation matching and, thereby, the perfor- mance of the permuted sparse model. We also demonstrate our hypothesis on the large-scale ImageNet dataset (Deng et al., 2009) using ResNet50, showing the efficacy our method across different models and datasets of varying sizes.
VGGIICIIFAR10. We utilize the modified VGGIL architecture implemented by Jordan et al. (2023) trained on CIFAR-10 (details in Appendix A.1). We observe that for moderate sparsity (80%) in Figure 8a, the gap
5

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
93
Test Accuracy (5)
Text Accuracy (%g
Test Accuracy (%)
51-
W
68-
294
Permuted
100
Nahoo
Pommueed
Pemuzed
Rewind Points
Rewind Points
Rewind Points
Rewind Points
(16)
gan.5
Accuracy
955.5
95.0
12
945
34..5-
F33.5
20
Salvo
00
Pe
60
100
174
20
60
80
Nerrated
155
93.0
20
40
60
Party
80
100
20
00
100
Rewind Points
Rewind Points
Rewind Points
20.0
10 10=33 8
46
296.0
$3.3
Teat Accuracy
Accuracy
45,
Accuraty
Accuracy
65.75
95.0
25.50
08.25
95.0
24.54
35.25
94.0
25.00
20
Permuted
db
100
94.75
94.04
131
Porretee
100
072
Nare
Perrutal
Rewind Points
Rewind Points
Rewind Points
Rewind Points
25.0-
Accuracy 053
Test Accuracy (15)
Accurra
8-6.2
96 25
36.0
25.5
16
96.0-
96
95.0
65
R3.55
95.0
100
Teat
44.5
$4.0
100
Rewlnd
Rewind
Rewind Points
Rewind Polnts
(a) Sparsity=0.88
(b) sparsity= 0.90 0.90
(c) Sparsitty = 0.95 0.95
(d) sparsity = 0.97
Figure ResNet20x (w /CCFAR-IO. Test accuracy sparse network solutions vs. increasing rewind points for different sparsity levels and widths, w. The dashed (--) line shows the dense model accuracy. The effect of the rewind point on the test accuracy for different sparsities shown. the width increases, the gap between training from a random initialization with the permuted mask and the LTH/dense baseline (dashed line) decreases, unlike training with the non-permuted mask (naive), showing model trained with the permuted mask generalizes better than naive.
between the permuted and the naive baseline is not large, however for a higher sparsity level (90%), the permuted solution significantly outperforms the naive solution as shown in Figure 8b. For the VGG11 model, on increasing the rewind point, the permuted solution closely matches the accuracy of LTH, while the naive solution significantly plateaus and does not improve on increasing the rewind point. For higher sparsities, the naive haseline was unstable in training as the modified VGG11 architecture does not have BatchNorm layers (loffe Szegedy, 2015); we omit those results in the discussion for a fair comparison. Detailed results presented in Table 9 in Appendix A.4.
observed in Figure 7, the permuted solution outperforms the naive solution across all sparsity levels, showing that our hypothesis holds true on large- datasets as well. While the permuted solution performs better than the naive solution, there is still significant gap between LTH and the permuted solution in the case of the ImageNet dataset compared to the CIFAR-10/100 dataset. This could be due to permutation matching not being accurate enough, as only small subset of the training dataset was used for activation matching. This can also be visualized in terms of the loss barrier in Figure 3c between the permuted model A and model B: the loss barrier after permutation is more prominent compared to the CIFAR dataset (Figures 3a and 3b). Thus, the permutation mapping π cannot match the models perfectly in the case of ImageNet since the permutation matching algorithm uses a greedy search algorithm to find the permutation mapping. However, given a better mapping, may be possible to further improve the performance of the permuted solution as discussed
ResNeet50/mmgeeee.. We also validated our hypothesis on the ILSVRC 2012 (ImageNet) dataset, which consists of 1. 1.28 million images across 1,000 classes (Deng et al., 2009). We used the ResNet50 model to evaluate the performance of the permuted mask different sparsity levels. As
6

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Tast Accuracy 00
Test Accuracy (%)
Accuracy (0
Accuracy
666
64
60
62
100
20
40
60
100
100
20
100
Points
Points
Test Accuracy (%5
Test Accuracy (%)
Test Accuracy
10
13
74+
Test
74
171
No.se
Permrtted
171
Nate
Parrsss
12
Nave
27
Termusss
20
20
100
20
40
6D
100
150
0
20
40
20
80
100
Rewind Points
Rewind Points
Rewind Points
Rewind Points
81.
Test Accuracy (h6]
Test Accuracy (%)
00
1 = 8
Test accuracy
7044
Accuracy
70-
70
764
14
74
ab
160
40
100
72
100
Rewind
Test Accuracy (%)
Accuracy 6%)
Test Accuracy (%)
W 16
30-
Phamaaed
Natvo
Parmuted
311
Makra
105
150
25
Rewind Points
Rewind Points
Rewind Points
Rewind Points
(a) sparsity 0.80
(b) sparsity = 0.90
(c) sparsity = 0.95
(d) sparsity = 0.97
Figure 6. ResNet20x (w)/CIFAR--00 Test accuracy of sparse network solutions increasing rewind points for different sparsity levels and widths, w. The dashed (--) line shows the dense model accuracy. The effect of the rewind points on the test accuracy for different sparsities shown. As the width increases, the gap between training from random initialization with the permuted mask and the L.HH/desse baseline (dashed line) decreases, unlike training with the non permuted mask (naive), showing model trained with the permuted model generalizes better than naive.
in Section 4.3. Detailed results are presented Table 10 Appendix A.4. As demonstrated in Table 10, the permuted solution outperforms the naive approach by nearly 2% at higher sparsity levels.
mask (permuted), the LTH mask (naive), LTH mask & init. and the original pruned solution (IMP) on which the L.TH is based. We follow the same analysis as Evci et al. (2022) and compare the diversity of the resulting models, over five different training runs, using disagreement score, KL diver- gence and JS divergence. We also compare with ensemble of five models trained independently with different random seeds. As shown in Table I, an ensemble of permuted models shows higher diversity across the metrics than the LTH, showing that the permuted models learn a more diverse set of solutions. We provide additional details in Appendix C.
4.2. Diversity Analysis of Permuted Models.
limitation of LTH is that it consistently converges very similar solutions the original pruned model (Evci al., 2022). Evci et al. (2022) speculate this occurs because the LTH is always trained with the same mitializaation/rewind point, and effectively relearns the same solution. Our hypothesis is that permuted LTH masks, trained with distinct initializaaion/rewindd points and subject to approximation errors permutation matching, may learn more diverse functions than the LTH itself. We analyze the diversity of sparse models trained at 90% sparsity, with either permuted
4.3. Effect of Model Width Multiplier.
Permutation matching an NP-hard problem; the activation matching algorithm proposed by Ainsworth et al. (2023) does not find the global optimum; rather, uses greedy search to

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Test Accuracy (9
Table Ensemble Diversity Metrics CIEAR-IOCIEAR- Although the mean test accuracy of LTH is higher, the ensemble of permuted models achieves better test accuracy due better func- tional diversity of permuted models. Here compare several measurements function space similarity between the models in- cluding disagreement, which measures prediction differences (Fort ct, 2020), and Beulbacck-Leibler (KL MKL.//eeon-Shamoon Shannon (JS) diver- gence, which quantify how much the output distributions of different models differ (Evci et al., 2022). As shown, the permuted masks achieve similar diversity as computational expensive IMP solutions, also resulting ensembles with a similar increase in generalization.
90
664
86
10
15
20
40
45
50
Rewind Points
sparsity 0.80
92.5
90.0
Test Accuracy
87.5
Mask
Test Accuracy
Ensemble Disagree- KL JS Acc. (%) ment
85.0
82.5-
ResNer20x (1000xX11XCCOO (1/CIFAR-10
10
15
20
Rewind Points
30
35
40
45
none (dense)
92.76 0.
IMP
91.09
93.25
0.093
0.352
0.130
(b) sparsity = 0.90
LTH
91.15 + 0.163
91.43
0,035
0.038
0.011
Test Accuracy (%)
permuted
89.38 + 0.170
91.75
0.107
0.273
0.091
naive
88.68 + 0.205
91.07
0.113
0.271
0.089
ResNer20x (4)/CIFAR-100)
none (dense)
78.37
80
IMP
7.326
79.27
0.259
1.005
0.372
Pormuood
10
15
20
25
50
35
40
45
50
LTH
75.35 + 0.204
75.99
0.117
0.134
0.038
Rewind Points
permuted
72.48 + 0.356
77.85
0.278
0.918
0.327
sparsity = 0.95
naive
71.05 + 0.366
76.15
0.290
0.970
0.348
Figure 7. Ressett50x (1 NemageNet. Top-5 test accuracy rewinds points of sparse network solutions at various sparsity levels.
91.5
We observe the permuted solution consistently performing better than the naive solution for all sparsities. The dashed (-)) line shows the dense model accuracy.
Test Accuracy
91.0
80.
00.0
80.54
45
explore a restricted solution space. Therefore, practice, per- mutation matching does perfectly align two models. How- ever, it has been observed that for wider models, the algorithm can more closely align two models (Ainsworth et al., 2023; Sharma et al., 2024). To understand how the performance of the permuted model is affected by the approximation error of the matching algorithm, we evaluated the LMC and the accuracy of the permuted solution on ResNet20 models with varying layer widths. As shown in Figure 3, increasing the layer width, the loss barrier of the interpolated network re- duces, showing that permutation mapping becomes more ac- curate and aligns two models better. Also, can be observed in Figures and 6 that the permuted solution becomes close to the LTH solution on increasing the model width, showing that as the permutation matching becomes more accurate, the gap between the LTH and the permuted solution reduces.
Rewind Points
Rewlind Points
(a) sparsity 0.80
(b) sparsity 0.90
Figure 8. VGG11 (1)/IIFAR-11 Test accuracy of sparse solutions at increasing rewind points different sparsity levels. The dashed (--) line shows the dense model accuracy. In Figure 8b, the permuted solution closely matches the LTH solution. However, beyond certain rewind point, i.c. for > 20 the performance of the naive solution plateaus. Resulting in a more noticeable gap between the permuted and naive solutions compared to Figure 8a.
Neural Networks (DNNss. Our empirical findings across various models and datasets support the hypothesis that mis- alignment between the mask and loss basin prevents effective use of LTH masks with new initialization. Although finding permutation to align dense models is computationally expensive, the goal of our work to develop insights into the working of LTH and how the sparse mask can reused, not to improve the efficiency of LTH. We hope that our work will spur future work in this direction and will be useful the research community working in the realm of sparse training.
5. Conclusion
In this work, we demonstrate new insights into sparse training from random initialization and the Lottery Ticket Hypothesis (L.H)) by leveraging weight symmetry in Deep
8

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Impact Statement
References
Our work focuses on improving sparse training and reducing the computational cost of training DNN. By reducing the computational and memory requirements of training and facilitate the deployment
Ainsworth, S. K.. Hayase, J., and Srinivasa. S. reassn:
Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, (ICLR). OpenReview.net, 2023.
inference, sparse training of deep learning models on resource-constraine devices. However, most current hardware cannot leverage unstruc- tured spansity, which currently limits the impact of this work. Furthermore, model pruning often introduces algorithmic bias in the model (Hooker et al., 2020), it is important to evaluate algorithmic bias of sparse models before deploying for real-world applications.
Bertsekas, D. Network optimization: continuous and discrete models, volume 8, Athena Scientific, 1998.
Deng. J., Dong, W., Socher, R., Li, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009. doi: 10. WWW.:CCCPR..00.5200000
Acknowledgements
We would like to acknowledge the assistance of Nayan Saxena with an early code prototype for the project, and Yigit Yargic with an early iteration of Figure
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Leurning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
We gratefully acknowledge the support of Alberta In- novates (ALLRP-577350-2,, ALLRP-222301502,, the Natural Sciences and Engineering Research Council of Canada (NSERC) (RGPIN-2022-03112 DGECR-2022-
Evci, U., Ioannou, Y. A., Keskin, C., and Dauphin, Y. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, February 2022. doi: 2.099aaai.v36666.
00358), and Defence Research and Development Canada (DGDND-2022-03120),
This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.cu) ca).. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Fort, S., Hu, H., and Lakshminarayanan, B. Deep ensembles: A loss landscape perspective. arXiv preprint, 2020. doi: 10.48550/arXii. 1912.00777
Frankle, J. and Carbin, M. The lottery tickethypothesis: Find- ing sparse, trainable neural networks. In 7th International Conference on Learning Representations, 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
MA supported by the NSERC Postgraduate Scholarship, RBC Borealis through the Borealis AI Global Fellowship Award and the Digital Research Alliance of Canada EDIA Champions program. ES is supported by the Vector Research Grant at the Vector Institute. RGK gratefully acknowledges support from Canada CIFAR AI Chair. YI is supported by a Schulich Research Chair.
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML' 20. JMLR. org, 2020.
Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks. arXiv preprint, abs/1902.09574, 2019. doi: 10.48550/arXiv.. 1902.09574.
Contribution Statement
All authors contributed to the writing of the paper. MA and RJ implemented the method, conducted most experiments, and contributed the majority of the writing. MA and YI played key roles in designing the methodology and experimental setup. ES contributed to the design of the experimental setup, provided key insights, hypothesized the connection to the conclusion drawn by Paul al. (2023), summarized in Fig- ure 4 and our contributions, and ran experiments to validate it. RGK and YI, as senior authors, provided feedback on the writing and methodology. YI conceptualized the research idea, designed Figures 1, 2 and 12, and also contributed feedback on code and experiments throughout the project.
Goodfellow, J., Bengio. Y., and Courville, A. C. Deep Learning. Adaptive computation and machine learning. MIT Press, 2016. ISBN 978-0-262-00561-33.
Han, S., Pool, J., Tran, J., and Dally, W. J. Learning both weights and connections for efficient neural network. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1 1135-11444 1355-1143, 143, 2015.

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Han, S., Mao. H., and Dally, J. Deep compression: Compressing deep neural network with pruning. trained quantization and huffmman coding. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
volume 44 of Proceedings of Machine Learning Research, pp. 196- 212, Montreal, Canada, 11 Dec 2015. PMLR.
Nagarajan, V. and Kolter, Z. Uniform convergence may be unable to explain generalization in deep learning. In Wal- lach, H., Larochelle, H.,, Beygelzimer, A., d'Alché-Buc. F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, IEEE Computer Society, 2016. doi: 10.1100.11000 100.11099CYP...1.....0
Paganini, M. and Forde. J. Z. Streamlining tensor and net- work pruning in pytorch. arXiv preprint, abs/2004.13770, 2020. doi: 10448550/r2xiv.204..11
Hecht-Nielsen, R. On The Algebraic Structure of Feed- forward Network Weight Spaces. North-Holland,
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.,, Desmaison, A., Kopf, A., Yang, E., Detooo, Z. Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035, Curran Associates, Inc., 2019.
Amsterdam, 1990. ISBN 978-0-444-88400-8.
4.
Hooker, Moorosi, N., Clark, G., Bengio, S., and Denton, E. Characterising bias in compressed models. arXiv preprint, 2020. doi: 110.0044550/XX1.220000000.6
loffe, S. and Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML'15, pp. 448-456. JMLR.org, 2015.
Paul, M., Chen, F., Larsen, B. W., Frankle, J., Ganguli, S., and Dziugaite, G. K. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? The Eleventh International Conference on Learning Representations, 2023.
Ito, A., Yamada, M., and Kumagai, A. Analysis of linear mode connectivity via permutation- based weight matching. arXiv preprint, 2024. doi: 200110448550/rrrv22422...44
Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. REPAIR: renormalizing permuted activa- tions for interpolation repair. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Sharma, E., Kwok, D., Denton, T., Roy, D. M., Rolnick, D., and Dziugaite, G. K. Simultancous linear connectivity of neural networks modulo permutation. Learning and Knowledge Discovery in Databases. Research Track, pp. 262-279, Cham, 2024. Springer Nature Switzerland.
Machine
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kuhn, H. W. The hungarian method for the assignment problem. In Jünger, M., Liebling, T. M., Naddef, D., Nemhauser, G. L.. Pulleyblank, W. R., Reinelt, G., Rinaldi, G., and Wolsey, L. A. (eds.), 50 Years of Integer Programming 1958-2008 From the Early Years to the State-of-the-Art, pp. 29 29.47. Springer, 2010. 1oi:
Simonyan, K. and aidsermmma A. Very deep convolutional net- works for sarge-scale image recognition. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Simsek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, J. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9722-9733 PMLR, 2021.
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hoperoft, J. Convergent learning: Do different neural networks In Storcheus, D.,
learn the same representations? Rostamizadeh, A., and Kumar, S. (eds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015,
10

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
S. S. P. Jaggi, M. Model fusion via optimal transport. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Tatro, N. J., Chen, P., Das, P., Melnyk, I, Sattigeri, P., and Lai, R. Optimizing mode connectivity via neuron alignment. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
A. Appendix
A.1. Implementation Details for ResNet22 & VGG11 on CIFAR-10 and CIFAR-100
Architectures For residual neural networks, we train the standard ResNet20 on CIFAR-10 and CIFAR- 100 with varying width, We implemented scalar, w, that adjusts the number of channels in each convolutional and fully connected layer:
First Convolution Layer: The number of output channels is scaled from 16 to x 16.
Layer 1,2,3: The number of output channels for the convolutional blocks in these layers are scaled from 16, 32, and 64 to x 16, w x2, and w x 64, respectively.
Fully Connected Layer: The input dimension to the final linear layer is scaled to wo x 64.
For convolutional neural networks, train modified version of the standard VGG11 implemented by Jordan al. (2023) on CIFAR-10. Primary differences are:
A single fully connected layer at the end which directly maps the flattened feature map output from the convolutional layers the 10 classes for CIFAR-10 classification.
The classifier set up for CIFAR-I0 with 10 output classes as originally VGGI1 was designed for ImageNet with 1000 output classes (Deng et al., 2009).
Each of our results for a given rewound point, k, is averaged over 3 runs.
Datasets For our set of experiments we used the CIFAR- 10 and CIFAR-100 datasets (Krizhevsky, 2009). We apply the following standard data augmentation techniques to the training set:
Randontiorizzontalliip Randomly flips the image horizontally with given probability (by default, 50%).
RandomCrop: Randomly crops the image to size of x 32 pixels, with a padding of pixels around the image.
Optimizers Table
We use the following hyperparameters for ResNet20 and VGG1 trained on CIFAR-10/100, as outlined
Hyperparameter
Value
Optimizer
SGD
Momentum
0.9
Dense Learning Rate
0.08
Sparse Learning Rate
0.02
Weight Decay
5 10
Batch Size
128
Epochs (T)
200
Table Hyperparameters dense and sparse training both ResNet20 and VGGII.
A.2. Implementation Details for ResNet50 on ImageNet
Architecture We utilize the standard ResNet50 implementation provided by torchvision and customize PyTorch's distributed data parallel codebase for training models on ImageNet (Paszke al., 2019).
Dataset For our set of experiments used the ImageNet dataset (Deng et al., 2009). We apply the following standard data augmentation techniques to the training set:
SanddmHorizentllPP Randomly flips the image horizontally with a given probability (by default, 50%).
RandomRes izeCrrop: Randomly crops a region from the image and resizes it 224 224 pixels.
12

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Optimizers
We use the following hyperparameters for ResNet50 trained ImageNet, outlined in Table
Hyperparameter
Value
Optimizer
SGD
Momentum
0.9
Dense Learning Rate
0.4
Sparse Learning Rate
0.4
Weight Decay
1 10
Batch Size
1024
Epochs (T)
80
Table 3. Hyperparameters for dense and sparse training of ResNet50.
A.3. Pruning
We apply standard Iterative Magnitude Pruning Fine Tuning (IMP-FT) (Frankle & Carbin, 2019; Han et et., 2015; Renda et al., 2020) to obtain our final mask, MA, producing sparse subnetwork w @ mA For pruning, we utilize Py PyTorch's torch. nn. www.nnnnnttitlll. uuullss..................... prune library (Paganini & Forde, 2020).
1. In unstructured, global manner, identify and mask (set to zero) the smallest 20% of unpruned weights based their magnitude.
2. This process is repeated for s rounds to achieve the target sparsity S, with each subsequent round pruning 20% the remaining weights.
3. During each round, the model is trained for
per prune epochs.
Hyperparameter
ResNet20/VGG11
ResNet50
train_ @eaiii__eochs_ee____
50
20
Learning Rate
0.01
0.04
Table 4. Hyperparameters used for pruning ResNet20/VGG11 CIFAR-10/100 ResNet50 ImageNet.
A.4. Results
Detailed results for ResNet20x /CIFAR- 10 are provided in Tables to for VGGII (1)CIA(I)/ in Table for ( in Table 10, and for ResNet20 ( (CIFFARR- in Tables 11 14.
A.5. Additional Plots
Refer to Figures 9 and 10 for additional accuracy-vs-sparsity plots for ResNet20 on CIFAR-10 and CIFAR-100. Refer to Figure 11 for Top-1 accuracy vs. rewind points for ResNet50 on ImageNet.
B. Early Matching
In our current methodology, train both models A and B to convergence, resulting in the weight configurations w and w respectively. However, it has been observed in (Sharma et al., 2024) that it is possible to find permutation mapping carlier in training. In effort to reduce the computational cost associated with our approach, we aim to find a permutation ı that allows us to suitably align the weights of model at convergence, w with the weights of model B at an earlier training iteration T, w tim In Table 15, we provide additional results on early matching with the CIFAR- dataset, which shows that models can be matched earlier in the training, thus reducing the computational cost of our method.
13

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table 5. ResNet20X CCIFARR-- Results using the ResNet20x {1} trained on CIFAR- IO0 from a rewind point k using various methods of sparse training with sparsity S. LTH trains within the original dense/pruned solution basin, while naive/permuted train from new random initialization.
Rewind Epoch i:
Method
k= 0
10
20
25
50
75
100
LTH
90.41 + 0.14
92.12 +: 0.25
92.08 4: 0.36
92.10 +: 0.27
92.25 :: 0.14
92.32 1: 0.26
92.15 = 0.13
92.26 +: 0.19
92.21 0.16
80%
naive
89.67 + 0.35
89.74 + 0.69
90.16 + 0.14
90.07 +: 0.09
90.13 + 0.11
90.40 + 0.11
90.66 - 0.12
90.31 + 0.27
90.45 + 0.22
perm.
89.74 0.05
90.15 + 0.16
90.26 +: 0.08
90.72 + 0.12
90.68 = 0.18
90.72 + 0.28
90.76 = 0.27
91.13 + 0.06
90.82 0.21
LTH 90%
89.45 + 0.10
91.27 + 0.37
91.34 + 0.29
91.34 + 0.09
91.18 + 0.27
91.43 + 0.22
91.44 + 0.12
91.36 + 0.18
91.68 + 0.28
naive
88.47 + 0.21
88,70 + 0.14
88.77 + 0.21
88.84 + 0.43
88.83 + 0.27
88.78 + 0.02
88.99 = 0.08
88.81 = 0.17
88.82 + 0.07
perm.
88,59 +: 0.11
89.09 +: 0.22
89,56 +: 0.28
89.71 +: 0.12
89,50 +: 0.27
89.97 + 0.13
89.84 = 0.15
90,03 1: 0.07
89.77 + 0.15
LTH
87.83
0.38
90.33 + 0.22
90.39 +: 0.28
90.37 1: 0.21
90.58 :: 0.26
90.43 +: 0.20
90.56 = 0.29
90.44 :: 0.26
90.40 0.19
95%
naive
86.89 0.21
87.01 0.23
86.88 4: 0.13
87.28 +: 0.19
87.31 +: 0.36
87,00 :: 0.19
86.88 = 0.08
86,99 : 0.29
86.
0.22
perm.
87.24 0.22
87.70 0.08
87.92 + 0.25
88.23 +: 0.52
88.29 0.52
88.24 + 0.20
88.21 0.30
88.21 +: 0.20
88.04 0.22
LTH
86.03 +: 0.22
88.00 + 0.02
88.73 + 0.05
89.00 + 0.24
89.21 + 0.23
89.27 +: 0.14
89.03 + 0.27
89.12 +: 0.25
89.06 0.21
97%
naive
85.60 + 0.38
85.43 + 0.40
85.89 + 0.37
85.48 + 0.13
85.36 + 0.14
85.70 + 0.21
85.30 = 0.32
85.14 + 0.29
84.64 + 0.34
perm.
85.61 + 0.48
85.93 + 0.34
86,26 +: 0.40
86.48 +: 0.39
86.12 + 0.27
86,16 + 0.14
86.43 = 0.27
86.06 + 0.26
85,95 + 0.14
Table 6. ResNet20 (1)/CIARA Results using the ResNet20x {4] trained on C10 from rewind point & using various methods of sparse training with sparsity S. LTT trains within the original dense/pruned solution basin, while naive/permuted train from a new random initialization. Note this table is the same setting as Table 5 except w =4.
Rewind Epoch
Method
k=0
5
10
20
25
50
100
LTH
94.67 :: 0.14
95.57
0.05
95.84
95.80 +: 0.12
95.88 + 0.20
95.72
0.09
95.81
0.10
95.83 : 0.21
95.71 0.16
80%
naive
94.36 + 0.04
94.55 + 0.14
94.59 + +-229
94.74 + 0.13
94.69 + 0.09
94.81
0.06
95.07 + 0.17
95.02 + 0.11
94.97 + 0.21
perm.
9..39=0..9 0.19
94.88
0,28
95.15
95,20 0.16
95.17 + 0.21
95.28 + 0.29
95.43 = 0.14
95.40 = 0.10
95.30 0.08
LTH
94.43 :: 0.17
95.53 :: 0.21
95.63 1: 0.07
95.65 +: 0.30
95.66 +: 0.07
95.61 + 0.14
95.56 + 0.16
95.62 = 0.14
95.50 0.04
90%
nuive
93.79 + 0.15
93,96 +: 0.05
94.09 +:101
0.29
94.35 + 0.25
94,20 0.13
94.27
+ 0.19
94.23 + 0.08
94,19 = 0,27
perm.
93.97 = 0.29
94.64 = 0.13
0.117
94.93
94,92 0.11
94,90 +: 0.07
95.04 = 0.14
95.07 = 0.18
94,91 = 0.19
LTH
93.65 = 0.12
95.26 0.08
95.39
95.32
95,26 :: 0.03
95.33 + 0.07
95.40 +: 0.14
95.19 = 0.05
95.37 0.21
95%
nuive
93.27 = 0.07
93.30 0.11
93.63 1.63+00.00
93.61 0.21
93.66:10.11
93.67 1.14
93.43
0.21
93.51 = 0.32
93.14 0.03
perm.
93.54 + 0.24
94.17 + 0.07
94.46 + 0.10
94.27 + 0.19
94.61 + 0.07
94.54 0.07
0.11 94.75
94.75 + 0.09
94.54 + 0.27
LTH
93.00 +: 0.11
94.77
0.09
94.86 +: 0.06
94.9440.11
94.96 + 0.06
94.89 + 0.21
95.00
0.24
94.94 + 0.10
94.97 0.13
97%
naive
92.63 : 0.12
92.80 0.10
92.85 0.21
92.66
92.69 0.11
92.28 - 0.09
92.02 - 0.18
91.87 0.10
perm.
92.81 = 0.27
93.54 0.08
93.83 0.12
93.75 0.34
94.00 0.33
9..12
94.07 - 0.31
94.32 = 0.24
94.14 0.04
Additional Details for Section 4.2
In Section 4.2, the IMP solution is trained independently over 5 different seeds with iterative pruning to obtain 5 different sparse/pruned solutions with different sparse maskshopologies (M1 M2, M3, M4, Ms). The LTT ensemble is trained using the same mask (M and initialization (w) over 5 different runs (with different data order). Random initialization wi defines the winning ticket for mask Mj. The permuted ensemble is trained using 5 different permutations (T1, T2, N3, T4. TT) of the same mask (M1) with five different random weight initializations (W1, Wg. W3, W0, Wg).
D. Computational Overhead of the Permuted Solution
The primary difference in computational complexity between the LTH, naive, and permuted solutions lies in the process of neuronal alignment, where weight/activation matching is used to locate permutations in order to bring the hidden units of two networks into alignment. To obtain the permuted solution, two distinct models must be trained independently to convergence, after which their weights or activations are aligned through a permutation-matehhie though relatively efficient, adds a small computational overhead compared to LTH and naive solutions, which do not involve matching steps. However, it's 's important to note that the primary goal of this study is not to improve training efficiency but rather to investigate why the LTH framework fails when applied to sparse training from new random initializations (not
process. This alignment,
14

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table XesNet20x (8) (S/CCAAR--0. Results using ResNet20x {8} trained on CIFAR- 10, from rewind point using various methods of sparse training with sparsity S. LTH triins within the original dense/pruned solution basin, while naive/permuted train from new random initialization. Note this table is the same setting as Table except w =8
Rewind Epoch k
S
Method
k=
10
25
50
100
LTH
95.35 + 0.07
95.98 + 0.14
96.12 + 0.04
96.10 0.20
96.21 + 0.06
80%
naive
95.17 + 0.17
95.32 + 0.13
95.63 + 0.13
95.62
0.08
95.79 + 0.15
perm.
95.36 + 0.14
95.60 + 0.15
95.89 + 0.19
95.94 + 0.17
95.94 + 0.06
LTH
94.96 + 0.18
95.97 0.15
96.02 + 0.05
96.00 + 0.19
96. + 0.10
90%
naive
95.05 = 0.07
95.12
9.22
95.44 # 0.14
95.06 + 0.25
perm.
95.05 = 0.05
0.06
95.78
95.87 0.13
95.85 + 0.11
LTH
94.86 + 0.08
95.90 0.15
95.93 + 0.26
96.07 + 0.25
96.00 + 0.25
95%
naive
94.60 = 0.14
0.13
9.17
95.01 = 0.33
94.59 + 0.52
perm.
94.85 = 0.19
95.63 0.27
95.67 + 0.16
95.59 0.22
LTH
94.54 + 0.23
0.11
0.03
95.78 0.21
95.90 + 0.04
97%
naive
94.39 + 0.04
0.04
0.118
94.19 + 0.11
93.83 + 0.08
perm.
94.46 + 0.14
0.100
95.16+0..0
95.56 0.06
95.45 + 0.05
Table 8. ResNet20x ResNer20>x16XXCIIAAR (16)/CIAA--1,, Results using the ResNet20 {8) trained CIFAR-10, from rewind point using various methods of sparse training with sparsity LTH trains within the original densc/pruned solution basin, while naive/permuted train from random initialization. Note this table the same setting as Table except w 16.
Rewind Epoch A
S
Method
k=
10
25
50
100
LTH
95.62 0.19
95.84 + 0.36
96.05 + 0.34
96.31 + 0.18
9
80%
naive
95.47 0.15
95.71 + 0.22
95.71 + 0.26
96.09 + 0.04
95.99
perm.
9.17
95.79 + 0.29
96.00 0.14
96.24 0.11
96.21
LTH
96.10
96.19 0.49
96.18 0.20
96.41
90%
naive
95.37 + 0.09
95.47 + 0.13
95.66 + 0.01
95.70 + 0.13
95.76 + 0.14
perm.
95.58 + 0.22
95.80 + 0.14
96.11 0.13
96.17 + 0.17
96.04 + 0.05
LTH
95.08 = 0.21
95.96 0.39
9
96.16 0.30
96.26 + 0.23
95%
naive
95.27 + 0.13
95.43 + 0.09
95.57 + 0.37
95.63 0.25
95.27 + 0.55
perm.
95.39 = 0.26
96.02 0.22
96.12 +.188
96.18 0.18
96.06 + 0.09
LTH
95.19 + 0.27
95.84 + 0.25
96.14 + 0.30
96.12 + 0.27
96.17 + 0.33
97%
naive
94.94 + 0.04
95.06 0.17
95.29 + 0.15
95.13 0.19
94.35 + 0.45
perm.
95.07 + 0.06
0.22
95.88 0.14
95.90 + 0.24
95.88 + 0.09
associated with the winning ticket's mask).
E. Full Symmetry Figure including Lottery Ticket Hypothesis In Figure we include the full version of Figure including an illustration of the LTH in Figure 12b.
15

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table VTGIII Results using the VGGII trained on CIFAR-10, from rewind point using various methods
sparse training with sparsity S. LTH trains within the original dense/pruncd solution basin, while naive/permuted train from a new random initialization.
Rewind Epoch
S
Method
k= 0
10
15
20
25
50
LTH
89.94 +: 0.06
90.44 0.17
90.91 + 0.12
90.87 0.16
91.14 + 0.28
91.11
0.08
91.22 + 0.08
80%
naive
89.70 + 0.13
89.90 = 0.18
90.04 + 0.07
90.34 + 0.16
90.48 + 0.19
90.55 + 0.17
90.87 + 0.19
perm.
89.94 + 0.1
90.18 + 0.08
90.52 + 0.17
90.71 + 0.22
90.77 0.19
90.81 + 0.19
91.07 + 0.21
LTH
89.33 + 0.16
90.82 0.09
90.97 + 0.14
91.05 0.04
91.15 0.11
90.91 0.17
91.08 + 0.31
90%
naive
89.17 -.7
09.55
89.81
0.05
09.68
89.80 0.03
89.80 = 0.05
perm.
89.30 0.02
90.33 0.08
90.44
9
90.75
90.76 0.12
91.01 + 0.06
Table 10. ResNet50 (1yMmagee Top-1 and Top-5 Accuracies of ResNet50x (1) trained on ImageNet, from rewind point k using various methods of sparse training with sparsity S.
Top-1 1 Accuracy
Top-5 Accuracy
S
Method
1: 10
25
50
k 10
25
50
LTH
72.87
72.
65.23
91.13
90.66
86.65
80%
naive
69.13
68.94
60.30
88.85
88.1
83.22
perm.
69.87
69.85
61.14
89.16
89.45
84.04
LTH
71.40
70.74
60.62
90.27
90.00
83.94
90%
naive 65,49
64.77
54.46
86.55
86.26
79.07
perm.
66.25
66.37
57.40
87.23
87.37
81.45
LTH 68.61
68.07
59.83
89.03
88.25
82.96
95%
naive 61.39
60.77
51.78
83.79
83.58
76.79
perm.
62.48
62.77
52.98
84.51
84.79
78.11
16

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table 11. ResNet20x (1 ICIFAR-100. Results using ResNet20x (1} trained on CIFAR-100, from a rewind point k using various methods of sparse training with sparsity S. LTH trains within the original densc/pruned solution basin, while naive/permuted train from a new random initialization.
Rewind Epoch l:
S
Method
k=
10
25
50
100
LTH
63.69 + 0.41
67.67 0.08
67.66 + 0.25
67.82 + 0.17
67.73 + 0.38
80%
naive
62.89 + 0.16
63.37 + 0.09
63.07 + 0.44
63.36 :: 0.27
63.33 + 0.35
perm.
63.04 + 0.24
64.07 + 0.15
64.71 + 0.10
64.52 + 0.78
64.57 + 0.49
LTH
59.81 0.29
65.21 0.17
65.15 + 0.28
65.10 0.30
65.17 + 0.21
90%
naive
58.77 + 0.28
59.59 + 0.18
59.44 + 0.27
59.19 + 0.41
58.58 + 0.16
perm
59.32 0.32
60.60 0.79
61.32 0.33
61.53 0.65
60.93 + 0.51
LTH
55.71 + 0.52
61.08 + 0.54
61.73 + 0.18
61.65 0.37
61.68 + 0.18
95%
naive
54.04 0.29
$5.20
54.65 + 0.38
54.96
0.57
53.97 + 0.91
perm.
55.12 0.17
56.93 0.26
57.64 + 0.36
57.47 0.66
57.13 + 0.34
LTH
51.10 0.34
56.14 0.56
56.92 + 0.25
56.94 + 0.13
56.93 + 0.06
97%
naive
49.70
0.25
49.49 0.32
49.16
47.70
perm.
5.234
51.55
51.88 + 1.08
52.64 0.34
50.96 + 1.15
Table 12. ResNet20x (1)/CIFAR-100. Results using the ResNet20x (1) trained CIFAR-100, from rewind point k; using various methods of sparse training with sparsity S. LTH trains within the original dense/pruned solution basin, while naive/permuted train from random initialization. Note this table the same setting as Table 11 except 4.
Rewind Epoch A
Method
k=
10
25
50
100
LTH
74.46 + 0.12
77.57 + 0.06
77.35 + 0.31
77.75 + 0.26
77.64 0.14
80%
naive
7..30
44110
74.98 0.17
75.21
0.12
15.20
perm.
73.68 - 0.09
7.324
75.74 0.41
76.12 0.37
+
LTH
72.54
76.56 0.11
76.80 0.34
76.80
90%
naive
7
72.56
72.89 0.27
72.59 + 0.15
72.54:0.33
perm.
72.18 + 0.23
74.17 0.35
74.21 + 0.23
74.45 + 0.27
74.89 + 0.47
LTH
7.216:0.25
75.41 0.18
75.68 0.17
75.76 0.17
95%
naive
70.17 + 0.47
70.95
+
71.21 + 0.26
69.95 + 0.42
perm.
70.41 + 0.07
7..70+00211
72.92 + 0.33
73.65 0.28
73.41 + 0.18
LTH
69.06 + 0.03
74.00
74.08 0.37
74.18 0.18
74.29 0.31
97%
naive
68.40 0.21
69.26
69.06
68.67 0.47
13.42+0.77
perm.
69.08 0.22
71.41 + 0.54
71.49 0
71.92 0.17
72.20 + 0.08
17

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table 13. ResNet20x (8)/0)/(8///AAAAA (8)/CIEAR-100. Results using ResNet20x (8) trained on CIFAR-100 100, from a rewind point k. using various methods of sparse training with sparsity S. LTH trains within the original dense/pruned solution basin, while naive/permuted train from a new random initialization. Note table the same setting as Table except =.
Rewind Epoch A
Method
k=
10
25
50
100
LTH
78.09 = 0.28
80.63 + 0.32
80.83 + 0.39
80.92 + 0.06
80.66 + 0.34
80%
naive
76.86 + 0.17
77.47 0.35
78.20 + 0.61
78.65
0.33
78.74 + 0.39
perm.
77.34 0.26
78.82 + 0.34
79.20 0.16
79.55 0.38
79.54 + 0.39
LTH
76.47 0.43
80.02 0.07
80.10 + 0.13
79.98 + 0.33
79.98 + 0.20
90%
naive
75.68 + 0.23
76.36 0.21
76.80 0.14
77.27 0.12
76.55 + 0.49
perm.
7
77.99
0.12
78.62 - 0.19
78.82 0.17
LTH
75.38 + 0.02
79.42 0.06
79.24
79.35 0.06
79.29 + 0.13
95%
naive
74.78 = 0.15
75.48 0.18
75.53 0.15
75.27 0.15
74.38 + 0.65
perm.
75.07 1.07=0000 0.14
76.97 0.46
77.80 + 0.14
77.74 0.51
78.04 + 0.42
LTH
78.63 0.297
78.65 0.50
78.74 0.49
78.47 0.16
97%
naive
73.13 0.26
73.73 + 0.12
73.76 0.27
73.26 + 0.07
72.79 + 0.46
perm.
73.81 0.67
76.29 0.14
76.38 + 0.57
76.57 + 0.29
76.79 + 0.76
Table 14. ResNet20x(16)/CIER. Results using the ResNer20x {16] trained on CIFAR- 100, from rewind point using various methods of sparse training with sparsity S. LTH trains within the original dense/pruned solution basin, while naive/permuted train from a new random initialization. Note table the same setting as Table except
Rewind Epoch k
Method
k=
10
25
50
100
LTH
80.21 0.18
82.32 + 0.34
82.40 + 0.26
82.48 + 0.38
82.16 0.30
80%
naive
79.31 + 0.06
79.50 0.09
80.24
81.02 0.11
81.01 + 0.07
perm.
79.35 0.11
8.444
81.15
81.57 0.38
81.81 + 0.21
LTH
82.14 0.18
81.95 + 0.03
82.11 + 0.12
90%
naive
78.78 + 0.37
79.26 0.11
79.42 + 0.51
79.56 0.26
79.57 + 0.13
perm.
9.09
80.49 0.32
80.59 0.15
81.12 0.05
81.24 0.09
LTH
78.32 0.34
81.57 0.09
81.57 + 0.32
81.47 0.25
81.63 + 0.07
95%
naive
78.01 + 0.02
78.53 0.10
78.45 0.21
78.38 + 0.43
77.49 + 0.06
perm.
78.25 + 0.20
79.76 0.20
80.50 + 0.04
80.47 + 0.08
80.25 + 0.21
LTH
77.49 0.27
81.07 + 0.07
81.06 0.11
81.11 0.18
81.14 + 0.32
97%
naive
76.46 + 0.44
76.71 + 0.41
77.19 0.09
76.93 + 0.36
75.53 + 0.40
perm.
77.04
79.30 0.11
79.62 + 0.14
79.63 + 0.06
18

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Test Accurass (%a
Test Accur acy (9))
Test Accuracy
00m
Narrated
0.00
3.85
5.80
6.55
0.30
0.35
0.30
3.55
6.00
0.55
0.30
0.35
5.00
6.85
6.50
0.55
Sparsity
Sparsity
Sparsity
Sparsity
(a) rewind=10
(b) rewind 25
(c) rewind 50
(d) rewind 100
96
964
96
Accuracy (%6
Accuracy (%6
Test Accuracy (%)
(16
100
Parmabed
13H
Habro
Parmuted
271
Mahra
Sparsity
Sparsity
Sparsity
Sparsity
(e) rewind=10
(f) rewind 25
(g) rewind 50
(h) rewind
66.0
47.0
96.0
Test Accuracy
25.5-
Aecuracy
05.55
Accuracy
35.5
Test Accuracy
35.3
W 8
35.0
65.0
05.0
445
94.5
24.0
635
67-4
0.35
Sadvoo
0.30
Perriced
0.55
030
1TH
0.30
Nytrated
0.55
94.0
020
17H
0.55
No KT
0.30
Porrated
0.35
0.50
13H
3.35
Nake
Permacod
0:55
Sparsity
Sparsity
Sparsity
Sparsity
(i) rewind=10
(j) rewind = 25
(k) rewindd = 50
(I) rewind 100
05.50
36.5
Test Accuracy (83
25
Test (4)
0.
Accuracy
Test Accuracy
10 = 16
85. 75
$3.50
85 50.00
35.25
95.00
94.0
Sparsity
Sparsity
Spaniit
Sparsity
(m) 10
(n) rewind=:dd00
(0) rewind 50
(p) rewind
Figure Accuracy sparsity trend for ResNet20x (w) (CI//AAAR--O.AA the width increases, the gap between permuted and naive solutions increases, showing permuted masks help with sparse training. With increased width, we observe a more significant gap seen throughout Figures 9h, 91 and 9p and the permuted solution approaches the LTH solution. The dashed (--) line shows the dense model accuracy.
19

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Tast Accuracy (80
Text Accuracy [%)
Test Accuracy (%)
Accuracy (%)
50
270
U7
Namuus
50
U24
31
Natro
0.50
0.55
3.50
0.55
0.30
0.25
0.30
0.55
0.50
0.55
0.50
0.55
0.00
0.85
0.50
0.55
Sparsity
Sparsity
Sparsity
Sparsity
(a) rewind=10
(b) rewind=25
(c) rewind=50
(d) rewind=100 100
Accuracy (%0
Accuracy (%6
Test Accuracy (%)
72+
72
11
68
0.30
0:30
0.90
Sparsity
Sparsity
Sparsity
Sparsity
(c) rewind=10
(1) rewind = 25
(g) rewind = 50
(h) rewind = 100
Test Accuracy 090
60
Test Accuracy [86]
Test Accuracy (%)
80-
78+
10 = 8
70
0.00
17H
0.05
3.50
Narrsted
0.55
0.30
17H
0.25
Nahi
0.30
2.55
74
0.50
UHH
0.55
0.30
Notroted
0.55
72
0.50
24
0.85
Matre
0.50
0.50
Sparsity
Sparsity
Sparsity
Sparsity
(i) rewind=10
(j) rewind = 25
(k) rewind =50
()) rewind 100
Accuracy 1%0
Test Accuracy (%)
Accuracy
51
11 16
754
78
77
0.30
Sparsity
Sparsity
Sparsity
(m) rewind=10
(n) rewind = 25
(0) rewind = 50
(p) rewind = 100
Figure Accuracy vs sparsity trend ResNet20 w /CIFAR-100 Similar the phenomenon seen in Figure 9, with higher width, the gap between permuted and naive solutions increases. As seen in Figures 10d, 10h, 101 and 10p and the permuted solution approaches the LTH solution. The dashed (- -) line shows dense model accuracy.
Test Accuracy (%)
72
Test Accuracy (%)
70+
Test Accuracy (%)
70+
70
6.5
66
60+
62*
Permuted
55
Permated
30
50
Rewind Points
Rewind Points
Rewind Points
(a) sparsity = 0.80
(b) sparsity = 0.90
(c) sparsity = 0.95
Figure ResNet50x Net. Top-1 test accuracy vs rewinds points of sparse network solutions various sparsity levels. We observe the permuted solution consistently peroforming better than the naive solution for all sparsitics. The dashed (--) line shows the dense model accuracy.
20

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Table 15. ResNet20x /CIFAR-O0 Results using the ResNer20x {1} trained on CIFAR-10, from a rewind point k= 20, using various methods of sparse training with sparsity S. The LTH and naive methods remain fixed they independent of matching. For the permuted method, the permutation, n, is obtained by matching fully trained dense model at t= (T = 200) with another model at an early point training at l =i, where i 55,0
Early Matching Point
S
Method
1=5
50
100
200
LTH naive
92.25 + 0.14
80%
90.13 + 0.11
perm. 90.49 + 0.37
90.34 + 0.63
90.42 + 0.29
90.42
0.25
90.68 + 0.18
LTH
91.18 + 0.27
90%
naive
88.83 + 0.27
perm. 89.16 0.51
89.23 + 0.59
89.39 + 0.69
89.31 0.60
89.50 + 0.27
LTH
90.58 0.26
95%
naive
87.31 0.36
perm. +7.37+0.33
87.68 + 0.77
87.43 + 1.00
87.54 = 0.43
88.29 + 0.52
LTH
89.21 = 0.23
97%
naive
85.36 = 0.14
perm.
85.77 0.44
85.93 0.94
86.09 0.51
85.88 0.47
86.12 0.27
21

---

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
W1
Initialization
tran
mask
solution
dense
sparse
high loss
low loss
whi
W
w
W
m,
Wg
w/
Dense training and pruning model A.
(b) LTH training model with pruned mask.
W1
w 0
1=0
(ma))
W @F mm.
WB [=0
& ma/
Wo
(c) Sparse training model B with A mask.
Figure Weight Symmetry and the Sparse Training Problem (Full Figure). model with single layer and only parameters, w (10),a1)), operating on single input Zo has the weight symmetry in the 2D loss landscape as illustrated above. In (a) the original dense model, w a. trained from a random dense initialization, w w== to dense solution, w which then pruned using weight magnitude resulting in the mask m = (1,0). In (b) we re-use the init. wxr to train model A with the pruned mask from (a). m.. in the LTH. (c), naively using the same mask to train model, B, from different random initialization will likely result the initialization being far from good solution. Permuting the mask to match the (symmetric) basin in which the new initialization is in will enable sparse training.
22


Reviews:
[{"review_id": "kzsEebIGIm", "reviewer": "Reviewer_oZBY", "summary": "The paper studies the property of winning tickets, where the combination of the sparse mask and initial weight values determines eventual generalization performance. When these networks are trained using the same sparse mask but different weight initializations, their performance deteriorates. This is a well known test for whether a sparse network at initialization is a winning ticket or not.\n\nThe authors hypothesize that this is due to the mismatch of the basins between different weight initialization and the winning ticket mask. This hypothesis was motivated by the recent discovery that dense networks trained from different random initializations find solutions within the same loss basin modulo permutation. The authors then test the hypothesis by finding a permutation function between two trained dense networks, and show that the permuted winning ticket mask corresponding to the first network can be applied to the second network at a rewind point with better performance as compared to naive application of the mask.\n\nThe analysis conducted demonstrates that sparse networks found in each iteration of the IMP are linearly connected in the loss landscape to the dense network when the variance collapse is accounted for. This allows for finding a permutation function after the dense networks are fully trained and not when both are fully pruned. The authors validate the hypothesis on a range of networks and datasets. The results show that the performance degradation of sparse networks trained with the permuted masks decreases with the width of the networks. Finally, it is also shown that the diversity of the solutions discovered is greater than the winning tickets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback and helpful insights. We have added new experiments based on your insights/questions and added our response below: \n\n> a natural question that arises is whether the permutation function can be obtained early in training eliminating the requirement for fully training the new model?\n\nIn our work, we used 2 trained dense models to find the permutation mapping, but we can also find the perm mapping early in the training, as noted by [Sharma, 2024]. We have added an additional experiment on early matching with the CIFAR-10 dataset, which shows that models can be matched earlier in the training, thus reducing the computational cost of our method. \n(https://imgur.com/a/BgiE4W3)\n\n>  For instance can those masks be reused across datasets ?\n\nWe conducted an additional experiment where we obtained a mask for the CIFAR-10 dataset and reused the mask with a new init on SVHN datasets. Permuted mask outperforms unpermuted mask (naive) at all levels of sparsities. Thank you for this interesting direction, we will add more experiments in the final version of the paper. (https://imgur.com/a/UDiMQHs)\n\n> Can the randomly initialized neural network be permuted\n\nYes, indeed, we can also apply the permutation, $\\pi$, to the random initialization instead of the mask; the resulting network remains functionally equivalent. In our work, we chose to permute the mask as we did not want to modify the new random init.\n\n\n> Can the authors clarify what they mean by rewind points? Is it simply the training iteration at which the masks were applied to various dense networks ?\n\nThat's correct! Rewind points is the training epoch at which sparse mask is applied to the dense model. We will make it more clear in the paper. \n\n> Is it also the point at which the weights were rewinded during IMP while obtaining the original mask ($\\textbf{m}_A$) resulting in different masks for different rewind points?\n\nWe used  IMP-FT, which does not use weight rewind to obtain sparse masks. We preferred IMP-FT over IMP with weight rewinding because IMP-FT is computationally less expensive, which allowed us to conduct extensive experiments. In our experiments, we obtain a mask from IMP-FT and apply the mask to the dense model at different rewind points. \n\n> in Figure 4, is the IMP used along with weight rewinding or is IMP-FT used?\n\nSince Paul et al. used IMP (with weight rewinding) for their analysis [1], which suggested only successive IMP iterations are linearly mode-connected, we used IMP (with weight rewinding) in Fig 4. to show that all sparse networks found in each iteration of IMP and dense networks are linearly mode-connected once the variance collapse is taken into account. We decided to use IMP with weight rewinding in Fig 4. for a fair and direct comparison to observations made in [1]. \n\n> If IMP-FT was used then does the linear mode connectivity exist between the dense network and the sequential winning tickets of lower density?\n\nWe conducted an additional experiment to confirm that linear mode connectivity exists between the dense network and sparse models at each iteration of IMP-FT. We appreciate your attention to detail; this additional analysis provides valuable insights that strengthen our paper's findings. (https://imgur.com/a/2l7khkN)\n\n> The results show a clear trend as the width of the networks increases. The authors show that this is due to better LMC. Can this be simply because the permutation matching is better because each unit has a larger pool of units to match from ? \n\nYes, that’s correct. We have the same intuition as yours that wider models (with $\\ell$ layers, each of width $n$) have more possible permutations (up to $(n!)^{\\ell}$), which makes matching two models more accurate. We show this by increasing the model width and comparing the LMC. As observed in Fig.3, on expanding the model width, LMC becomes better, suggesting permutation matching is better for wider models. We will add this explanation/intuition to the final manuscript. \n\n> Additionally, the variance in the unit activations can be studied, larger width may result in lower variance and better match.\n\nThis is an interesting insight that could explain why activation (permutation) matching works better for wider models. We analyzed the variance across all layers and observed that the variance for each layer significantly decreases with increasing the width. For example, variance for the *second conv layer in third block* decreases from *0.012* to *0.0001* on increasing width from 1 to 4. Other layers follow the same trend. We will add plots for all the layers in the manuscript. \n\nWe hope that we have answered all your questions/concerns; we would really appreciate it if you could update your final score!\n\n[1]  Paul et al., Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask", "claims_and_evidence": "The claims regarding misalignment between weight initialization-based optimization basins and winning ticket masks are well-supported by empirical evidence. The paper provides novel insights into the relationship between winning tickets and their original dense networks.", "methods_and_evaluation_criteria": "The presented methods and evaluation criteria are well-suited for testing the hypothesis.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments are very well set-up with clear conclusions. However, there is a lack of clarity regarding the iterative magnitude pruning process used to identify the winning ticket mask. In the appendix the authors specify the use of IMP-FT which does not include a weight rewinding step.", "relation_to_broader_scientific_literature": "The key contribution of the work is showing that winning ticket masks can be used for networks with different weight initialization provided a suitable permutation function is available. The results provide insights into winning ticket sparse masks and the correlation between the specific weights at initialization. This has been previously used as a test for other pruning at initialization methods [Lee2019, Wang2020, Tanaka2020], without a degradation in performance.\n\nThe idea that the entire SGD trajectory and the sparse networks obtained can be aligned for two networks via the same permutation has been explored previously [Sharma2024]. The authors build upon those findings and show that winning ticket masks can be reused for different weight initializations.", "essential_references_not_discussed": "The paper is quite extensive in its discussions of relevant literature and it is one of the strengths of the paper.", "other_strengths_and_weaknesses": "Strengths: \n\n* The paper is very well written, the arguments made and evidence provided is very convincing. Please see the response to previous sections\n\nWeaknesses:\n\n* The contributions, while valuable, are primarily extensions of closely related previous work.\n\n* Permuted masks are obtained through a function that is optimized between two trained dense networks and is computationally expensive. The authors acknowledge this, however, a natural question that arises is whether the permutation function can be obtained early in training eliminating the requirement for fully training the new model ?\n\n* To what degree the permuted sparse masks can be reused is unclear. For instance can those masks be reused across datasets ?", "other_comments_or_suggestions": "Please see the Questions section below and Weaknesses section above.", "questions_for_authors": "* Can the randomly initialized neural network be permuted instead of the mask to obtain the same results, given that a permutation is found that transforms model B to match model A ?\n* Can the authors clarify what they mean by rewind points? Is it simply the training iteration at which the masks were applied to various dense networks ? Is it also the point at which the weights were rewinded during IMP while obtaining the original mask (m_A) resulting in different masks for different rewind points ?\n* In Figure 4, is the IMP used along with weight rewinding or is IMP-FT used ? If IMP-FT was used then does the linear mode connectivity (accounting for variance collapse) exist between the dense network and the sequential winning tickets of lower density?\n* The results show a clear trend as the width of the networks increases. The authors show that this is due to better LMC. Can this be simply because the permutation matching is better because each unit has a larger pool of units to match from ? Additionally, the variance in the unit activations can be studied, larger width may result in lower variance and better match.\n\n## Update after rebuttal\n\nI thank the authors for their response and for generating the requested results. I believe the paper makes several significant contributions, I have improved the rating. However, I request the authors to make the requested changes in the next version of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SPl2xf5anF", "reviewer": "Reviewer_nWHn", "summary": "This paper hypothesizes that Iterative Magnitude Pruning (IMP) fails to generalize its sparse mask to other random initialization because the basin in which other random initialization resides does not match the basin constructed by the IMP sparse mask.\nTo address this, the authors propose to permute the IMP mask to align with the basin of other random initialization.\nThe authors evaluate the proposed method on CIFAR10/100 and ImageNet datasets with VGGNet and ResNet.\n\n## Update after rebuttal\n\nThe authors' response did not effectively address the raised concerns, particularly about the gap between the authors' claim and the proposed method. Thus, the reviewer still hesitates to accept this work because the current version is confusing and unconvincing. However, the reviewer has decided to raise the original rating to 'weak reject', expecting the authors to accept the suggestions detailed in the comments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The authors claim that the IMP sparse mask may not match the basin in which other random init resides. evidence for this claim is just analysis in Figure 1 with assumption of a single layer with two parameters case.\n\nWe validate our hypothesis through comprehensive experiments conducted across multiple datasets and model architectures. Our findings are substantiated by similar observation in [3], which demonstrated that IMP sparse masks only work when dense and sparse models are within the same loss basin and linearly mode-connected.\n\nOur hypothesis is intuitive: by matching loss basins of two models with different initializations, we can permute and reuse the IMP mask from a rewind point. \nOur experimental results provide empirical evidence supporting this claim, showing consistent improvement across different model arch and datasets.\n\n> Thus, the assumption that an initialized model is supposed to be in a specific basin may not be true.\n\nWe agree that initialization alone does not determine the basin. However, **we do not claim this anywhere in the manuscript**. Models trained from the same initialization can end up in different basins [1]. That’s why using a rewind point for Lottery Tickets is necessary, and we use a rewind point with our method as well. We will add a note on this in final version of manuscript to make it clearer. \n\nOur method builds upon the fact that models trained using LTH mask (with rewind points) always land in the same basin, as the authors observed in [2]. Our key claim is that the winning mask can be used with a rewind point obtained from different init, provided we account for weight symmetry to align the basins. We empirically demonstrate this through extensive experiments across multiple datasets and model architectures.\n\n> If the proposed sparsification method is effective with larger models, it would be meaningless.\n\nWe *respectfully* disagree with this statement. Even with width=1,  we can observe statistically significant improvement for different datasets as shown in Tables 5, 9, 10, 11.\n \n* **CIFAR-10** (at 97% sparsity, width=1) - Improvement of **1%**\n* **CIFAR-100** (at 97% sparsity, width=1) - Improvement of **3.5%**\n* **Imagenet** (at 95% sparsity, width=1) - Improvement of **2%** (top-1)\n\nThese improvements are statistically significant and demonstrate the efficacy of our method. In the manuscripts, we added experiments with varying widths to get more insights about the accuracy of permutation matching and show that once permutation matching gets better on increasing the width, the gap between LTH and permuted mask (our method) reduces. This experiment provides more insight into the role of permutation matching for our proposed method. You can find more details in **Sec 4.3**.\n\nWe would also like to highlight **our work aims at better understanding of lottery tickets and winning masks, not just improving the accuracy**.  As noted by reviewer **oZBY**,  our work \"provides novel insights into the relationship between winning tickets and their original dense networks.” We believe our findings will be useful for the sparse training research community.  \n\n> In the main manuscript, the authors use 'LTH mask' repeatedly. However, it is difficult to find the details of which methods the authors use in the main manuscript and I managed to find the details in the Appendix. Similarly, in experimental sections, It is difficult to find what 'Naive' refers to. Overall, a clearer and more reader-friendly presentation seems to be required.\n\nWe appreciate the suggestion; we will add a separate paragraph to define naive, LTH and permuted masks for an easier understanding.\n\nHowever, we would like to point out that we have defined the *LTH*, *naive*, and *permuted mask* at multiple places in the manuscript (**first one at L94-96**; see more below). Moreover, **Figure 2** in the manuscript explains the differences between LTH, naive and permuted baselines.  \n\n* **Line 94-96**: \"Permuting the LTH sparse mask to align with\nthe new random initialization improves the performance\nof the trained model (**permuted**), compared to the model\ntrained without permuting the sparse mask (**naive**).\"\n\n* **Line 190-192**: \"We denote training with the permuted mask, $\\pi(\\textbf{m}_A)$\nas **permuted** and with the non-permuted mask, $\\textbf{m}_A$ as **naive**\"\n\n* **Line 212-214**: \"To evaluate the transferability of the permuted LTH mask we train, a different random initialization $\\textbf{w}_B^{t=0}$, the LTH sparse mask $\\textbf{m}_A$ and permuted LTH mask $\\pi(\\textbf{m}_A)$, which we denote the **naive** and **permuted** solution\nrespectively.\"\n\nIf you're satisfied with our reply, we'd appreciate if you can update your score. \n\n[1]. Frankle et al., Linear mode connectivity and the lottery ticket hypothesis\n\n[2] Evci et al, Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win\n\n[3]. Paul et al., Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask", "claims_and_evidence": "The authors claim that the IMP sparse mask may not match the basin in which other random initialization resides.\nThe evidence for this claim is just the analysis in Figure 1 with the assumption of a single layer with two parameters case.\n\nHowever, previous work [1] discovered that an initialized model often exhibits insufficient stability to SGD noise, meaning that it is not in a basin of attraction.\nThus, the assumption that an initialized model is supposed to be in a specific basin may not be true.\nAlso, it is difficult to agree that a sparse mask determines a specific basin of attraction according to the findings in [1].\n\nTherefore, the reviewer is not convinced by the claims of this paper.\n\n[1] Frankle et al, \"Linear mode connectivity and the lottery ticket hypothesis\", ICML2020.", "methods_and_evaluation_criteria": "The authors propose permutation matching to match the basin of an initialized model and a sparse model.\nHowever, this method is just adopted from previous works without any significant modification.\nThus, this paper seems to lack technical novelty.\n\nFor evaluation, the authors use CIFAR10/100 and ImageNet datasets, which are commonly used benchmark datasets.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The authors show the experimental results with varying widths of networks, showing that the proposed method is more effective with large widths.\nThis claim seems to be contrary to the goal of network sparsification.\nIf the proposed sparsification method is effective with larger models, it would be meaningless.", "relation_to_broader_scientific_literature": "The contribution of this paper seems to be limited.\nThe authors' assumption is not convincing and the experimental results are also not significant.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "All are mentioned in other sections.", "other_comments_or_suggestions": "In the main manuscript, the authors use 'LTH mask' repeatedly.\nHowever, it is difficult to find the details of which methods the authors use in the main manuscript and I managed to find the details in the Appendix.\nSimilarly, in experimental sections, It is difficult to find what 'Naive' refers to.\nOverall, a clearer and more reader-friendly presentation seems to be required.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SieuJCJZNB", "reviewer": "Reviewer_yfeq", "summary": "The authors investigate the problem of LTH masks not being compatible with random initializations. They find that by aligning the loss basins via a matching permutation, an LTH mask can be used with a random initialization, not associated with the mask.\nThis work shows that LTH masks can be reused with random initializations via permutation matching (to some extent).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the detailed feedback provided; we provide more details below: \n\n1. \n\nIn our work, we used 2 trained dense models to find permutation (perm) mapping, as primary aim of the paper was to understand why winning tickets don't work with different random init. \nOne can also find perm mapping early in training, as noted by [Sharma, 2024]. We have added an additional experiment on early matching with CIFAR-10, which shows that models can be matched earlier in training, thus reducing computational cost of our method. (https://imgur.com/a/BgiE4W3)\n\nWe don't think sparse models can be used for finding perm mapping as the mask projects the model into a different sub-space, thus cannot be matched with a perm. \n\n2. \n\nYour observation is indeed correct! As noted in manuscript (L188), the perm matching algo uses a greedy approach and thus finds an approximate solution, i.e., the perm matching is not good enough (as you noted). As discussed in Sec 4.3, perm matching works better for wider models [2]. \nThis can be observed in the LMC plot (Fig 3.), where the loss barrier decreases when the model width is increased, showing that the perm matching finds a better solution as we increase the width.\nOur experiments in Sec 4.3 show that as we increase the model width, gap between LTH and permuted mask decreases, which suggests that the permuted solution will closely match LTH performance if given an accurate permutation mapping.\n\n3. \n\n>What is the difference between IMP and LTH in Table 1? Is IMP trained over 5 different seeds and LTH uses the same mask and init over 5 different run\n\nYes, IMP is trained independently over 5 different seeds with iterative pruning to obtain 5 different sparse/pruned solutions with different sparse masks/topologies ($M_0$, $M_1$, $M_2$, $M_3$, $M_4$, $M_5$). \nLTH ensemble is trained using the same mask ($M_0$) and init ($w_0$) over 5 different runs (with different data order). Random init $w_0$ defines the winning ticket for mask $M_0$. \nThe permuted ensemble is trained using 5 different permutations ($\\pi_1$, $\\pi_2$, $\\pi_3$, $\\pi_4$, $\\pi_5$) of the same mask ($M_0$) with five different random inits ($w_1$, $w_2$, $w_3$, $w_4$, $w_5$). \n\n>  The permutations however, have different random initializations. In spite of this, the permuted solutions are similar or worse than IMP.\n\nThis is an interesting question! Our intuition is that in the case of IMP,  models are trained with different random initializations, and they discover different sparse topologies, which helps in learning more diverse solutions but is computationally expensive and not practical to create ensembles. The IMP baselines serve as an upper bound for diversity metrics as both training from different random initializations and learning different topologies introduce high randomness/stochasticity to learn very diverse solutions.\n\nLTH, as noted in prior work, does not learn diverse solutions as they are trained using the same mask and init but over different runs [1]. Thus, LTH is not suitable for generating ensembles.\n\nIn contrast, our proposed method allows us to reuse the LTH mask with different random inits, introducing more source of randomness than the LTH baseline and thus improving the diversity of the solution. However, since we reuse the mask to train permuted ensembles, the diversity will be less than IMP (which uses both different init and sparse topologies) but good enough for making ensembles. This can be observed in Table 1., where the LTH ensemble does not improve the accuracy compared to a single model, while the permuted ensemble significantly improves the performance compared to a single model. It is worth noting that for CIFAR-100 datasets, the permuted ensemble (77.85%) surpasses the LTH ensemble (75.99%), demonstrating that the permuting mask can help train more diverse solutions with less computational cost incurred as compared to IMP.\n\nWe will add more explanations and define LTH, IMP, and Permuted ensembles for an easier understanding.\n\n4. \n> \"Low width networks observe a smaller improvement with permutation\" \n\nThe perm matching algorithm doesn't work well with lower widths, which can be observed in the loss-barrier plots (high barrier --> poor perm matching). We still observe smaller but statistically significant improvements with the permuted mask, as discussed below. As we increase the width, perm matching becomes better, and the loss barrier decreases (fig 3.); our proposed method with permuted mask becomes closer to LTH.\n\nEven at width=1, we can see significant improvements at 97% sparsity:\n\n**CIFAR-10**: +1%\n\n**CIFAR-100**: +3.5%\n\n**Imagenet** (at 95% sparsity, width=1):  +2% (top-1)\n\nWe have added more experiments in our reply to reviewer **oZBY**, which you may also find interesting.\nIf you're satisfied with our explanation, we'd greatly appreciate you updating your score.\n\n[1] Evci et al, Gradient Flow in Sparse Neural Networks\n\n[2]. Ainsworth et al., Git Re-Basin", "claims_and_evidence": "The claims made in the paper are supported empirically via experiments on CIFAR10, CIFAR100 and also the ImageNet datasets.", "methods_and_evaluation_criteria": "Yes, the evaluation methods (generalization performance) and criteria (image classification datasets) are appropriate for this problem setting.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The conclusions drawn from the ensemble experiments are a bit unclear to me (see questions below).", "relation_to_broader_scientific_literature": "This paper addresses the problem of making LTH masks useful for other random initializations, which is a step in the direction to understand how sparse networks can be trained from scratch. This is an active area of research.", "essential_references_not_discussed": "The authors discuss the relevant literature.", "other_strengths_and_weaknesses": "Strengths\n1. Simply permuting the mask can help match the LTH mask to any random initialization.\n2. This improves the performance of the random initialization with the random mask.\n\nWeaknesses (and questions)\n1. It seems that the permutation can only be identified by training two dense networks, is this necessary or can this be done on a sparse network as well?\n2. The hypothesis that the difference between a random initialization and the LTH is due to the misalignment of the loss basin seems limited. Because via permutation matching, the random init with the LTH mask is only slightly better than the naive baseline. Significant improvements are only seen on wider networks, even in the case of CIFAR10 and CIFAR100. This suggests that there might be more than a misalignment of basins or the permutation matching is not good enough, maybe the authors could investigate this?\n3. What is the difference between IMP and LTH in Table 1, is IMP trained over 5 different seeds and LTH uses the same mask and init over 5 different runs? This will naturally have a smaller ensemble effect since the only randomness is the stochastic noise compared to different initializations in the case of IMP. The permutations however, have different random initializations, which would help ensembling. In spite of this, the permuted solutions are similar or worse than IMP. Can the authors explain the experimental setup in more detail to highlight the differences between the permuted ensemble and the LTH ensemble. Otherwise, the functional diversity of permutations is unclear to me.\n4. Low width networks observe a smaller improvement with permutation, is there a reason for this? Is there a tradeoff between the alignment of the mask and initialization (via permutation) and the amount of overparameterization (width).", "other_comments_or_suggestions": "I appreciate the authors’ insight on using permutations to make LTH masks flexible and reusable with random initializations; the improvements in performance seem limited to me. I believe the paper will benefit from an explanation of why the performance gains are limited (is it due to the limitation of permutation matching or due to the fact that the hypothesis of misaligned basins is not enough). Hence I lean towards a reject at the moment, but I am happy to increase my score after a discussion with the authors during the rebuttal.", "questions_for_authors": "See weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethical concerns."}}]

Author rebuttal:

Meta-review: The paper proposes a method to enhance the generalization of Lottery Ticket Hypothesis (LTH) sparse masks by aligning them via permutation to the optimization basin of a new random initialization. While one reviewer raised concerns about clarity, technical novelty, and the strength of claims—particularly regarding initialization versus rewind points—these were addressed effectively in the rebuttal through additional experiments and clarifications. Two other reviewers strongly supported the acceptance of the paper, highlighting its novel insights, empirical robustness across multiple datasets (CIFAR-10/100, ImageNet), and relevance to the sparse training community. 

The authors convincingly demonstrated that permuted masks significantly improve performance compared to naïve application. Although computational efficiency and generalization to arbitrary random initializations remain areas for future improvement, the consensus is that the paper makes valuable contributions and addresses important questions in sparse neural network training. Overall, AC recommends acceptance.
Decision: Accept (poster)