Find A Winning Sign: Sign Is All We Need to Win the Lottery
Authors: Junghun Oh
Sungyong Baik
Kyoung Mu Lee
Venue: ICLR 2025 Poster (2025)
Keywords: lottery ticket hypothesis
network pruning
linear mode connectivity
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We demonstrate that an effective signed mask can allow any randomly initialized network to win the lottery.

Abstract: The Lottery Ticket Hypothesis (LTH) posits the existence of a sparse subnetwork (a.k.a. winning ticket) that can generalize comparably to its over-parameterized counterpart when trained from scratch.
The common approach to finding a winning ticket is to preserve the original strong generalization through Iterative Pruning (IP) and transfer information useful for achieving the learned generalization by applying the resulting sparse mask to an untrained network.
However, existing IP methods still struggle to generalize their observations beyond ad-hoc initialization and small-scale architectures or datasets, or they bypass these challenges by applying their mask to trained weights instead of initialized ones.
In this paper, we demonstrate that the parameter sign configuration plays a crucial role in conveying useful information for generalization to any randomly initialized network.
Through linear mode connectivity analysis, we observe that a sparse network trained by an existing IP method can retain its basin of attraction if its parameter signs and normalization layer parameters are preserved.
To take a step closer to finding a winning ticket, we alleviate the reliance on normalization layer parameters by preventing high error barriers along the linear path between the sparse network trained by our method and its counterpart with initialized normalization layer parameters.
Interestingly, across various architectures and datasets, we observe that any randomly initialized network can be optimized to exhibit low error barriers along the linear path to the sparse network trained by our method by inheriting its sparsity and parameter sign information, potentially achieving performance comparable to the original.
The code is available at https://github.com/JungHunOh/AWS_ICLR2025.git.

Published as a conference paper at ICLR 2025
FIND A WINNING SIGN: SIGN IS ALL WE NEED TO WIN THE LOTTERY
Junghun C Sungyong Baik3.4, and Kyoung Mu Lee
l Dept. of ESE& IIAII Seoul National University
3 Dept. of Artificial Intelligence, 'Dept. of Data Science, Hanyang University [dh6dh, kyoungmu @snu.ar ac kr, dsybaikehanyang .c.kr
ABSTRACT
The Lottery Ticket Hypothesis (LTH) posits the existence of sparse subnetwork (ak.a. winning ticket) that can generalize comparably to its wver-parameterized counterpart when trained from scratch. The common approach to finding a winning ticket is to preserve the original strong generalization through Iterative Pruning (IP) and transfer information useful for achieving the learned generalization by applying the resulting sparse mask to an untrained network. However, existing IP methods still struggle generalize their observations beyond ad-hoc initialization and small- scale architectures or datasets, or they bypass these challenges by applying their mask to trained weights instead of initialized ones. In this paper, we demonstrate that the parameter sign configuration plays a crucial role in conveying useful information for generalization to any randomly initialized network. Through linear mode connectivity analysis, we observe that a sparse network trained by an existing method can retain its basin of attraction its parameter signs and normalization layer parameters are preserved. To take a step closer to finding a winning ticket, we alleviate the reliance on normalization layer parameters by preventing high error barriers along the linear path between the sparse network trained by our method and its counterpart with initialized normalization layer parameters. Interestingly, across various architectures and datasets, we observe that any randomly initialized network can be optimized to exhibit low error barriers along the linear path to the sparse network trained by our method by inheriting sparsity and parameter sign information, potentially achieving performance comparable to the original. The code is available at https://gthub..com///uuuuuuoon/ JCLR2025.g0
INTRODUCTION
In the field of deep learning, over parameterization is viewed as a key to enhancing network capacity and improving generalization (Neyshabur et al., (elkkn et al., 2019). It well known that after training
parammteameee dense network, many redundant parameters arise that can be removed without affecting performance, leading to the emergence of network pruning techniques (Liu et al.. 2017: Lin et al.; 2020). However, pruning an initialized dense network before training often leads to a sparse network that is difficult to optimize and fails to match the original generalization (Li ct al. 2017: Evci et al. 2022). This phenomenon led to question, first posed by Frankle & Carbin (2019) as the lottery ticket hypothesis (LTH): Is there a sparse subnetwork (i.e., a winning lottery ticket) that achieve generalization comparable its dense counterpart when trained from initialization? This challenging research question has garnered significant attention and inspired many follow-up studies.
Iterative magnitude praning (Frankle & Carbin, 2019) (IMP) is of the representative methods to identify a winning ticket through iterating three phases: training, pruning, and rewinding. Many researchers have sought to understand how IMP finds a winning ticket. Among several insightful findings, perspectives from the loss landscape have provided valuable insights. Frankle et al. (2020);; Evci al. (2022): ; Paul al. (2023) have shown that IMP can find a winning ticket only when the network obtained after the training phase maintains its basin of attraction after the pruning and
define basin of attraction as set of points connected by low-loss paths, where gradient descent from any point converges the same minimum.
FFollowing previous works (Evci et al. 2022 Paul et al. 2023

---

Published as a conference paper at ICLR 2025
Subnetwork
AWS Suanetwork
Preserved Signs & Randomly Initialized
LMC Region
LRR Solution
AWS Solution
Preserved Signs & Randomly Initialized + and y
Optimization Direction
Engo Berter
Test
High Eroo Barter
Error
Dense network error
AWS (ours)
Figure 1: Illustration of our motivation and method. ?/b and & denote network parameters of normalization layers and parameters excluding those of normalization layers, respectively. The 'LMC region' refers to region of solutions that are linearly mode-connected to the LRR or AWS solution.
rewinding phases, thereby preserving strong generalization potential (i.e., generalization ability after training) throughout subsequent iterations. Since this condition is difficult to satisfy in relatively large-scale settings, variants of IMP bypass the challenges by either rewinding to warm-up trained parameters (weight rewinding) (Frankle et al. 2020a) or skipping the parameter rewinding phase (learning rate rewinding) (Renda et al.] 2020). : Although they found subnetwork that performs comparably to a dense network after training, it is not at initialization, and thus not a winning ticket.
In this paper, we empirically show that effective signed mask, a sparse mask with parameter sign information, is a key to satisfying the challenging condition for finding a winning ticket. Specifically, we leverage learning rate rewinding (LRR) for its ability to find effective parameter sign and sparsity configuration (Gadhikar& & Burkhhll 2024). Then, with a slight modification to LRR, we demonstrate that if parameter signs are preserved, the subnetwork obtained through our LRR variant remains within its basin of attraction even after randomly initializing its parameters. This implies that the generalization potential of the subnetwork can be transferred to any randomly initialized network via the signed mask, possibly allowing it to generalize comparably to the dense network after training.
We observe that the original LRR fails to achieve this. As illustrated on the left side of Figure [ we observe that the LRR subnetwork leaves its basin after randomly initializing its parameters while preserving their signs, indicated by the red ball and the high error barrier between the yellow and red balls, similar to the findings in Frankle et al. (2220b). This failure stems from the significant influence of initializing normalization layer parameters. Interestingly, when we exclude normalization parameters from initialization and only randomize the other parameters while preserving their signs, the resulting network retains the original basin. This, in turn, leads to convergence to a solution with low error barrier along the linear path to the LRR solution (i.e., the solution obtained by training the LRR subnetwork), as illustrated by the green ball moving towards the blue ball on the orange line. These results suggest that the signed mask and normalization parameters of the LRR subnetwork enable any randomly initialized network to inherit its generalization potential.
To take a step closer to finding a winning ticket, we eliminate the need for trained normalization parameters by addressing the adverse effects of initializing them. To this end, we propose AWS, a variation of LRR to find A Winning Sign, which prevents high error barriers along the linear path between the AWS subnetwork and its counterpart with initialized normalization parameters. During training, AWS randomly interpolates between the current and initialized normalization parameters linearly, using the interpolated values for each forward pass. As illustrated on the right side of Figure we argue that the AWS subnetwork stays within its basin even after randomizing parameters while preserving their signs, indicated by the low error barrier between the yellow and red balls. This helps the resulting network converge to a solution linearly mode-connected to the AWS solution, with performance close the dense network (gray dotted line). Experiments across architectures and datasets show that the signed mask from the AWS subnetwork enables any randomly initialized network to generalize comparably the dense counterpart after training.
We summarize the contributions of our work as follows:
We observe that any randomly initialized network can inherit the generalization potential of the LRR subnetwork through its signed mask and the normalization layer parameters.
We propose AWS that alleviates the dependence on trained normalization parameters by pre- venting high error barriers between the AWS subnetwork and its counterpart with initialized normalization parameters.
In contrast to existing methods that are limited to finding a winning ticket with an ad-hoc initialization, we show that any randomly initialized network can generalize comparably to a dense network after training by applying the AWS-driven signed mask to it.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Lottery Ticket Hypothesis (LTH),, Frankle Frankle & Carbin (2019) propose LTH which states that within a dense network, there exists sparse subnetwork that, when trained from initialization, can achieve performance comparable to the dense counterpart. To find such a winning lottery ticket, the authors proposed iterative magnitude pruning (IMP) and demonstrated that IMP successfully finds winning ticket in a relatively small-scale setting. Follow-up works have delved into a broad range of topics related to LTH, such as theoretical support for the existence of the winning ticket (Malach et al.
2020;
Orseau et al.) 2020] Burkholz 2022 da Cunha et al., 2022). efficient alternatives to IMP ( You
et al.
2020), searching for a winning ticket without weight training (Chen et al. 2022 Sreenivasan
et al.
2022 Koster et al. [022). ). and empirical analyses on winning ticket (Zhou et al F Frankle
ct al.
2020a Ma et al. 2021 Sakamoto & Sato
2022
Evei al. 2022
Paul et al. 2023).
Insights into a Winning Ticket. One of the most important topics is investigating what makes
sparse network win the lottery. Frankle et al. (2020a) introduced the notion of mode connectiv-
ity (Freeman & Bruna 2017 Nguyen 2019: Draxler et al.) 2018: Garipov et al., 2018 Lubana ct al. (023)) into LTH to investigate the conditions under which IMP finds winning ticket. They consider a network stable against SGD noise if, under different SGD randomness, it converges to a region of solutions that exhibit low error barriers along the linear path connecting them. Based on this definition, they demonstrated that IMP succeeds only when the rewound network is stable against SGD noise. [Evci et al. (2022) found that a winning ticket can be found only when it resides in the same basin as the pruning solution used to obtain the sparse pruning mask. Paul et al. (2023) demonstrated that a sparse mask obtained in an IMP iteration guides the subsequent pruning solution to be linearly mode-connected to the previous IMP solution, leading the consecutive pruning solutions to be piece-wise linearly mode-connected. In summary, these findings suggest that IMP can identify winning ticket only when the network obtained from the training phase remains within its basin of attraction after the pruning and rewinding phases, thereby preserving the generalization of the original dense network throughout all iterations. Variants of IMP, such as weight rewinding (Frankle) itt al. 2020a) and learning rate rewinding (Renda et al. (020)). bypass this challenging condition by applying the found sparse mask to trained parameters rather than initialized ones, failing to find a winning ticket.
Significance of Parameter Signs in LTH. Recently, several works reported the importance of parameter signs from the perspective of representation capacity (Wang et al. 2023a b). /Zhou et al. (2019) are the first to discover the role of parameter signs the context of LTH. They empirically showed that in the parameter rewinding stage of IMP, rewinding parameter signs has greater impact on the performance than the magnitudes. By contrast, Frankle et al. (20205) showed that transferring the signed mask obtained by IMP to the original initialization performs worse than transferring them along with the respective magnitudes. Gadhikar & Burkholz (2024) demonstrated that IMP can fail to find winning ticket because it loses crucial sign information during parameter rewinding and struggles to learn an effective sign configuration again due to reduced network capacity. They claimed that learning rate rewinding (Renda et al. 2020) (LRR), on the other hand, identifies more performant sparse networks by finding and maintaining the effective sign configuration during training.
We observe that the ineffectiveness of transferring parameter signs to an initialized network, as noted in FraFran et al. (2020b), is due to the adverse effect of initializing the normalization layer parameters. To address this issue, we propose a slight variant of LRR and demonstrate that the challenging conditions for finding a winning ticket suggested by Frankle et al. (2020a); [Evci et al. (2022); Paul et al. (023) can be satisfied using the effective signed mask acquired by our LRR variant.
3 METHOD
3.1 NOTATIONS AND BACKGROUND
Lottery Ticket Hypothesis. Let 0 E IRd denote the parameters of a neural network. We use to also represent the neural network parameterized by 0. Consider binary mask m E {0, 1] having the same shape as 0. Note that the mask values for parameters not targeted for pruning, such as biases, are fixed to be 1. Then, the mask defines sparse subnetwork of the original dense network as 0 @ m. The lottery ticket hypothesis (LTH) (Frankle & Carbin 2019) posits the existence of a mask with non-trivial sparsity (i.e., E; m & d) that allows Omni ? m, where O.n
3

---

Published as a conference paper at ICLR 2025
represents initialized parameters, to perform comparably to the dense network after training. Such a sparse network at initialization is referred to as a winning ticket. Iterative magnitude pruning way to find the winning ticket through iterative training + pruning + rewi rewinding procedures. Let A(O, u) represent an SGD learning algorithm with initialized learning rate scheduler that updates 0 until convergence using SGI) randomness u ~ U (e.g, randomness from a data loader or data augmentations), We omit u from A(O, u) if unnecessary. At training phase of the t-th iteration, IMP trains the masked initial parameters, obtaining 0.MP @ m
(IMP) (Frankle & Carbin 2019) was suggested
IMP denotes the mask obtained IMP A(0)MP @ m.MP). where m
from the (t- 1)-th iteration. At pruning phase, IMP produces the t-th mask by removing a portion of the non-zero weights (commonly 20%) in 0.MP @ m
IMP with the smallest magnitudes:
m. = prune (0|MP) @ m )). At rewi nd inding ng phase, O.MP is rewound to the initial parameters, OfNir and the entire process is repeated until t = T. Finally, IMP produces O @ m!MP referred to as the IMP subnetwork, and after training the IMP subnetwork, IMP obtains A(O)air @ m!
referred to as the IMP solution.
Variants of IMP. Several variants of IMP have been proposed to address the failure of IMP in generalizing to more challenging settings, especially focusing on the rewinding phase. Based on the analysis of stability against SGD noise, Frankle moisk,Frankke et al. (2020a) proposed weight rewinding (WR) that rewinds 0.MP the warm-up trained parameters rather than to 0 Learning rate rewinding (LRR) (Renda et al., 2020) is another variant of IMP that skips parameter rewinding and instead rewinds only the learning rate schedule. At the training phase of the 1-th iteration, LRR trains OL @ LİR obtaining 0L @ LRR = A( @ m(=1). where LRR represents the mask m
obtained from the (t - 1)-th iteration. After obtaining m, LRR = prune @ m. ) at the pruning phase, LRR skips the rewinding phase and continues the subsequent iterations until t t= T. Finally, LRR produces 0LRR @ mm..R LRR refeereed to as the LRR subnetwork, and after training the LRR subnetwork, LRR obtains A(O) & m!RR), ), referred to as the LRR solution.
Linear Mode Connectivity and stability against SGD Noise. Linear mode connectivity and stability against SGD noise have been adopted as useful tools for analyzing winning tickets from the loss landscape perspective (Frankle et al. 2020a; Exci et al. 2022 Paul et al. (023). Let {()) denote the test error of O. We define the error barrier between two parameters, O and 0' when interpolating them by factor of a as the difference between the error of the interpolated network and the mean error:
(a = Ela0 + a1-a)0') - (E()) + E/22))
(1)
Then, we define E(0, 0*) = sup,, (0, 0*) If If 6') is smaller than a sufficiently small value e, we consider O and 0' to be linearly mode-connected (LMC). e is often determined empirically such as the standard deviation of errors of dense network across different training seeds (Paul et al., (023)) Based on the definition of LMC, we define the stability against SGD noise as the condition where pair of networks are LMC when trained from the same initial parameters but with different SGD randomness (Frankle et al. 2020a). Formally, 0 is considered stable against SGD noise if A(0, u) and A(O, ut) are LMC with u. u² U.
As demonstrated by previous works (Frankle et al. 2020a Evei et al. 2022 Paul et al. 2023 J IMP can identify a winning ticket only when at the t-th iteration, 0 C m
IMP still resides in the basin of attraction of the (t- 1)-th solution after the training phase, PIMP @ mIMPP even after updating mMM to M.M and rewinding D.MP to 0MMP, Paul et al. (2023) claim that if this condition is satisfied at every iteration, the consecutive solutions after the training phase are piece-wise LMC, allowing the final IMP solution generalize comparably to the dense network. To examine this condition, previous works investigate whether a rewound network is stable against SGD noise and converges to solution with linear mode connectivity to the network before the rewinding and pruning phases. WR and LRR bypass the challenging condition by rewinding to warm-up trained parameters or skipping the rewi sewinding phase. In contrast, this paper argues that any randomly initialized network can inherit strong generalization potential through a mask with parameter sign information, progressing toward the goal of LTH.
3.2 MOTIVATION
Gadhikar & Burkhhlzz (024) demonstrated that learning rate rewinding (LRR) successfully main- tains the performance of dense networks by learning and maintaining an effective parameter sign
4

---

Published as a conference paper at ICLR 2025
A0 my
A(sbel, fhan. o
A(shol(dan shaal) @ e
Aljdhan Parl ml A(Oua))
ResNet50
MobileNetV2
ShufflaNet
EffieenNet
70
Test Accuracy (N)
12
70-
22
61
611
66
0.02
0.06
0.18
0.53
0.5
0.25
0.13
0:33
0.05
0.13
0.35
0.8
0.02
0.06
2.13
0.33
0.5
Remaining Parameter Ratio
Remniining Parameter Ratio
Remaining Parameter Ratio
Remaining Parameter Ratio
(a) Test accuracy
Al(a)a (QR O my
Alahs[[ (
) @ #
A(aba( (oun, thaa)) @ a
ResfNe:50
MobileNetV2
ShuffleNet
EthcientNet
01)
33
30
27
20
10
Interpolation
Interpolation
Interpolation Ratio
Interpolation
(b) stability against SGD-noise
Alie] oi maanl
Alabs((Pann () a
A(abel thual) an
ResNet50
MobilsMerV2
ShufllaNeet
EflicientNee
70
(0-
50
00
50
301
10
01
0.0
0.2
056
0.8
3.0
0.0
0.2
04
0.5
0.8
1.0
0.0
0.2
0,4
266
0.8
0.0
0.2
04
08
Interpolation Ratio
Interpolation Ratio
Interpolation Ratio
Interpolation Ratio
(c) Linear mode connectivity with A((ØtR BDLKK
LRR
x.)
Figure 2: Motivational experiments on CIFAR-100. We investigate the effect of parameter initialization in the LRR subnetwork while preserving their signs with respect to (a) test accuracy, (b) SGD-noise stability, and (c) linear mode connectivity with the LRR subnetwork. In (b) and (c), we use a pruned network with a remaining parameter ratio of approximately 0.09. We show mean (each point) and standard deviation (shaded area) across 3 trials.
configuration while pruning unimportant parameters, which iterative magnitude pruning (IMP) and weight rewinding (WR) fail. Several other works (Zhou et al.) 2019; Frankle al.) 2020b Chen ct al. 2022 Sreenivasan et al. 2022 Koster et al. 2022 Wang et al. 2023b a) also found the impor- tance of parameter signs in the context of LTH representation learning- In this work, motivated by these studies, we hypothesize that the parameter sign information obtained through LRR can transfer the generalization potential of the LRR subnetwork to any randomly initialized network. Let (-) denote a function that outputs the sign of each input element if it is non-zero, and otherwise. Then, siRR sign, (OLRR LRR) represents the signed mask of the LRR subnet- ( work. More specifically, our hypothesis states that applying IRRRR to a randomly initialized network, Ouit will enable the resulting network match the performance of the LRR solution after training: A(abs(Om.. @ sLRR) ~ A(0)RR @ mfRR) where abs(.) denotes the modulus function.
similar idea was studied by Frankle et al. (2020b). They showed that replacing the signs of initialized parameters with those of the IMP subnetwork does not improve performance compared to using magnitude information in conjunction. We point out that this failure attributed to ignoring the impact of parameters that may rely more on magnitudes than their signs. In the case of weight parameters, such as the weights a convolutional layer, the sign configuration has critical role in determining the functional mechanism of the layer discussed in Wang et al. (2023b a): Gadhikar & Burkholz (2024). By contrast, for parameters in a normalization layer, such as the batch normalization (loffe & Szegedy (015)) layer normalization (Ba al. 2016), the magnitude may be much more important

---

Published as a conference paper at ICLR 2025
Algorithm 1 AWS: slight modification to LRR to find winning sign. The modification is highlighted in red.
Require: Initialize Φu. sbu. and mo (1,..., Rd
1: for t = 1 to do
2: while not converge do
3: (abt-1, shma)a = a -b + (1 a) Viii where a ~ (O, 1)
> Interpolating vb:-1 and shimi
4:
Forward pass using 11 (1ht-1 1, 4hmn )a] mm-1 Forward pass with the interpolated parameters
5:
Update 1 and - via gradient descent
6:
end while
7: Φ. - Φ:-1 and bb + 4/211
8: m. t prune((or, t/) O m)-1)
Update the sparse mask
9: Rewind learning rate scheduler
10: end for
11:
return (ign,((ør, xbr) @ m.)
Obtain the signed mask
than the sign since the scaling parameter, initialized to 1, is nearly always positive after training, and the bias parameter loses the replaced signs since it is initialized to O. We also observe that transferring the signs of parameters of normalization layers is not beneficial as analyzed in Appendix A
Let & and +/ denote the parameters excluding those of normalization layers and those of normalization layers, respectively. Then, we represent a randomly initialized network and the LRR subnetwork as muit = Mmit Mmmil and 0 = [Ø v,.RRR, respectively. To test conjecture, we com- pare two cases: in the LRR subnetwork, randomly initializing the magnitudes of both ΦLRK and dbLRR versus only dLRR while maintaining the signed mask (i.e., A(abs( ((hinn: v/imi)) s(RR) vs. A(abs(( (oiit vol )) @ s).R)). Figure ssows the results on CIFAR- 100 with various architectures. For details on the experiments, please refer to Section 4.1 Figure Za shows that similar to the results in Frankle et al. (2020b), preserving the signed mask while initializing the magnitudes of all parameters randomly (indicated by the orange plots) results in performance similar the case where sign information is not used (indicated by the purple plots), lagging far behind the performance of the LRR solution (indicated by the blue plots). This result indicates that sign information not beneficial when all parameters are randomly initialized. On the other hand, interestingly, when the parameters of normalization layers are kept intact (indicated by the red plots), the performance is comparable to the LRR solution after training, indicating that using the sign information of the LRR subnetwork is beneficial when excluding the influence of initializing the normalization layer parameters. To examine whether the resulting networks reside in the basin of attraction of the LRR subnetwork, analyze their stability against SGD noise and linear mode connectivity with the LRR subnetwork. Figure Figure22]demoooe demonnstass that while randomly initializing the magnitudes of all parameters significantly ruins the stability of the LRR subnetwork (indicated by the orange plots), it is effectively preserved when ignoring the influence of initializing the normalization layer parameters (indicated by the green plots). As shown in Figure 2c we also observe that the preserved stability, in turn, leads the resulting network, abs( ([mmn (b[RR @ ST to converge to a solution with low error barrier along the linear path connecting it to the LRR solution (indicated by the green plots), suggesting they are in the same basin of attraction. This contrasts with the high error barrier observed when initializing all parameter magnitudes (indicated by the orange plots).
The left side of Figure illustrates the observations from our motivational experiments:
the LRR subnetwork, randomly initializing all parameters while preserving their signs causes the resulting network to lose SGD noise stability of the LRR subnetwork, potentially leading to a suboptimal solution, as indicated by the red ball.
On the other hand, when the normalization layer parameters are kept intact, the resulting network, abs([ 26 C sy exhibits significant stability against SGD noise and converges to a solution with low error barrier along the linear path connecting it to the LRR solution, resulting in performance comparable to that dense network, as indicated by the green ball.
Our observations suggest that when the parameter signs are preserved, the LRR subnetwork stays within its basin of attraction even after randomly initializing the parameters, excluding those of normalization layers. In other words, randomly initialized network can inherit the generalization potential of the LRR subnetwork through the signed mask and normalization layer parameters of the LRR subnetwork.

---

Published as a conference paper at ICLR 2025
3.3 AWS: FINDING A WINNING SIGN
Our observations provide valuable insights into the role of the signed mask in transferring strong generalization potential to any randomly initialized network. However, the need for the trained normalization parameters still limits LRR in achieving the goal of LTH. In this subsection, we introduce a method that addresses the adverse impact of initializing the normalization layer parameters to further progress toward the goal of LTH. Our goal is to maintain the basin of attraction in which the LRR subnetwork resides when the normalization layer parameters are initialized. Two networks residing in the same basin may indicate that no high error barrier exists along the linear path connecting them (Evci et al., 2022). Thus, we propose a simple variation of LRR to find a winning sign, referred to as AWS, that prevents any high error barriers along the linear path connecting the LRR subnetwork and its counterpart with initialized normalization parameters. Specifically, AWS randomly and linearly interpolates the parameters of normalization layers with their initialization and uses the interpolated parameters instead of the original parameters during training. Formally, at every network forward pass during the I-th iteration, AWS obtains
(WAS ( a + A,W + (1-a)- Viiitt
where wbAwW denotes the parameters of normalization layers during the 1-th iteration of AWS and a ~ U (0, 1). Then, AWS uses (ys,Ws, ii instead of wtAw for network forwarding, We present the pscudo-code of AWS in Algorithm[] I omitting the superscript 'AWS' for simplicity. After all iterations, we transfer the resulting signed mask, s,WS to a randomly initialized network, obtaining abs (mi)) O AWS and train using normal training set-up until convergence.
The right side of Figure ! Iillustrates the effectiveness of AWS. In contrast to LRR, AWS can allow a randomly initialized network to lie in the basin of attraction of the AWS subnetwork through the learned signed mask, stYS,, possibly leading it to converge to solution that is linearly mode- connected to the AWS solution (indicated by the red and blue ball). Thus, we argue that sAWs can transfer the generalization potential of the AWS subnetwork to any randomly initialized network, possibly resulting in performance comparable to dense network after training.
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Datasets and models. Following the previous works (Ma et al.
2021
Gadhikar & Burkholz 2024 Tiny-ImageNer (Le & Yang
we conduct experiments on CIFAR- 100 (Krizheskyy & Hinton 2015), and ImageNet (Russakovsky et al. 2015). For both CIFAR- 100 and Tiny-ImageNet, we adopt ResNet-50 (He et al., 2016). MobileNetV3 (Sandler et al. 2018). ShuffleNet (Zhang et al., 2018), and EfficientNet (Tan & Le 2019) to validate our method across various architectures, For ImageNet experiments, we use ResNet50, MobileNetV2, and MLP-Mixer (Tolstikhin et al., (021). We adopt MLP-Mixer, which includes layer normalization, to demonstrate the generalization of our method to different types of normalization layers rather than the batch normalization layer.
2009).
Implementation. In both learning rate rewinding (LRR) and the proposed AWS method, we observe that many training epochs and learning rate scheduling are unnecessary during the r i ning phase, as the network converges quickly due to the absence of parameter rewinding, and learning rate scheduling has little impact on the performance. Thus, during the t training phase for both LRR and AWS, we train a network for 10 epochs for CIFAR- 100 and Tiny- Tiny-ImageNet experiments, and 5 epochs for ImageNet experiments without learning rate scheduling. To ensure a network can converge at the early iterations, we perform warm-up training before the first iteration: 10 epochs for CIFAR-100 and Liny-ImageNe and 20 epochs for ImageNet. After the T-th iteration, conduct final training of 100 epochs for CIFAR- 100 and Tiny- Ting-ImageNet, 120 epochs for ResNer-50 and MobileNetV2 on ImageNet, and 200 epochs for MLP-Mixer.
Optimization. For the CIFAR- 100 and Tiny-ImageNe experiments, we use the SGD optimizer with a momentum of 0.9, weight decay of 5e-4, and an initial learning rate of 0.1, except for MobileNetV2. which uses a learning rate of 0.05. The learning rate decays by a factor of 0.1 at the 50th and 75th epochs over 100 epochs. For ImageNet experiments, we use the Adam optimizer with 31 = 0.9, 32 = 0.999, and an initial learning rate of 0.001 for both ResNet-50 and MLP-Mixes and the SGD optimizer with a momentum of 0.9 and an initial learning rate of 0.05 for Mob. The weight decay is set to 5e-5 for MobileNetV2 and MLP-Mixer and le-5 for ResNet-50. We use cosine annealing for learning rate scheduling. The batch size is set to 128 for CIFAR- and Tiny-ImageNet, 256 for ResNet-50 and Mob on ImageNet, and 512 for MixeMixer.
7

---

Published as a conference paper at ICLR 2025
A miy
Alegws mxp
A(aka(O.x) almh
+ A(aln(0.xx) 64
Altaa m
A(Bum)
ResNet50
MobileNerV2
ResNeti0
MobielletV2
Tex Accurayy (N)
Twl Aronay (N)
501
05
0.00
0.08
0.33
0.8
0.56
2.13
0.33
0.02
0.06
0.11
0.8
0.25
0.31
ShuffleMet
EfficientNet
ShuffteNet
Efficientiiet
07.51
Tex Actuscy (5)
-2.0
90.0
47.5
40.0
42-
50-
51-
02:
Tee:
$7.5
40.5
40.0
43-
42-
63
Remaining Parameter Ratio
Remaining Parameter Ratio
Remaining Parameter Ratio
Remaining Parameter Ratio
CTF AR-100
Tiny-ImageNet
(a) Test accuracy
Aio my
Ala)a(O af
Alabs(Ous) a
ReaNet50
MohisNleiV2
ReeNes50
MuzielketV2
(N)
01
15
10-
3.0
22
0.6
055
0.5
1.2
200
02
0.6
03
12
0.3
0.2
04
00
0.
1.0
0.2
0.2
0.4
06
2.0
ShuffeNitt
EffeientNet
Shufflelet
Efficinthket
80-
5.10 0' (N)
21
55
01
05
Interpllation
Interpoiaton
CIFAR- 100
Tiny-ImageNet
(b) stability against SGD noise
A10 mi Atggan 13R
Alsbo(O.az) w
A10 m
AlabolOzz) @ a
ALOS mm!!!
RenNesS0
MobisttetV2
ResNett0
MobileNetV2
23
20
10
20-
00
02
04
36
35
LG
00
03
04
26
35
10
ShuffdNist
EfficinnNitt
StuffeNNet
EfficienNNet
50-
23-
20
22
Interpoaaion Ratio
2.
00
Interpolation Rario
2.4
2.
66
1.0
000
Interpolation
85
CIFAR-100
Imng-ImageNNe
(c) Linear mode connectivity
Figure 3: Main results CIFAR-100 and Ting-ImageNet. (a): Test accuracy of the LRR solution (blue), the AWS solution (green), a randomly initialized network trained with the LRR-drived signed mask (orange), and randomly initialized network trained with the AWS-driven signed mask (red). (b) and (c): Analysis of SGD noise stability and linear mode connectivity, respectively. A randomly initialized network trained with the AWS-driven signed mask exhibits high SGD noise stability and low error barriers along the linear path to the AWS solution (green), contrasting to the case of LRR (orange). In (b) and (c), we use pruned network with remaining parameter ratio of approximately 0.09. We report the mean (each point) and standard deviation (shaded area) across 3 trials.
8

---

Published as a conference paper at ICLR 2025
4.2 RESULTS ON CIFAR- 100 AND
Test Performance. In Figure 3a we report the test performance on CIFAR-100 and Tiny-ImageNet.
We compare the performance of six networks after training: (1) initialized dense network, A A(Omi,,
(2) LRR subnetwork, A(0), @ mlRR), (3) AWS subnetwork, A(0 @ m² (4) randomly initialized network masked with a signed mask of a LRR subnetwork, A(abs(Omil @ s488), (5) randomly initialized network with a signed mask of a AWS subnetwork, A(abs(Omma) imi) @ sf and (6) randomly initialized network masked with a mask of a LRR subnetwork, A(Oii @ ml(#). Note that A(-) indicates a normal training algorithm without interpolating normalization layer parameters. First, we note that in most cases, the AWS solution (indicated by the green plots) achieves performance comparable to or better than the LRR solution (indicated by the blue plots). This addresses the potential concern that randomly interpolating normalization parameters in our method could adversely affect the performance of the AWS solution. Then, we transfer the signed mask from LRR, $/ and AWS, sAWS to a randomly initialized network 0mi In the case of LRR, the performance of a randomly initialized network masked with syKe after training (indicated by the orange plots) is similar to the case when the sign information is not used (indicated by the purple plots), lagging far behind the performance of the LRR solution. On the other hand, we observe that in most cases, the performance of a randomly initialized network masked with s after training (indicated by the red plots) is comparable to that of the AWS solution. Finally, We observe that the signed mask obtained through AWS allows a randomly initialized network to perform comparably to a dense network (indicated by the dotted line) after training at non-trivial sparsity as long as the AWS solution performs comparably to the dense network.
stability against SGD-Noise and Linear Mode Connectivity. We further compare the effectiveness of saWS and slRR by examining whether they can transfer the strong generalization potential of the AWS or LRR subnetwork to a randomly initialized network. To this end, we first examine the SGD noise stability of abs O mit) @ syth and abs( @ s The results in Figure 3b demonstrate that minn)) @ s|/R shown by the orange plots, exhibit significantly high error barriers between a pair of networks trained with different SGD randomness. On the other hand, (6m)) O s/WS, represented by the green plots, exhibits comparably low error barriers between pair of networks trained with different SGD randomness, relative to the reference stability of ØLRR ( s7RR shown by the blue plots. Moreover, we examine the lincar mode connectivity between A(abs (0.i) @ s4WS) and A(abs (0AWS) o sAWS), as well as between A(abs(O O s confirm whether they lie in the same basin. The results in Figure Be demonstrate that transferring sJRR to a randomly initialized network causes the network to converge to a solution with significantly high error barriers between the LRR solution, as indicated by the orange plots. By contrast, random initialized network masked with s exhibits relatively low error barriers between the AWS solution, as indicated by the green plots, compared to the reference shown by the blue plots.
and (((abs(OLka) @ sLRR) to
Discussion. Our motivational experiments in Figure 2 show that the signed mask of the LRR sub- network alone cannot transfer the generalization potential of the LRR subnetwork to a randomly initialized network; it is also necessary to transfer the normalization layer parameters of the LRR sub- network. For the goal of LTH, we propose AWS that eliminates the need for the trained normalization parameters. The test performance in Figure 3a and the analysis of the SGD noise stability and linear mode connectivity in Figure 33b and Figure 3c show that a randomly initialized network masked with the signed mask from the AWS subnetwork lies in the basin of attraction of the AWS subnetwork. This demonstrates that sAWS itself can transfer the generalization potential of the AWS subnetwork.
4.3 GENERALIZATION TO IMAGENET AND LAYER NORMALIZZATION
To validate the effectiveness of AWS on a more challenging task, we evaluate AWS on ImageNet dataset. The results in Table [ show that across various model and sparsity (i.e.. 1- 'Remaining Params"), a randomly initialized network masked with the AWS driven signed mask achieves perfor- mance comparable to the AWS solution (i.e., A(abs(
@ sAWS) ) vs. A(0AWS @ mAWS), )), whereas masking with the LRR-driven signed mask fails to reach the performance of the LRR solution (i.e., A(abs(0 (ini) O sJRR) ) vs. A(OLRR O mLRK)), )). Notably, we observe similar trend in the results on MiX-Mixer, which uses only layer normalization for normalization. These results show that the signed mask of the AWS subnetwork can transfer the generalization potential of the AWS subnetwork to a randomly initialized network on a more challenging dataset and with layer normalization, further demonstrating the effectiveness of AWS.

---

Published as a conference paper at ICLR 2025
Table 1: Experimental results on ImageNet. Onii indicates randomly initialized parameters and
"Remaining Params.' refers to the remaining parameter ratio. We present more results in Table 2
Model
Method
Use AWS subnewoork Trained from Top-1 (1) Remaining
Dense Network
75.26 0.27
AiokRR majass
75.11 + 0.27
A10 @ ma
74.48 0.16
Alabui 6 a
77320 1'0.1
ResNet50
Alabsi 0
74.65 0.20
A10 mm m
X
74.58 0.25
A1p à m NWS,
Mlsbii mid graee "raeeer
"11911 0.22
73.63 4: 0.21
0.14
Alabai wws
73,65
Dense Network
69.42 0.41
Aio ma
69.25 0.33
68.75 4: 0.22
0.4
Alahui and JRR
67.99 1
MobileNeiV2
Alalsi ows)
68.66 0.24
Aig mase
0.17
Ateaws & maws
66.17 0.20
0.21
Maih @ AWs
66.02
Network
Aid mi
Aio @ Aww
59,23 1.0.0.
0.21
Atabai ( 1 ste
$7.34 0.300
MI P-Mier
Ala a
Mi ms
58.61 0.23
Aioaws & mews
Atshsi @ aut
56.91 + 0.21
0.11
Atshui @ wws)
58.62 0.35
4.4 COMPARISON TO RELATED WORK
There are existing works that also aim to search for an effec- tive signed mask (Koster et al. 2022 Sreenivasan et al.) (022).. Among them, we compare our AWS to GM (Sreenivasan et al. 2022), as GM is evaluated on a more diverse and large-scale architecture than the method of Koster et al. (2022). Figure 4
75
Test Accuracy (%)
demonstrates that while GM performance falls short of that of the dense network across all levels of sparsity, randomly initialized network masked with sAWS can perform comparably to both the AWS subnetwork and the dense network until sparsity about 0.9. We also highlight that our work provides valuable insights into the role of parameter signs and the in- fluence of normalization layers concerning finding a winning ticket. Moreover, it is noteworthy that the signed mask obtained by AWS can be applied to any random initialization. In contrast, GM trains signed mask tailored for specific initialization, which likely limits its applicability to other initialization.
65
Aroii misis
60
Alabo(f.aa) 28th) CM
Al00al
0.0 0.01 0.02 0.06 0.13 0.33 0.8 Remaining Parameter Ratio
Figure 4: Comparison to GM CIFAR-100 with ResNet-32. The results of GM are approximated from (Sreenivasan et al. 2022).
5 CONCLUSION
Finding a winning ticket is still an open problem in the field of the lottery ticket hypothesis (LTH). In this work, show that an effective signed mask, sparse mask with sign information, crucial for conveying information that enables an initialized network to achieve strong generalization. We observe that the signed mask and normalization parameters from the subnetwork trained by learning rate rewinding (LRR) can transfer the generalization potential of the LRR subnetwork to any randomly initialized network. To progress towards the goal of L.TH, we propose AWS, slight variation of LRR, that mitigates the reliance on the trained normalization layer parameters by encouraging low error barriers between the AWS subnetwork and its counterpart with initialized normalization parameters. In contrast to the existing methods limited to finding winning ticket with ad-hoc initialization, we demonstrate that the signed mask from the AWS subnetwork can allow any randomly initialized network to reside within the basin of the AWS subnetwork, possibly leading the resulting network to generalize as well as the dense network. For future work, we will investigate the effectiveness the signed mask acquired through AWS in the transfer learning scenario.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported part by the IITP grants [No.2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National University), No. 2021-0-02068, and No.2023-0-00156, and the NOTIE grant (No. RS-2024-004524410) by the Korean govermment.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https://arxe httgg/bb//lll 06450
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias variance rafe-off. Proceedings of the National Academy of Sciences, (16633):15849-11311
2019. doi: 10. (0073//aa..9000000. 16. BtLhttps:: /www 1073 pnas 1903070116
pnas .mas.org/den /ttgg/doi//
Rebekka Burkholz. Most activation functions can win the lottery without excessive depth. In NeurlPS, 2022.
Xiaohan Chen, Jason Zhang, and Zhangyang Wang. Peck-a-boo: What (more) is disguised randomly weighted neural network, and how to find it efficiently. In ICLR, 2022.
Arthur da Cunha, Emanuele Natale, and Laurent Viennot. Proving the lottery ticket hypothesis for convolutional neural networks. In ICLR, 2022.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. ICMI,, 2018.
Utku Evci, Yani A. Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In AAAI, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: finding sparse, trainable neural networks. In ICLR, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. ICML, 2020a.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In ICLR, 2020b.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In ICLR, 2017.
Advait Gadhikar and Rebekka Burkholz. Masks, signs, and learning rate rewinding. In ICLR, 2024.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NIPS, 2018.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Nils Koster, Oliver Grothe, and Achim Rettinger. Signing the supermask: Keep, hide, invert. In ICLR, 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
11

---

Published as a conference paper at ICLR 2025
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: filter pruning using high-rank feature map. In CVPR, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Ekdeep Singh Lubana, Eric J. Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In ICML, 2023.
Xiaolong Ma, Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin. Sijia Liu, Zhangyang Wang, and Yanzhi Wang. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? In NeurIPS, 2021.
Eran Malach, Gilad Ychudai, Shai Shalev-Shwarta and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In ICML, 2020.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrrizatio in generalization of neural networks. In ICLR, 2019.
Quynh Nguyen. On connected sublevel sets in deep learning. In ICML, 2019.
Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. In NeurIPS, 2020.
Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in winning ticket's mask? In ICLR, 2023.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. IJCV, 2015.
Keitaro Sakamoto and Issei Sato. Analyzing lottery ticket hypothesis from pac-bayesian theory perspective. NewrIPS, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: inverted residuals and linear bottlenecks. CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Kartik Sreenivasan, Jy yong Sohn, Liu Yang, Matthew Grinde, Alliot Nagle, Hongyi Wang. Eric Xing, Kangwook Lee, and Dimitris Papailiopoulos. Rare gems: Finding lottery tickets at initialization. In NeurlPS, 2022.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, 2021.
Qingyang Wang, Michael A. Powell, Ali Geisa, Eric Bridgeford, Carey E. Priebe, and Joshua Vogelstein. Why do networks have inhibitory/negative connections? In ICCV, 2023a.
Qingyang Wang, Michael A. Powell, Ali Geisa, Eric W. Bridgeford. and Joshua Vogelstein. Polarity is all you need to learn and transfer faster. In ICML, 2023b.
Haoran You, Chaojian Li. Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing carly-bird tickets: Towards more efficient training of deep networks. In ICLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In NeurIPS, 2019.
13

---

Published as a conference paper at ICLR 2025
A IMPORTANCE OF PARAMETER SIGNS IN NORMALIZATION LAYERS.
In Section 3.2 we observe that preserving the signs of parameters in the LRR subnetwork while initializing their magnitudes, excluding those in normalization layers, allows the resulting network to stay within the basin of attraction of the LRR subnetwork, but it fails when the normalization layer parameters are initialized together. We claim that this occurs since the parameters in normalization layers rely more on the magnitude of parameters rather than their signs. The scaling factor in normalization layer is nearly always positive, thus its sign information may be useless. In the case of the bias factor, it loses its sign information after initialized to O, but it does not necessarily mean its sign information not beneficial. To further validate our claim, we also compare the case where the bias factor in normalization layers is initialized to constant, thus their sign information does not disappear after initialization. In Figure 5 we observe that maintaining the signs of bias factors (indicated by the purple plots) is not beneficial compared to the original initialization case (indicated by the orange plots), demonstrating our claim.
Allafan of @ m
Alabuldduan a ) m
A(abol( (huel) 5 2 - Alshalithua Vhall) @ aaa) A(0
ResNet50
MobileNetV2
ShuffleNet
EfficeettNet
Accutocy (N)
69
13
2
67-
66
5: 5.
65-
64
0.5
3.06
0.02
Remaining Parameter Ratio
Remaining Parameter Ratio
Remaining Parameter Ratio
Remaining Parameter Ratio
Figure 5: Effect of transferring the sign of normalization layer parameters. ai denotes the initialized normalization layer parameters whose scaling and bias factors are set to I and 0.1. We conduct the experiments on CIFAR-100.
B Is AWS ALWAYS BETTER THAN LRR?
In the main manuscript, we show that after training, the post-training performance of randomly
initialized network masked with syR (i.e., A(abs(Omm)) O SLRR)) lags far behind that of the LRR solution. However, we found that for several network architectures, the performance of A(abs(B (0 O ST $LRR) is similar to that of the LRR solution. In Figure 64. we present the test performance on VGG11- bn (Simonyan & Zisserman 2015). We observe that A(abs(0mm)) @ s)RR) (indicated by the orange plots) achieves performance similar to that of the LRR solution (indicated by the blue plots). Thus, the performance difference between A(abs( (Om)) C sJRR) and ((abs(Omii)) 3 sAWS) (indicated by the red plots) is trivial. We also investigate the SGD noise stability of A(abss( ((ab)(@mm)) @ s ..RRR and A(abs(O 0 mit) @ 8/7 s:AWS) and their linear mode connectivity the corresponding LRR AWS solution in Figure 60 and Figure Gc. respectively. We observe that the signed masks from both LRR and AWS enable randomly initialized network remain stable against SGD noise and converge to a solution with linear mode connectivity to the corresponding LRR or AWS solution. Thus, in some cases, LRR can yield an effective signed mask that transfers the generalization potential of the LRR subnetwork to randomly initialized network. However, we argue that AWS is more effective and generalizable to a wider range more complex architectures as demonstrated in Figure 3
71.0
Test Accuracy (%)
70.5
70.0
NA onith
Aitht oodf ai Aid wrofh Auseibal to o with
69.5
0, 07) (%)
Altua,
07 (%)
Alsaisaal saatt Ald mi
A. mis")
68.5
E.00.0
A mite
08.0
67.5
A.B.al
08 Remaining Parameter Ratio
0.02
0.06
0.13
0.33
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Interpolation Ratio
Interpolation Ratio
(a) Test Accuracy
(b) stability against SGD noise
(c) Linear mode connectivity
Figure 6: Results of LRR on CIFAR-100 with VGG11-bn.
14

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results on ImageNet. mmn indicates randomly initialized parameters and "Remaining Params." refers to the remaining parameter ratio.
Model
Method
Use AWS subnetwark Trained from Oar Top- Acc.
Remainnin Params.
Dense Network
75.26
0.27
A( @ mLaR)
A.BAWS maws
74.48 0.16
0.27
K
73.20 0.021
Alabs (muir) AWs)
74,65 0.20
A @ m
74.92 0.10
A(gAws mwss
Alassi tone g
72.96 0.20
ResNet50
Alabsi0 mad ww
A @ m
x
74.62 =: 0.12
Alg mws
Alabs Omid LRR
72.46 0.18
73.76 0.26
0.17
Afabul Omid) WW))
73.9911 0.23
A @ m
X
74,58 + 0.25
A mes
5
73.63 1. 0.21
0.14
Alabs (mir) a
x
71.94 0.22
Alabs Giid (WS)
73.65 0.12
Dense Network
69.42 0.41
Alo @ m
69.25 0.33
ALOAWs mws,
68,75 0.22
0.41
Alabsi
67.99 0.11
Alabs ( AWs
68.66 0.24
Al A(O milkk
Al nawss maws,
0.33
Alabsi Ouad
67.01 0.19
MobileNetV2 Alabs mane Aws
68.04 0.28
AtgERa @ m
68.36 0.20
Alo7ws maws
67.39 0.24
Alabol Ommd L
64.92 0.25
Alabs (mi Aws)
ACDERR @ m
x
X
67.76 1: 0.17
AtoAws maws
Alabsi Omid LRR
66.17 1: 0.20
64.03 = 0.21
0.21
Alabs Omid @ AW))
66.02 0.29
Donse Network
59.37 : 0.12
Aio @ mLRR,
59.18 0.26
AloAws maWs,
Alabsi Omii af R
57.54 = 0.30
59.23 0.18
0.21
Alabsi Omid Aws)
59.17 0.19
ALoLRR milk,
Aooss co
mAws,
0.17
57.21 0.21
Alaba Omii) Aws
MLP-Mixer
$8.93 0.19
AtoLRR @ miakk
58.83 0.23
Ale Aws Aws,
0.24
Alabsi (iid ggar
37.09 "0.19"
Alabs (mii) aws
A(OLR @ ml
x
58.61 1: 0.23
AlaAws ma
58.77 +: 0.11
0.11
Afahi Ommi @ sERR)
56,91 0.211
Alabs Ouar @ swWs)
58.62 0.35
15


Reviews:
[{"review_id": "Tsigc7UPjQ", "reviewer": "Reviewer_tJqq", "summary": "The paper explores the lottery ticket hypothesis (LTH), which suggests that a sparse network, or \"winning ticket,\" can generalize as well as a dense network when trained from initialization. Previous methods struggled to find such tickets, especially in large-scale settings. The authors propose using a \"signed mask,\" a binary mask with parameter sign information, to transfer generalization potential to a randomly initialized network. They introduce AWS, a variation of learning rate rewinding (LRR) to find A Winning Sign, which addresses the influence of normalization layers and maintains low error barriers. This approach enables a randomly initialized network to perform comparably to a dense network across various architectures and datasets, advancing the search for a true winning ticket in LTH.", "questions": "Please refer to details of above \"Weaknesses\" sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed AWS method leverages a signed mask (a sparse mask with sign information) to transfer the generalization potential of a subnetwork to a randomly initialized network. This approach help addresses a critical challenge in LTH by maintaining the network's basin of attraction, even after initializing all parameters, thus taking a meaningful step towards identifying a true winning ticket.\n\n+ AWS effectively eliminates the reliance on trained normalization layer parameters, which has been a major limitation in previous methods. By encouraging linear mode connectivity between the AWS subnetwork and its counterpart with initialized normalization parameters, the method ensures that the subnetwork remains within its basin of attraction, which avoids the need for parameter retaining.\n\n+ The paper provides empirical evidence demonstrating that the signed mask can enable a randomly initialized network to generalize comparably to a dense network across various architectures and datasets. This shows the method's applicability and effectiveness in various settings.", "weaknesses": "- The proposed method heavily relies on the effectiveness of signed mask in preserving the generalization potential of the subnetwork. It is not clear whther the proposed method can be generalized to more model architectures and tasks. If the signed mask does not perform well across different architectures, tasks or datasets, the method's applicability could be limited.\n\n- The method requires careful handling of normalization layer parameters, which adds complexity. The need to interpolate normalization parameters during training may bring challenges when scaling up the proposed method. its scalability to large scale models/datasets is unclear as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lT4Z5ZmGX3", "reviewer": "Reviewer_3AkZ", "summary": "The Lottery Ticket Hypothesis (LTH) assumes that there exists a sparse subnetwork inside a dense one that can be trained from initialization to reach a comparable performance to the full model; the aim is to find such a subnetwork, called the winning ticket. \nThis work focuses around LTH and introduces a novel approach based on a sparse mask with sign information, while considering  a variation of learning rate rewinding (LRR) that encourages linear mode connectivity to bypass issues tied to the initialization of normalization layers. The experimental results vouch for the efficacy of the proposed method; across various architectures and datasets, the proposed sign mask allows for a randomly initialized network to perform comparably to a dense network.", "questions": "Why did the authors choose only the MobileNetV2 and MLP-Mixer for the Imagenet setting? Were there any issues with the other architectures?\n\nCan the authors provide a corresponding table (similar to Table 1) for the CIFAR-100 and Tiny-ImageNet experiments?\n\nWhat are the sparsity levels for these experiments? Do these correspond to the remaining parameter ratio in Fig. 3?\n\nWhat is the behavior of the proposed approach in different sparsity levels in the case of ImageNet? How does it compare to standard LRR?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is overall well-written and easy to follow. The authors clearly present the motivation of the paper while providing some insights into the proposed approach right from the start. \nThe related work section is clear giving a good overview of the LTH field and its issues. The proposed LRR variant is simple, considering a random an simple interpolation of the normalization parameters.\n\nThe experimental evaluation includes a variety of datasets and architectures with the experimental details clearly stated and the experimental results vouch for the efficacy of the proposed approach.", "weaknesses": "I find that the main novelty of the proposed approach is the insights into the normalization issues and the introduced AWS variant. In this context, I do not consider the contribution to be of the highest novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJ3krWvAUl", "reviewer": "Reviewer_Kwwx", "summary": "This paper aims to find lottery ticket hypothesis for neural network models, especially from the perspective of parameter sign. It proposes to slightly change the LRR based on an interpolation method to make the sparse network with good trainability for generalization. Experiments are based on CIFAR and imagenet dataset across different model architectures.", "questions": "Please check the weaknesses section above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Find the winning ticket is still an open question in the sparse network training area, such topic is a valuable one to explore.\n2. Experimental results shown in the draft support the claimed statement from the paper.", "weaknesses": "1. Compared with originla LRR, the change is relatively trivial with not enough research novelty.\n2. Used datasets and model architectures are not large-scale like mentioned goal in the abstract. Adding more larger datasets and models will be more supportive.\n3. The training configurations for cifar and imagenet may not reach the fully convergence for model training, where 100 and 120 epochs are used for these two, respectively.\n4. Overall, the paper is not very clear to read, such as the figures on the main results section. Which line represents the proposed method and which one is for baselines? how to indicate the proposed method outperforms others?\n5. In addition, in table.1, more sparsity ratios for experiments will be helpful. And using more other and larger networks will further support the paper claim. More comprehensive empirical studies are needed to support the draft conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VDpGnAqmOw", "reviewer": "Reviewer_q2c9", "summary": "This paper suggests that a signed mask (a binary sparsity mask with sign) is necessary to find the winning tickets in the field of the Lottery Ticket Hypothesis (LTH). Specifically, this paper founds that equipped with the signed mask as well as normalization layer parameters of a subnetwork trained with learning rate rewinding (LRR), a random initialized network can also achieve comparable performance with learning rate rewinding (LRR) trained networks. Extensive experiments on CIFAR-100, Tiny-ImageNet, and ImageNet with various networks demonstrate the effectiveness of signed mask.", "questions": "Please address and discuss the weaknesses above in author response.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a pioneering work that studies the importance of signed mask in the field of LTH. The LTH suggests that the weights initialization, either the original random initialization from the vanilla LTH or some early checkpoints from the weight rewinding, is necessary to win tickets, while this paper suggests that we can rely on the signed mask with random initialization to achieve comparable performance. \nFrom my personal perspective, finding the signed mask is easier in practice than finding the ad-hoc initialization.", "weaknesses": "1. This paper only evaluate some lightweight networks on ImageNet such as MobileNet and MLP-Mixer, with an accuracy below 70%. Thus, it's unclear whether the proposed method still generalize well on more powerful networks such as ResNet and Vision Transformers. At least, it would be better if this paper could follow the setting in LRR paper that applies Resnet-50 on ImageNet (see Table 1 of Renda et al., 2020). \n\n\n2. This paper assumes that networks always include normalization layers and its parameters is necessary, which could limit its generalization. Although normalization is pretty common in modern networks, some early networks don't have normalization layers, such as LeNet and VGG16 without batch normalization [1]. Therefore, it would be better to examine the performance of the signed mask of the LRR subnetwork on these pioneering models to determine if the conclusions hold consistently across different architectures.\n\n\n3. This paper claims that \"an effective signed mask is ALL WE NEED to win the lottery\". However, this work still requires the learning rate rewinding to obtains the signed mask with multiple ($T$) iterations. Ideally, the signed mask can be directly obtained from a pre-trained dense networks in the future work. \n\n4. Some mathematical notations seem inconsistent. For example, if I understand correctly, in Line 184, the LRR subnetwork should be $\\mathcal{A}( \\theta_T^{LRR} ,  m_T^{LRR})$ (same as Line 436), rather than $\\theta_0^{LRR}$.  Besides, is $\\theta_0$ (Line 172) same as $\\theta_{init}$. \n\n\n\n\n[1] https://pytorch.org/vision/main/_modules/torchvision/models/vgg.html#VGG16_Weights", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)