Sign-In to the Lottery: Reparameterizing Sparse Training
Authors: Advait Gadhikar
Tom Jacobs
Chao Zhou
Rebekka Burkholz
Venue: NeurIPS 2025 poster ()
Keywords: pruning at initialization
sparse training
lottery ticket hypothesis
mirror flow
reparameterization
sign flips
Primary area: deep_learning
TL;DR: Reparameterizing sparse training for improved PaI by enabling sign flips

Abstract: The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.

Sign-In to the Lottery: Reparameterized Sparse Training
Rebekka Burkholz CISPA Helmholtz Center for Information Security, Saarbrücken, Germany (advait, gadhikar, tom, com.jacobs, chao. zhou, burkholz)@cispa..o
Advait Gadhikan Tom Jacobs' Chao Zhou
Abstract
The performance gap between training sparse neural networks from scratch (Pal) and dense-to-sparse training, presents a major roadblock for efficient deep learn- ing. According the Lottery Ticket Hypothesis, Pal hinges finding problem specific parameter initialization, given a sparse mask. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to Pal. To address this issue, we propose Sign-In, which employs a dynamic reparameteriza- tion that provably induces sign flips. Such sign flips are complementary the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of Pal, they also carve out the main open challenge to close the gap between Pal and dense-to-sparee training.
Introduction
As neural network sizes scale to billions of parameters, building efficient training pipelines is of paramount importance. Sparsification plays a pivotal role in enabling such efficient pipelines (Frantar Alistarh (023). State-of-the-ar sparsification methods rely eliminating parameters during training (Kuznedelev et al.] 2004).. The success of these methods can be attributed to finding an optimal combination of parameters and sparse masks (Paul et al. 2023). However, identifying them requires training dense network before it can be sparsified such that information from the dense network can be inherited by the subsequent sparse network (Gadhikar & Burkholz/ 2024).
is then no surprise that the most successful sparsification methods rely on dense training phases. Dense to-sparse methods like gradual pruning (Han et al. 2013) continuous sparsification (Kusu) pati et al.] (020) explicitly start from a dense network. However, this comes at the cost of high computational and memory demands during the dense training phase. Moreover, the benefits of sparsification become mostly available towards the end of training or during inference.
Is there way to train sparse networks from the start, given sparse mask? The Lottery Ticket (LTT) Hypothesis (Frankle Carbin (019) posits that there exist sparse networks random initialization which enjoy optimal parameter-mas mask coupling and can achieve the same performance their dense counterpart. Finding such sparse networks at initialization would unlock the benefits of sparse training from scratch. A plethora of pruning at initialization (Pal) methods have attempted to find such sparse networks at initialization (Lee et al. 2019 Tanaka et al. 2020 Wang et al. 2020). however, they struggle to match the generalization performance of dense-to-spane sparse training methods.
Understanding and overcoming the limitations of Pal remains an open research problem and the focus of our work. The main challenge for Pal methods is to find highly performant parameter-me mask combinations by leveraging task- task-specific information efficiently (Kumar et al.) (0244] Frankle et al.) 2021). The problem is thus comprised of two parts: a) learning mask and b) optimizing the
Equual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

weights corresponding to the mask, which is hard if the mask is extremely sparse. Here we are less concerned about learning mask, but instead focus on the hard problem of optimizing parameter-mask combinations, given a mask at initialization. At the center of our investigation is the question:
What hinders training a sparse mask, and can the gap between Pal and feess--tosppars be closed?
The key to addressing this question lies in promoting parameter flips. Echoing the analysis of learning rate rewinding (LRR) by Gadhikar & Burkholz (2024), through a broader empirical investigation of dense-to-sparse training methods, we identify a critical phenomenon responsible for finding optimal parameter-mase combinations: the abilitt to flip parameter signs during training. We term this phenomenon sign alignment, as the signs encode all relevant information additionally to the mask (and data) to enable effeetive training from scratch. Moreover, occurs early in the dense-to-sparse training process, promoting convergence toward flatter minima. In contrast, Pal faces challenges in stabilizing sign flips. In line with this observation, it shown in theoretically tractable setting of one- hidden-nnuuon that training a sparse network from scratch fails to flip signs from a bad initialization (Gadhikar& & Burkholz 2024).
show, sign alignment
Motivated by this insight, we propose a way to flip and learn correct signs during sparse training from scratch in higher number of cases (see Figure 2). We achieve this via Sign-In, a reparameterization m @ w of the non- mon-masked parameters, providing them with an inner degree of freedom (where m and continuous parameters). Note that this reparameterization depending on the inner scaling, has a bias towards sparsity. This has been used by the continuous sparsification methods Spred (Ziyin & Wang 2023) and PlLoT (Jacobs & Burkholz 2025). In contrast, we do not the reparameterization for sparsification but for improving the optimization geometry to enable sign alignment and thus seek to counteract its sparsity bias. We propose an inner scaling that promotes sign flips by rehalancing network layers and, correspondingly, the speed in which they learn.
This provably recovers a different sign case than dense training and could thus be seen as comple- mentary (see Figure Moreover, combination with a moderate degree of over-parameterization which tends to be common in Pal, Sign-In fully solves the one-hidden h-whhh-dddddoo neuron problem. In line with this insight, we verify in experiments that existing dense-to-spars. methods can be improved by utilizing Sign-In. Most importantly, we demonstrate across multiple experiments that Sign-In significantly enhances the performance of sparse training from scratch, including those using random masks. We attribute this improvement to better sign alignment and the ability discover flatter minima. However, as we show, Sign-In cannot replicate the same sign flipping mechanism as that of initial dense training, and is still not competitive with the accuracy of dense-to-sparse methods.
This poses the question: Could there exist another unknown parameterization that could replace the sign flipping mechanism of dense training? Unfortunately, the answer is negative, as our proof of an impossibility statement reveals. While we can report progress in the right direction, it remains an open problem completely close the gap between Pal and dense-to sp-sse methods.
Contributions 1) We empirically show that dense-to-spars, training methods identify trainable parameter-mask combinations via early sign alignment allowing them to find flatter minima and achieve state-of-the-ee performance. 2) We provide evidence for the following hypothesis: Across dense-to-sparse training masks, the learned signs define an initialization that enables competitive sparse training from scratch. A fully random initialization (that is decoupled from the mask) performs on par or worse than Pal (including random pruning). 3) To learn signs more effectively, we propose Sign-In, a weight reparameterization, which can provably uusng Riemannian gradient flow and dynamical systems analysis recover correct signs in case that is complementary to overparameterization. 4) Extensive empirical evaluation shows that Sign-In can significantly improve the performance of sparse training from scratch by improving sign alignment and finding flatter minima.
2 Related Work
Lottery Ticket Hypothesis (LTH) Training sparse neural networks from scratch is hard problem (Kumar et al. 2024), but |Frankle & Carbin (2019) posit that there exist initializations that make sparse networks trainable, i.e., Lottery Tickets. However, their approach to find such initializations by
iterative magnitude pruning (IMP) computationally expensive and its performance does not transfer to larger models, even though lottery tickets should exist in theory da Cunha et al. (2022) Burkholz

---

(2022a b Burkholz et al. (2022): Burkholz 2024). Renda et al. (2020) improved its accuracy by Weight Rewinding (WR) and Learning Rate Rewinding (LRR). Both WR and LRR are similarly expensive, but inherit the initialization from denser networks.
Dense-to-sparse methods While dense training has been conjectured to be critical factor enter good loss basin Paul et al. (2023) or quadrant Gadhikar & Burkholz] (2024), other methods aim to reduce its duration. Gradual pruning methods like AC/DC (Peste et al. 2021). CAP (Kuznedelev et al. 2024), and WoodFisher (Singh & Alistarh 2020) prune based on parameter importance score, interspersed with training. Continuous sparsification methods such as STR (Kusupati al. 2020). spred (Zyiin & Wang 2023), and PILoT (Jacobs & Burkholz, (025) reparameterize dense neural network induce sparsity. Although such dense training phases seem to be critical to learn good parameter signs Gadhikar& & Burkholz (2024), we propose complementary method, which also works for pruning initialization.
An orthogonal class of algorithms, dynamic sparse training, includes RiGL (Evci et al. 2020 Lasby al. 2023 Chen et al. 202I), SET (Mocanu al. 20188) MEST (Yuan et al.) 2021). CHTs Zhang ct al. 2025 which dynamically modify a sparse mask during training. However, our work focuses on training a fixed mask from scratch.
Pruning at initialization (Pal) A variety of methods have been proposed to sparsify networks at initialization based different scores that assess parameter importance like saliency in SNIP (Lee ct al. 2019) and GraSP (Wang et al. 22020), signal flow in Synflow (Tanaka et al., (2220) and maximal paths (Pham al., 2023). Other methods include, DPal (Xiang al.] 2025), ) Phew(Patil & Dovrolis 2021]. ProsPr (Alizadeh et a..). However, randomly pruned sparse network can perform at par with
these methods (Liu al. [202T] Gadhikar al. 2023 Ma et al.] 2021 Su al. 2020), , calling our
ability to identify problem specific masks into question (Frankle et al. 2021; Jain et al. (222).. Kumar
lens, showing that initial dense training, in contrast to Pal, increases effective model capacity. Our goal is not to address the mask identification problem but to improve sparse training given mask, by reparameterization.
al. 2024) characterize the challenge from an information-theoredt
Reparameterization Reparameterizations have been studied from the perspective of implicit bias and gradient flow (Chizat & Bach 20200 Li et al. 2022
Woodworth et al. 20200 Gunasekar et al.
2020
2017 Chou et al.) 2024] Vaškevičius et al. 2019 Li et [003] Zhao et al. 2022 |i et al.
2021
Dominé et al. 2024 Jacobs & Burkholz, 2025 Jacobs al. 2025 Marcotte et al. 2023
2024
Liang & Montufar (025). It explains how overparameterizatioo can improve generalization
and how this is affected by various training factors such as the initialization, learning rate, noise, and explicit regularization. An important assumption in these works is that the activation function linear. This simplification enables the reparameterization be expressed using mirror flow or Riemannian gradient flow frameworks (Li et al. 2022). In contrast, we employ a reparameterization specifically induce sign flips in the presence of non-linear activation function. Our proposed reparameterization has been applied to induce continuous sparsification (Ziyin & Wang [203]
[023] 30023Jaccbb Jacobs
& Burkholz 2025 Kolb et al. 2025) with the help of explicit regularization. To combat this trend.
we rescale the parameters dynamically exploiting rescale invariance.
3 Correct Signs Facilitate Sparse Training
0.5
04
Sign Flygs
Init In Warnig
0.1
Whrmup in Fiual
ACIDC
mark
STR mark
Romdom
Soip
Synflm
Figure 1: Sign learning a 90% sparse ResNet50 on ImageNet. The majority of signs are flipped early for dense-to-sparss methods, upto warmup. Moreover, signs stabilize after warmp-up. In contrast, signs do not stabilize after warmup for sparse training from scratch for different masks.
3

---

Paul et al. (2023) have established that iterative pruning methods like Learning Rate Rewinding (LRR) and Weight Rewinding (WR) rely on the effective exploration of the loss landscape during dense overparameterized training. Dense-to-sparse training inherits its performance by training overparameterized in the beginning, identifying a good loss basin. Gadhikar & Burkholz] (2024) have shown that this loss basin can be characterized by the weight signs and thus the quadrant of the solution. Our first objective is to show that this finding is not unique to LRR. In fact, it is a common phenomenon that points to critical limitation of Pal to learn signs.
Our experiments reveal two key findings: (i) Initializing network with an aligned sign and mask is sufficient for successful sparse training. (ii) Sign alignment happens early during training for dense-to-spane -o-arss methods and guides the training trajectory towards flatter minima. Specifically, we study three well-established sparsification methods to draw insights on sign alignment: magnitude based pruning with AC/DC (Peste et al. 202T), continuous sparsification with STR (Kusupati et al.) 2020), and dynamic sparse training with RiGL (Evci et al. 2020). The first two methods leverage information from dense training by alternating dense and sparse training and gradual sparsification by soft thresholding respectively. Although, RiGL does not involve a dense training phase, it retains dense gradient information, which we observe serves a similar purpose. For comparison with sparse training. we train a randomly pruned sparse mask from scratch, which has been shown to be competitive with (and even better than) other Pal methods (Liu et al., 202I) and serves as a strong baseline. Noteworthy is the fact that the random mask contains no information about the task. Even though other masks should therefore have an advantage, as we will see later, they cannot shine through because of limited trainability.
Signs are learned early To study the alignment between the sparse mask and parameter signs, we track them during dense-to-sparse training of ResNet50 on ImageNet (see Section 6 for details). As shown in Figure T the initial signs and final learned signs are significantly different since approximately 50% of the signs are flipped for all methods (blue bar), i.e., the signs initializa- tion have a close to random overlap with the final learned signs. As training pro- gresses, upto the warmup phase (10-th epoch), a significant number of signs flip, as shown by the red bar in Figure I After the warmup phase, fewer signs flip in addition (green bar in Figure []. In comparison, training the same masks from scratch starting from a random re- initialization (which is decoupled from the conse-to-sparse training procedure) does not identify stable signs early. The performance of these masks is at par with random mask trained from scratch (see Table in the appendix) in the absence of sign alignment. Moreover, the sign flip statistics show similar pattern as training a random mask from scratch. A significant number of sign flips occur after warmup, and the overlap between initialization, warmup and the final signs is closer to random (see Figure TT. Across all three dense-to-spane -o-arse baselines, we observe that signs are learned early during training and remain largely stable after warmup. In contrast, sparse training does not find stable signs to enable sign alignment, reflecting in its worse performance.
Table I: Sign alignment for a ResNet50 trained on Ima- geNet with 90% sparsity. Baseline performance is recov- ered when the learned mask is initialized with the learned parameter signs, but not with the learned parameter magni- tude and random signs.
Method
AC/DC
STR
RiGL
Baseline
74.68
73.65
73.75
Epoch 10 sign learned mask
73.97
71.7
73.32
Epoch 30 sign learned mask
74.89
73.91
73.7
Final sign + leamed mask
74.88
73.77
73.74
Epoch 10 mag + leamned mask
70.93
68.01
72.30
Final mag leameed mask
70.94
68.35
72.40
Random init leamned mask
70.6
68.38
71.89
Signs are more informative than magnitude As we verify next, the critical information about the task that is missing from the mask are the parameter signs. To do so, we use signs learned at different stages during training, the learned mask and random magnitudes for each method to initialize a sparse network. As reported in Table [ final signs combined with the mask are sufficient for sparse training and achieve performance comparable to the baseline. Moreover, since the signs are identified as early as the 10-th epoch during training, these early signs (combined with the mask) recover baseline performance with an error of R 1%. Full baseline performance can be recovered by using signs from the 30-th epoch. In contrast, if we initialize the sparse network with magnitudes learned at

---

intermediate epochs, learned mask and random signs, we barely improve over training randomly intialized sparse mask from scratch Table !
Our results not only suggest that sign alignment is sufficient for sparse training, but it also happens early during dense-to-sparse training. Sparse training from scratch with falls short in comparison.
Flatter loss landscapes with dense-to-sparse methods To dive deeper into the benefits of sign alignment for trainability, we track the maximum eigenvalue of the Hessian of the loss as a proxy of sharpness (Jiang et al., 2019). can also be interpreted as a proxy measure for the tractability (or conditioning) of the optimization problem. As shown in Figure 4(b) the dense-to-spars. baselines AC/DC, STR and RiGL have smaller sharpness indicating a flatter loss landscape once signs align during training. In comparison, the random sparse network converges to sharper minimum.
Our observations lead to the conclusion that dense-to- dense-to-sparse methods are effective because they inherit critical information from dense training in the form of parameter signs during the early training phase. These signs play crucial role in dictating the training trajectory of the sparse network. contrast, this mechanism is absent in training fixed mask from scratch, resulting in poor performance and sharper loss landscape during training, which indicates convergence to an inferior loss basin.
A natural question arises: Can we achieve similar sign alignment in sparse training from scratch? While it seems impossible to replicate the benefits of initial dense training while starting sparse, we demonstrate that sign alignment can be improved using an orthogonal sign flipping mechanism, which we induce by a network reparameterization. As we show, this reparameterization helps to learn correct ground truth signs in theoretically tractable case of single neuron two-layer network.
4 Sign-In: Learning Sign Flips
To improve the learning of task relevant sign flips, introduce a reparameterization. Every weight parameter is replaced by the product of two parameters m @ w, where C denotes a pointwise multiplication (Hadamard product). This introduces determined by the initialization and captured by m² w² = 31, where 3 > 0 and 1 = (1, ..., 1). According to Lemma 4.8 (Li et al. (202), the reparameterization induces Riemannian gradient flow in the original parameter 0:
additional degree of freedom 3, which is
d0,- V0 08+ VL)
0g= 0init
(1)
This holds for general continuously differentiable objective function L R + RR. To induce helpful sign flips, we have to deal with two factors: stochastic noise (Pesme et al. 2021 and weight decay (Jacobs & Burkholz 2025). Both factors have the same effect on the training dynamics: They induce sparsity by shrinking B. Yet, if / = O, we cannot sign flip due the v 0 in the Riemannian gradient flow Eq. (T. We address this issue with Algorithm| in Appendix B by resetting the inner scaling / without changing the weights of the network every p epochs up to epoch T2. In all our experiments, we choose the same scaling of B = 1 for all parameters, as this leads to stable results. We employ an analytic solution to apply the rescaling on both m and w, as given in Appendix C This gives the weights more opportunities to align their signs correctly, as proven for a simple example in Section 5
Computational and memory cost Sign-In doubles the amount of parameters relative sparse training. This is also the case for continuous sparsification methods that utilize the implicit bias of m towards sparsity, like spred (Ziyin & Wang 2023) and PILoT (Jacbss & Burkholz] 2025). Nevertheless, according to Ziyin & Wang (2023). the training time of a ResNet50 with m O w parameterization on ImageNet increases roughly by 5% only and the memory cost negligible if the batch size is larger than 50. Furthermore, at inference, we would return to the representation by merging m w. Therefore, Sign-In is able to improve sparse training with minimal overhead as also highlighted by FLOP comparison in Table [5
5
Sign Learning in Theory
Gadhikar & Burkholz 2024) proposed simple testbed to compare two iterative pruning algorithms, IMP and LRR. L.RR's superior performance explained by its ability to flip the sign of a critical parameter during dense training phase. As we show, the same mechanism distinguishes sparse
5

---

training from scratch with random initialization and dense training. The next section is dedicated to the question how Sign-In changes this picture. Remarkably, it provably resolves another case than dense (overparameterized) training. In combination with dense training, it can flip even all relevant signs and solve the learning problem (see Figure
Sparse Sign-In
Overparam + Sign-In
Sparse
Overparam
X
V
a
a
x
X
X
x
X
Figure 2:
Sign-In recovers the solution in a different case than overpameterization for a two-layer network by sign flipping. Combining the two solves all cases (empirically).
Single neuron setting Let {z: be data set of i..d .V random variables. Consider two-layer neural network consisting of single hidden neuron f IRd R such that f(z a, w) ac (w 2), where (-) maxf:, 0} denotes the Rectified Linear Unit (ReLU). The labels (ya ) are generated from a sparse teacher yı = f(z a, w) with ā > and w = (1/ā, O, 0, ... 0). On this data, train student two-layer neural network with gradient flow and randomly balanced parameter initialization, i.e., mm [iniil 11- The objective is minimize the mean squared error 2n
(MSE), which is given by L(a, w) = 1 Di (f (zi a, w) - y1)² The training dynamics with loss function L is described by the gradient flow:
da; = - L(a, wildt, dw; = Va Wt
4 = am
(2)
WD Win-
In this setting, are interested in comparing two cases: a) We call the multi-dimensione case with > the overparameterized case, as we could also set the weight parameters Wy with index j > 1 zero. We also denote the student f1. Dense-to-sparse training would typically first train all parameters, then prune all weights with j > 1, and continue training only wi and a after pruning. b) The one-dimensional case d = 1 can be from the start. is equivalent to considering a 1-dimensional problem with 1. We also write fo for the student, which faces generally harder problem, as establish next.
special case where mask the components j
Sparse training If the student one-dimensional (d = 1), can only recover the ground truth if is initialized with the correct signs, i.e. when 0;i, and Win 0.
Theorem 5.1. (Theorem 2.1 (Gadhikar & Burkholz 2024)) In the one- me-dimensional single neuron the student
recover the ground truth given sufficiently many samples, if ain > and W1,in > 0. In all other cases (asn > O, Wi,in < 0), (asn < O, Win > 0) and (amn O, Wi,in 0), it will not attain the correct solution.
setting, Eq. (2) with
Sign-In reparameterization The Sign-In reparameterization, which we have introduced in Section 4 can resolve the additional case ain. < O, Wi,in > O when d 1. Note that both and are reparameterized with different inner scalings denoted by 31 and Bg. The corresponding Riemannian gradient flow in the original parameters a and wi is:
da = - V + B.O.T(a, d1, a0 Oin
dw = w + 1,000, Iaa, w1,e)dl, 11.0 = W1.in-
If Wi, > for all / >> 0, is continuously differentiable.
Theorem 5.2 (Sign-In improves Pal). Let and w, follow the Sign-In gradient flow dynamics in Eq. (3) and let 31 > 32 0. If W1,in > 0, then the student fo can learn the correct target with probability 1 - ()) If Wi.in 0, it still fails recover the ground truth.
The proof given after Theorem A. in the appendix. The main steps are: a) Characterization of the gradient field at balanced initialization, breaking the coupling that causes it to approach zero. b) Determining the direction of the stable and unstable manifold at the origin (saddle point). c) Demonstrating that the dynamical system must converge towards stationary point.
Proof.

---

Remark 5.3. The proof combines tools from dynamical systems theory and mirror flow theory to show convergence. which might be of independent interest.
To illustrate Theorem 5.2] we simulate the toy setup for both the standard gradient flow and the reparameterized gradient flow. Figure 3(a) shows that the standard gradient flow converges only when the signs are correctly initialized, whereas the reparameterized model also succeeds if dgn < O, W1.in > O. Other initializations lead the student to converge to the origin and thus function that constant at 0, i.e. training fails.
Note that the condition 31 > $2 > O in Theorem 5.2 is necessary. In practice, this suggests that each layer would require different scaling, which can be hard to tune. We show next that this is not needed and the same scaling can be used for each layer in practice.
Multi-dimensions input For multi-dimensional input (d > 1), the outer layer parameter a cannot sign flip like case of d = 1 (Lemma 2.2 (Gadhikar & Burkholz 2024g). However, we have shown that when the rescaling of is larger then that of W1, L.e,, 31 B2 O. the sign of is recovered, if W1 initialized with the correct sign. This is impractical for larger neural networks with more layers, since we have to choose a different scaling & for all layers. Nonetheless, for multi-dimensionn input layer, different rescaling is not necessary, as we verify empirically next.
We train the student network with d 5 for time T = 200 with gradient descent and a learning rate 7j > 0 such that total steps are T/n. The experiment is repeated for 100 different initializations. The success percentages training runs presented in Table 2 We observe that Sign-In consistently recovers the sign of a, with same scaling 3 I.
A potential explanation is that finite learning rate 1) > 0 induces overshooting, leading to sign flips. However, gradient flow with the standard parameterization does not recover the correct sign of a due to this overshooting. Accordingly, the success of Sign-In cannot be attributed en- tirely to this factor. The key factor is that the balance between a and ||w]|z, is broken. In- tuitively, for to sign flip, it needs move faster than ||welle I, as in the one-dimensional
Table 2: Percentage of sign recoveries if ag 0 for standard and Sign-In gradient descent using different learning rates n.
Method
GF
Sign-In GF
0.001
0%
100%
7 = 0.01
0%
100%
|/ in l1. 2 wg for all le E [d]. Therefore, the reparameterization induces relative speed up for compared || i.e.,
case (see Figure ((a).. We observe that |a 12
+ 3 in
2
|/ I.. + 3
w kiin B
(4)
This indicates a can potentially sign flip without different rescaling in-between layers.
No replacement for overparameterization While it remarkable that the combination of Sign-In and overparameterized training can solve our theoretical toy problem, ideally, we would like to facilitate training from scratch (d = 1) without utilizing any dense training. Sign-In is not able to achieve this without dense training but does there exist another reparameterization that could? Unfortunately, the answer is negative, as we establish next.
Consider the one-dimensional problem again, with continuous differentiable reparameteriza- tion 91 E (M1,, R) and 92 E CM2, R) where M1 and M2 are smooth manifolds. Then we reparametrize both a and to such that gi (a) and w = g2()).
Theorem 5.4 (No reparameterization replaces overparameterization). If < O, then there exists no continuous parameterization (21,92) for which the resulting gradient flow reaches the global optimum.
Proof. The flow has to move through wi = O. We can assume there exists an w" E M2 such that g(w") = 0. otherwise it is impossible to move through wN 0. This implies that O, Oa, L 0.
Thus, no continuously differentiable reparameterization can recover the sign of w 0
According to Theorem 5.4 from optimization perspective, we can not replace overameterization with reparameterizzaiion to recover the second case in Figure This impossibility further emphasizes the difficulty of optimizing sparse networks without dense training phase.
7

---

Multiple neurons Do our insights still hold if we increase the number of neurons? To verify the
broader validity of our claims, we consider neural network f2 R + given by fz a, W)
E:=1 a,o (w: 2)) The data is generated from a similarly structured teacher network f2 with & neurons and input dimension d The weights are initialized and trained outlined in Appendix D
A difficult sign initialization is chosen as a, >> 0 for all E [3]. Figure (aaure3iillan illlustrates each neuron by vector |a W,. The teacher neurons vectors touching the unit eircle and student neurons
denoted with markers.
+ sign
sign
-1
-55
Final
(a) Single neuron.
(b) Gradient flow.
(c) Sign-In.
Figure 3: Student-teacher setup (a) Training one neuron for single input d = 1, when wi S both methods fail to reach the ground truth. If w/ 0, the Sign-In gradient flow succeeds, whereas gradient flow fails in this case when additionally a < 0. (b) Representation of a two layer student- teacher neural network with multiple neurons inspired by Chizat et al. (2020) for sparse training from scratch fails with bad signs. (c) Sign-In enables learning the representation with bad signs.
6
Experiments
We show that Sign-In provably improves verify that its merits also translate to larger scale settings. While our main focus on Pal and thus training sparse networks from scratch, our theory implies that Sign-In could also boost cense-to-sparss training performance, their combination can resolve cases where either would fail. However, do not expect training from scratch with Sign-In to outperform training starting from well aligned signs, Sign-In acts orthogonally dense-to-spand to-sare training in theoretical setting. Sign-In is expected to facilitate different sign flips.
small scale example (Theorem 5.2). now we have to
We perform experiments standard vision benchmarks including CIFARIO. CIFAR 100 (Krizhevsky et al. 22009), and ImageNet (Deng et al. (009) training a ResNet20, ResNet18. and ResNet50 model, respectively. For architectural variation, we also train vision transformer DeiT Small (Touvron et al. 202T on ImageNet. All training details are provided Table 4 in Appendix
Experimental Setup
Table 3: Sparse training with Sign-In.
Dataset
Method
= 0.8
= 0.9
0.95
Random
38.25(+-0.35)
86.25(+00.3
33.56(+0.22)
CIFARIO + ResNer20
Random + Sign-In
87.83(+0.11) (9.17
84.74(=0.00
Random
73.95(+0.1)
72.96(+0.24)
71.36(++.10)
CIFAR100 ResNet18
Random Sign-In
75.32
(3.94
72.51 72.51(+0.14)
Random
73.87(1:0.6)
71.56(110.03)
(8.72(+
ImageNet, ResNet50
Random Sinn-In
74.12(=#.09)
72.19(+0.18)
69.38(+0.1)
Scaling reset An integral part of Sign-In is that we dynamically reset the internal scaling to combat the implicit bias towards sparsity of the parameterization. To ablate its contribution, we report results for Sign-In In with and without resetting the scaling for random mask in Table in the appendix. Indeed, without the scaling reset, the performance gain less significant. This confirms that resetting mitigates the effect of stochastic noise and weight decay, enabling better sign flips.
Sign-In improves sparse training Table results for sparse training with random mask. The layerwise sparsity ratios chosen balanced such that each layer has the same number of nonzero

---

parameters (Gadhikar et al. 2023). Sparse training with Sign-In as per Algorithm consistently outperforms standard training, substantiating our theoretical results. A similar improvement is observed for other Pal methods in Table T6 in Appendix H We apply Sign-In also to different sparse masks identified with Pal or dense-to-sparg, methods as reported in Table Appendix H While layerwise sparsity ratios determine the overall trainability of sparse network, Sign-In improves performance in better trainable settings.
Vision transformer with Sign-In Sign-In is also able to improve sparse training of a randomly masked vision transformer. A DeiT Small architecture (trained for 300 epochs from scratch) gains more than 1% in performance with Sign-In, as reported in Table 10 Appendix
Improved sign alignment Figure 4(a) validates the Sign-In In mechanism by tracking the percentage of sign flips during training for a 90% sparse ResNet50 on ImageNet. Across the full duration of training, Sign-In enables strictly more sign flips than sparse training. This improves sign alignment, reflected in the improved performance and the flatter minima (see Figure 4(b)). Since Sign-In employs an alternate mechanism for sign flipping, cannot learn stable signs early like dense-to-sparse methods. Instead, it enables more sign flips throughout training as compared to standard sparse training, to aid alignment by better exploration of the parameter space.
Sign-In improves other sparse training methods Only a combination of Sign-In and overparame- terization is able to solve all sign flip cases in our theoretical setting, as also confirmed by numerical experiments in Table Appendix [ Therefore, we expect the combination of dense-to-spare or dynamic sparse training methods with Sign-In to achieve further gains in performance. Appendix K shows that both AC/DC and RiGL. are improved with Sign-In.
Implications of theory The surprising suc- cess of Sign-In combined with overparame- terization in our theoretical setting relies on ble in the same setting without Sign-In. w1 can follow once the sign of a is correctly learned. We conjecture that weights in lay- ers close to the input would have a harder time flipping in deep neural networks con- sisting of more layers and neurons, as other layers on top slow its learning down. In line with this narrative, Sign-In promotes more sign flips in layers closer to the out- put, as shown in Figure in the appendix. To understand how these sign flips relate to generalization, we must analyze more complex scenarios in future and see whether more effective sign learning could further boost dense-to train- ing. The primary focus of this work has been to improve sparse training from scratch and Sign-In overcomes one of its major limitations regarding learning parameter signs.
the last layer a learning at higher speed than the first layer W1. This enables flip its sign if necessary, which is impossi-
Sigh Tile
Bandee: Sips
Racker MC0C STR
Akll Remdom
Epochs
Mathoda
(a) Sign Flips
(b) Sharpness
Figure 4: Sign flips with Sign-In Training a random, 90% sparse ResNet50 ImageNet with Sign-In in- duces (a) more sign flips during training and finds (b) a flatter minimum.
Conclusion
Parameter signs that aligned with a given mask are sufficient train sparse network from scratch. We have empirically verified this conjecture for masks that result from dense-to-sparse training algorithms. Regardless of the sparse mask, learning highly performant parameter signs is great challenge for Pal. To aid parameter sign flipping. we have proposed to reparameterize neural network by Sign-In. For single neuron network, we have proven with theory from Riemannian gradient flows and dynamical systems that Sign-In resolves a case that is complementary to overparameterized training which translates largescale performance. Ideally, we would like to make dense training phases obsolete to save computational resources. Although, as we have shown, reparameterization alone will not be able to achieve this. Closing the gap between training sparse, randomly initialized networks from scratch and dense-to-sparse training remains a hard open problem. Our work brings us closer to this goal with two key insights: a) sign alignment is enough for training sparse initialized network and b) reparameterization improve sign alignment.

---

8 Acknowledgements
We gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (SSC)Jllic Supercomputing Centre (202T). We are also grateful for funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116395 SPARSE-ML.
References
Alizadeh, M., Tailor, S. A., Zintgraf. L. M., van Amersfoort, J., Farquhar, S., Lane, D., and Gal, Prospect pruming: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations.
Burkholz, Convolutional and residual networks provably contain lottery tickets. In International Conference Machine Learning, 2022a.
Burkholz, R. Most activation functions can win the lottery without excessive depth. In Advances in Neural Information Processing Systems, 2022b.
Burkholz, R. Batch normalization is sufficient for universal function approximation in CNNs. In International Conference on Learning Representations, 2024.
Burkholz, R., Laha, N., Mukherjee, R., and Gotovos, A. On the existence of universal lottery tickets. In International Conference on Learning Representations, 2022.
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., and Wang. Z. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34: 19974-19988, 2021.
Chizat, and Bach, Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Abernethy, J. and Agarwal, S. (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Re- search, pp. 1305-1338. PMLR, 09-12 Jul 2020. URL httpps://proceede/ v125/chizat20a. html
mlr presss/
Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. 2020.
Chou, H.-H., Maly, J., and Stöger, D. How to induce regularization in linear models: A guide to reparametrizing gradient flow, 2024.
da Cunha, A., Natale, E., and Viennot, L.. Proving the lottery ticket hypothesis for convolutional neural networks. In International Conference on Learning Representations, 2022.
Deng, J.. Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A Large-Scall Scale Hierarchical Image Database. In CVPR09, 2009.
Dominé, C. C. J., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A. M., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. URL https //openreview.oo //pperreree...et/foouumdccceeee www..e//ooouuriicaAA33j......
Evei, U., Gale, T., Menick, J., Castro, S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952, PMLR, 2020.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021.
Frantar, E. and Alistarh Sparsegpt: Massive language models can be accurately pruned one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
10

---

Gadhikar, A. and Burkholz, R. Masks, signs, and learning rate rewinding. In Twelfth International Conference on Learning Representations, 2024. URL ULL_atpe://ppeennrrr...o/////oop @ODvxQ8TXW
Gadhikar, A. H., Mukherjee, S., and Burkholz, R. Why random pruning all we need to start sparse. In International Conference on Machine Learning, 2023.
Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., and Srebro, N. Implicit regularization in matrix factorization, 2017.
Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias terms of optimization geometry, 2020.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Jacobs, T. and Burkholz, R. Mask in the mirror: Implicit sparsification. In The Thirteenth International Conference on Learning Representations, 2025. URL [tLLaappes//pppeeereeeeee///oore U477yTS33tt]
Jacobs, T., Zhou, C., and Burkholz, Mirror, mirror of the flow: How does regularization shape implicit bias? In Forty-second International Conference on Machine Learning. 2025. URL
attpps:/oppenreviet http://ppnrrviie.eeeettrrrrrrrme net./oorumidd-MiRRR1
Jain, R., Adnan, M. Sharma, E., and Ioannou, Y. Winning tickets from random initialization: Aligning masks for sparse training. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. URL BRLhhttps:cm httbtp:://openreviee.... wwt.://wwweeeeeeeee..eeeeeeeeeee penee//formmid=xxi000
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, Fantastic generalization measures and where to find them. In International Conference Learning Representations, 2019.
Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities, 7:A183, 2021.
doi: 10. 1..11785/jssf-7-1888
Kolb, C., Weber, T., Bischl, B., and Rügamer, D. Deep weight factorization: Sparse learning through the lens of artificial symmetries. In The Thirteenth International Conference on Learning Representations, 2025. URL https:///ppenreviee
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Kumar, T., Luo, K., and Sellke, M. No free prune: Information-theerrtic barriers to pruning at initialization. In Forty-first International Conference on Machine Learning, 2024.
Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade. S., and Farhadi, A. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, July 2020.
Kuznedelev, D., Kurtić, E., Frantar, E., and Alistarh, D. Cap: Correlation- aware pruming for highly- accurate sparse vision models. Advances in Neural Information Processing Systems, 36, 2024.
Lasby, M., Golubeva, A., Evei, U., Nica, M., and Ioannou, Y. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.002299, 2023.
Lee, N., Ajamthan, T., and Torr. H. S. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Li, J., Nguyen, T. V., Hegde, C., and Wong, R. K. W. Implicit sparse regularization: The impact of depth and early stopping, 2021. URL https::tp/// //attpss///rxi.....m wttg/aa////....0.....
Li, J., Nguyen, V., Hegde, C., and Wong, R. W. Implicit regularization for group sparsity, 2023. URL https: / w/tp///aivvvv/////////eee www.iii..org///ooo abs/2301 10000011111111000

---

Li, Z., Wang, T., Lee, J. D., and Arora, S. Implicit bias of gradient descent on reparametrized models: On equivalence mirror descent. ArXiv, abs/2207.04036, 2022. URL https::/api semanticscholar. org/ Serg/CorpusID 250407876
Liang, S. and Montufar, G. Implicit bias of mirror flow for shallow neural networks in univariate regression, In The Thirteenth International Conference on Learning Representations, 2025. URL httpps://pperrevieu net/ http://ooommdd-IPP0K/X0.0
Liu,, Chen, T., Chen, X., Shen, L.,, Mocanu, D. C.. Wang, 7.,, and Pechenizkiy, M. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference Learning Representations, 2021.
Ma, X., Yuan, G., Shen, X., Chen, T., Chen, X., Chen, X., Liu, N., Qin, M., Liu, S., Wang. Z., et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems, 34:12749-12760, 2021.
Marcotte, S., Gribonval, R., and Peyr'c. G. Abide by the law and follow the flow: Conservation laws for gradient flows. ArXiv, abs/2307.00144, 2023, URL http:p////appp.emmmiiiccccoooole CorpusID:2293166537
www.seemimnnnlllaa......a .rg/
Marcotte, S., Gribonval, R., and Peyré, G. Keep the momentum: Conservation laws beyond euclidean gradient flows, 2024. URL https: //arxiv. org/ barrgg/bb///e abs/2405. 12888
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibeseu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications,, 9(1), jun 2018. doi: 10. 440008461446--1114444133333. URL https
//doi.org// 1roooooog/11...1////181.00.. /tt////001000/1111111770--1----0 043-66--33
Nelaturu, S. H., Gadhikar, A., and Burkholz, R. TurboPrume: High-Speed Distributed Lottery Ticket
Training. htt////cte..gggttbbb../////aaaas
Patil, S. M. and Dovrolis, C. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 8432-8442. PMLR, 2021.
Paul, M., Chen, F., Larsen, B. W., Frankle, J., Ganguli, S., and Dziugaite, G. K. Unmasking the lottery ticket hypothesis: What's encoded in winning ticket's mask? In The Eleventh International Conference on Learning Representations, 2023. URLLattp:///pppnriiiv..eee/ wwwp:rrviiii.ntt//ouu/mraa. XSsW2Am -ukZ
Pesme, S., Pillaud- Vivien, L., and Flammarion, N. Implicit bias sgd for diagonal linear networks: provable benefit of stochasticity, 2021.
Peste, A., Iofinova, E., Vladu, A., and Alistarh, D. Ac/de: Alternating compressee/decoopressed training of deep neural networks. 2021.
Pham, H., Liu, S., Xiang, L., Le, D., Wen, H., Tran-Thanh, L., al. Towards data-agnostic pruning at initialization: What makes a good sparse mask? In shirth-seventh Conference on Neural Information Processing Systems, 2023.
Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Singh, S. P. and Alistarh, D. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 83:18098-18109 2020.
J., Chen, Y., Cai, T., Wu, T., Gao, R., Wang. L.,, and Lee, J. D. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in neural information processing systems, 33: 20390-20401, 2020.
Tanaka, H., Kunin, D., Yamins, D. I.., and Ganguli, Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
12

---

Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou. H. Training data-efficienn image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMIR, 2021.
Vaškevičius, T., Kanade, V., and Rebeschini, P. Implicit regularization for optimal sparse recovery, 2019. URLpttts:://axxy org/ absp/909.00.... 05122
Wang, C., Zhang, G., and Grosse, R. B. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020.
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E.,, Savarese, P., Golan, I., Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models, 2020.
Xiang. L.. Nguyen- Tri, Q. Nguyen, L.-C., Pham, H., Than, K., Tran-Thani L., and Wen, H. Dpai: Differentiable pruning at initialization with node-patl balance principle. In The Thirteenth International Conference Learning Representations, 2025.
Yuan, G., Ma, X., Niu, W., Li. Z., Kong. Z.. Liu, N., Gong, Y., Zhan, Z.. He, C.,, Jin, Q. al. Mest: Accurate and fast cemmorr-cconomic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:2083-220851 2021.
Zhang, Y., Zhao, J., Wu, W., Liao, Z., Michieli, U., and Cannistraci, V. Brain-inspired sparse train- ing enables transformers and llms to perform as fully connected. arXiv preprint arXiv:2501.99107, 2025.
Zhao, P., Yang. Y., and He, Q.-C. High-dimensional linear regression via implicit regularization. Biomelrika, (0)94)):333-4) 1046, February 2022. ISSN 1464-3510. doi: 10. 10.1099//ioomee/aaac0000.. URL httt:://dxx doi. org/ 100 1ww111999////////eee asac010
Ziyin, L. and Wang, Z. spred: Solving l penalty with sgd, 2023. URL bttts:://axxiiii abs/ 2210.01212
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims the introduction and abstract match the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Theorem 5.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
13

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Theorem 5.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
14

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have submitted code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gwwss CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
15

---

Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each run repeated thrice.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
16

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics Sttnggg.rrrrr/eoooo....////////a
Answer: [Yes]
Justification: We follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have submitted our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have submitted code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
18

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy ottteper/eeerrrsssccc////eeerrre for what should or should not be described.
19

---

A Proof of Theorem 5.2
In this section we show the main theoretical result of the paper. We drop the dependence the index for w as we are only concerned with the one dimensional case here.
Theorem A.I. Let f be the teacher and f be the student network such that and follow the gradient flow dynamics in Eq. ( with random balanced parameter initialization. Moreover, let
31 B2 > Wen > O, then I can learn the correct larget with probability (2) In the other case (Win < 0) learning fails.
Proof. The proof idea is to show that for a balanced initialization with Win the flow for /. always enters the open set
:o = {(a, w) E RR2 < ww, w 0}.
Furthermore, we show that the flow stays in the open set To. The system is Riemannian gradient flow which implies that flow converges towards a stationary point ro It remains to be shown that the stationary point at the origin saddle and the stable manifold of the origin is not in To- Thus, the remaining stationary points are the global optimizers.
First we show that for balanced initializations |ain Win 0, we have (ain: Win) o- In case -ain Win > O, we have that (Ain: Win) E To \ To i.e. the boundary of To- Therefore, we need to show the gradient field at (ain, Win) points into To.
> 0 enter the region Fo- In case
The balanced initialization implies that
O₂ L(ain Win) - 1. Win).
Moreover, since à > 0, 02 L(ain Win) > Using that B1 > B2 O have that the gradient field satisfies:
dain
10g + B,000 Ma W4n dt
w? 1.000.
Win) )
w + 120m Win)dt = -dwin
Therefore there is > 0 such that dto w 0. Thus there is tu such that (ato Wio)
To.
We have entered the set To. we have to show that cannot leave the set To- This can be shown by computing the gradient field at the boundaries. The boundary can be split up into three cases:
. Bi (aa, w) E E - w 0}
B3 == (aa w) E R²: 0, 0}
The origin (0,0))
The first case of B1 is covered by the balanced initialization. For the second case of B2 we can compute the gradient field again. We now only need that dw, >> We linearize dwy.
du; = C Ca..v/Baau B2agdt > 0,
=1)² > O with probability 1 1- --))". The last case is the saddle point where C= + max (0,
the origin which show not possible be reached from the open set To- Thus for all (oin, Won) E To we have that for all t 2 0, (at, we) E To or lim; (ar, w.) = (0,
In the case that 0 the flow can be written dynamical system Riemannian manifold. This allows us to guarantee convergence to a stationary point. The flow is given by
do: -C of + 31 (arw? we) dt dw: = -C CC + 32 (azw: - az) dl
an Oin
Wo Wins
where C -= Ei stationary points at the origin and the set aw 1. The dynamical system is Riemannian gradient
En max (0, }2 > with probability 1 1 (1) This dynamical system has
20

---

flow system therefore the flow converges to a stationary point. The stationary point at the origin saddle point. Therefore, the only way of getting stuck the origin when we initialize on the associated stable manifold. We show that this not possible for the balanced initialization. We calculate the linearization of the stable manifold and use that the balanced initialization stays To- The linearization at the origin (0,0) is given by
day CVB widt dw: = B2
By a direct calculation of the eigen vectors the lincarization of the stable manifold is given by the vector (-v 1i [B.T. .1). It follows from 31 32 that the lincarization of the stable manifold is outside of To- Suppose that from To the stable manifold is reachable. Then there is continuous differentiable curve "/r with initialization "Yo = (amn, Win) E Tn such that lim ?tt (0, 0). This is not possible it violates the gradient field at the boundaries of To. Thus, the flow does not converge to the stationary point at the origin. This concludes the first part, since the only set of stationary points the set of global optima.
The other two remaining cases fail as the boundary at w is not differentiable and the gradient flow stops there. D
21

---

B Sign-In algorithm
We provide pseudocode for Sign-In for Section 4 with rescaling as explained in Section
Algorithm Sign-In
Input: objective L, scaling B, frequency p. epochs T, mask S, stop rescaling epoch T- Initialize L(S @ mo @ wo) such that ma C Wh = 0b and ma w SI. f T do
if i mod p = 0 and i < T2 then
Rescale my 3 wy such that m w 31
end if
M4+1, W = Optimizer (L (S O m. ① w;)) end for
Return Model parameters 0r = mr O WT
C Algorithm Initialization
We derive here an exact analytic formula for the initialization of the reparameterization, Since the reparameterization is pointwise we can restrict ourself to the one dimensional case. For this, we need to solve the system of equations for arbitrary æ E R and 3 > 0:
mgws- mg w B
We find that solution is given by:
x
ma = u and Wp
where V 2x² + with
Note that the solution also be written
mo v + and wy /2
2
v2
where u = V + vx² with
22

---

D Multiple Neurons Visualization
We also explore the multi neuron case. The neural network R likd x R² E. is given by fa W, z) == Ei= 0,a (w? 2). The data is generated from similar structured teacher network f2 with A = 3 neurons and input dimension d 2. Furthermore, à Uni((-1111 1}) i.c. i.id. Rademacher random variables and Wi, ~ (0, 1) for all i E [k; d] and normalized over the input dimension. We initialize the dense network with A: 20 and the so-called COB initialization based on the rich regime in (Chizat et al. 2020). All weights are initialized with / (O, 1/d) and we ensure that a, 14+10 for E [10] In the sparse cases initialize it 3 neurons. In the good signs we initialize such that the signs of the weights align with the teacher network. In contrast for the bad signs
[3]. The learning rate used is 7 1 for the sparse setup and = 2.15 for the dense setup. Moreover, the scaling for the reparameterization is 3 = 2. This results in Figure 5 where represent each neuron |a;] w;- The teacher neurons vectors touching the unit circle and student neurons are denoted with markers. We illustrate both the benefit of correct sign alignment (Fig. 5(b) and the reparameterization in case of bad alignment (Fig. ((d)) This is contrasted with dense training (Fig. ((a) and bad alignment (Fig. 5(c).
choose a; 0 for all
Init
Pinal
(a) Dense.
(b) Good signs.
Bad signs.
(d) Sign-In.
Figure 5: Representation of a two layer student-teacher neural network inspired by Chizat et al. (2020). Signs crucial in learning sparse representations. A dense neural network can easily learn the representation in Fig. 5(a) Sparse networks can learn when signs are correct Fig. 5(b) and fail with bad signs Fig. 5(c) The reparameterization enables learning the representation with bad signs in
Fig. 5(d)
23

---

E Training Details
In this section we provide the details of the additional hyperparameters used for experiments. Table 4 we provide the standard hyperparameters such as learning rate and weight decay. We note that the loss used for both CIFARIO and CIFARI00 the cross-entropy loss and for ImageNet the label smoothing loss with confidence 0.9. For the m O w parametrization we also apply Frobenius decay i.e. (m @ w with the same strength as weight decay for CIFARIO and CIFARIOK We provide The code used is based on TurboPrune as in (Neaauuru et al.).
these extra details in Table 5
Table 4: Training Details for all experiments presented in the paper.
Dataset
Model
LR
WD
Epochs
Batch Size
Optim
Schedule
CIFARIO
ResNet20
0.2
le 4
150
512
SGD
Triangular
CIFARI00
ResNet18
0.2
le - 4
150
512
SGD
Triangular
ResNet50
0.25
5e 5
100
1024
SGD)
Triangular
ImageNet
DelT Small
0.001
300
1024
AdamW
Triangular
Table 5: Training Details for the additional Sign-In parameters. The weight decay in this table overides the weight decay in Table
Dataset
Model
B
T2
Frobenius Decay
Weight decay
CIFARIO
ResNet20
1
75
le -
CIFAR100
ResNet18
1
75
1e -
0
ResNet50
1
75
0
5e 5
ImageNet
DeIT Small
1
225
0
le 1
24

---

Experiments on two-layer networks with Sign-In
We empirically verify the ability to recover correct signs for dense-to-sparse methods and Sign-In on the two-layer network with a empirically evaluate the effect of combining dense-to-sparee methods with Sign-In on the two-layer network. We repeat 100 runs for each case, with data generated from a teacher
setup described in Section 5 Further, we also
1, Ø [1. 0, 0),
such that the labels
y == fo tū, 2). For the multidimensional
use d 5. Each setting
trained with Gradient Descent for 2e iterations with learning rate 0.01. Results reported in Table 6 are the success fraction over 100 runs. We empirically observe that it is possible to successfully recover all cases when combining overparameterized training with Sign-In. Even the most difficult case that is unreachable for both their i.e. and < 0. Now it is possible due to Sign-In being able to flip a, whereas overparameterized training can flip w after that. This is also observed with improved performance of AC/DC with Sign-In (see Table 11
Table 6: Experiments two-layer networks in the student-teacher setup presented in Theorem 5.2
Initial Signs
Method
0, w > 0
0, 0
0,
w
0
a < 0, w
<
Sparse
1
Overparam
Sign-In
0
Overparam Sign-In
1
1.9
Gread Trai
Sigt
1.00
0.50
-15
-11
-0.
0/0
Figure 6: Student-teacher setup: training neuron with multidimensional inputs d 5. Combining the overparameterized case with Sign-In can recover all cases.
Failure for Overparam Sign-In We find that Overparam Sign-In does not always succeed in recovering all the cases, and requires sufficient overparameterization to correctly flip a <
choose the teacher network (ō w, 2) with = and Wij ~ N(0, 1), and train the student with d = 2 for the case with < 0. We find that Overparam Sinnn-n is not able resolve this case every time and requires additional overparameterization improve success rates.
25

---

G Sign-In flips signs in later layers
Here validate that the sign flips enabled by Sign-In different from those enabled with dense- to-sparse training. For the case = 0.9 reported in Table [1 for AC/DC and AC/DC Sign-In, we plot the layerwise sign flips for three stages, initialization to warmup (10-th epoch), warmup final and initialization to final. Additionally, we also compare the layerwise sign flips of sparse training randomly reinitialized AC/DC mask. While AC/DC identifies the correct sign flips upto warmup, Sign-In enables more sign flips during subsequent training for later layers in the network (see orange line in Figure 7(b)).
055
045
0.00
B35
0.6
0,5
041
0.0
042
048
916-
sot-
038
Layer
(a) Init to Warmup
(b) Warmup to Final
(c) Init to Final
Figure 7: Layerwise sign flips for Sign-In for AC/DC, AC/DC + Sign-In and sparse training with an AC/DC mask ResNet50 with 90% trained ImageNet.
26

---

H Sign-In with other Pal methods
We report sparse training performance with Sign-In across other Pal methods including SNIP (eee et al. 2019) and Synflow (Tanaka et al. 2020) in Table 16 We use the same setup as described in Appendix
We observe in Table 16 that Sign-In improves all Pal methods. Moreover, in most cases the random mask performs best all.
Table 7: Sign-In with Pal methods CIFAR1O and CIFAR100. denotes single run, as the other seeds crashed).
Dataset
Method
0.8
0.9
0.95
Random
88.41(+0.23)
86.41(+0.1)
03.62)+0.28)
Random Sign-In
89.37(+0.14)
87.83(+0.11)
44.74(+0.06
SNIP
88.12(+0.5)
85.82(+0.2)
81.98 +0.32
CIFAR10 ResNet20
SNIP Sign-In
89.88(=0.09)
87.27(#0.13)
(3.37
Synflow
18.15(+:.17))
85.32(11.24)
73.22*
Synflow Sign-In
89.26(+0.05)
86.47(+00.6)
80.13*
NPB
87.30(+0.39)
85.98(10.32))
82.06(10.9)
NPB + Sign-In
88.87(#0.02)
86.88(#0.51)
83.37(=0..55
Random
74.01(+0.1)
73.07(+0.19)
71.41(+0.18)
Random + Sign-In
75.32(+-0.25)
73.94(+0.13)
SNIP
74.08(+0.28)
72.73(+0.32)
71.21 (1.21(+0.15)
CIFARION ResNet18
SNIP Sign-In
75.16(+0.15)
73.64(10.32)
72.03
Synflow Synflow + Sign-In
73.41(H0.28)
71.37(10:42)
68.56(
75.16(+0.20)
73.64
72.03(
NPB
74.92(+#.63)
72.79(#0.56)
(0.29
NPB + Sign-In
(5.59(=0.15)
74.72(=0.18)
73.28(+0.4)
We perform sparse training with random reinitialization masks identified by dense-to-sparse and compare them with Pal methods. Once reinitialized, cense-to-spars, masks loose alignment with parameters and are at par with Pal methods (Jain et al. 2024 Frankle et al. 2021) These experts reveal two key insights:
Different masks
Layerwise sparsity ratios determine trainiability of sparse networks in the absence of sign alignment.
Sign-In benefits from low sparsity in initial layers.
First, we observe that in case of masks identified by STR and Pal with Snip and Synflow achieve poor performance for both standard training and Sign-In. The layerwise sparsity ratios of these masks reveal sparse initial layers (> 90%) or highly sparse (> 99%) later layers, which potentially affect the overall trainability of the network. In contrast, the RiGL mask, which
random mask with balanced layerwise sparsity ratio (Gadhikar et al. 2023), performs the best over all other masks with Sng Here initial layers are dense while later layers always have sparsity < 99%.
Second, we observe that when initial layers have low sparsity, Sign-In can outperform standard training (RiGL, AC/DC, Snip) but struggles if this is not the case (STR, Synflow). A potential cause would be the inability of initial layers to flip w (see Theorem 5.2)
27

---

Table 8: Sign-In with different masks for ResNet50 trained on ImageNet with 90% sparsity and random initialization. (* denotes a single run, as the runs for other seeds crashed.
Init
Mask
Random
Random Sign-In
AC/DC
70.66-0.12
70.9640.09
RiGL
72.02
72. 72.48_:2 19
STR
68.36-0.17
67.81 +0.34
Snip
52.9"
54.27*
Synflow 60.66+0.2
60. 60.59
Random
71.56-0.03
72.19
10
Deusity
AC/DC STR
RIGL
10
Snip
Synflow
10
201
30
50
Layer Index
Figure 8: Layerwise sparsity ratios at 90% sparsity for a ResNet50.
I Sign Importance
In order to identify the amount of correct signs required for training with the sparse mask to match the respective dense-to-sparse baseline (via agn-aaiigmmnt), we perturb an increasing fraction of signs reported in Table Perturbing more than 10% of the signs starts degrading performance rapidly.
Table 9: Ablation study the importance of weight signs.
Method
0.9
AC/DC baseline (from Table 1)
74.68
AC/DC Mask + 100% correct signs (from Table 1)
74.88
AC/DC Mask 95% correct signs
74.44
AC/DC Mask + 90% correct signs
73.54
AC/DC Mask 80% correct signs
71.81
AC/DC Mask + random signs (from Table 1)
70.88
Vision transformer with Sign-In
Sign-In is also able to improve sparse training of randomly masked vision transformer. A DeiT Small architecture (trained for 300 epochs from scratch) gains more than 1% in performance with Sign-In, reported in Table 10
K
Sign-In improves other sparse training methods
Only combination of Sign-In and overparameterization able to solve all sign flip challenges in thcoretical setting, also confirmed by numerical experiments, which are provided in Table
28

---

Table 10: DeïT Small on ImageNet, trained from scratch.
Method
0.5
Random
69.31 ++.7)
Random + Sign-In (ours)
70.62 +0.44)
in Appendix F Therefore, we expect the combination of dense-to-sparse methods with Sign-In to achieve further gains performance. Taablii]confrrmm that AC/DC improves with Sign-In. However, the improvement is less significant compared to Pal.
Table 11: Sign-In improves AC/DC for a ResNet50 trained on ImageNet.
Method = 0.8
8 = 0.9
0.95
AC/DC 75.83(+0.02) 74.75(#0.02) 72.59(=0.11)
AC/DC 75.9(+0.14) 74.74(++011) 72.88(+0.13)
Sign-In
Sign-In In also improves dynamic sparse training with RiGL and MEST by enabling better sign flips.
Table 12: Sign-In improves RiGL for a ResNet50 trained on ImageNet.
Method 8 == 0.8
8 == 0.9
=
0.95
RiGL 75.02(+0.1)
73.7(+0.2)
71.89(=0.07)
RiGL
+ Sign-In 75.02(+0.1)
74.27 73.07(10.17)
Table 13: Sign-In improves MEST for ResNet50 trained on ImageNet.
Method 8 0.8 8 0.9
0.95
MEST
74.67
73.1
70.84
MEST
74.74
73.32
71.41
Sign-In
Rescaling m O w in Sign-In
While proposed pointwise reparameterizatioo with w allows better sign flips for sparse training, we find that the rescaling step outlined in Appendix is essential to maximize the benefits of the reparameterization show with improved performance of Sign-In over O in Table 14
29

---

Table 14: Effect of Rescaling Sign-In.
Dataset
Method
$ 0.8
= 0.9
= 0.95
Random
88.41
86.41(#0.1)
83.62(=0.28))
CIFARIO ResNet20
Random Sign-In
87.83(#0.11) (9.37
84.74
Random + m O w
89.11(#0.08)
87.13(#0.23)
83.88 +0.022)
Random
74.01(+0.11
73.07
71.11#0.18)
CIFAR100. ResNet18
Random + Sign-In
75.32(=0.25)
73.94(+0.13)
(2.51
Random @ 74.62(=0.24)
73.18(#0.01)
72.08(#0.2))
Random
73.87(+0.1)
71.55(+0.04)
(+..03)
ImageNet, ReaNer50
Random + Sign-In
74.12(=0.09)
72.19
69.38
Random @ w
(3.96(+0.04)
71.83(10.1)
69.15(#0.09)
M Code for Sign-In in PyTorch
We provide example of integrating Sign-In any training setup in PyTorch by simply replacing the Conv (or Linear) layer with [
import torch
import torch, nn as an
class ConvNaskMi Conv2d):
Conv2d layer with paranatrization of for the veights. Additionally nask is appliad during the forvard paas to the veighta of the layer.
Arge:
rekuargs: Keyzord argunents nn Coav2d.
def ...nit_ (self, **kuarge); super i...init__. (selvarss)
self. self. ral.rrrgiateerbbufee.e ("mask", torch. owee_ (aelf (aw..
celf. alpha - 0.5
to (self. eeight device) torch, aqrt reight. self weight. devica) seight teight. (swlf. weigct. + alphavalpha))
worch..grt (self to
selfim an Par slpha/u)/up... sgrt #2) requires_grad-True)) self. seight.data - (u alpha/u) sqrptt///ppp.r.reeee...o eqrt
def aerge(self)) during pruning. self weight.i data self mif.m.tte
combines
to give effactiva seight,
to solself.weighte veight.device) device) self weight
def rescale (excaleeav(self):
telf...to(self veight device) neight (self reight.device) torch. (srl.
device)
self alphatself alpha)
self.m.data
self. weight. x/u
def forvard (self, x):
"""Forvar pass with macked veighte.
sparsaWeight
co(self.reight seight weight.device,, device) self wtw.lole....eeee device) self seight to (aell weight.
return (.conv2d
x,
sparseWeight,
self,
self.stride.
self.padding,
self.dilation.
self.groups.
Listing 1: Conv2d layer with Sign-In parametrization of m @ w.
30

---

N FLOPS required for Sign-In
We perform a FLOP comparison for Sign-In vs. other Pal methods. The FLOP overhead due to the reparameterization in Sign-In is negligible and only required during training, as m O w can be merged after training, during inference.
For example, the FLOPs for a convolutional layer are:
(2 x au Wont Cout x K K Cin) + (Cou Ci K K)
An additional term (Cout Cin K K) introduced by the elementwise product O w, but it negligible with respect to the other term.
We report the FLOPs per forward pass for ResNet50.
Table 15: FLOP Comparison of Sign-In In vs. other Pal methods on ResNet50, Sign-In introduces minimal overhead while improving accuracy.
Method
Training FLOPs (G)
Inference FLOPs (G)
Acc
Dense
8.2
8.2
76.89
Sparse (s = 0.8)
4.99
4.99
73.87
Sign-In (s 0.8)
4.99
74.12
Sparse 0.9)
3.89
3.89
71.56
Sign-In (s = 0.9)
3.9
3,89
72.19
High sparsity regime ablation
In this section we provide results for the high sparsity regime.
Table 16: Sign-In In with Pal methods on CIFARIO and CIFARI00 in the high sparsity regime.
Dataset
Method
0.98
0.99
Random
88.41(+0.23)
86.41(=00.1)
Random Sign-In
89.37
87.83(10.11)
SNIP
88.12(=0.5)
85.82(+0.2)
CIFARIO ResNer20
SNIP Sign-In
89.88 (0.09)
87.27(#0.13)
Synflow
88.15(+0.17)
85.32(+0.24)
Synflow + Sign-In
89.26(+0.05)
86.47
Random
74.01(=0.1)
73.07
Random + Sign-In
75.32(=0.25)
73.94
SNIP
74.08(+0.28)
72.73(11.32)
CIFAR100. ResNer18
SNIP + Sign-In
75.16(+1.15)
73.64(10..2)
Synflow
73.41
71.47
Synflow + Sign-In
75.16(+0.20)
73.64(=0.33)
31


Reviews:
[{"review_id": "5l7uWx7o6p", "reviewer": "Reviewer_A4ar", "summary": "This paper investigates the limitations of Pruning at Initialization (PaI) for sparse neural network training and proposes Sign-In, a novel reparameterization strategy designed to induce sign flips in parameters to improve the alignment between sparse masks and learned weights. The method introduces a dynamic reparameterization which facilitates better optimization geometry and aids in recovering correct parameter signs, especially in sparse settings. The paper supports its claims with both theoretical analysis and empirical evaluations on CIFAR-10/100, ImageNet.", "questions": "Please refer to the strength and weaknesses.", "limitations": "Yes.", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper provides provable guarantees that Sign-In enables successful learning in previously hard-to-optimize configurations.\n\n2. The authors evaluate their method on multiple scales and architectures across different sparsity, which demonstrate robustness.\n\n**Weaknesses**\n\n1. Although the author of the paper claims that the focus of the proposed method is on PaI, but the whole PaI research domain still falls in the degraded performance on model accuracy. My concern in on the motivation of the paper: does improving such method really necessary? Since the improved accuracy by Sign-In is still not comparable to many sparse training methods, the motivation is questionable.\n\n2. The result in the paper is limited. And the existing results show limited improvements against baselines.  In the main paper, it only shows one small table of results, with 3 network structures corresponding to 3 datasets. The baseline method is Random initialization, instead of other more comprehensive PaI methods. For the baseline, it is very obvious that Random initialization has worst performance, such that the proposed method may have the best improvement performance. But if switching to a different method (e.g., SNIP), the results may not that good. Therefore, the demonstration of the results is not convincing. \n\n3. For PaI, the author only shows SNIP, Synflow plus the proposed method. There are many other PaI method, for example, the concurrent work with SNIP, GraSP, is not compared. And many following works are also not included. \n\n4. For Dynamic Sparse Training method like RigL, the improvement is limited (72.02->72.48). The improved accuracy is still not better than MEST (72.58) at same training budget and sparsity. I wonder if applied on even better sparse training method (PaI or DST), does the proposed method still work? This is another concern about the proposed method.\n\n5. The motivation of the method is based on the Lottery Ticket Hypothesis, which is already proved to be falsified by many later works (also cited by the author of the paper in line 110). I may disagree with using it as the motivation or develop rigorous method based on it. The author of the paper should consider using another more rigorous theory.\n\n6. Minor, line 37, concered -> concerned.", "was_revised": true, "final_justification": "The author does not address my concern on research motivation. The necessity of the research is not strong enough. PaI is not a good start point that initialize efficient training. Therefore I don’t support acceptance of this paper.", "rebuttal": "We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.\n\n**Rebuttal of Weaknesses** \n\n1. Pruning at Initialization (PaI) is an active research field (see e.g. [1,2,7]), as solving this problem would be considered a significant breakthrough for multiple reasons: \n    - From a practical side, achieving competitive performance when training a fixed mask from scratch would enable training neural networks in low resource settings (like edge devices) - even lower than what is possible with dynamic sparse training (DST). \n    - Training models locally could also have positive implications for data privacy. \n    - Training a sparse mask would cost less memory and compute (in particular on specialized hardware such as [5]) in comparison to training denser models (or using denser gradients or larger batch sizes as in some DST approaches). The fact that the mask is fixed and known from the start in contrast to DST, would also allow for utilizing optimized kernels (or even hardware) that exploit the specific sparsity patterns to gain computational speedups during forward and backward propagation.\n    - From a conceptual side, improving PaI likely requires novel optimization approaches that provide fundamental insights into the interplay between overparameterization or sparsity and neural network optimization. Our method Sign-In, for instance, addresses the limitation of standard optimizers to effectively learn to switch parameter signs. In overcoming this limitation, it provably solves an optimization problem that is hard in the standard setting. \n\n2. We would like to highlight Table 7 in the appendix, which presents results for Sign-In applied to SNIP and Synflow alongside random masks. In addition to these PaI methods, we also report new results for NPB (a recent PaI method) and RiGL (a dynamic sparse training method) below. \n\n3. Many other methods do not outperform random pruning at initialization (PaI) as shown in [3,4]. For that reason, we focused on demonstrating the effectiveness of Sign-In on random pruning in the main manuscript. With an extra page of space, we would be happy to move our results on SNIP and Synflow to the main part of the paper.\nIn addition, as suggested by the reviewer, we perform experiments on PaI with the NPB method [7], results shown below on both CIFAR10 with a ResNet20 and CIFAR100 with a ResNet18:\n\n| Model              \t| s = 0.8            \t|s =  0.9           | s =  0.95             \t|\n|------------------------|-------------------|------------------|--------------------|\n| NPB C10 Res20      \t| 87.30 ± 0.39  \t| 85.98 ± 0.32 \t| 82.06 ± 0.9 \t|\n| NPB+Sign-In C10 Res20  | 88.87 ± 0.016 \t| 86.88 ± 0.51 \t| 83.37 ± 0.65   \t|\n| NPB C100 Res18     \t| 74.92 ± 0.63  \t| 72.79 ± 0.56 \t| 71.49 ± 0.24   \t|\n| NPB+Sign-In C100 Res18 | 75.59 ± 0.15  \t| 74.72 ± 0.18 \t| 73.28 ± 0.40   \t|\n\n\n4. We assume your comment refers to Table 8 in the appendix. \n    - The purpose of Table 8, is to investigate how Sign-In performs for different fixed mask topologies that are derived from running different sparse training methods. Each method in Table 8 is initialized with a fixed mask which has been identified by a specified pruning method and then trained from scratch. The experiment is designed like this to assess how close we come to solving PaI (where only the mask is given). Matching a target performance (obtained by the sparse training method) would mean solving PaI. Hence, the numbers in Table 7 highlight this gap between PaI and other sparse training methods like RiGL or ACDC. The actual performances for these methods are reported in Table 1 for comparison (which for RiGL is 73.75%).\n    - As per your comment, we can still ask the question if our method Sign-In could improve dynamic sparse training. To this end, the table below for ResNet50 on ImageNet shows that Sign-In can boost RiGL especially at high sparsity. \n    - It can also improve Mest, as we demonstrate with a ResNet50 trained on ImageNet.\n\n| Method     \t| s = 0.8        \t| s = 0.9           | s = 0.95          |\n|----------------|---------------|-------------------|------------------|\n| RiGL       \t| 75.02 ± 0.1   | 73.70 ± 0.2   \t| 71.89 ± 0.07 \t|\n| RiGL + Sign-In | 75.02 ± 0.1   | 74.27 ± 0.08 \t| 73.07 ± 0.17 \t|\n\n\n| Method     \t| s = 0.8       | s = 0.9       | s = 0.95          |\n|----------------|---------------|-------------------|------------|\n| Mest       \t|  74.67  | 73.1  | 70.84 |\n| Mest + Sign-In |  74.74 | 73.32  | 71.41 |\n\n\n5. Neither the premise of our work nor our theory is based on the lottery ticket hypothesis (LTH). \n    - We cite the LTH as an important line of work that highlights the difficulty of training sparse neural networks from scratch. The fact that lottery tickets might not exist or are more difficult to find in large-scale settings supports our point. [6] hypothesizes that the main challenge for lottery tickets is in fact to learn parameter sign flips. It argues that learning rate rewinding (LRR), which gives up on the lottery ticket idea, succeeds by improved sign identification. Table 1 demonstrates that the superior sign learning ability is actually a more universal property of contemporary sparsification and DST methods, which do not find lottery tickets but utilize different forms of overparameterization. \n    - Our proposed parameterization Sign-In further boosts sign learning. Our theory even proves this in the setting of [6].\nAnother way to frame our innovations could be to cast them as exploration of controllable Riemannian gradient descent for better generalization. While we demonstrate its utility for PaI, the tools can also be used for different purposes.\n\n6. We will correct the typo in the revised manuscript. Thank you for pointing it out.\n\nIn summary, we have argued why PaI is a worthwhile endeavour to enable computing on edge devices and gain theoretical insights into the role of overparameterization in (sparse) deep learning. \nMoreover, we have referred to more results with other PaI methods (Sign-In applied to SNIP and Synflow in Table 7) and added experiments with NPB and the dynamic sparse training (DST) method RiGL, finding that our method Sign-In leads to consistent improvements.\nFurthermore, we have clarified our theoretical foundation and motivated it from an optimization angle.\n\nWe sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications upon request.\n\n\n[1] Iurada, Leonardo, Marco Ciccone, and Tatiana Tommasi. \"Finding lottery tickets in vision models via data-driven spectral foresight pruning.\" CVPR 2024.\n\n[2] Adnan, Mohammed et al. “Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry.” ICML 2025\n\n[3] Liu, Shiwei, et al. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. ICLR 2022\n\n[4] Gadhikar, Advait Harshal, Sohom Mukherjee, and Rebekka Burkholz. \"Why Random Pruning Is All We Need to Start Sparse.\" ICML 2023.\n\n[5] Thangarasa, Vithursan, et al. \"Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency.\" International Conference on Machine Learning. PMLR, 2024.\n\n[6] Gadhikar, A. H., & Burkholz, R. (2024). Masks, signs, and learning rate rewinding. ICLR 2024.\n\n[7] Pham, H., Ta, T. A., Liu, S., Xiang, L., Le, D., Wen, H., & Tran-Thanh, L. Towards data-agnostic pruning at initialization: What makes a good sparse mask?  NeurIPS 2023"}, {"review_id": "0mNSNwnTJm", "reviewer": "Reviewer_sXcm", "summary": "The paper studies the performance gap between pruning at initialization (PaI) methods and methods termed dense-to-sparse, in the sparse training problem domain. The authors identify sign stabilization as a distinguishing factor between the two classes of methods, where sign flips occur consistently throughout training in PaI methods while they stabilize in dense-to-sparse methods. Additionally, the authors show that the signs (identified early) alone with the final mask (fixed) is enough to recover the performance in dense-to-sparse methods, while the mask with random weights is not. \n\nSing-in is proposed as a method to reparameterize the fixed PaI masks to find the optimal sign assignment during training as compared to the standard training procedure. The experiments and results show that sign-in results in significant improvements in test accuracy for the PaI masks. Additionally, the authors show that the use of sign-in results in flatter minima by computing the sharpness of the loss landscape.", "questions": "- Can it be the case that the ability to change the connectivity and thereby starting from 0 in the newly added weights, allows the network to implicitly change or shift the signs in dynamic sparse training methods such as RiGL?\n\n- Does sign flipping help in a very high sparsity regime? (0.98, 0.99) It may be so that sign flipping is a helpful tool when even the sparse mask is over-parameterized, but not in those high sparsity regime where mask structure may become very important.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is very well written and easy to follow, while making significant contributions towards the issue of the performance gap between PaI methods and dense-to-sparse methods. \n\n- The experiments are very well conducted, including the ones that show preliminary results of the sign stabilization, the importance of signs learned against the weight magnitudes in recovering the performance. \n\n- The proposed method results in significant gains over the PaI methods, which is a strong result. \n\n**Weaknesses:**\n\n- Study of signs and their importance in lottery ticket initialization has been previously studied, which limits the contributions. However, I do note that the authors make significant contributions in terms of signs learned, the associated phases, and the proposed method itself. \n\n**Comments:**\n\n- Additionally, regarding “dense-to-sparse” methods. The experiments and results provided in the paper are for sparse training methods, the difference being the ability to modify or update the mask during training, albeit while using some form of dense gradients or training iterations - AC/DC being alternate, and RiGL using dense gradients. The authors should move to standard nomenclature in the literature being PaI-Pruning at Initialization and sparse training. \n\n- There seems to be a missing link between PaI masks at initialization as compared to the result shown that signs are more important than weight magnitudes in dense-to-sparse methods in section 2. The authors should consider an experiment where PaI masks are initialized with some “good signs” to show a clear connection between signs at initialization improving performance. \n\n- I appreciate the authors' discussion on computation and memory costs associated with sign-in. The authors should consider the following plot, x-axis being the computation budget in terms of FLOPs, and y-axis being the test accuracy, and scatter plot the results for all sparsity levels and all the methods. This gives a better picture of the budget as compared to the performance trade-off.  \n\n- The authors should consider adding the comparison with dense-to-sparse methods in the main part with dense-to-sparse+Sign-in, PaI, and PaI+Sign-in. This is a requirement since all those methods operate in a “similar” problem space where the training efficiency is paramount. \n\n- Minor: The earlier parts of the paper including the introduction can be made much simpler, coherent and less dense. A small suggestion would be not to highlight all the results, (theoretic or empirical) as it is uneasy to follow those if the reader is already not familiar with the entire paper.", "was_revised": true, "final_justification": "I support accepting this paper based on the authors' rebuttal, additional results and their proposed changes. My previous concerns have been addressed through: clarifying the proposed method as an orthogonal tool, and adding discussion about optimal sparsity levels including discussion and ideas to identify effective sparsity thresholds. These modifications will significantly enhance the paper's clarity and highlights the contributions.", "rebuttal": "We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. We appreciate that you find our work well written, easy to follow and consider our results to be strong.\nPlease find a detailed point-by-point response below. In case of any open questions, we would be happy to discuss them.\n\n**Rebuttal of Comments:**\n1.  We are happy to switch to the more common terminology of the field and mainly distinguish sparse training methods and pruning at initialization.\n\n2. By choosing the signs corresponding to warm-up or some preliminary epoch we are effectively selecting partially correct signs. To analyze the dependence on partially correct signs in more detail, we have performed additional experiments where we vary the number of correct signs for a mask identified by ACDC on a ResNet50 for ImageNet. The table below shows that fewer correct signs at initialization progressively degrade performance.\n\n| Method                     \t| s=0.9   |\n|--------------------------------|---------|\n| ACDC baseline (from Table 1) \t| 74.68\t|\n| ACDC Mask + 100% correct signs (from Table 1) | 74.88 |\n| ACDC Mask + 95% correct signs  |  74.44 |\n| ACDC Mask + 90% correct signs  | 73.54    |\n| ACDC Mask + 80% correct signs  | 71.81 |\n| ACDC Mask + random signs (from Table 1)  | 70.88 |\n\n3. We would be happy to include such a figure in the main manuscript. Table 12 shows that Sign-In’s added overhead is negligible compared to the method it is applied to. Therefore, Sign-In pushes the Pareto frontier of PaI, as it achieves significantly better generalization with a similar overhead.\n\n4. The table below presents the effect of combining Sign-In with RiGL for ResNet50 on ImageNet. We conclude that also dynamic masks can benefit from improved sign learning at higher sparsity. \n\n| Method     \t| s = 0.8        \t| s = 0.9           | s = 0.95          |\n|----------------|---------------|-------------------|------------------|\n| RiGL       \t| 75.02 ± 0.1   | 73.70 ± 0.2   \t| 71.89 ± 0.07 \t|\n| RiGL + Sign-In | 75.02 ± 0.1   | 74.27 ± 0.085 \t| 73.07 ± 0.17 \t|\n\n5. We would be happy to sparsify the introduction by putting less emphasis on all the individual results in the revised manuscript. \n\nQuestions:\n\n1. That is a really interesting idea. We also believe that improved sign learning could be an important mechanism contributing to the success of RiGL. To analyze this hypothesis, we have perform RiGL with two ablations in addition to re-initializing weights with 0. They are a) initializing with the weight value at pruning and b) switching the sign of the weight upon re-initalizing. We find that both perform worse than initializing at 0. This is in line with the findings of the authors of RiGL and in line with your hypothesis that initialization at 0 promotes flexibility of parameters to learn signs. (Furthermore, it perturbs the model less.) Interestingly, reinitializing with the opposite sign performs better that the original sign, suggesting that a sign flip could be helpful. We report these results for a ResNet50 on ImageNet at 90% sparsity.\n\n| RiGL ablation           | s=0.9   |\n|------------------------|---------|\n| Reinit with Zero (baseline)   \t| 73.75   |\n| Reinit with Opposite Sign | 71.6  |\n| Reinit with Same Sign    | 69.51 |\n\nIn addition to the mechanism of pruning and regrowing in RiGL, our results suggest that training RiGL in combination with Sign-In can lead to additional gains especially at higher sparsity (see previous section on weaknesses), as also the weights that are not rewired learn signs more effectively.\n\n\n2. Sign-In also helps in the high sparsity regime. Inevitably, there will be a point where the connectivity between layers will break down depending on mask topology. As we see in experiments (Table 3 and 7), the performance gains are comparable for different sparsity levels. Sign-In cannot counteract the connectivity issue. However, as Sign-In is agnostic to the mask structure and the correct signs become more important for sparser masks, Sign-In should lead to similar gains in the high sparsity regime. To verify this intuition, we have conducted an additional experiment for random pruning at initialization, SNIP, and Synflow on CIFAR10 and CIFAR100. According to the table below, Sign In still outperforms the baselines. However as expected at 99% sparsity, due to mask topology instability, the differences between Sign-In and the baseline are within noise.\n\n\n| Method              | s=0.98               | s=0.99               |\n|---------------------|------------------|------------------|\n| Random c10          | 78.44 ± 0.13     | 68.56 ± 0.49     |\n| Random c10 Sign-In  | 78.78 ± 0.14     | 69.17 ± 0.29     |\n| SNIP c10            | 74.08 ± 0.72     | 65.83 ± 0.83     |\n| SNIP c10 Sign-In    | 74.22 ± 1.31     | 64.93 ± 1.13     |\n(Synflow failed)\n\n| Method                 | s=0.98               | s=0.99               |\n|------------------------|------------------|------------------|\n| Random c100            | 69.44 ± 0.47     | 67.13 ± 0.30     |\n| Random c100 Sign-In    | 70.14 ± 0.23     | 68.23 ± 0.67     |\n| SNIP c100              | 68.54 ± 0.42     | 64.74 ± 1.45     |\n| SNIP c100 Sign-In      | 69.73 ± 0.77     | 64.81 ± 0.63     |\n| Synflow c100           | 64.82 ± 0.35     | 59.57 ± 0.46     |\n| Synflow c100 Sign-In   | 65.92 ± 0.30     | 59.56 ± 0.80     |\n\n\n\nWe sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications."}, {"review_id": "BBtzJdZwzP", "reviewer": "Reviewer_G1i7", "summary": "The paper demonstrates the importance of learning signs in training a pruned model through theoretical analysis and experiments. It also proposes a method that promotes sign learning via reparameterization. Experiments show that this method improves performance across datasets and confirms that it encourages sign flips during training.", "questions": "As the authors mentioned, the proposed method is orthogonal to existing techniques. Therefore, it would be worthwhile to explore how much additional gain it can provide when combined with some of the most advanced methods, which can help assess its potential for further improving the sota in practice.", "limitations": "Yes. The authors have discussed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1. The paper presents experiments showing evidence for the importance of learning signs in sparse training. It also shows that, in dense-to-sparse training, signs tend to stabilize early, whereas in sparse training they do not.\n\n2. It proposes a new method that encourages the learning of signs and demonstrates through experiments that this method improves performance.\n\n\nWeaknesses:\n\n1. There has been prior work investigating the role of signs as early as 2019 [1], with findings that appear highly relevant to the observations made in this paper. There may also be more recent follow-up studies on the topic. However, the paper does not discuss any connections to these earlier works, making it unclear whether the findings—particularly those in Section 3—are novel or what they contribute beyond known results.\n\n2. The motivation behind the proposed method lacks clarity, and the logical connection to the preceding analysis is not well established. Specifically, there appears to be a discrepancy between the problem characterization in Section 3 and the justification for the proposed approach. Section 3 highlights that, in sparse training, signs do not stabilize after the warm-up phase and continue to flip—unlike in dense-to-sparse training, where signs stabilize early. This suggests that the core issue lies in the lack of sign stability. However, the intuition provided—both in the text and in the theoretical analysis—for the proposed sign-in method is that it encourages sign flipping, as also shown empirically in Figure 4. This seems to contradict the earlier analysis, which identifies excessive sign flipping as a problem. It remains unclear how encouraging more sign flips helps address the issue of instability. There needs to be a clearer explanation or analysis of how the observations regarding sign instability is leveraged/how the proposed method connects to these observations, and why promoting sign changes would be beneficial in this context.\n\n\n3. In the current experiments, it is unclear how effective the method is in advancing state-of-the-art performance or how it compares to results reported in prior work, as the only comparison is against a naive baseline. Although the method is orthogonal to existing approaches for sparse training, it would still be valuable to include comparisons with stronger baselines, or explore combinations with other methods, and conduct ablation studies to better understand its position and practical utility. This is particularly important given that the primary focus of the paper is to introduce a new method. Demonstrating its practical value more convincingly would strengthen the contribution.\n\n[1] Zhou, Hattie, et al. \"Deconstructing lottery tickets: Zeros, signs, and the supermask.\" Advances in neural information processing systems 32 (2019).", "was_revised": true, "final_justification": "My main concerns have been adequately addressed during the rebuttal. I also reviewed the other reviewers' feedback. The third point raised in Reviewer sXcm's last comment—regarding performance in the very high sparsity regime—could be a limitation. Other than that, I did not notice any points that I consider to be significant issues. Therefore, I have adjusted my score accordingly.", "rebuttal": "We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Please find a detailed point-by-point response below. In case of any open questions, we would be happy to discuss them.\n\n**Rebuttal of Weaknesses**\n\n1. We thank you for pointing us to the related work [1]. While [1] draws the insight that the signs learnt by Iterative Magnitude Pruning (IMP) and thus the initial signs are informative on small scale experiments (CIFAR10), our work is concerned with the question when signs become informative during (and the end of) training, how something similar can be achieved by PaI also in the context of larger scale experimental settings (like ImageNet). Our work is closer to [4], which analyzes how this sign learning mechanism enables LRR to outperform IMP. As mentioned, we show that this insight is more universal and holds for various mask topologies and sparsification methods. We will be happy to add this discussion.\n\n2. As you have pointed out, dense to sparse methods flip parameter signs early in training, which stabilize during subsequent training. \n   - However, this is not the case for PaI methods as highlighted by Theorem 5.4, due to the lack of overparameterization. To mitigate this, according to Theorem 5.2, Sign-In introduces an orthogonal sign-flipping mechanism that enables useful sign flips also in the absence of overparameterization. \n   - Hence, in the case of PaI, where early sign flipping followed by sign stabilization is not possible, Sign-In induces a mechanism to learn the correct sign flips (as shown in Figure 2). \n   - Specifically, Sign-In facilitates more useful sign flips, as shown in the toy example by recovering the ground truth. \n   - To motivate this further, consider our theoretical example, where we want to train a single neuron $a \\phi( w x)$, where the sign of $a \\in \\mathbb{R}$ is wrongly initialized. The parameter $a$ cannot be flipped due to the balancedness property. Nevertheless, all parameters in the weight vector $w \\in \\mathbb{R}^d$ have a chance of flipping while the activation stays non-zero for some data points. This will not necessarily bring us close to the ground truth but will lead to more sign flips (in $w$)  as in line with the general instability of the sign flipping in PaI, whereas Sign-In allows for a correct additional sign flip.\n   - Therefore we use the amount of sign flips as a proxy for the amount of additional correct sign flips on top. Further experiments with ACDC + Sign-In (Table 10) and Rigl + Sign-In (shown below) show that Sign-In performs useful sign flips which are orthogonal to the sign stabilizing mechanism in dense-to-sparse methods.\n   - So the flow of argumentation is that we first showed empirically that correct signs plus mask is enough for recovering dense performance then this is followed by a method that can provably induce a part of these correct sign flips.\n\n3. The additional experiments below highlight the utility of Sign-In for state-of-the-art dynamic sparse training methods like RiGL and Mest. On Imagenet with a ResNet50, we see relevant improvements starting from the 90% sparsity level, where the sign optimization starts to matter more.\n\n| Method     \t| s = 0.8        \t| s = 0.9           | s = 0.95          |\n|----------------|---------------|-------------------|------------------|\n| RiGL       \t| 75.02 ± 0.1   | 73.70 ± 0.2   \t| 71.89 ± 0.07 \t|\n| RiGL + Sign-In | 75.02 ± 0.1   | 74.27 ± 0.085 \t| 73.07 ± 0.17 \t|\n\n| Method     \t| s = 0.8       | s = 0.9       | s = 0.95          |\n|----------------|---------------|-------------------|------------------|\n| Mest       \t|  74.67  | 73.1  | 70.84 |\n| Mest + Sign-In |  74.74 | 73.32  | 71.41 |\n\n\nFurthermore, we show the utility of Sign-In with other PaI methods such as SNIP and Synflow in Table 7 in Appendix H. Note that [2,3] showed that the random baseline often performs better, which was the main reason to highlight it in the main part of the manuscript. We would be happy to move the additional results on SNIP and Synflow to the main part of the manuscript given an extra page.\nAdditionally, we have conducted experiments with the recently proposed PaI method NPB [5], as reported below.\n\n| Model              \t| s = 80%            \t|s =  90%           \t|s =  95%             \t|\n|------------------------|-------------------|------------------|--------------------|\n| NPB C10 Res20      \t| 87.30 ± 0.39  \t| 85.98 ± 0.32 \t| 82.06 ± 0.9 \t|\n| NPB+Sign In C10 Res20  | 88.87 ± 0.016 \t| 86.88 ± 0.51 \t| 83.37 ± 0.65   \t|\n| NPB C100 Res18     \t| 74.92 ± 0.63  \t| 72.79 ± 0.56 \t| 71.49 ± 0.24   \t|\n| NPB+Sign In C100 Res18 | 75.59 ± 0.15  \t| 74.72 ± 0.18 \t| 73.28 ± 0.40   \t|\n\nQuestion\n\nWe report results with additional PaI methods as well as highlight that Sign-In can improve dynamic sparse training methods like RiGL via its sign flipping mechanism. Similar improvements on sparsification with ACDC are also reported in Table 10.\n\n\nWe sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications.\n\n[1] Zhou, Hattie, et al. \"Deconstructing lottery tickets: Zeros, signs, and the supermask.\" NeurIPS 2019.\n\n[2] Liu, Shiwei, et al. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. ICLR 2022.\n\n[3] Gadhikar, Advait Harshal, Sohom Mukherjee, and Rebekka Burkholz. \"Why Random Pruning Is All We Need to Start Sparse.\" ICML 2023. \n\n[4] Gadhikar, A. H., & Burkholz, R. (2024). \"Masks, signs, and learning rate rewinding.\" ICLR 2024. \n\n[5] Pham, H., Ta, T. A., Liu, S., Xiang, L., Le, D., Wen, H., & Tran-Thanh, L. (2023). Towards data-agnostic pruning at initialization: What makes a good sparse mask? NeurIPS 2023."}, {"review_id": "liEzKuTlwt", "reviewer": "Reviewer_DCG2", "summary": "This paper explores the role of sign-flipping dynamics in sparse training, proposing a reparameterization approach to enhance early, stable sign alignment. The authors argue that such alignment is a crucial factor distinguishing dense-to-sparse methods from sparse-from-scratch approaches. Their empirical findings demonstrate substantial performance gains in classification tasks across CIFAR-10, CIFAR-100, and ImageNet, while a theoretical analysis of a toy model further reinforces the connection between sign flips and convergence in overparameterized networks.", "questions": "In Table 1, please confirm that the reported numbers are indeed the final Top-1 classification accuracies for ImageNet with 90% sparsity.\n\nCould you elaborate on the behavior of Equation (1) (See weaknesses)?\n\nDo you have experiments that show the behavior for different values of $\\beta$?\n\n​While the paper shows that early sign alignment correlates with better performance, what happens when you directly manipulated sign flips to see how it affects accuracy?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "$\\textbf{STRENTHS}$\n\n-Novel Insight: The paper offers an interesting and novel perspective on why dense-to-sparse methods outperform sparse-from-scratch, highlighting the role of early sign alignment in sparse networks.\n\n-Empirical Validation: The extensive experiments across multiple datasets convincingly support the claim that early sign flipping is critical for performance in sparse settings.\n\n-Theoretical Support: The use of a toy single-neuron model in Theorem 5.1 provides a clear and intuitive illustration of the sign-flipping dynamics and their potential benefits.\n\n-Clarity in Main Sections: The introduction and main experimental results are generally well written and easy to follow.\n\n$\\textbf{WEAKNESSES}$\n\n-Clarity of Key Results: Figure 1 does not clearly state that it reports sign flip dynamics, not accuracy values, which could confuse readers. Additionally, Table 1 does not explicitly label its entries as final Top-1 classification accuracies on ImageNet (90% sparsity), making it harder to interpret.\n\n-Classification of Methods: The distinction between dense-to-sparse and sparse-from-scratch methods in Figure 1 (the first three bars vs. the rest) is not explicitly named, limiting interpretability.\n\n-Acronym Usage: The acronym “PaI” is introduced in the title and abstract without being defined, which could be confusing for readers unfamiliar with the term.\n\n-Sign Flipping Mechanism (Equation 1): There seems to be a potential issue when the parameter $\\phi_t$  is large and negative, and the gradient is positive. In this case, the reparameterized update direction appears to move  $\\phi_t$\n  further negative, which seems counter to the intended sign-flipping mechanism. Clarification on how this behavior is mitigated in practice would strengthen the paper’s claims.\n\n-Single Neuron Clarification: In Theorem 5.1 and its discussion, it would be helpful to explicitly state that both the student and teacher networks are single-neuron (1–1–1) networks, to avoid any ambiguity and help readers understand the simplicity of the toy model used.\n\n-I feel like there would have been better experiments than the one in Fig 1 (see questions section).\n\n-The paper has very little theory.", "was_revised": true, "final_justification": "I think it's a really nice paper. Based on the rebuttal I'm happy to change to accept.", "rebuttal": "We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. We appreciate your positive comments regarding the clarity of our work, theoretical support and extensive experiments. Please find a detailed point-by-point response to your comments below. In case of any open questions, we would be happy to discuss them.\n\n**Rebuttal of Weaknesses**\n\n1. In the revised manuscript, we would be happy to extend the table and figure description as suggested to highlight sign flipping dynamics and that the reported results are for Top-1% accuracy on ImageNet.\n\n2. To make the distinction between DST and PaI in Figure 1 clearer, we will add two headers saying DST and PaI. Moreover, we will make the current vertical line that indicates the distinction bolder.\n\n3. We agree with the reviewer and will replace “training sparse neural networks from scratch (PaI)” with the full name Pruning at Initialization (PaI) in the abstract and define it early on. \n\n4. If the gradient is positive ($\\nabla_{\\theta}f > 0$) and the parameter theta is negative ($\\theta < 0$), this means that the sign is perceived to be already correct (assuming that the gradient is pointing in the correct direction). This is also true for standard gradient descent. Note that Sign-In controls the relative magnitude but not the direction itself. In practice, we would work with stochastic gradient estimates. Noisy gradients could push the parameter away from zero wrongly then, as in case of standard gradient descent. Sign-In would simply not add merit in this case. \nThe main benefit of Sign-In is realized in the case when the gradient and parameter are both pointing in the same direction (eg: $\\nabla_{\\theta}f > 0, \\theta > 0$), which would imply that the gradient is pushing the parameter towards 0 (because $\\theta \\leftarrow \\theta - \\eta \\nabla_{\\theta}f$). Sign-In speeds up this movement towards 0 and promotes thus a sign flip.\n\n5. We will explicitly state that both the teacher and student are single neuron neural networks in a revision of our manuscript.\n\n6. To strengthen the our conclusions from Figure 1, we provide additional experiments by directly manipulating the sign as suggested by the reviewer. We also report results where Sign-In improves state-of-the-art pruning methods like dynamic sparse training with RiGL. See tables below.\n\n7. Our theory combines insights from two lines of work, namely, Riemannian gradient flows induced by overparameterization and dynamical systems theory. The combination of both is a novel idea and requires non-trivial proof steps. Moreover, the theory captures and provides insights into a novel practical algorithm, Sign-In, which translates to large scale experiments and presents a step forward towards solving the PaI problem. For that reason, we see potential that our conceptual insights could inspire further theoretical investigations and new method design.\n\n**Answers to Questions:**\n\n1. Yes these are indeed the top 1% accuracy values. We will make sure that this is clear from the table description in the revised manuscript.\n\n2. Please see Point 4 in weaknesses (above).\n\n3. Different $\\beta$ would lead to different effective learning rates (see Eq.(1) for example). Setting $\\beta =1$ allows us to keep the effective learning rate close to the original learning rate of standard optimizers. To illustrate this point, we have conducted an experiment for different $\\beta$ on CIFAR100 with a Resnet18 for random pruning with $90$% sparsity:\n\n| $\\mathbf{\\beta}$ | s=0.9 |\n|------------|---------------|\n| 0.2    \t| 73.26 ± 0.17  |\n| 0.5    \t| 73.95 ± 0.04 |\n| 1      \t| 73.94 ± 0.13  |\n| 4      \t| 73.99 ± 0.38  |\n| 20     \t| 73.03 ± 0.29  |\n\nWe observe that higher $\\beta$ could lead to slightly better performance, most likely due to more opportunities for correct sign flips. Yet, it also induces a higher variance due to the higher effective learning rate. Across different experiments, $\\beta=1$ required less tuning of other hyperparameters. Hence, we choose this value for $\\beta$ in all our experiments.\n\n4. In Table 1 in the main text, we have studied the effect of manipulating signs at initialization. In principle, it might also be possible to provide the right signs later during training, as parameters need to move less given the right signs. However, this is an orthogonal question to solving PaI.\n\nIn addition, we report results for manipulating the signs as suggested by the reviewer, extending the results from Table 1 for ACDC. We find the reducing the number of correct signs progressively worsens performance.\n\n| Method                     \t| s=0.9   |\n|--------------------------------|---------|\n| ACDC baseline (from Table 1) \t| 74.68\t|\n| ACDC Mask + 100% correct signs (from Table 1) | 74.88 |\n| ACDC Mask + 95% correct signs  |  74.44 |\n| ACDC Mask + 90% correct signs  | 73.54    |\n| ACDC Mask + 80% correct signs  | 71.81 |\n| ACDC Mask + random sign (from Table 1)  | 70.88 |\n\nTo strengthen our point that Sign-In improves contemporary sparse training methods, we provide experiments with RiGL + Sign-In. This highlights that learning with a dynamic mask as in RiGL, Sign-In also helps with the sign flipping mechanism, which can improve performance at high sparsity, especially. \n\n| Method     \t| s = 0.8        \t| s = 0.9           | s = 0.95          |\n|----------------|---------------|-------------------|------------------|\n| RiGL       \t| 75.02 ± 0.1   | 73.70 ± 0.2   \t| 71.89 ± 0.07 \t|\n| RiGL + Sign-In | 75.02 ± 0.1   | 74.27 ± 0.085 \t| 73.07 ± 0.17 \t|\n\n\nWe sincerely thank you for your thoughtful feedback and constructive suggestions, which have improved our paper. We are looking forward to the discussion period and would be happy to address further questions and provide additional clarifications."}]

Author rebuttal:

Meta-review: The authors explore the role of signs in both sparse training and pruning at initialization (PaI), and propose a method (Sign-In) to induce sign flips in parameters to improve the alignment of masks for sparse training. While there is previous literature exploring signs and mask alignment in the context of sparse training, the authors explore a different context than previous work and provide analysis on PaI and Dynamic Sparse Training (DST) methods. 

The reviewers outlined the strengths of the work in the experimental evaluation, the improved performance of Sign-In over existing PaI baselines, and the strong theoretical backing for the method. While the reviewers did initially note several important weaknesses in the work, the discussion and rebuttal addressed almost all of the reviewers' concerns, with the exception perhaps of those of Reviewer A4ar. I would like to thank all the reviewers and the authors for their productive and extensive discussions, and their genuine engagement, during the rebuttal period which it appears to have improved the work substantially, which is the best outcome one can hope for from peer review.

It appears Reviewer A4ar's main issue with the work that remains unaddressed through the rebuttal is with the potential impact of the PaI research field itself in the context of much stronger sparse training (e.g. DST) baselines, and by extension, a lack of impact and audience for the author's work. The potential impact of PaI is of course debatable, and Reviewer A4ar's skepticism of PaI in general is no doubt shared amongst many within the research community (perhaps even myself). However, regardless of where one stands on PaI, I believe it is clear that there is a significant audience within the ML research community interested in PaI, and this work shows significant improvements on existing PaI baselines which will be of interest to many. Moreover, I believe it's important to point out that the authors demonstrate insights and improvements to DST/sparse training methods including RiGL and MEST, showing the work has a much broader reach and potential impact than PaI alone.
Decision: Accept (poster)