Title: Sign-In to the Lottery: Reparameterizing Sparse Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: pruning at initialization

sparse training

lottery ticket hypothesis

mirror flow

reparameterization

sign flips
Abstract: The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the role of sign-flipping dynamics in sparse training, proposing a reparameterization approach to enhance early, stable sign alignment. The authors argue that such alignment is a crucial factor distinguishing dense-to-sparse methods from sparse-from-scratch approaches. Their empirical findings demonstrate substantial performance gains in classification tasks across CIFAR-10, CIFAR-100, and ImageNet, while a theoretical analysis of a toy model further reinforces the connection between sign flips and convergence in overparameterized networks.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{STRENTHS}$\n\n-Novel Insight: The paper offers an interesting and novel perspective on why dense-to-sparse methods outperform sparse-from-scratch, highlighting the role of early sign alignment in sparse networks.\n\n-Empirical Validation: The extensive experiments across multiple datasets convincingly support the claim that early sign flipping is critical for performance in sparse settings.\n\n-Theoretical Support: The use of a toy single-neuron model in Theorem 5.1 provides a clear and intuitive illustration of the sign-flipping dynamics and their potential benefits.\n\n-Clarity in Main Sections: The introduction and main experimental results are generally well written and easy to follow.\n\n$\\textbf{WEAKNESSES}$\n\n-Clarity of Key Results: Figure 1 does not clearly state that it reports sign flip dynamics, not accuracy values, which could confuse readers. Additionally, Table 1 does not explicitly label its entries as final Top-1 classification accuracies on ImageNet (90% sparsity), making it harder to interpret.\n\n-Classification of Methods: The distinction between dense-to-sparse and sparse-from-scratch methods in Figure 1 (the first three bars vs. the rest) is not explicitly named, limiting interpretability.\n\n-Acronym Usage: The acronym “PaI” is introduced in the title and abstract without being defined, which could be confusing for readers unfamiliar with the term.\n\n-Sign Flipping Mechanism (Equation 1): There seems to be a potential issue when the parameter $\\phi_t$  is large and negative, and the gradient is positive. In this case, the reparameterized update direction appears to move  $\\phi_t$\n  further negative, which seems counter to the intended sign-flipping mechanism. Clarification on how this behavior is mitigated in practice would strengthen the paper’s claims.\n\n-Single Neuron Clarification: In Theorem 5.1 and its discussion, it would be helpful to explicitly state that both the student and teacher networks are single-neuron (1–1–1) networks, to avoid any ambiguity and help readers understand the simplicity of the toy model used.\n\n-I feel like there would have been better experiments than the one in Fig 1 (see questions section).\n\n-The paper has very little theory.\n\nQUESTIONS:\nIn Table 1, please confirm that the reported numbers are indeed the final Top-1 classification accuracies for ImageNet with 90% sparsity.\n\nCould you elaborate on the behavior of Equation (1) (See weaknesses)?\n\nDo you have experiments that show the behavior for different values of $\\beta$?\n\n​While the paper shows that early sign alignment correlates with better performance, what happens when you directly manipulated sign flips to see how it affects accuracy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates the importance of learning signs in training a pruned model through theoretical analysis and experiments. It also proposes a method that promotes sign learning via reparameterization. Experiments show that this method improves performance across datasets and confirms that it encourages sign flips during training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The paper presents experiments showing evidence for the importance of learning signs in sparse training. It also shows that, in dense-to-sparse training, signs tend to stabilize early, whereas in sparse training they do not.\n\n2. It proposes a new method that encourages the learning of signs and demonstrates through experiments that this method improves performance.\n\n\nWeaknesses:\n\n1. There has been prior work investigating the role of signs as early as 2019 [1], with findings that appear highly relevant to the observations made in this paper. There may also be more recent follow-up studies on the topic. However, the paper does not discuss any connections to these earlier works, making it unclear whether the findings—particularly those in Section 3—are novel or what they contribute beyond known results.\n\n2. The motivation behind the proposed method lacks clarity, and the logical connection to the preceding analysis is not well established. Specifically, there appears to be a discrepancy between the problem characterization in Section 3 and the justification for the proposed approach. Section 3 highlights that, in sparse training, signs do not stabilize after the warm-up phase and continue to flip—unlike in dense-to-sparse training, where signs stabilize early. This suggests that the core issue lies in the lack of sign stability. However, the intuition provided—both in the text and in the theoretical analysis—for the proposed sign-in method is that it encourages sign flipping, as also shown empirically in Figure 4. This seems to contradict the earlier analysis, which identifies excessive sign flipping as a problem. It remains unclear how encouraging more sign flips helps address the issue of instability. There needs to be a clearer explanation or analysis of how the observations regarding sign instability is leveraged/how the proposed method connects to these observations, and why promoting sign changes would be beneficial in this context.\n\n\n3. In the current experiments, it is unclear how effective the method is in advancing state-of-the-art performance or how it compares to results reported in prior work, as the only comparison is against a naive baseline. Although the method is orthogonal to existing approaches for sparse training, it would still be valuable to include comparisons with stronger baselines, or explore combinations with other methods, and conduct ablation studies to better understand its position and practical utility. This is particularly important given that the primary focus of the paper is to introduce a new method. Demonstrating its practical value more convincingly would strengthen the contribution.\n\n[1] Zhou, Hattie, et al. \"Deconstructing lottery tickets: Zeros, signs, and the supermask.\" Advances in neural information processing systems 32 (2019).\n\nQUESTIONS:\nAs the authors mentioned, the proposed method is orthogonal to existing techniques. Therefore, it would be worthwhile to explore how much additional gain it can provide when combined with some of the most advanced methods, which can help assess its potential for further improving the sota in practice.\n\nLIMITATIONS:\nYes. The authors have discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the performance gap between pruning at initialization (PaI) methods and methods termed dense-to-sparse, in the sparse training problem domain. The authors identify sign stabilization as a distinguishing factor between the two classes of methods, where sign flips occur consistently throughout training in PaI methods while they stabilize in dense-to-sparse methods. Additionally, the authors show that the signs (identified early) alone with the final mask (fixed) is enough to recover the performance in dense-to-sparse methods, while the mask with random weights is not. \n\nSing-in is proposed as a method to reparameterize the fixed PaI masks to find the optimal sign assignment during training as compared to the standard training procedure. The experiments and results show that sign-in results in significant improvements in test accuracy for the PaI masks. Additionally, the authors show that the use of sign-in results in flatter minima by computing the sharpness of the loss landscape.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper is very well written and easy to follow, while making significant contributions towards the issue of the performance gap between PaI methods and dense-to-sparse methods. \n\n- The experiments are very well conducted, including the ones that show preliminary results of the sign stabilization, the importance of signs learned against the weight magnitudes in recovering the performance. \n\n- The proposed method results in significant gains over the PaI methods, which is a strong result. \n\n**Weaknesses:**\n\n- Study of signs and their importance in lottery ticket initialization has been previously studied, which limits the contributions. However, I do note that the authors make significant contributions in terms of signs learned, the associated phases, and the proposed method itself. \n\n**Comments:**\n\n- Additionally, regarding “dense-to-sparse” methods. The experiments and results provided in the paper are for sparse training methods, the difference being the ability to modify or update the mask during training, albeit while using some form of dense gradients or training iterations - AC/DC being alternate, and RiGL using dense gradients. The authors should move to standard nomenclature in the literature being PaI-Pruning at Initialization and sparse training. \n\n- There seems to be a missing link between PaI masks at initialization as compared to the result shown that signs are more important than weight magnitudes in dense-to-sparse methods in section 2. The authors should consider an experiment where PaI masks are initialized with some “good signs” to show a clear connection between signs at initialization improving performance. \n\n- I appreciate the authors' discussion on computation and memory costs associated with sign-in. The authors should consider the following plot, x-axis being the computation budget in terms of FLOPs, and y-axis being the test accuracy, and scatter plot the results for all sparsity levels and all the methods. This gives a better picture of the budget as compared to the performance trade-off.  \n\n- The authors should consider adding the comparison with dense-to-sparse methods in the main part with dense-to-sparse+Sign-in, PaI, and PaI+Sign-in. This is a requirement since all those methods operate in a “similar” problem space where the training efficiency is paramount. \n\n- Minor: The earlier parts of the paper including the introduction can be made much simpler, coherent and less dense. A small suggestion would be not to highlight all the results, (theoretic or empirical) as it is uneasy to follow those if the reader is already not familiar with the entire paper.\n\nQUESTIONS:\n- Can it be the case that the ability to change the connectivity and thereby starting from 0 in the newly added weights, allows the network to implicitly change or shift the signs in dynamic sparse training methods such as RiGL?\n\n- Does sign flipping help in a very high sparsity regime? (0.98, 0.99) It may be so that sign flipping is a helpful tool when even the sparse mask is over-parameterized, but not in those high sparsity regime where mask structure may become very important.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the limitations of Pruning at Initialization (PaI) for sparse neural network training and proposes Sign-In, a novel reparameterization strategy designed to induce sign flips in parameters to improve the alignment between sparse masks and learned weights. The method introduces a dynamic reparameterization which facilitates better optimization geometry and aids in recovering correct parameter signs, especially in sparse settings. The paper supports its claims with both theoretical analysis and empirical evaluations on CIFAR-10/100, ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper provides provable guarantees that Sign-In enables successful learning in previously hard-to-optimize configurations.\n\n2. The authors evaluate their method on multiple scales and architectures across different sparsity, which demonstrate robustness.\n\n**Weaknesses**\n\n1. Although the author of the paper claims that the focus of the proposed method is on PaI, but the whole PaI research domain still falls in the degraded performance on model accuracy. My concern in on the motivation of the paper: does improving such method really necessary? Since the improved accuracy by Sign-In is still not comparable to many sparse training methods, the motivation is questionable.\n\n2. The result in the paper is limited. And the existing results show limited improvements against baselines.  In the main paper, it only shows one small table of results, with 3 network structures corresponding to 3 datasets. The baseline method is Random initialization, instead of other more comprehensive PaI methods. For the baseline, it is very obvious that Random initialization has worst performance, such that the proposed method may have the best improvement performance. But if switching to a different method (e.g., SNIP), the results may not that good. Therefore, the demonstration of the results is not convincing. \n\n3. For PaI, the author only shows SNIP, Synflow plus the proposed method. There are many other PaI method, for example, the concurrent work with SNIP, GraSP, is not compared. And many following works are also not included. \n\n4. For Dynamic Sparse Training method like RigL, the improvement is limited (72.02->72.48). The improved accuracy is still not better than MEST (72.58) at same training budget and sparsity. I wonder if applied on even better sparse training method (PaI or DST), does the proposed method still work? This is another concern about the proposed method.\n\n5. The motivation of the method is based on the Lottery Ticket Hypothesis, which is already proved to be falsified by many later works (also cited by the author of the paper in line 110). I may disagree with using it as the motivation or develop rigorous method based on it. The author of the paper should consider using another more rigorous theory.\n\n6. Minor, line 37, concered -> concerned.\n\nQUESTIONS:\nPlease refer to the strength and weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-02"}