Title: Hyperflux: Pruning Reveals the Importance of Weights
Venue: NeurIPS 2025
Accepted: no
Keywords: Computer Vision

Architectures

Algorithms

Sparsity
Abstract: Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most existing methods use ad-hoc heuristics, lacking much insight and justified mainly by empirical results. We introduce Hyperflux, a conceptually-grounded $L_0$ pruning approach that estimates each weight’s importance through its *flux*, the gradient's response to the weight's removal. A global *pressure* term continuously drives all weights toward pruning, with those critical for accuracy being automatically regrown based on their flux. We postulate several properties that naturally follow from our framework and experimentally validate each of them. One such property is the relationship between final sparsity and pressure, for which we derive a generalized scaling-law equation that is used to design our sparsity-controlling scheduler. Empirically, we demonstrate state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Hyperflux, a framework for pruning weights in neural networks through the gradient response to weight removal (flux). The framework dynamically prunes and regrows weights based on a 'pressure' term. In addition, the framework allows using a scheduler for controlling sparsity. The method is well-motivated and performs well on CIFAR10/100. On ImageNet, Hyperflux lags slightly behind state-of-the-art methods in accuracy and training efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper presents an intuitive idea of estimating weight importance by observing the gradient flow after pruning.\n2. The method allows for dynamic pruning where pruned weights can 'regrow' at a later stage.\n3. Competitive results on CIFAR10/100 at high sparsity.\n4. The proposed scheduling mechanism provides a principled approach to controlling sparsity during training.\n\nWeaknesses:\n1. While the idea of 'flux' is well-articulated, it overlaps significantly with other gradient-aware methods. Also, the 'Topology' definition shares many similarities with [1]. The main difference seems to lie in the framing rather than the fundamental technique. This limits the novelty of this work.\n2. On ImageNet, Hyperflux results fall slightly behind AC/DC and Spartan, while requiring higher training costs. This is due to the additional parameters and calculations revolving around flux.\n3. The evaluation is limited to image classification on CIFAR and ImageNet, and with old network architectures (ResNet will soon be 10 years old). It would strengthen the paper to include more complex tasks (such as semantic segmentation, object detection, and depth prediction) with newer architectures (E.g., ConvNeXt, ViT).\n\n\n[1] Savarese et al. Winning the Lottery with Continuous Sparsification. NeurIPS 2020.\n\nQUESTIONS:\nI would appreciate addressing the concerns I've raised in the weaknesses section.\n\nLIMITATIONS:\nThe authors mention the limitations of the method, which lie mainly in the additional computational cost during training.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper presents Hyperflux, a L0 pruning approach that estimates each weight’s importance through its flux, the gradient’s response to the weight’s removal. A global pressure term continuously drives all weights toward pruning, with those critical for accuracy being automatically regrown based on their flux. The paper's evaluation state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.

STRENGTHS AND WEAKNESSES:
Strengths:
+ The idea of pruning model weights based on flux is original and interesting.
+ The paper is able to explain why the proposed method works and experimentally validate the claims.
+ The paper is well written.

Weaknesses:
- The considered architectures include only ResNet-50 and VGG-19 and are not diverse enough. These models are also both quite outdated.
- The improvement over SOTA is small. Given the large number of related pruning works with similar performance, this paper appears to be just yet another pruning work and its significance is questionable.
- All the considered baselines were published before 2022.
- The proposed method in fact performs worse than the related work AC/DC.
- As a unstructured pruning method, the resulting models do not run well on commodity hardware.

QUESTIONS:
- Why does HyperFlux perform worse than AC/DC?
- How long does HyperFlux take to prune a model like ResNet-50?

LIMITATIONS:
Can the proposed solution generate real societal impact to widely deployed real-world applications and systems?
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hyperflux, which prunes a network by adding a single pressure term γ that keeps pushing every weight toward zero.\nWhen a weight is set to zero, the method looks at the new gradient—called flux—to see how badly the loss wants that weight back; big flux means the weight will grow again. Because pruning (pressure) and regrowth (flux) run together at every step, the model keeps only the weights it truly needs. A simple rule links γ to the target sparsity, so a built-in γ-scheduler can hit any desired sparsity without manual tuning. On CIFAR-10/100 and ImageNet-1k, this approach matches or beats RigL, GraNet, and AC/DC while staying up to 96 % sparse and using just 6–15% of the dense FLOPs.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n1. I found the flux–pressure idea very interesting. By defining “flux” as the gradient that appears when a weight is gone, the method ties importance directly to loss change, giving a more interpretable alternative to magnitude, Taylor expansion, or Hessian heuristics.\n\n2. The experiment results look good. For example, on CIFAR-10/100 and ImageNet-1k, the method matches or surpasses strong baselines such as RigL, GraNet and AC/DC at sparsity levels up to 96 %, while cutting inference FLOPs to as low as 6–15 % of the dense model.\n\n3. The comprehensive ablation study further validate the effectiveness of the proposed method.\n\n\nCons\n1. My main concern is that the novelty of the proposed method might be relatively limited.\nFlux is essentially a first-order Taylor term already explored in earlier Taylor-based pruning work [1], the author also shows that the proposed method has a substantial link to the Taylor expansion based existing works in Appendix. The prune-regrow loop follows the Dynamic Sparse Training (DST) line started by SET [2], RigL [3] and GraNet [4]. The paper mainly combines these two ideas (though with some incremental but solid improvement) rather than introducing a new paradigm.\n\n3. I think the training cost of the proposed method is quite high.\nTraining still requires about 0.5–0.6× the dense FLOPs on ImageNet, which is higher than GraNet and similar DST baselines; the paper gives no wall-clock or memory numbers.\n\n4. All results focus on image classification, so the claimed generality remains unknown.\n\n\n[1] Molchanov, Pavlo, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. \"Importance estimation for neural network pruning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11264-11272. 2019.\n\n[2] Mocanu, Decebal Constantin, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\" Nature communications 9, no. 1 (2018): 2383.\n\n[3] Evci, Utku, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. \"Rigging the lottery: Making all tickets winners.\" In International conference on machine learning, pp. 2943-2952. PMLR, 2020.\n\n[4] Liu, Shiwei, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. \"Sparse training via boosting pruning plasticity with neuroregeneration.\" Advances in Neural Information Processing Systems 34 (2021): 9908-9922.\n\nQUESTIONS:\nMy main concern is that the novelty of the proposed method is not very clear to me (see Cons #1 above). If the authors can clarify this point (as well as other cons listed above), I would be willing to raise my score.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a pruning algorithm called hyperflux. The method works by adding an learnable 'importance' parameter $t_i$ for each weight $w_i$ which is activated by a step-function and multiplied in its corresponding weight as $w_i \\times H(t_i)$. During backpropagation the gradient of $H$ is taken to be identity (which can be considered as a kind of straight-through estimation). The authors derive the gradients and use it to describe the behavior of the algorithm. A term (called 'pressure') proportional to $\\sum t_i$ is added to the loss function to drive weights towards pruning. They demonstrate state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n* The paper is clearly written and the mechanism with which pruning is achieved is clearly described.\n* I find the method to be very novel and creative (but note that I'm not very familiar with the area of pruning). For example, it is interesting that they have two variables for each weight. Commonly the magnitude of a weight is assumed to be its importance but, here, the two are separated. Novel ideas can potentially be very impactful.\n* Strong empirical performance: state-of-the-art on CIFAR10 and CIFAR100 and competitive with state-of-the-art on ImageNet.\n\n## Weaknesses\n\n* I wouldn't say that the method is \"conceptually grounded.\" Deriving gradients isn't a theoretical justification for the method. But it's fine to leave a more theoretical understanding for future works.\n\nQUESTIONS:\n* Why do the 90% and 95% pruned networks with hyperflux consistently beat the original dense network on CIFAR10 and CIFAR100?\n* The three row groups in Table 3 aren't labeled. What are they?\n* Are you reporting test accuracy or training accuracy?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}