Hyperbolic Aware Minimization: Implicit Bias for Sparsity
Authors: Tom Jacobs
Advait Gadhikar
Celia Rubio-Madrigal
Rebekka Burkholz
Venue: ICLR 2026 Poster (2026)
Keywords: Sparsity
Implicit bias
Sign flip
Exponential update
Training dynamics
Bregman function
Primary area: optimization
TL;DR: We propose Hyperbolic Aware Minimization (HAM), a method that improves sparse training by preserving the benefits of hyperbolic implicit bias while avoiding slowdown caused by the vanishing inverse metric for parameter updates.

Abstract: Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise overparameterization promotes sparsity, but also yields a small inverse Riemannian metric near zero, slowing down parameter movement and impeding meaningful parameter sign flips. To overcome this obstacle, we propose Hyperbolic Aware Minimization (HAM), which alternates a standard optimizer step with a lightweight hyperbolic mirror step. The mirror step incurs less compute and memory than pointwise overparameterization, reproduces its beneficial hyperbolic geometry for feature learning, and mitigates the small–inverse-metric bottleneck. Our characterization of the implicit bias in the context of underdetermined linear regression provides insights into the mechanism how HAM consistently increases performance --even in the case of dense training, as we demonstrate in experiments with standard vision benchmarks. HAM is especially effective in combination with different sparsification methods, advancing the state of the art.

Published as a conference paper at ICLR 2026
HYPERBOLIC AWARE MINIMIZATION: IMPLICIT BIAS FOR SPARSITY
Tom Jacobs Advait Gadhikar Celia Rubio-Madrigal Rebekka Burkholz
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany (t.m.. jacobs, burkhol 1z z @ c de
ABSTRACT
Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise overparameterization promotes sparsity, but also yields a small inverse Riemannian metric near zero, slowing down parameter movement and impeding meaningful parameter sign flips. To overcome this obstacle, we propose Hyperbolic Aware Minimization (HAM), which alternates standard optimizer step with a lightweight hyperbolic mirror step. The mirror step incurs less compute and memory than pointwise overparameterization, reproduces its beneficial hyperbolic bottleneck.
geometry for feature learning, and mitigates the umall-inverse-mett Our characterization of the implicit bias in the context of underdetermined linear regression provides insights into the mechanism how HAM consistently increases performance even the case of dense training, as we demonstrate experiments with standard vision benchmarks. HAM is especially effective in combination with different sparsification methods, advancing the state of the art.
INTRODUCTION
The success of modern deep learning relies on large amounts of overparameterriation, which has led to computationally demanding trend to increase the size of models, and thus the number of trainable parameters by orders of magnitude (Hoffmann et al., 2022 Kaplan al. (222)) A common explanation for this phenomenon are implicit biases that originate from combination of the optimizer and the overparameterization (Pesme et al., 2021 Gunasekar al. 201 Woodworth et al. 2020).
which regularize the training dynamics and thus improve the generalization performance.
Training sparse models instead leads to suboptimal performance (Li et al., 2017 Frankle & Carbin
2018). This fact has limited pruning at initialization (Pal) approaches (Tanaka al. 22200] Lee et al.
2019 Liu et al. 2021a) that aim to reduce the heavy computational and memory demands by masking the network before training the remaining parameters. In contrast, state-of-the-ard sparsification methods utilize overparameterization in some capacity, as they either gradually prune parameters in Dense-to-Sparse (DiS) training (Peste al. 202T Kuznedelev et al., 2024 Kusupati al. 2020
Jacobs & Burkhhlz 2025 Kolb et al. 2025) or dynamically explore multiple sparse masks to find high-performing sparse networks with Dynamic Sparse Training (DST) (Evei et al. 2020; Lasby et al. 2023 Chen et al. (2021). Key observations regarding these algorithms are that a) mild sparsity (which does not degrade performance relative dense baseline) (Jin al., 2022) and b) longer training with standard optimizers can improve generalization performance significantly. The latter indicates that sparse models are difficult train and take longer converge (Kuznedeley Consequently, sparse training ideally leverages overparameterization to improve generalization.
(023).
recent development to improve sparse training is the pointwise overparameterization proposed in PIL.T (Jacobs& & Burkholz, 2025) and Sign-In (Gadhikar et al. 2025). All parameter weights E IR" are replaced by a pointwise product of parameters m O w, with both m, w E IR". This changes the implicit bias of the optimization process and leads to substantial generalization benefits for sparse training. In PILoT, a continuous sparsification method, the overparameterization is used to jointly learn the mask m. Meanwhile, the Pal method Sign-In uses it to increase the plasticity of non masked parameters and facilitate sign flips, which was shown to be a major obstacle in sparse training (Gadhikar & Burkholz 2024a). Both methods, PILoT and Sign-In, achieve state-of-the-an

---

Published as a conference paper at ICLR 2026
No hard perturbations
No extra parameters
Implicit spansity bias
Sign flips
Dense training
PILoT (Jaooss & Burkholz 2025
Sign-In Gaddikar et al. 2025
(less strong) (less strong)
HAM (OUTS)
Table HAM induces a less strong implicit sparsity bias (moderating between Lz2 and L1) and flips parameter signs more easily due to its inverse metric (see Fig. I. which together lead to boosting sparse training without explicit overparameterization.
results in their respective categories. However, on their own, they fall short of baseline methods of sparse training that do not utilize this form of overparameterization, such as AC/DC (Peste et al.) 2021) and RiGL (Evci et al. 2020). To understand this gap, we investigate their training dynamics.
The dynamics of the overparamettriiztion m @ can be derived within the mirror flow (Li et al.) 2022) or time- dependepndd mirror flow framework (Jacobs et al.) 2025). It is associated with the hy- perbolic mirror map /Woodworth et al. 2020) and, depending on initialization, learning rate, and reg- ularization, it changes from an implicit L2 (Dense) to L1 (Sparse) bias during training. The mirror flow induced by m @ u can also be characterized by Riemannian gradient flow with associated metric (Li et al. 2022). Comparing these met- rics highlights a problem: m w suffers from a small inverse metric g g0) near the origin, where g Riemannian metric tensor (Jacobs) & Burkholz 2025) (see Fig. and Appendix F As consequence, parameters can get stuck at O, preventing effective sign-flips. Sign-In partially mitigates this issue by itcratively re-initializing m and w such that m O remains fixed. We set = (m² - 2 2 >> O. However, the remedy is unstable and introduces hard perturbation to the training dynamics, limiting its positive effects.
HAM
3
Acceleration Zone
GD
m O W
16
Stuck
The inverse metric ( of HAM is above the one of gradient descent (GD), while the overparameterizzaioo
Figure 1:
@ w is below for small y. This enables moving from the initial- ization O, to the optimum O" instead of getting stuck. Therefore, HAM fixes the vanishing in- verse metric. Note the hyperbolic geometric structure of HAM and m @ w compared to the flatness of GD.
In this work, we propose capture the essential structure of the two methods PILoT and Sign-In and thus the implicit bias of pointwise overparameterivation m ( w. which provably aids in finding generalizable sparse solutions. At the same time, we are able avoid their drawbacks: their slow down near zero and their need for explicit overparameterization that negatively impacts memory and compute (see Table MT. We do this by deriving plug-and-play hyperbolic optimization step, which we alternate with gradient descent or any other first-order optimizer. Our alternating method is called HAM: Hyperbolic Aware Minimization ($ 3 3.1 HAM mitigates the small inverse metric problem of m @ w and keeps similar but fully controllable implicit bias, as shown in $ $ We evaluate HAM on standard vision benchmarks and find that it consistently improves generalization, especially of sparse training ($ Remarkably, HAM tends to enhance generalization complementary to sharpness aware minimization (SAM) (Foret et al. 2021). yet incurs only negligible computational overhead. be explained by two major mechanisms: a) It accelerates training around 0, thus improving sign learning. This facilitated by its geometry and a larger inverse metric. b) The implicit bias towards sparsity regularizes training inducing mild sparsity. Both mechanisms boost generalization performance of sparsification techniques, such as AC/DC (Peste et al., 2021) and RiGL (Evci et al. 2020), and even of dense model training. In summary, our contributions are:
These improvements
We introduce HAM, a lightweight, plug-and- and--la general purpose optimization step that integrates with any optimizer at negligible computational cost.
We provide a theoretical analysis of HAM's training dynamics using Riemannian gradient flow for linear regression ($ 4 characterizing its implicit bias and sign-flipping mechanism (Appendix E
2

---

Published as a conference paper at ICLR 2026
HAM inherits the geometric benefits of recent sparsity parameterizations while mitigating their vanishing inverse metric problem (see Figure and Appendix(F). The benefits are a implicit sparsity bias which facilitates mild sparsity and complementary sign flips to those of dense training.
Empirically, HAM improves state-of-the-ar sparsity methods (AC/DC, RiGL, STR), enhances standard dense training. and is also compatible with optimizers like SAM
2 RELATED WORK
Sparsification Sparse training methods can be categorized into three broad classes: Pruning at Initialization (Pal), Dense-to-Sparse training (DIS), and Dynamic Sparse Training (DTS). Pal methods identify sparse mask at initialization and train the remaining parameters to convergence. They
include methods like SNIP (Lee et al. 2019). Synflow (Tanaka et al. 2020), NPB (Pham et al. 22023 J.
PHEW (Patil& & Dovrolis 2021). GraSP (Wang et al.) (020) and random pruning (Liu et al. 202 la
Gadhikar et al.] 2023). Their primary limitation is that standard optimizers do not find generalizable
solutions on these fixed masks, as they struggle to effectively learn parameter signs (Gadhikar &
Burkholz 2024b) Gadhikar et al. 2025). In contrast, DiS methods learn the mask via a dense or denser
phase of training. followed by any kind of pruning step and possibly more training. This includes
iterative pruning methods like IMP (Frankle & Carbin| 2019). LRR (Renda et al.. 2020 Han et al.
2015), AC/DC (Peste et al. 2021). CAP (Kuznedelev et al. 2024). and WoodFisher (Singh & Alistarh
2020). Continuous sparsification methods, which start from a dense network and gradually sparsify
with a learnable mask, also fall under this category. They include PILoT (Jacobs Burkhhlz
2025), STR (Kusupati et al. (020). CS (Savarese et al.) (021) and spred (Ziyin & Wang, [2ng] The
third class of methods, Dynamic Sparse Training, start from an already sparse mask but dynamically update it during training, and in this sense utilize a form of (dynamic) overparameterization (Liu
jet al. 2021b). Examples include RiGL (Evci et al. 2020). MEST (Yuan et al. 202I), and SET (Mocanu al.) 2018). While Pal methods cannot compete with the generalization performance of DtS and DST methods, Sign-In (Gadhikar et al. 2025) improves on Pal by using the pointwise overparameterization
@ u, which leverages hyperbolic mirror map facilitate sign flips. In this work, we propose instead a simpler, more powerful hyperbolic optimization step to leverage a similar mirror map without doubling the number of parameters and solving an issue with an associated inverse metric.
Implicit bias and mirror flow The implicit bias of neural networks is well studied topic that
aims to explain the regularization benefits resulting from overparameterization (Woodworth et al.
2020 Gunasekar al. 20176 2018 Li et al. 2022). It is primarily characterized within the mirror
flow framework, a well esttabiiihd concept in convex optimization (Alvarez et al. [0004] Beck &
Teboulle, 20033 Rockafellar Fenchel 1970, Boyd & Vandenberghe |209 | et al. (022). A mirror flow can be seen as gradient flow a Riemannian manifold (Li et al.) 2022; Alvarez
ct al. 2004) with the metric tensor being the Hessian the Legendre function, which has also been extended to cover stochastic gradient descent (SGD) (Pesme et al. 2021 Even et al.) 2023 Lyu &
Zhu 2023) and more recently to explicit regularization (Jacobs et al. (2025) ). This framework allows us characterize the implicit bias. The main observation is that large learning rates, stochastic noise from SGD, and regularization can benefit generalization by implicitly inducing sparsity. However, overparameterization can also lead to small inverse metrics, slowing down convergence and potentially hampering generalization (Jacobs & Burkholz 2025), which we can successfully avoid in HAM.
Related optimizers The mirror flow framework also enables through the lens of natural gradient descent. Accordingly, the inverse metric is adapted due to (an approximation of) the Fisher information matrix, which captures second-order information (Martens) 2014; Amari 1999). A more general Bayesian framework (Khan & Rue (02T)) has been used to gain insights into invariant distributions by using Lie groups (Kiral et al. (023) and to develop the IVON optimizer (Shen et al.] 2024). Within this framework, HAM, our proposal that alternates exponential updates with gradient descent steps, can be interpreted as a mapping of the Fisher information (metric) to known posterior distribution, as derived in Appendix C Moreover, our proposed hyperbolic update is reminiscent of exponentiated gradient descent (Kivinen & Warmuth 1994). Distinctly from HAM, it optimizes probability distributions and thus includes normalization to stay on a probability simplex, as seen, for example, in Chapter 7 of Vishnoi (202I). Exponential gradient descent, which has recently been applied reweighting batches (Majidi et al. 2021) or
view our algorithm HAM
3

---

Published as a conference paper at ICLR 2026
augmenting ADAM (Bernstein et al. 2020), also utilizes similar exponential updates but does not rely on an alternating scheme like HAM. This prevents it from facilitating more suitable sign flips than gradient descent. More advantages of HAM are analyzed in Appendix D and E
Two-step and alternating schemes Various previous works have explored alternating training schemes including proximal methods, soft thresholding, ADMM, alternating least squares, and expectation maximization, among others (Parikh & Boyd, 2014, Boyd et al.] 2011: Cichooki et al. 2009; McLachlan & Krishnan 1996). The most related alternating algorithms HAM based on birth-death dynamics at a neuron level in two-layer neural network training (Rotskoff et al. 2019) or variational inference (Mielke & Zhu 2025 Gladin al. 2024, Yan al.) 2024). An important difference is that we work on a weight level while other approaches work on neuron or distribution level and serve an entirely different purpose. Furthermore, one of the most well-known two-step approaches is sharpness aware minimization (SAM) (Foret et al.) 2021), which promotes the search for flat solutions at the expense of almost doubling the compute of one optimization step. In contrast, HAM encourages implicit sparsity bias and acceleration around 0, which are complementary mechanisms. Our experiments (Table B demonstrate that our proposed hyperbolic step can be effectively combined with SAM to further boost generalization.
3 MOTIVATION AND DERIVATION OF HAM
We derive our novel optimization step by building on insights from the implicit bias of recently developed sparse training methods. These methods have exploited a reparameterization of the neural network: They replace each weight & with a product of two weights m O w, where @ is the Hadamard product, i.e., a pointwise multiplication. For this reparameterizzttion it known that stochastic noise and weight decay induce sparsity by an implicit L1 penalty (Pesme et al. 202T Jacobs & Burkholz, 2025). The next paragraphs restate the induced gradient flow of this reparameterization (where the learning rate 7) + 0). Note that u² abreviates u 02
Gradient flow training Consider a continuously differentiable and L-smooth loss function f R" + E. It can be trained by means of gradient descent: 0x+1 O₂ 1) f(xx), initialized at 0 = Oinits where > 0 is the learning rate. Taking - O, we obtain its gradient flow: d0, V f )tt. Its integral form is used in the mirror flow analysis and descriptions of the
implicit bias: 02 0g = - fo (O,)))
Reparameterized gradient flow Li et al. (2022) derive a similar formulation for the reparame- terization m @ w trained with gradient descent, while Theorem 2. in Jacobs & Burkholz (2025) integrates weight decay with strength / in the analysis resulting of the following gradient flow:
ddm; = w. C VS 23m,
We Winits
dw: = m, C Vf(O,) 2,3w,dt
me = mini-
This corresponds to the integral equation for 0: m. ww:
0: = u @ exp (2 ff(O.)ds 48t o O exp V(((.)ds 43t
(1)
where un := ma + wD and vo = mo wi for /ool < mg are chosen such that un - v/ Og. 3 > 0
/2
v2
is the strength of weight decay. This results in time varying Riemannian gradient flow for 0,:
d0,- = VO3- 2 @ V/(0,) - 2330,d,
1o = O
(2)
where Aug o exp (43)). Eq. implies that we cannot move through zero when 7 + 0.
Exponential gradient descent The hyperbolic gradient flow in Eq. (1) not only corresponds to the gradient flow of m O w, but also to exponential gradient descent. This is presented by Wu & Rebeschini (202I) for matrix forms in a matrix sensing task without regularization (3 0). We use this connection to derive an update without the need for the reparameterizzatoon (or regularization).
The update captured by the following theorem:
See Definition A.1 in the appendix for definition of L-smoothness.
4

---

Published as a conference paper at ICLR 2026
Theorem 3.1 If mo sign(Ou)wx Vol. then
0 = 02 exp (-17 (2 sign(0 + 43))
(3)
is equivalent to Eq.
7 up-to first order, i.e., the discretization ervor is O
Proof. See proof of Theorem B.1 in the appendix.
Note that a more general update for product of matrices is provided by Wu & Rebeschini (2021).
Their exponential update suffers from the same problem as the parameterization m O w, as it corresponds y 0 and thus completely preventing sign flips (see Corollary B.2). Our proposal HAM overcomes this obstacle by alternating a gradient step with an exponential update step.
Derivation of HAM The novelty stems from alternating the new hyperbolic update in Eq. B with another optimizer. This forms the basis of our proposal HAM. We derive its explicit form for gradient descent as follows:
0 x 0z (V(())...)
(GD)
0 0x+1 =0 0x+y @ expp(-1) (a: sign(0x) ())-
(HYP)
Note it is not necessary to the same learning rate for gradient descent and for the exponential update. In fact, the learning rates control the strength of the implicit bias towards sparsity, for which we have introduced an additional hyperparameter a R. The exponential update now more closely resembles the hyperbolic gradient update in Eq. T). as it can switch the sign in the exponential. $ studies the resulting gradient flow.
Interpretation The exponential update (HYP) introduces a weight scaling which correspond to a metric (0) = 1//6 0) for a Riemannian gradient flow, as we will see in $4 This changes how parameters evolve compared to standard gradient descent. When the sign of parameter is correct, the update refines its magnitude. If the sign is incorrect, it drives the parameter exponentially fast toward However, on its way a sign flip, the parameter gets stuck in 0 because the parameter update is proportional to f = 0 at the origin. To facilitate the sign flip, we need the intermediary gradient step GD which explains the advantages of HAM over pure exponential updates. summary, our combined update can be interpreted as: Learn the magnitude when the sign is correct; otherwise, move rapidly to zero to enable sign correction. This mechanism crucial for enabling sparse training (Gadhikar & Burkholzz 2024a Gadhikar et al. (2025), as shown in our experiments ($ 5 where HAM significantly boosts performance compared to standard optimizers.
Note that the second step HHP depends both on 0g and 0 which would require twice the memory of gradient descent. To avoid this, we replace sign(Ox) with sign(0 (x+))))+)))) We restate the second HAM step actually deployed (HYPa) in $3.11 It also has benefits for the optimization itself, promoting more stable sign flips, as we discuss Appendix SaddiixDDaan and Figure Building on recent work on sign flips (Gadhikar et al. (225), we argue that the sign should be aligned with the gradient evaluation to assess whether the step should be accelerated or not, to promote meaningful sign flips. This insight is tightly linked to our choice of sign(@ k+ instead of sign(f;) in the HYP* step.
Remark 3.2
3.1 ALGORITHM: HYPERBOLIC AWARE MINIMIZATION (HAM)
We propose HAM (Algorithm D. which alternates between any standard optimizer step and hyperbolic (signed) mirror map to improve the general trainability of neural networks. The proposed method is inspired by recent sparsification methods, theoretically justified in $ 4 We next state the main algorithmic innovations.
Hyperbolic step Let r) > denote the learning rate and a, B 2 0 be positive constants. The hyperbolic step deployed with parameters 0₂ E R2"h is given by
0x = 0x+ O exp (-n(a sign 4+ ( + +3))
(HYP*)
where 0x+y is the step of any other optimizer. controls the convergence speed and hyperbolic awareness of the method, and 3 induces explicit regularization similar to that of PILoT (Jacobs &
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 HAM
Require: steps T, schedule 77, initialization Oinit, constants a, 3 2 for k E O.. . do
0x+ OptimizerStep( (((()))))))) 0x+11 = HyperbolicStep((1 (0++ +++.. (f())), 0, B))) according to formula HYP*)
end for return Model weights 0r
Burkholz 2025).
Note that we have replaced sign(0;) with sign(O*+ }). as mentioned in Remark Our ynil of implicit bias $ still remains valid with this change, as we show in Appendix
3.2
Theorem B.6
Memory and compute overhead HAM does not incur any memory overhead, as it reuses the known gradient and current signs of the weights. In contrast, the pointwise overparameterization m & w doubles the number of parameters, which would only be negligible in case of large batch sizes where activations dominate the memory requirements (Ziyin & Wang 2022 Jacobs & Burkholz 2025 Kolb et al., 2025). Moreover, the additional extra flops during training are negligible, as they are linear the number parameters.
THEORY: GRADIENT FLOW ANALYSIS
Our theory (Eqs. (GD HYP identifies the implicit bias of HAM's Riemannian gradient flow in parameter space (Thm. (.2) and provides a convergence analysis (Thms. [43 and Accordingly, HAM solves the vanishing inverse metric problem of m w, and thus converges faster, while retaining the same asymptotic implicit sparsity bias. In this section, we assume that the objective function f I + R continuously differentiable, i.e., f E C²,, and L smooth Appendix E proves that HAM also induces meaningful sign flips like m O w (Gadhikar et al.) (025)) We focus on the case of nonzero 3 are highlighted in Section B.1
to simplify the exposition. Theorem 4.2 derives the flow for general B. The effects
Riemannian gradient flows In order to concisely study the behavior of HAM, we consider a gradient flow formulation. Gradient flow (flat) and m @ w (hyperbolic) flow can be described as Riemannian gradient flows depending on a general metric g()):
d0, 9 V
0g 0
We refer to the quantity g - (0) For @ Jacobs & Burkholz (2025) have derived 9 (0) 0 + + where 7 depends the initialization scale and can change due to noise and regularization. In contrast, the inverse metric of HAM is not changed by these factors. To give overview, the inverse metrics are also reported in Table
(0) as the inverse metric. It trivial for gradient flow, since 90D 1.
The vanishing inverse metric problem For small y and small weights 0, the inverse metric /0² + 72 of m O
Table 2: Inverse metrics of gradient de- scent, the overparameterization m C w, and HAM.
can get much smaller than 1 (see Figure D This implies that learning close to is slowed down, which makes transitions through O (and sign flips of 0) much harder, slowing down convergence.
GD
O
HAM
( 1
0 + 72 1 + a
Theorem 4.1 I is L-smmothh (Definition A.7y satisfies
the PL-inequality (A.2) or is convex and arg min (f((): RR) is non-empty, then the iterates O. converge to minimizer of for both metrics 9GD and 9mo Under the PL-inequality, the linear convergence rates are respectively and A min; 7/a where A > is the PL-inequality constant.
Proof. We can apply Theorem A.3 in Jacobs & Burkholz (2025) (Thm. A.7 in the appendix) and Theorem 4.14 in Li et al. (2022) (Thm. A.6Iin the appendix). For the convergences rates it is sufficient
to bound the inverse metrics from below such that have gGD > 1 and Imow min; Ti-
0

---

Published as a conference paper at ICLR 2026
Riemannian gradient flow of HAM In comparison, HAM speeds up learning around O. To show this and characterize HAM's dynamics, we derive its gradient flow from Eqs. (DHHYP) by writing out the iterates in sum notation and taking the learning rate 17 0.
Theorem 4,2 The Riemannian gradient flow (1) - 0) of Eqs. GD HYP) is: d0; (1+ 0, @ V (f(O:) 30,dt,
0o
0 Oinit,
(4)
where| . is applied pointwise. Moreover, if 3 = 0, the inverse metric is 9HMM (0) + alo).
Proof. This follows from writing out the update and then taking the limit to get an integral
equation. The gradient flow then follows from the Leibniz rule. See Theorem B.3 in the appendix. D
Convergence of HAM We analyze the inverse metric and convergence behavior of HAM when
3 = 0. In this case, the inverse metric is given by (HAM (0) = + a Table 2). indicating that HAM can converge faster than gradient descent depending on a and the magnitude of the weights. This stands in stark contrast with sparsification methods, where a decaying y << 1 slows down movement. We formalize this behavior under the same conditions as Theorem
Theorem 4.3 Under the same setting as Theorem 4.1 the iterates of HAM in Eq. (G with 0, 0, converge to a minimizer of f. Moreover, the linear convergence rate is under the PL-inequality.
Proof. Similar as in Theorem 4.1 we again can lower bound 9HAM 2 for the convergence rate.
This proves that HAM avoids the vanishing inverse metric problem from the pointwise overparame- terization m @ w. while keeping the geometric benefits as we will see next. Furthermore, we discuss the case for B > 0 in SB.1
Implicit bias of HAM We characterize the implicit bias of HAM by analyzing its associated Riemannian gradient flow. This confirms that HAM not only speeds up convergence with respect to m @ w and small 7, but also influences the nature of the solution. To show this, we compute the Bregman function R2 (see Definition A.4) such that its Hessian yields the required metric, i.e.,
ПНAM = V Ra.
Lemma 4.4 The function for a E R is given by
Ra (0) = Σ (a 1) In (a 00,1 + 1) aa 10,1
0 sign
log(1 + al0,,o).
a
If a 0, Ra is Bregman function (Definition
A.4
Proof. See Lemma B.4 in the appendix.
Concretely, we can use Lemma 4.4 to characterize the implicit bias for under-determined linear regression. Let {zz, }:) C IK" x R be a dataset of size d. The output of a linear model on the i- th data x? e. The goal therefore solve the regression for the target vector y (1/1 V/2 ... ... ya) and input vector Z (21, 22,..... Za).
Theorem 4.5 Consider the same setting as Theorem 4.3 with 3 = O. Then, if [ = f(Z 0 y). the gradient flow of O2 in Eq. (4) converges to the solution of the optimization problem: Ozo arg minze=y Rag (0).
Proof. Apply the mirror flow part of Theorem 4.17 (Li al. 2022) for Bregman functions.
Theorem 4.5 provides an intuition about the type of solutions to which HAM converges. We are particularly interested in the shape of R2 when Og 0 (to understand sign flips). Note that the gradient flow is well defined at 0g if 3 0.
Theorem 4.6
Let Og O. Then, R. 10112 if 0. and R. |10||2, if cc, where
indicates proportionality, i.e., there exists some positive functions by R + R and l2 : IR + R in the neighborhood of the limiting point such that for all 0 E R". lim by (o. = =10 and
limg h2(a)) (0) = ||0||,
7

---

Published as a conference paper at ICLR 2026
Proof. See proof of Theorem B.5 in the appendix.
Theorem 4.6 is illustrated in Figure 2(b) Ra of HAM induces an implicit bias that interpolates
between L2 and L1. similarly to
(Jacobs & Burkholz 2025).
Remark 4.7 the - 0o setting. HAM induces an L1 bias. However, practice, due to discretizations, this setting would require a much smaller learning rate to ensure convergence. This makes HAM less suited to fully induce sparsity on its own than the related sparsification methods. Therefore, HAM is best used in combination with other methods to find spurse solutions, acting guide for sparse geometry during training.
Remark 44.7 emphasizes that HAM needs a substantially large a to induce sparsity on its own. This is
in line with our takeaway from } Our hyperbolic step primarily contributes learning the correct magnitude of weight and promotes sign flips. This differs from m O w overparameteriiztion,, where sparsity emerges due the inherently small inverse metric.
In practice, we apply additional weight decay with strength 3 >> 0. This promotes sparsity but does not change the inverse metric for HAM. In contrast, for m ) it worsens the vanishing inverse metric problem (see Appendix( F. as we learn from comparing gradient flows (Eq. 2 and Eq. 4). HAM has the advantage that we can freely tune a for the right amount of implicit sparsity.
Remark 4.8
For details on how 3 > O influences Thms 4.3 and 4.5
see B.
5 EXPERIMENTS
Our main goal is to highlight the versatility of our novel optimizer step, HAM. and verify our theoretical insights into its mechanisms. To this end, we compare HAM to two algorithms that explicitly utilize the parameterization m ④ wt a) PiLoT (Jacobs Burkholz, 2025), continuous sparsification method, and b) Sign-In (Gadhikar et al. 2025), an optimization approach designed to improve training sparse masks (especially in the context of Pal). Sign-In promotes sign flips complementary to dense training by rescaling to in intervals, partially mitigating the vanishing inverse metric problem but inducing frequent perturbations to the optimization (see Appendix FT.
HAM, due to its implicit sparsity bias (Theorem 4.7p and improve plasticity, is particularly compatible with sparse training methods, as we showcase in multiple scenarios. We choose hyperparameters (a, 3) based grid search for dense training (see Figure II and Figure 12p. The chosen hyperparameters are transferred to all dense and sparse training methods. For ImageNet and Vision Transformers, we use (a, 3) (200, le 3) and for CIFAR100 (),3)= (200, 16c 3).
HAM improves generalization in a way that is complementary to Sharpness Aware Minimization (SAM) (Foret al.] 2002). We also apply HAM other tasks such as pre-training vision transformers, LLM fine-tuning and graph and node classification in Appendices G.3 G.6 and G.7 This demon- strates the general utility of HAM as an optimization principle. In addition, Appendix G.I verifies the improved (sparse) implicit bias proven in Theorem 4.5 for underdetermined linear regression.
Dense training Table 3 demonstrates that HAM improves dense training for ResNet50 on Ima- geNet (Deng al. 2009). Moreover, HAM works complementary to Sharpeness Aware Minimization (SAM) (Foret et al. 202T). Combining both algorithms (SAM-HAM) achieves the best overall per- formance. Table further highlights that training HAM longer (using similar compute budget as SAM, whose iterations are twice as expensive) achieves a similar improvement. Figure tracks the total L1 norm of the parameters during training to illustrate the complementary mechanisms of HAM and SAM. The same conclusions hold for similar experiments with the smaller vision benchmark CIFAR 100 (Krizhevsky et al.) 2009) (see Table 7). Furthermore, Table 8 showcases performance gains also for the transformer architecture DeiT (Touvron et al., 2021) trained on ImageNet with AdamW. A grid search for HAM's hyperparameters a and 3 on ImageNet (Deng et al. 2009) and
CIFAR100 00 (Krizhevsky et al. 2009) is visualized in Fig. 1T and T2 The best values for stable
across different tasks, while 3 needs tuning similar to weight ht decay.
Sparsification We demonstrate that HAM improves state-of of-tee-a----r pruning methods AC/DC and RiGI, as well random pruning initialization with the same hyperparameter configuration used
8

---

Published as a conference paper at ICLR 2026
HAM Sigu-li
d=2
0.8
Pervetiags
2-10
60
0.6
Interval
(a) Sign flips during training.
(b) HAM's Bregman function.
Figure 2: Demonstration of HAM's mechanisms. (a) The percentage sign flips during training for Random Pal with sparsity level 90% trained for 100 epochs, where each interval correspond to ten epochs. HAM able to consistently perform more sign flips than both the baseline and Sign-In. (b) Plot of the normalized Bregman function Ra, where increasing leads to L1 shape.
Table 3: HAM improves dense training of a ResNet50 ImageNet.
100 epochs
200 epochs
SAM, 100 epochs
+ SAM, 200 epochs
Baseline
76.72 + 0.19
77.27 + 0.13
77.10 + 0.21
77.94 + 0.16
HAM
+
77.86 87 0.05
77.92 0.15
78.56 + 0.12
in dense training. Table 4ilie44illuuaaaa illustrates that dense-to-spars, training becomes significantly better with HAM. Improvements are most significant for AC/DC, which uses dense training phases effectively. We attribute this also to the fact that AC/DC turns parameters indiscriminately, while RiGL does so based on gradient information. The improvements over PILoT and Sign-In show that we successfully extract the main beneficial mechanism of the hyperbolic geometry while mitigating the downsides.
Sign flip mechanism We show that HAM outperforms Sign-In (Gadhikar al. 2025), which promotes sign flips complementary to dense training and tries to mitigate the vanishing inverse metric problem by repeated parameter rescaling. HAM still induces more sign flips than Sign-In and standard training, as demonstrated by Figure 2(a) which in line with our theory (see Appendix E. Supporting this, show in Appendix G.4 the improvement for training with various fixed masks.
Table 4: Dense-to-sparse training and pruning at initialization with HAM on ImageNet with ResNet50.
Pruning type
Method
= 0.8
= 0.9
8 0.95
Random
73.87(++0.6)
71.56(=0.03)
68.72(+0.00)
Pal
Random + Sign-In
74.12(++0.9)
72.19(=0.18)
69.38(+0.1)
Random + -HM
74.84(=0.09)
72.72(+0.03)
70.05(=0.06)
AC/DC AC/DC + Sign-In
75.83(10.02)
74.75(=0.02)
72.59(40.11)
DIS
75.9(+0.14)
74.74(=0.12)
72.88(+0.11)
AC/DC HAM
77.2(10.14)
76.66(+00.1)
75.45(10.13
RiGL
75.02(+L0.)
73.7(=0.2)
71.89(+0.07)
DST
RiGL Sign-In
75.02
74.27(=0.08)
73.17(=0.17)
RiGL + HAM
76.22(=0.07)
(4.00(+
72.93(+0.1)
spred PILoT
72.64
71.84
69.47
Cont. spars.
75.62
74.73
71.3
STR
75.49(++0.1)
72.4(+0.11)
64.94(+0.07)
STR HAM
76.37(=0.18)
75.01(+0.02)
71.41

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We propose new hyperbolic update step that can be combined with any first-order optimizer and that improves generalization of dense and sparse training, making it suitable as a general purpose optimizer. Our algorithm HAM (Hyperbolic Aware Minimization) mitigates the vanishing inverse metric of the pointwise overparameterization m 3 w used in recent sparsification methods, while inducing similar implicit bias. Due to discretization, it is more suitable to control the strength and shape of the bias-ant and accordingly improve generalization in general. especially for dense-to-sye dense-to-sparse training. The main mechanisms how HAM achieves this are implicit bias towards sparsity and an acceleration of learning that promotes parameter sign flips. It remains an interesting open question different mirror maps could create better task and optimizer-speeifie awareness. For example, for some tasks one might want to take into account robustness; for the optimizer, it might be the momentum or normalization. This could lead more algorithmic advances improve generalization via implicit bias control and to new theory for understanding the success of deep learning algorithms. In particular, optimizers with implicit biases that emulate the positive effects of other types of overparameterization. without explicitly requiring huge models, may represent an important leap in reducing the high computational costs associated with deep learning.
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). We also gratefully acknowledge funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101 16395 SPARSE-MI.
REPRODUCIBILITY STATEMENT
For the theory, detailed proofs have been provided for the main statements in Appendix and used previously known statements have been provided in Appendix A For the experiments, the details provided in Appendix G with each experiment having its own subsection with accompanied specifics. The code use for the experiments is also attached.
LLM STATEMENT
To improve fluency of the text sentence level editing has been done using large language models.
REFERENCES
Felipe Alvarez, Jérôme Bolte, and Olivier Brahic. Hessian riemannian gradient flows in convex programming. SIAM Journal on Control and Optimization, 43(2):477-501, Jamuary 2004. ISSN 1095-7138.
doi: 10.1137740063111922499999
URL http:/ do.. www.:o........111..33111.1. /0t.
5036301 902419977
Shun-ichi Amari. Natural gradient works efficiently in learning. In Unsupervised Learning: Foundations of Neural Computation. The MIT Press, 05 1999. ISBN 9780262289033. doi: 10.755 047755//mppres/7011..0..000 h /tL wtt.ei551mmmmmmm http:///o.....o.........///trr/p
003. 0010
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175. 2003. ISSN 0167-6377. doi: http:://kaiiggy//oooooooooooooo0 com/ http://cciiooooarrr...e////////3
URLhttps: www. wwt.w.ccienneieee...o
Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Ye Yu Liu, Anima Anandkumar, and Yisong Yue. Leaminng compositional functions via multiplicative weight updates. In
10

---

Published as a conference paper at ICLR 2026
H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 13319-13330. Curran Associates, Inc., 2020. URL https: neurips www.ppprreepppprprppppppppppp files/papez/2020) F1le/9a32ef66ll f1le/444724666c22555577777555555
.pdf
Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. 2009. URL https://web
stanford edu webb//bboy/ccwxbook/
Stephen P. Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3:1-122, 2011. httthtts:://at semant i ccccoll ar. www./orrgs.sooooooooooooo ID:
51789432
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In Interna- lional Conference on Learning Representations, 2022. URL
net/
Tianlogg Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988 2021.
A. Cichocki, R. Zdunek, A.H. Phan, and S. Amari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. Wiley, 2009. ISBN 9780470747285. URL htttt::t:.. //bokk/....kkooo gwwgle.ooooooooooooooooooooooo.. twt.de//boooo wwloooosiddaaassstiii WgswC
J. Deng, W. Dong, R. Socher, L.-J Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) over di- agonal linear networks: Implicit bias, large stepsizes and edge of stability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. http://www...............o..aaaa httppeer__ieeeeeee/222320 ConpperreCofefence..oo .ddf
Curran Associates, Inc.,
2023. URL |attps: www.wnn.uiiicc.oo. cc/pppperrccppppp...pppppp. leee/
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awai minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https: htt////peeeeeieeeeeeeeeeeee wtt.://wwwweeeeeeeeeeeeeeeeeeeee www.eee/tfooooo wtt.ee/ooorrmm...m.mm..m....
1rM.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv: Learning, 2018. hhttts:://aaat wemanticscholaaa org/
CorpusID::338888
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Advait Gadhikar and Rebekka Burkholz. Masks, signs, and learning rate rewinding. In Twelfth International Conference on Learning Representations, 2024a. URL /tttps:///peerreleee
net/Eorum? Late/Eoorum1GGg000000888881
Advait Gadhikar, Tom Jacobs, Chao Zhou, and Rebekka Burkholz. Sign-in to the lottery: Reparame- terizing sparse training. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL h/tp:::://pppeeee /ttp://ieeeeeiieeeeeeeee net/forum?in www.iwKT77y 72E
Advait Harshal Gadhikar and Rebekka Burkholz. Masks, signs, and learning rate rewinding. In International Conference on Learning Representations, 2024b.
Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz. Why random pruning is all we need to start sparse. In International Conference Machine Learning, 2023.
11

---

Published as a conference paper at ICLR 2026
Egor Gladin, Pavel Dvurechensky, Alexander Mielke, and Jie Zhu. Interaction-foree transport gradient flows. ArXiv, Abs//2005.17775, 2024. URL httpt::t//aatt www..p...ssmannnco..lsas..a
org/CozpusID:27000 22:00163111700
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprini urXiv:2407.2178. 2024.
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization, 2017a. URL https://arxxi www. abs/
1705. 09280
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL ht httpp::///.. |tt.attts:://ppeeee fil elee//815 d2a 91
neurips. cc/paper.pppppeeepee files filess/paee//2117. Paapepadd
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1841.1841. PMLR, 10-15 Jul 2018. URL http:tt:/pppccceddi
mlr. wwwssppseeeeeppp v / gww.s//ggnaaaaaaaaaa.. html
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL http:/EEL/caaaaSSSaaaaaaaaaaaaa3 2017 htt//file////
www.:ss....e..p............e..ee cc/paper files/paper/
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: datasets for machine learning on graphs. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN
Zhiqiang HU, Yihuai Lan, Lei Wang, Wanyu Xu, Ec-Peng Lim, Roy Ka-We Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023.
Tom Jacobs and Rebekka Burkholz. Mask in the mirror: Implicit sparsification. In The Thirteenth International Conference on Learning Representations, 2025. URL httt:tttt///opereeeee
net/Eorum?ide 747ym1S3ut
Tom Jacobs, Chao Zhou, and Rebekka Burkholz. Mirror, mirror of the flow: How does regularization shape implicit bias? In Forty-second International Conference on Machine Learning, 2025, URL
net forumm wtw..rrmmmm..M..............
Adarsh Jamadandi, Celia Rubio-Madrigal, and Rebekka Burkholz. Spectral graph pruning against over-squashing and over-smoothing. In The Tiirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL
net http/formmmmmm
EMKrwJY2del
Tian Jin, Michael Carbin, Dan Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Pruning's effect on generalization through the lens of training and regularization. Advances in Neural Information Processing Systems, 35:37947-37961, 2022.
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar XXtv:200100836111 2020.
Mohammad Emtiyaz Khan and H. Rue. The bayesian learning rule. J. Mach. Learn, Res., 24:281:1- 281:46, 2021. URL https://aph) semant wwwcocar. waw.hhoaa...rrg/orrrsttD---.222. 0
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL [tt/stttss///pperrrrii net
Jyrki Kivinen and Manfred Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Technical report, USA, 1994.
Chris Kolb, Tobias Weber, Bernd Bischl, and David Rügamer. Deep weight factorization: Sparse learn- ing through the lens of artificial symmetries. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL
net. forum ? 7mn5
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateck Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity, Proceedings of the International Conference on Machine Learning, July 2020.
Denis Kuznedelev, Eldar Kurtic, Eugenia lofinova, Elias Frantar, Alexandra Peste, and Dan Alistarh. Accurate neural network pruning requires rethinking spanse optimization. Trans. Mach. Learn. Res., 2024, 2023. URL htttt: //api. semant ics wwwcchoolllac.. :tt..ooooo///
Denis Kuznedelev, Eldar Kurtić, Elias Frantar, and Dan Alistarh. Cap: Correlation pruning for highly-accurate sparse vision models. 36, 2024.
Eren Mchmet Kıral, Thomas Möllenhoff, and Mohammad Emtiyaz Khan. The lie-group bayesian learning rule. In International Conference on Artificial Intelligence and Statistics, 2023. URL
https: /api. semant ies.hnncccccllla wtagg/orppsslic ID: :2::2:574 257404948
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305..m 02299, 2023.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. Torr. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
Zhiyuan Li, Tianhao Wang, Jason D. Lee, and Sanjeev Arora. Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent. Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps:///ppeeeeeeeeeeee
net forumm id==21kd=s< 2OV
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: deight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- pwerpparamtterization in-time over over- prraaattetttiiitoooonn in sparse training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6989-7000. PMLR 18-24 Jul 2021b.
13

---

Published as a conference paper at ICLR 2026
Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic GNNs are strong baselines: Reassessing GNNs for node classification. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:/ httpss///pppeeeeee.e.oo wwwp://eteeeettttetaeea forum?id-
xk 4K
Yuankai Luo, Lei Shi, and Xiao-Ming Wu. GNN+:
Can classic GNNs be strong baselines for graph-level tasks? International Conference on Machine Learning (ICMI.), 2025. URL
htttps://axxiv. org/
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman, and Eric-Jan Wagenmakers. A tutorial fisher information, 2017. URI. htt.tt::///:/xxiiiiii http:///aarrioooo/bbbbbba//a//a/
org/ abs /175 00000000066
Bochen Lyu and Zhanxing Zhu. Implicit bias of (stochastic) gradient descent for rank-1 linear neural network. In seventh Conference on Neural Information Processing Systems, 2023. URL
htttss:/oppeneviiee net fonnt/oorrmmidee 2ttp://errreeee..eettoouuuunnne / |BEUTVZee
Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K. Warmuth. Exponentiated gradient
reweighting for robust training under label noise and beyond. ArXiv, 0bs/2104.01493 2021. URL
https://api. www.ntiisnccceeeo..... www.ehhoo...rr..oooooooeoeo wwwp/orpusc ID: :3300242882
James Martens. New insights and perspectives on the natural gradient method. J. Mach. Learn.
Res., 21:146:1-116776 2014. URL httpssts: seman ccccoollcr
10284405
Geoffrey J. McLachlan and Thriyambakam Krishnan. The algorithm and extensions. 1996. URL
https:://ppi.smman wew.mmannicccoo...ooooooo..
ID: 122530182
Péter Mernyei and Cätälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks,
https://aaaxv..or///bb////////// 02900
Alexander Mielke and Jia-Jie Zhu. Hellinger- gradii of entropy functionals. 2025. URL
gradient flows: Global exponential decay
275932494
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse con- nectivity inspired by network science. Nature Communications, 9(1), jun 2018. doi: 10.1038/ 1038:20....3.2011 44144444677770
CRLLhttps://doi.oor
Sree Harsha Nelaturu, Advait Gadhikar, and Rebekka Burkholz. TurboPrune: High-Speed Distributed Lottery Ticket Training. URL http:p////ggiiua........aaaaaaaa
Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim.,
2014. ISSN 2167-3888. doi: 10.1561/2400000000. URL 1 1(3):127-33)) Jammury 24C0000003
Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 8432 8442. PMLR, 2021.
Scott Pesme, Loucas Pillaud-Vivii and Nicolas Flammarion. Implicit bias of SGD for diagonal linear networks: a provable benefit of stochasticity. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL T
hittps::/openrevieet net www.:/ert/ooom/m/o
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating com- pressed/decompresssd training of deep neural networks, 2021.
Hoang Pham, The-Anh Ta, Shiwei Liu, Lichuan Xiang, Dung D. Le, Hongkai Wen, and Long Tran-Thank Towards data-agnostic pruning at initialization: What makes good sparse mask? In Thirth- Conference on Neural Information Processing Systems, 2023. URL https //openrevic net./ http://rwwmmmoxxxoooooxo////e
14

---

Published as a conference paper at ICLR 2026
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In The Eleventh International Conference on Learning Representations, 2023. URL https:
//operreview ew net. ftrrfoorrmmmm// dtt.rrmmmddt.tofofoo-oooo-oo
Chendi Qian, Andrei Manolache, Kareem Ahmed, Zhe Zeng, Guy Van den Broeck, Mathias Niepert, and Christopher Morris. Probabilistically rewired message-passing neural networks. In The Twelfth International Conference Learning Representations, 2024, URL https: httptt:///peeeeeeetee.e /wwp/reveeeeeeeeeeeeeeeeeeeeeeee
6Wcx 7gVk
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Tyrell R Rockafellar and Werner Fenchel. Convex Analysis. 1970. URL hctps://api
198122200199
Grant M. Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden Eijnden. Global convergence of neuron birth-death dynamics. ArXiv, abs/1992.00143
2019.
URL https://api
www.aaa.o.rg/ooooooeee ID: ID:
Celia Rubio-Madrigal, Adarsh Jamadandi, and Rebekka Burkholz. GNNs getting comfy: Community and feature similarity guided rewiring. In The Thhrteenth International Conference on Learning Representations, 2025. URL ptttttps://pperrvviiie
net/ wtte://ooommmmnig55500000
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification, 2021.
Oleksandr Shchur, Maximilian Mumme, Alcksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation, 2019. URL
org/ awsssasss...... 05511
05868
Yuesong Shen, Nico Daheim. Bai Cong. Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. In International Conference Machine Learning (ICML), 2024. URL htt.h:tts//// httpvvxxvvvvv.////v// org. /20p://.2///2221766
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-18109 2020.
Haoyuan Sun, Kwangjun Ahn, Christos Thrampoulidis, and Navid Azizan. Mirror descent maximizes generalized margin and can be implemented efficiently. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL httpps://openreviieet
Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Nisheeth K. Vishnoi. Algorithms for Convex Optimization. Cambridge University Press, 2021.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings Thirty Third Conference Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 3635-36733 3673, PMLR, 09-12 Jul 2020. URL wttpt:tp:/// wtttts:ss/ppoeeeeeeeeeeeee ngs. ml www.preaapp. serrss/v1255 wttp://owwwoo.oooooo h20a html
15

---

Published as a conference paper at ICLR 2026
Fan Wu and Patrick Rebeschini. Implicit regularization in matrix sensing via mirror descent. In Neural Information Processing Systems, 2021. URL httpss//app semanticscholaa.
orq/CozpusID:2300 235244812
Yuling Yan, Kaizheng Wang, and Philippe Rigollet. Learning gaussian mixtures using the wasserstein--isherrraa gradient flow. The Anmals of Statistics, 2024. URL https://apl
benanticschola..org wwt.:o/uusro/oooouuusooooou 2773139388
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge, Advances in Neural Information Processing Systems, 34:20838-20850, 2021.
Liu Ziyin and Zihao Wang. spred: Solving II penalty with sgd. In International Conference on Machine Learning, 2022. URL http::ttt/ssstt //api. semanticscho ar Sarg/Corpusl00
259075663
16

---

Published as a conference paper at ICLR 2026
A OPTIMIZATION DEFINITIONS AND RESULTS
In this section we recall some basic definitions from convex and non-convex optimization.
Definition A.1 (L-smooth) A differentiable function f _ + R is said to be L-smooth if its gradient is Lipschitz continuous with constant L 0. That for all 0, IR"
[V - [(( S 1.100 E|1
or equivalently,
((E) f(() (V 0) + 2 - 011
or equivalently,
|( - I ( f).
where f minøeR" f(0).
Definition A.2 (PI-inequality) A differentiable function f R -> satisfies the PL inequality with parameter A > 0 for all 0 E IR",
2A f*)
where f minnEr" f(0).
Definition A.3 (Legendre function Definition 3.8 ( et al. 2022 Let R: R uu be
differentiable convex function. We say R is a Legendre function when the following holds:
R is strictly convex on int(dom R).
For any sequence (0,) going to the boundary of dom R, lime- [R(O0))
In order to recover the convergence result in Theorem 4.14 in (Li et al. 2022) the function R also needs to be a Bregman function, which we define in Definition A.4 First, let us denote with DR denote the Bregman divergence with respect the generator function Rt:
DR(01 02) = R(O,)) - R((2) - (V (V))),), 0, 02)
for 0 02 E dom R.
Definition A.4 (Bregman function Definition 4.1 (Alvarez et al. 2004)) A function is called
Bregman function if it satisfies the following properties:
dom R is closed. R is strictly convex and continuous on dom R. is C² on int It).
For any 0 dom R and y E R. {E dom R)Dn((,E) S 7 is bounded.
For any 0 dom R and sequence (0.) C int(dom R) such that lim; 0: 0, it
holds that lim (R(0, 0,)
Theorem A.5 (Theorem 4.7 Alvarez et al. (2004)) If R is Legendre function with dom IR.", then if the domain of the convex conjugate dom R² _R" implies that R is Bregman function
From now on let R. be Bregman function A.4 Consider the Riemannian gradient flow:
d0, = ( f dt,)
6o = Oxrit-
This covers all settings considered in the main text: gradient descent, m O w and HAM as shown in Lemma B.4
Theorem A.6 (Theorem 4.14 (i et al. 2022) Assume that R is a Bregman function and that is
quasi-convex, f locally Lipschitz and argmin { ( 0 E IR" is non-empty. Then as t - oo, O, converges critical point 0*, Moreover, f is convex O, converges minimizer of f.
17

---

Published as a conference paper at ICLR 2026
Theorem A.7 (Theorem A.3 Jacobs & Burkholz 2025 Assume R satisfies for all 0 R"
Consider the same setting Theorem A.6
(V²R())) = 2 =
V2 R"
(5)
Furthermore, assume f satisfies the PL-inequality (A.2) Furthermore, the loss converges linearly with rate aA.
Then O, converges minimizer of f.
18

---

Published as a conference paper at ICLR 2026
B PROOFS OF THEORETICAL STATEMENTS
Here we provide detailed proofs of the main statements in the paper. The theorems correspondence
is:
The proof of Theorem 3.1 is
B.1
The proof of Theorem 4.2 is B.3
The proof of Lemma 4.4 is in B.4
The proof of Theorem 4.6 is in B.5
Theorem B.1 (Theorem 3. 1.1 If mg = sign w00
Vol. then
0x+1 02 exp (-1) (i 48))
(6)
equivalent to Eq. (7) up-to first order Taylor approximation.
First we use the Taylor approximation of the exponential function exp + O(22) to get the update:
0 D++1 0g Ox (f(),)) 4v,30, + (17²)
We show this is equivalent up to first order to the gradient descent of the overparametterzaaion:
x = M C Wk+1 02 (m + wf) vf(0+) 4n00,,
To do so, we show that m + w 2 2 for & [T] up zeroth-order approximation by induction. For k 0, the statement holds per assumption on the initialization. The induction step is:
m w m + 1 V 4n30; + n² (mk + w vf()))² ~ +0())+((m))
This concludes the induction and the proof.
Corollary B.2 Exponential gradient descent can not move through zero, preventing sign flips.
Proof. The operation exp(-) is always non-negative. Therefore multiplying with it will always keep the same sign since the sign operator is pointwise distributive:
sign(0x+1)
(0g)(0:) exp (-n (2 0x (2ssgng() (f:)))(())))))) +
+,8))
s0gn(0;) (exp (-n (2 sign 0g ((){) - 46)))
sign(Ox))
Note we use L- smoothness and sufficient small learning rate to ensure bounded gradient preventing (Ox))) oo. If the gradient explodes we still can only end up in zero leading sign(O so also no sign flip that case.
Theorem B.3 (Theorem 4.2) The gradient (n + 0) of Egs.
GD HYP is given by:
d0, - 10 (aV sign(0,)) dt,
1, 0
Writing out the computation of iterates 0₂ give us:
0: = 0y cxp
na sign sign((,) nB
vf(i)) exp
na soginn((0) (()g)()))(()))))) nn3
This allows take 0 and get integral equation for the dynamics:
0, = 0, exp (- J a sign ( ((..) + 3ds 1 vf(0.) exp (-/, (ing)() + 3de ds.
(7)
19

---

Published as a conference paper at ICLR 2026
Differentiating the first term under the Leibniz rule gives:
d (-% sign(0.)V{(O)) +dds)
00 exp 00exp
dl
0o exp (-/o a (isinnn,)n)))m (.) - Sds (-a
+
Next, differentiate the second term under the Leibniz rule:
d V5(0.) exp
a asign((.)V//(() + .dde ds)
dt 0 d V[(O,) exp
-V/(0))
(isign(O.) + 3dc ds
dl
(O, ! (f.())))x)) exp
sign 0 f(.)) + Bdc ds (-asign((0))) (,) -
Combining gives by noticing the form of 0::
d0, - f(O,)dt 0, (0. + 3) dt
= V f(O))dt
|0 (aV + sign 3)) dt
Lemma B.4 (Lemma 4.4) The Bregman function R. for a > 0 is given by:
(a [0:0:1 + 1) (a [0,| 1) [0:| 0, sign(Or.o)
(0) =
log(1 + al0,,ol)
a²
Proof.
We first construct the mirror map R. by using the corresponding Hessian 9HAM- Next we check that R₂ is a Bregman function. The Hessian of the mirror map R., (0) is:
1
v R. (0) + o0
Moreover we need (O,) = 0. Therefore by integrating twice, Ra for a > 0 is given by:
(a( = (a |0 + 1) In
[0 + +1 |9:|
(0gn( log (1 + alO,,o)
a²
It remains to be checked if Ra is Bregman. For this use relationship between Legendre and Bregman functions. We first show that R₂ is Legendre and its convex conjugate as well. Then follows from Theorem Ah5 Ra is Bregman for 0.
Note that we have dom = int = R. R. is strictly convex as for all E R the Hessian is positive definite. This shows the first statement. Next, since R. is separable we can show the second statement for each parameter separately. Take a sequence {0t, ) for coordinate j E [n] such that [0,y] oo then by construction of R& we have
lim Oy (0i.)²² ao
and log(+) increasing functions. Therefore Rr, is Legendre.
The convex conjugate gradient dom R (rangeV = R Therefore since R dom v VR2 domB, we can apply Theorem A.5 This concludes the result.
Theorem B.5 (Theorem 4.6 Let 0p O, then R., ~ 110112 La if 0 and R.. ||0 L, if a Co,
where ~ indicates proportionality i.e. there exists some positive functions h IR - R and hy R -
R in the neighborhood of the limiting point such that for all IR" lim h R2. 0 2
and lim (2( R. (0)=
||0 L
20

---

Published as a conference paper at ICLR 2026
Proof. The first statement follows from the Taylor approximation:
Ra "
02 10.1
00 1
which is valid if [0.] << so h1 (a) 1.
For large a >> O we have that Ra = Ei 10.l log(a/0:)). Therefore, we have
a log(a)
Ra (0) ~ Σ 00:1 = |00|
s0 l2
Tog(a)) which is positive 1. This concludes the proof.
The proof of Theorem B.5 follows similar steps as that of Woodworth et al. (2020).
Sign discrepancy In our implemented HAM algorithm (HYP# we use sign ( instead of (0g from the derived step (HYP). We now argue why this does not change the implications of our theory, as their gradient flows are equivalent. We show in Thm. B.6 in continuous time, the iterates (YP follow a jump process. For this jump process, the jump vanishes in the flow setting, leading to no discrepancy between using either version. In discrete time, however, this may not be the case. We provide an explanation for the effect discrete time Appendix [ The main implication of the flow being the same is that at the end of training the implicit bias is the same, as the end corresponds to smaller learning rates.
Theorem B.6 Initialize Oinit + O. Then the gradient flow of HAM iee Eqs. GD HYPE (1) -> 0)
given by: do, = -V)-(((() - |0.1 (O. sign(0.))) dt
0o Oinit
(8)
Proof. The statement follows from making the observation: Sign flips can only occur near zero; further away the processes equivalent.
Let θg denote the resulting process with - 0 and 9 the signed gradient flow. O, does not have to be gradient flow as can have discontinuous jumps due to the sign inconsistency. We can write the update as follows:
0 = Oo exp
na sign n1 - na (sign(0,+)) sign(0,)) v)(0,)
k-1
na sign ( (())) - 175 na (sign(0)+)) sign((,))))) v)(0,)
f(,)) exp
0=
Then the discrepancy i.c.
[sign( (0)4)) - sign (0,)) between the signs becomes a IR + R function
Of 0, +
8(0)= =
2 0; = 0+
for e [n]
-2 if 0, = 0-
The process for 0, with v - O can be written as
0. 0, exp (-f S/
Vf(O.) exp (-/, sign (a.nn + - - ds.
Differentiating under the Leibniz rule gives:
dō, -V - -10 (aVI((.) + sign + dl
(
0 =0= 0
21

---

Published as a conference paper at ICLR 2026
where 8: Rd + R" is a delta function for every coordinate. Therefore, the jumps vanish as they
multiplied with 0 we have that
d0, V f(
|0:l (Q.) sign(0))) dt 0. Oiitt
which is equivalent to the gradient flow of Theorem B.3
B.1 THE EFFECT OF NON-ZERO B
The convergence and implicit bias results Thms 4.3 and 4.5 focus on the case / = 0. In the follow, we discuss general the case of B 2 O. First, the flow takes the general form:
d0t 9 (f.) -0,dt,
(9)
We have for m O w that mt. (0.) - / 0 + 7/ and for HAM we have THAM = 1 a0 in
line with Table 2 For 3 >> 0, we can define the maanfold-regulariieee the corresponding explicit regularization.
This quantity determines
Definition B.7 For a time varying Riemannian gradient flow with off manifold weight decay Eq. ( the mamamioold is given by
M,(0)
Σ 8EIM]
where 9t is the i-th component of the seperable metric tensor.
Using Definition
B.7
we can compute Mj in both cases. This gives
a 10,1| In (la [0,] + 11)
Mrm (0) = Σ
03 + 22 and MHAM
(10)
a²
iE|n]
<E(x)]
Knowing this, we converge to the minimizer of the objective function f + 3M, assuming that there exists an M such that Mt M for 1 + xc. Note that, for m @ w. we have that M, - - for L- oc. This matches the LASSO optimization objective derived in spred (Ziyin & Wang 2022). Additionally for HAM, we have that aHMA > ||,, for + oo, indicating that we induce less sparsity, as we rescale with a. Concretely, for fixed 33 and large a, we approximately solve the LASSO objective with regularization coefficient /. Note that for large a, the explicit regularization strength decays while the implicit regularization gets closer to T.,.
convex, adapt the convergence result. As both on-mannfolddreguulrzzaaooo
Furthermore, obtain the implicit bias result, we use the mirror flow formulation. We know from (Jacobs & Burkholz] (025) that O corresponds to time-varying mirror flow for which we need > to recover optimality. contrast, for HAM we get the following mirror flow:
dv R, (0.) = (VS(0)) --((==)))))))
O,
dt.
0
6 initt
It a 0
This follows from the new objective function f 3.MHMMM To fulfill the optimality condition in the implicit bias result for linear regression, we would need to show that the mirror flow is in the span of z? to satisfy the KKT condition. This can only be guaranteed when the regularization turned off. Therefore, similarly to m C w. we would need to turn-off the regularization at the end of training to obtain optimality.
22

---

Published as a conference paper at ICLR 2026
C FISHER INFORMATION DERIVATIONS
Our algorithm can be interpreted through the lens natural gradient descent (Amari
1999 Martens
2014). Each optimizer step corresponds natural gradient update 0 0 79 T6p V (0ks).
with is now the Fisher information. The key insight is that parameters follow a known parameterized distribution that is learned by the optimizer. For gradient descent, (GD to normally distributed random variable @ with unit variance and learnable mean; i.e., 0 = E[X].
which corresponds
Thus we can interpret e as the mean of a normal distribution whose position is learned.
In contrast, the hyperbolic step (HYP) on its own corresponds to (0) 1/ which directly follows from the first order approximation of the exponential function. Similarly, we can match random variables Fisher information to the metric Suyp (f). It corresponds to random variable X parameterized as normal distribution with unit variance: /22 In this view, weights sign f [E by using E E
are recovered via magnitude of the expected position. Furthermore, if the sign is not correct, the hyperbolic step (HPP) will move the parameter exponentially fast towards zero, facilitating its sign flip. We provide derivations in next paragraph. To summarize, our combined update can be interpreted as follows:
20/0). This means that we learn the
Learn the position
GD and then the magnitude if the sign is correct; else move fast to zero HYP.
This mechanism is crucial to facilitate sparse training (Gadhikar & Burkholz, 2024a Gadhikar et al. 2025 as portrayed in our experiments (S 5) where HAM considerably boosts its performance.
Derivations of the Fisher information We provide here the Fisher information I == g calculations for one dimensional random variables N 1) and (( 1). The Fisher information is defined as see for example Definition 1.1 in (Ly et al. 2017):
(
log (x:00 ) 27
I(0) = Ex
(11)
where f is the probability density function of the random variable X and E is the expectation with respect to the random variable.
Let X 1), where 0 E IR is the mean parameter. The likelihood function is:
1
(- -092
f(X:0)
exp
22
Taking the natural logarithm:
1
((0) = log ((X:0) 1-71==1100000000 (og((T) マ (X - 0)2
The score function is:
de
= (X
dB
Then the Fisher Information is given by:
IGD (0) = df 27
= E((X 0)2 = Var(X)
d0
Let X ~ 11) The likelihood function is:
1 (x-211111)
f(x:)) = 22x
exp
The log- likellhooood function is:
{(0) = llg((7))
2. (X - 2//1112
23

---

Published as a conference paper at ICLR 2026
Differentiate to get the score function:
de
(- sign(0)
sign(0) X - 2v0
(X -v
d8
v10
v0
Now square the score and take the expectation:
dl 27 d0
X
0
27
1
[0Y E
E
TO
(S(XX-2100)) = [0 Var(X) = 0
Note that instead of being part of the exponential family of distributions this distribution part of the curved exponential family distributions.
24

---

Published as a conference paper at ICLR 2026
D DIFFERENT SIGNS IN THE EXPONENTIAL UPDATE
In this section, we show that using the updated signs (ign(0++)) ) in the hyperbolic step HHYP)) instead of the original ones sign(0,) is actually beneficial for performance. The main difference occurs when a sign flip takes place due to the gradient step (that is, sign (0+ & sign(0r)). which can lead to a discrepancy in discrete time. Then, updating the sign leads to an acceleration away from zero, the gradient f does not change and still points in the same direction. This further aids in preventing parameters getting stuck at zero, apart from the benefits of the hyperbolic step on its own. We present the different cases in Figure 3
new sign (9
++
Accelerate away from 0
Different
yes
sign( 3 of (Og) training)
Sign used HYP
Decelerate
old sign(
Sign flip? (0,)
GD see
sign O
Same
sign (0 2++) = sOg (at end of training)
Same implicit bias (Thm. B.6
Figure 3: The difference between using sign(O (+++ and (0) in HYP The main change we incur by using the new sign is that accelerates away from zero when a sign Hlp occurs. Thus, when parameters small, we can be more certain that they actually redundant. Furthermore, when sign flips become less frequent due to decreasing learning rate at the end training, we get the same implicit bias regardless, shown in Theorem B.6
25

---

Published as a conference paper at ICLR 2026
E ONE-NEURON TOY EXAMPLE SIGN FLIPS
We show with a similar arguments as in Gadhiikrr et al. 2025 that HAM allows for sign flips in toy example. For this argument similarly set 3 and have layerwise different a. In the same vein we argue that in presence of more overparameteriiation using the same
constant is fine. This is also empirically substantiated by observing more sign flips with HAM.
One dimensional neuron Consider a Gaussian i..d. data set 2, ~ (N( 1) with [d] Let f: IR + R be our objective function described by:
1
f(a, w) 2d
2 (v= awx(2)))²
where a(-) max (0, - the Rectified Linear Unit (ReLU), We want to learn target one dimen- sional neuron äo (z-), which generates the outputs y; for i [d]. Then gradient flow dynamics for HAM is described by:
do: = - (ailasl | + +1)++10000 1) we)dt,
Ginit
(12)
dw, - (a2 we/ + 1) 0w, f(a, wy wy)dt, Wo Winit
Note that standard gradient flow would get stuck at zero, as has to satisfy a balance equation in Lemma E. which is based on Theorem 2.1 in (Gadhikar al., 2025). The balance equation implies that if a5 wh C, then for all / 2 0 we have that of w C'Inn our case: C = 0). In order that the student with parameters (a, w) learn the ground truth, the parameters have to be able to sign flip when they don't i match the ground truth's 's sign. This can be divided into four cases, i.c. the total amount of sign cases. In the balanced setting for gradient flow, the ground truth is recoverable only the parameter signs align. Results of these four cases are shown in Figure
Sparse
Dense + Sign-In
Sparse
Dense
Sign-In
W
W
X
X
x
X
Figure 4: (Figure 2 from Gadhikar et (0025) ) showing sign flipping benefits achieved with pointwise overparameteriizttion m @ w, for the sparse and dense case on a single-hidden neuron model.
ā (w.) with ã be the teacher and the student network objective such that and w follow the gradient flow dynamics in Eq. (72) with random balanced parameter initialization. For 0 a2 O (standard gradient flow) flow) sign(ā).
Lemma E.1 Let ğ(a w)
the student only can learn one of four cases
i.e. when Winit > 0 and sign
Proof. If Wense is negative then it needs to flip its sign which is prevented by the ReLU activation. We know from the balance equation that for 2
Jall = [wel-
This implies that if a, then w, implying sāgn(ā). O
also not recover the case where sign(ainat) #
We now show that Eq. 12 can find the ground truth, even if the sign of is misaligned, similarly as in (Gadhikar et al. 2025) for the m @ w reparameterivation.
Theorem E.2 Let g(a, w) = (w:) with w > be the teacher and f be the student network objective such that and w follow the gradient flow dynamics in Eq. (22) with random balanced parameter initialization. Moreover, let on > 022 > O. If Winit > 0, then f can learn the correct target with probability 1 (1)² In the other case (Winit 0) learning fails.
26

---

Published as a conference paper at ICLR 2026
Proof. The proof idea is to show that for balanced initialization with Wintt > 0 the flow for t >> 0
always enters the open set
To := (a, w) 12 w. > 0}.
Furthermore, we show that the flow stays in the open set To. The system is Riemannian gradient flow which implies that the flow converges towards stationary point in Pp. It remains to be shown that the stationary point at the origin a saddle and the stable manifold of the origin is not To- Thus, the remaining stationary points are the global optimizers.
First we show that for balanced initializations [dimit] = Winit > enter the region To. which can be divided into cases. In case ainit Wanit O, have (Asnit: Winit) To- In case -ainit- Winit > O, we have that (Amitt Wiist) E To \ To i.e. the boundary of T'. Therefore, we need show the gradient field at (ainit, Winit) points into To.
The balanced initialization implies that
Oa Ja (ainity Winit)
Ow (dinity Wanit)-
> O. Using that 01 > 02 > we have that the gradient Moreover, since à > O, (ainstt Winse) field satisfies:
doinit
- (o01 |aiiir| + 1) (ainitt Winit dt
(a1 Wiisel + 1) f (amit Winit
< Cogluin + 1) a f (Asist: Winst )tt = -dwin
Therefore there is a to > 0 such that ato ~1 0. Thus there is to such that (atas Wto) L'.
We have entered the set ro have show that cannot leave the To- This can be shown by computing the gradient field at the boundaries. The boundary can be split up into three cases:
Bi = ((a, w) E R² -a = w > 0}
B3 {(a, w) E R2
0, 0}
. The origin (0,
The first case of B3 is covered by the balanced initialization. For the second case of B2 we can compute the gradient field again. We now only need that dwy, 0. We linearize dwy;
du, = Ca,dt > 0,
where -= - (0, 2,)² 2 > O with probability 1 1- -- The last case is the saddle point at the origin which we show not possible to be reached from the open set To- Thus for all (ainity Wanit) E To we have that for all 2 (a,, w.) To lim, (a,, w.) (0, 0).
In the case that w > the flow can be written as a dynamical system on a Riemannian manifold. This allows to guarantee convergence stationary point. The flow is given by
{
da C (as a.l + 1) (arwî we) dt dw; C (a [uy + 1) (afw, ; a.) dt
ag Ainit
Wo Wiit
where -= = EE max(0, 2, 2 > with probability 1 1- (2) This dynamical system has stationary points at the origin and the set aw = 1. The dynamical system is Riemannian gradient flow system therefore the flow converges to a stationary point. The stationary point the origin is saddle point. Therefore, the only way of getting stuck at the origin is when we initialize on the associated stable manifold. We show that this not possible for the balanced initialization. We calculate the lincarization of the stable manifold and use that the balanced initialization stays in l'o The linearization at the origin (0, 0) is given by
{
da= = Cw.dl
dw Ca,dt.
By a direct calculation of the eigen vectors the lincarization of the stable manifold is given by the vector (-1, 1). This is the exact boundary of To, for which showed that for finite Winit and Osnit
27

---

Published as a conference paper at ICLR 2026
enter Tg. Suppose that from Tg the stable manifold is reachable. Then there is continuous differentiable curve ") with initialization "o = (aini, Winat) E To such that lim "/t = (0, 0). This is not possible it violates the gradient field at the boundaries of To- Thus, the flow does not converge to the stationary point at the origin. This concludes the first part, since the only set of stationary points the set of global optima.
The other two remaining cases fail as the boundary at w = 0 is not differentiable and the gradient flow stops there. D
Theorem E.2 highlights a benefit of HAM over gradient flow. A key difference with the proof in (Gadhikar al. 2025) that now the stable manifold is exactly the boundary at To. Therefore, are relying on the non-linearity of the model to push us into the open set To. A similarity between the proofs is that rely on a > Q2 O. In the next part we argue that for multidimensional inputs this is not necessary and we can use single constant a.
Multidimensional neuron We can consider the gradient field at a balanced initialization for multidimensional input case. Then we have the following inequality:
(atlainit) + 1) = (Pmtteant + +1 2 Vn
(clleWsnielll,, + 1) 2 Σ (alwin,d +1)
(13)
where we used the relation between the L1 and Le norm. Note that now there is significant gap due to switching between /n and n. This inequality ensures that at initialization the gradient field is pointing in a similar direction as for the one dimensional case, promoting useful sign flips (Gadhikar
et al. 2025).
28

---

Published as a conference paper at ICLR 2026
F ILLUSTRATION OF VANISHING INVERSE METRIC
We track the average inverse metric coefficient at 0. This implies for HAM we get (0) = by definition of its inverse metric. For m w get g (0) m w We track the average during training in the first layer of ResNer50 trained on Imagenet. We consider 4 scenarios: HAM, Sign-In for 90% sparsity according (Gadhikar al. 2025). dense training m 3 ur with and without weight decay. The weight decay selected for m @ w IS set to 2e - 5, which is less than half of case of dense training. Note that both cases Frobenius decay i.e. I| @ w Itz, is applied in accordance with (Jacobs & Burkholz, 2025
the strength it would
In Figure [ we observe that the inverse metrics of m w decays severely when weight decay is applied. For the reparameterization, weight decay is needed to induce sparsity, so in order to use for sparsity needs to be used. Furthermore, note that even though Sign-In manually resets the rescaling at the start of training the metric decays at the end of training.
1.0
HAM
Sign-In, 90% m m w
0.8
[0)
m O W wd
0.6
0.4
0.2
0.0
0
25
50
75
100
Interval
Figure 5: The first layer of a Resnet50's average inverse metric at zero reported at every tenth epoch.
29

---

Published as a conference paper at ICLR 2026
1|x 1- X 1112
10a
GD
HAM
HAM signed
10
10"
10
10
10
Steps
Figure 6: Gradient flow simulation of HAM, HAM with corrected sign and gradient descent. Observe the slight benefit of using sign( (+++ instead of (Ox)
G EXPERIMENTAL DETAILS
We present the additional hyper parameters and other details of the experimental setup. Furthermore we provide ablations on various setups. In general HAM is applied to layers except the batchnorm or layernorm layers.
G.1 OVERPARAMETERIIZD LINEAR REGRESSION
We illustrate Theorem 4.5 by considering a under-determinet linear regression setup, similar that of Pesme et al. [2021]: Jacobs & Burkholz (2025). We consider a sparse groundtruth 0* and initialize Og Moreover, we use the mean squared error loss function. We generate data by sampling z; (o, I) i..d. for [d]. with d 40 and
100. We compare gradient descent with and without HAM. Moreover we also show what happens if we replace sign x++/ (0x++)))) with sign(Os
denoted with HAM signed. The learning rate is r) 10- 4, and both algorithms
for 10e + 6 steps. We track the distance to the ground truth during training. Figure 6 we observe that HAM gets closer to the ground truth and converges faster then both gradient descent and HAM signed, where we set a = 1000. This corresponds to less strong sparse implicit bias than L1-
In the same setting, illustrate why need both steps GYHHP
We do this with an ablation, i.e. by using the hyperbolic step (HYP*) and gradient descent (GD) on their own. We initialize 10e
1. This means initialize with the opposite signs compared with the ground truth. In Figure 7 we observe that HAM reaches close to the ground truth, while the exponential step diverges, can not reach the ground truth due having no sparse implicit bias. Moreover, gradient descent can also not reach the ground truth on its own. Therefore, both the hyperbolic and the gradient steps are necessary.
Furthermore, if we increase and decrease the learning rate we can recover the ground truth solution. Concretely, consider HAM with the following configurations (a, m) (103+j, +j 10-4-14) for j E [3]. In Figure 8 we observe that we get closer to the ground truth.
G.2 DENSE AND SPARSE TRAINING ON VISION TASKS AND ABLATION
We provide additional results on CIFAR100 (Krizhevsky al. 2009) in Table 7 Furthermore, we train a small DeiT Touvrro et al.] 2021) with Adam in Table All results are for 3 seeds. We provide the hyperparameter grid search for CIFAR 100 and Imagenet (Deng et al. 2009) in Figures 11 and 12 We find that the grid search is consistent i.e. there is a global best configuration. This implies is easy to tune for specific tasks. We illustrate the convergence and implicit bias behaviour by tracking the training loss and L1 norm in Figure We also track the L, norm when comparing with SAM to show that SAM and HAM exploit different principles in Figure I0] The additional hyper-parameters of the experiments be found in Table 5 The same parameters used for the
30

---

Published as a conference paper at ICLR 2026
10
1|122
10
GD
10
HAM
Hyp
10
10
10
10
10"
10
Steps
Figure 7: HAM vs hyperbolic step HYP under the incorrect sign initializations.
10
10
10
a
1000
10
a
10000
10
a
100000
1000000
10
10
10
10
10
Steps
Figure 8: Gradient flow simulation of HAM with corrected sign for different a. Larger a leads to closer ground truth recovery.
31

---

Published as a conference paper at ICLR 2026
500000
Baseline
Triin Loss
LJ norm
400000
200,
a 0 -3
300000
200, B 3
2
2000000
25
50
75
100
0
25
50
75
100
Epochs
Epochs
Figure Training dynamics of HAM compared the baseline for ResNet 50 on Imagenet. Observe in the first figure (left) that the average training loss converges faster with higher given the same B. This illustrates the convergence speed up predicted by our developed theory. Moreover, in the second figure (right) the average L1 norm decays more due the regularization constant B, whereas larger leads to larger initial increase in the average L1 norm it decays faster in the end. This is in line with being more uncertain about the sign of the weights in the beginning of training.
sparse training setup. To reproduce sparse training methods including AC/DC, RiGL and STR we use hyprraarameters prescribed by the authors. Each experiment was run on A100 GPUS. The code used is based on TurboPrune in (Nelaturu et al.).
Parameters for m @ w The m @ w parameterization is not regularized with weight decay for the dense scenario, as this induces sparsity. Instead, weight decay is applied on the product ||m @ w 2 with strength 5, the same strength as for dense training (Gadhikar al. 2025).
HAM optimization To optimize with HAM for dense and sparse training setups on vision tasks, we use a 200 and le based on our grid search in Figure 12 Additionally, we clamp the exponent in the HAM step (see Equation HYP* between [-5, to avoid exploding gradients. Note, all experiments, HYP* is not applied BatchNorm LayerNorm layers.
Table 5: Training Details for the dense vision experiments presented in the paper.
Dataset
Model
LR
Weight Decay
Epochs
Batch Size
Optim
LR Schedule
CIFARI00
ResNet18
0.2
150
SGD, m 0.9, SAM
Triangular
ResNe150
0.25
100, 200
SGD, m = 0.9, SAM
Triangular
ImageNet
DelT' Small
0.005
300
1024
Adam
Triangular
Table 6: HAM improves dense training of ResNet50 on Imagenet (Deng al. 2009).
Dataset
Baseline (no HAM)
a a= le
== 200, 0 a
200, B le
HAM, 100 epchs
76.72 + 0.19
77.01 0.14
16.72=0.00
77.51 0.11
HAM, 200 epchs
77.27 + 0.13
77.48 1 0.09
7..24
77.86 1 0.05
SAM-HAM, 100 epchs
77.10 0.21
77.53 + 0.16
77.21 0.09
77.92 0.15
SAM-HAM, 200 epchs
77.94 0.16
78.17 0.16
77.60 0.03
78.56 + 0.12
Table 7: Dense training with HAM on the CIFARI00 vision benchmarks.
Dataset
Baseline
= 0,3 = 16c 3
a= =2200, 8 = 0
200, 3 16e 3
HAM
75.25 + 0.24
75.36 + 0.04
75.31 + 0.30
76.12 + 0.27
SAM-HAM
75.12 + 0.68
76.30 + 0.11
75.25 + 0.20
76.65 + 0.23
32

---

Published as a conference paper at ICLR 2026
500000
Baseline
200, B = a = 0, le 3
400000
LI norm
a 200, B 1e 3 Baseline, SAM
300000
= 200, B = 0, SAM
a 0, B le 3, SAM
a = 200, B = le - 3, SAM
200000
0
50 Epochs
100
Figure Training dynamics of HAM with and without SAM for ResNet50 on Imagenet. Observe that the choice of our hyperparameters and 3 determine the general trend of the average L1 norm while the choice between SGD and SAM make less a difference. This provides additional evidence for their complementary working.
74.96
75.13
75.43
75.09
74.29
73.44
76
Ie-3
75.35
75.23
75.60
75.50
75.01
73.83
75
lc-2
75.46
75.91
75.52
75.72
75.13
75.21
Accuracy
B
lc-1 32c-3 16c-3
74
75.4
75.98
75.97
76.30
76.10
74.74
75.74
75.78
76.37
75.75
75.92
74.55
733
73.58
73.57
73.38
73.72
73.77
72.02
100
500
1000
Figure 11: One seed hyperparameter search for ResNet18 on CIFAR100.
33

---

Published as a conference paper at ICLR 2026
0
76.61
77.02
76.89
76.47
77
le-2 1e-3 1e-4
76.86
77.16
77.00
76.86
B
76
77.03
77.34
77.57
77.52
75
74.68
74.90
74.49
74.65
0
100
200
300
a
Figure 12: One seed hyperparameter search for Resnet 50 on Imagenet.
G.3 HAM FOR VISION TRANSFORMERS
We also verify that HAM can boost pre-training performance for ViTs trained with AdamW, for both dense training as well as sparse training with AC/DC shown in Table [ and Table 9 respettielly We use the same hyperparameters as for the ResNet-50 trained on Imagenet: (a, 3) = (200, 1e 3).
Table 8: Pre-training vision transformer from scratch for 300 epochs on ImageNet.
Setup
AdamW HAM
AdamW
ImageNet, DelT Small
72.62:0.22
72.31
Table 9: Sparse pre-training of vision transformer with AC/DC for 300 epochs on ImageNet at 50% sparsity.
Setup
AC/DC HAM
AC/DC
ImageNet, Deil Small
73.24
72.5+0.16
G.4 TRAINING WITH HAM AND DIFFERENT SPARSE MASKS.
HAM can be used to optimize sparse networks with different mask topologies. We train the nonzero weights of sparse mask topologies identified by different sparse methods including AC/DC, RiGI, STR and Pal masks. The weights are randomly initialized and optimized with HAM.
Note, this is equivalent to pruning at initialization with the mask obtained from the listed methods. We see consistent improvement across all topologies with HAM except the SNIP mask, which was unstable to train also without HAM. HAM performs best for sparse masks identified by RiGL and random pruning. potentially due to better trainability and good layerwise sparsity ratios identified by these methods, which influences performance when the mask does not change during training. Results provided in Table 10
34

---

Published as a conference paper at ICLR 2026
Table 10: HAM with different masks for ResNet50 trained on ImageNet with 90% sparsity and random initialization. (* denotes a single the runs for other seeds crashed.)
Init
Mask
Base
Sign-In
HAM
AC/DC
70.66.0.11
70.96
71.84
RiGL
72.02:0.23
12.48
:3.01:0.01
STR
68.36-10.17
68.75 68.75-0016 67.31
Snip 52.9*
54.27*
44.4840.57
Synflow 60.66.0.2
60.5944.07
62.4
Random
71.5640.03
72.1940.18
72.72+0.03
G.S PRUNING DENSE MODELS
Here we magnitude prune the best dense models acquired from both HAM and standard training. We observe in Table TT that HAM's implicit sparsity bias makes the model perform relatively better then the standard dense model.
Table 11: Validation accuracy comparison between Pruned HAM and Dense models at different sparsity levels.
Sparsity
HAM
Dense
0.9
1.384
0.104
0.8
59.008
0.424
0.7
71.940
37.776
0.5
76.824
73.404
G.6 FINETUNING LLMS
As we show, HAM can also boost the performance of LLM finetuning. We evaluate on the common- sense reasoning benchmark (Hu al., 2023) finetune LlaMA 3.2 models (Grattafiori al. 2024) and report accuracies across eight benchmarks in Table 12 On average, HAM improves on this task, demonstrating its compatibility with the optimizer ADAM and the LoRA architecture.
Table 12: Performance of LoRA
HAM the commonsense reasoning (Hu al.
2023)
benchmark.
LlaMa 3.2
Size
HS
WG
PQ
AE
AC
OB
SQ
BQ
Avg
LoRA
63.8
65.8
74.04
67.63
55.88
63.6
70.98
64.25
65.74
LoRA + HAM
64.8
68.35
74.21
68.39
51.79
65.8
71.2
62.17
65.83
LoRA
3B
88.8
80.66
83.73
73
82.65
66.89
76.8
78.19
69.44
78.39
LoRA + HAM
3B
89.40
80.58
82.69
81.77
68.43
80.2
78.25
69.48
78.85
Each experiment was run on 4 A100 GPUs. We use th experimental setup of DoRA et al. (024) to finetune I.aMA 3.2 models of size 1B and 3B with a 200, B = 1e-3 and = 100, 3 le respectively. We find that larger models benefit from less strong regularization which is consistent with our different regularization strengths 3 for ResNet18 and ResNet50.
Experimental details
35

---

Published as a conference paper at ICLR 2026
G.7 GRAPH AND NODE CLASSIFICATION
We report the experimental details of the 4 graph classification benchmarks from Table 13
We also include results on 13 node classification benchmarks, and an ablation on a < 0. Each experiment was
A100. We consider 4 graph classification tasks on the GCN architecture (Kipf & Welling 2017) in Table 13, and 13 node classification tasks on GCN, GATv2 (Brody et al. [022]] and GraphSAGE (Hamilton al. 2017) in Tables T5 T6. We also include the hyperparameter grid search, as well as an ablation on negative a. The success of negative values indicates that node classification prefers different type of implicit bias. Nonetheless, we see consistent improvements across almost all datasets and architectures with a > 0.
Table 13: Evaluation of HAM on 4 graph classification benchmarks from OGB (Hu et al. 2020
Dataset
ogbg-ppo
ogbg-code2
Metric
Accuracy 1
Avg. Precision t
AUROC t
Fl score f
GCN
75.48 + 0.15
27.57 + 0.04
82.37 = 0.29
13.89 + 2.11
GCN + HAM 75.72 = 0.24
27.81 + 0.22
82.50 + 0.69
13.96 + 2.06
G.7.1 GRAPII CLASSIFICATION
We report the performance of HAM on four graph classification datasets from Open Graph Benchmark (OGB) (Hu et al. 2020). The code to run these benchmarks is based on (Luo et al. 2025), using their choice of hyperparameters and ADAM as the optimizer. We use their GCN+ architecture, which a GCN equipped with edge features, normalization, dropout, residual connections, forward networks, and positional encodings. In order implement HAM in combination with dropout, mask the regularization term 33 with (grad * 0). We only apply HAM on the weights and biases associated with the convolutional layers. The results, shown in Table 13 are averaged over seeds. We report the best validation metric for the best values of and 3 for HAM, the selection of which is displayed in Table 14 The tuning range a E {1, 10, 100, 200), and 3 E {0. 0.01, 0.1}. Table 14 also includes the size of the datasets in terms of number of graphs. Note that for three datasets, is the same as for the vision tasks.
Table 14: a and 3 best values for the graph classification tasks.
Dataset
ogbg-ppa
ogbg-molpeba
ogbg-molhiv ogbg-code2
# graphs
158,100
437,929
452,741
41,127
200
200
200
1
0.1
0.1
0.1
0.01
G.7.2 NODE CLASSIFICATION
We furthermore report the performance of HAM on thirteen node classification datasets: Cora, CiteSeer, and PubMed (Kipf & Welling 2017). Wiki-CS (Menyyi Cangeal 2022), Coauthor-CS,
Coauthor-Physics, Amazon-Computerr and A Amazon-Photo (Shchur et al. 2019) (homophilic):
Amazon- Amazon-Ratings, Squirrel, Chameleon, Minesweeper, and Roman -mmare (Platonov et al. (023) (heterophilic). We evaluate over GCN, GATv2, and GraphSAGE. We only apply HAM on the weights and biases associated with the convolutional layers and on the attention parameters. The code to run these benchmarks is based on (Luo et al. 2024). using their choice of hyperparameters and ADAM as the optimizer. The results, shown in Tables 15 (homophilic) and 16 (heterophilic), are averaged over different runs according to the original setup. We report the best validation accuracy for the best values of a and | for HAM, the selection of which is displayed in Table 17 The tuning range is {1, 10, 100, 2003. and 3 {0, 0.01, 0.1). The best value per architecture in bold, and ties underlined. Tables [5 and 16 also include the size of the datasets in terms of number of nodes and edges.
36

---

Published as a conference paper at ICLR 2026
Table 15: Evaluation of HAM homophilic node classification benchmarks.
Dataset
cora
citeseer
pubmed
wikses
coauthor-cs es
coauthor physics
amazzon compuier
umazon photo
2,708
3,327 4,522
18,333
34,493
13,752
7,4%0
edges
245,861
GCN
81.32 2: 0.30
68.60
11:600::00 0.94
77.96
0.16
80.71
0.29
95.32 0.1.2
97.17
0.01
83.47
0.70
94.13 0.61
GCN+HAM
81,44 0143
68.64
0.97
78.16 4: 0.65
80.84 1: 0.27
95. 52 4: 0.04
97.18
0.02
83,93 1: 0,58
95.33 0,23
79.00
0.95
95.33
GAT-HAM
81.56
0.67
68.92 0.27
79.08 0.88
82.47 1: 0.38
95.33 0.07
97.16 0.01
84.40 1: 0.92
94.00 0.69
SAGE
80.44
1.03
67.44
0.26
79.
0.67
X1.72 + 0.46
95. 50 0.08
97.01
0.10
83.07 + 0.90
94. 0.3.1
SAGE-HAM
67.48
79.96
0.62
81.78
0.41
95.58 0.09
97.03
83.60 1,11
95.40 0.40
Table 16: Evaluation of HAM on 5 heterophilic node classification benchmarks.
Dataset
amazon-rating
squirrel
chameleon
minesweeper roman- amaan-empire
# nodes
24,492 93,050
2,223 46,998
890 8,854
10,000
22,662
# edges
39,402
32,927
GCN
53.23 0.54
44.52 + 1.12
46.12
2.38
97.46
0.24
90.96 + 0.33
GCN+HAM
53.43 + 0.44
44.55 + 1.28
46.78 + 2.22
97.78 + 0.53
91.22 + 0.40
GAT
55,47 + 0.20
42.22 + 1.73
45.84 + 3.02
97.98 + 0.21
90.58 + 0.91
GAT+HAM
55,58 + 0.47
43.17 + 1.37
46.37 + 3.32
98.37 0.46
90.83 + 0.89
SAGE
55.05 + 0.50
40.91 + 1.27
42.80 + 2.90
97.02 + 0.59
90.51 + 0.33
SAGE+HAM
55.50 0.55
40.85 + 1.16
43.07
2.84
97.77 0.16
90.57 + 0.44
Table 17: and 3 best values for the node classification tasks.
GCN
GAT
SAGE
Dataset 4
B
B
a
B
200
0.1
200
0.1
200
0.1
citeseer
0.01
10
0
200
0.01
pubmed
200
0.1
100
0
10
0.01
wikics
10
0.01
1
0.1
1
0.1
coauthor-es
0.1
10
0.1
10
0.1
souuthor-physice
10
0
10
0.01
1
0
amazon-computer
10
0.01
10
0
200
0
amazon-photo
10
0.01
10
0
1
0.01
amazon-ratings
10
0.01
200
0.01
200
0
squirrel
10
0.01
200
0
200
0.1
chameleon
200
200
0
200
0.1
minesweeper
200
0.1
100
0.1
1
0.01
roman-empire
10
0.1
10
0.1
10
0.1
Node classification with O We perform ablation the node classification tasks by assigning negative values to a, denoted as nHAM. Tables 18 ((ommppiilicc) and 19 (heterophilic) show the results of this ablation, while Table displays the optimal pair of (a: B3) hyperparameters. Note that the baselines never perform better than HAM or nHAM. Surprisingly. heterophilic datasets appear be able to benefit more consistently from nHAM, especially for GCN and GraphS In homophilic datasets and GAT's, it still provides consistent but smaller improvements, or matches > 0. This intriguing phenomenon requires further investigation, as may
the best performance indicate the need for different kind of implicit bias in certain graph- based architectures. Other methods particularly effective heterophilic settings, such as modifying the adjacency matrix (Qian) et al. 2024 Jamadandi et al. 2024 Subio-Madring et al. 2025), may offer insight into these results.
37

---

Published as a conference paper at ICLR 2026
Table 18: HAM (a > 0) compared to nHAM (a <0) on 8 homophilic node classification benchmarks.
pubmec
wikiss
couuthrrphysics
ummoo--cmmplkr
amazon-ppoot
GCN
0.30
60
0.94
77.96
90.71
0.29
95 35.32 0.12
97.
83,47
0.70
0.61
GCN+HAM
$1.44 1: 0.43
68.64 2: 0.97
78.16
0.65
30.84
0.27
95. 4: 0.04
97.18
0.02
83.93 = 0.58
95.33 1: 0.23
GCN+aHAM
$1.44 1: 0.26
68.68
0,99
0.58 78.08
0.30 80.89
95.34 0.09
97.17 0.00
84,20 0.40
95.27 0.50
$2.35
GAT-HAM
$1.56 0.67
0.27 68.92
68.92
79.08
82.47
0.38 $2.45
0.07
97.16
84.40 0.92
94,60
0.69
GAT+aHAM
$1.56 4: 0.43
0.18
79.08 0.88
0.41
95.33 4: 0.07
97.16 0.01
83.93 0.31
94,90 1: 0.72
SAGE
50.44 1: 1.03
67.44
0.26
79.36
81.72
0.46
95.50
0.08
97.01 0,10
83.07 0.90
95.33 0.31
95.58
SAGE+aHAM
$1.00 +: 0.35
67.48 0.23
79.84 0.93
$1.82
0.45
95.56
0.12
97.02 0.10
83.60 = 1.04
95.67 :: 0.61
Table 19: HAM (a> 0) compared to nHAM (a <0) on heterophilic node classification benchmarks.
Dataset
amazon-ratings
squirrel
chameleon
minesweeper
roman-empire
GCN
53.23 + 0.54
44.52 1.12
46. 2.38
97.46 + 0.24
90.96 + 0.33
GCN+HAM
53.43 + 0.44
44.55 + 1.28
46.78 = 2.22
97.78 + 0.53
91.22 + 0.40
GCN+*HAM
53.43 + 0.25
44.58 1.09
46.78 = 1.85
97.85 + 0.10
91.23 + 0.36
GAT
55.47 + 0.20
42.22 + 1.73
45.84 + 3.02
97.98 0.21
90.58 + 0.91
GAT+HAM
55.58 + 0.47
43.17 1.37
46.37
3.32
98.37 0.46
90.83 0.89
GAT+1HAM
55.76 + 0.55
42.84 + 1.25
46.23 + 2.91
98.53 + 0.25
90.74 + 0.82
SAGE
55.05 + 0.50
40.91 1.27
42.80 + 2.90
97.02 0.59
90.51 + 0.33
SAGE+HAM
55.50 + 0.55
40.85 1.16
43.07
2.84
97.77 + 0.16
90.57 + 0.44
SAGE+1HAM
55.25 + 1.00
41.11 1.61
43.32 + 2.92
97.81 + 0.14
90.64 0.56
Table 20: and /3 best values for the node classification tasks with 0.
GCN
GAT
SAGE
Dataset 4
8
8
cora
-10
0.01
-10
-200
0.01
citeseer
-1
0.01
-10
0
-100
0
pubmed
-00
0.01
-100
wikics
100
0.1
-10
0.01
-10
0.01
coauthor-cs
-10
0.01
-100
0.01
-100
0.01
coauthor-physies
-1
0.1
-1
0.01
-10
0.1
amazon-computer
200
0
0.01
-1
0.01
amazon-photo
-200
0
200
-1
0.01
amazon-ratings
-10
0
-200
0.1
-1
0
squirrel
-10
0
-20
0.01
-200
0.1
chameleon
-20
0
-100
-200
0.1
minesweeper
-100
0.1
-10
0.1
-200
0.1
roman-empire
-200
0.1
-10
0.1
-1
0.1
38


Reviews:
[{"review_id": "wMlaXHK2yB", "reviewer": "Reviewer_PXeq", "summary": "- This paper studies the training of sparse networks, specifically those which learn the mask as $\\theta = m \\odot w$\n- This paper builds on previous work which identify a limitation of these models in learning sign flips due to the small inverse Riemannian metric as $m$ or $w$ approaches zero\n- Motivated by the reparameterized gradient flow of these masked models, the authors propose interleaving hyperbolic updates between regular gradient steps.\n- The hyperbolic updates inherit the favourable sparsity inducing dynamics from training masked models, while the regular gradient steps handle sign flips\n- Empirically, HAM outperforms methods which only use the base optimizer on ImageNet and CIFAR-100 on both dense and sparse training", "questions": "- I'm generally interested in evaluating whether the regular optimization step is strictly to facilitate sign flips, or offers more performance beyond that. To this end, I was wondering, (1) what is the effect of only the hyperbolic step, and (2) what is the effect of only applying the regular gradient step for small magnitude weights (i.e. ones whose signs could be flipped).\n- What kind of sparsity does HAM trained models get (without combining with any sparse training). I.e. is direct magnitude pruning at the end of initialization more successful with HAM than with standard training due to HAM's implicit bias to sparse networks?\n- Can the authors offer any intuition why HAM improves performance in dense training? Given that the motivation of the paper was directed towards sparse training, the improvement in dense training is surprising.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear motivation and presentation\n- The method is compatible with existing optimizers (such as SAM), and requires little computational overhead\n- Strong empirical results - with the method surprisingly also improving dense training\n- Thorough theoretical analysis", "weaknesses": "- The method additionally adds two hyperparameters ($\\alpha$, $\\beta$). Based on figure 8 and 9 in the appendix, the performance of HAM is quite sensitive of these hyperparameters, and optimal hyperparameters seems to differ between models, making it difficult to tune\n- It seems that although HAM is motivated from the perspective of sparse training, it doesn't actually improve sparse training directly without pairing it with another method (remark 4.7 and 4.8). Given this, the motivation behind HAM seems less clear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnzimLowDT", "reviewer": "Reviewer_vz7i", "summary": "This paper proposes a new optimizer step: Hyperbolic-Aware Minimization (HAM), where first a nominal step is taking by the base optimizer (e.g. Gradient Descent), then a \"hyperbolic\" step analogous to exponential gradient descent. By taking the sign of the previous iterate into account, this method reduces the updated iterate's magnitude for the components of the previous iterate and the gradient that agree, otherwise the exponential step blows up the updated iterate's magnitude to exaggerate the descent step on disagreeing components. The optimizer behavior is supported by deriving the corresponding Riemannian gradient flow, and applying it to the toy problem of underdetermined linear regression. Therein, it is found that HAM induces an in-between of $L^1$ and $L^2$-flavored regularization, with the behavior approaching $L^1$ when the hyperparameter $\\alpha$ is large. It is noted that this implicit bias toward sparsity is rather weak. The efficacy of HAM is then demonstrated on a wide range of training tasks.", "questions": "On the note of the hyperparameter scale, in line with recent lines of literature studying the effective learning rate induced by various hyperparameters, is it possible that the \"correct\" scale of $\\alpha$ (and $\\beta$) actually has an implicit dependence on the scale of the parameter block (e.g. width)? For example, the Maximum Update Parameterization framework (see e.g. [1]) suggests that layer-wise descent methods require a per-layer adjustment based on the dimensionality in order to guarantee stable evolution across model scales. It seems that an entrywise modification like HAM should also naturally admit some layerwise dimensionality scaling. In particular, did the authors experience the optimal choice of $\\alpha, \\beta$ changing across different model sizes?\n\nThe HAM step compares the sign of the gradient with the iterate. When using a different descent method than GD, is the hyperbolic step adjusted accordingly, or does it still compare to the gradient?\n\n[1] Yang et al. \"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper proposes an interesting optimizer modification that attempts to reap the benefits of \"point-wise\" overparameterization via parameter-wise multiplication without introducing a memory overhead. The sign-adjustment using the previous iterate in the hyperbolic step seems to be novel, and has an interesting interpretation via Riemannian gradient flow. The experiments seem to demonstrate that HAM provides modest improvements across many scenarios as a drop-in adjustment, which seems to imply its general-purpose nature. Furthermore, it seems that the \"sparsity-friendly\" nature of HAM synergizes well with pruning or sparse training methods.", "weaknesses": "Overall, to me it is not made clear in Sections 3 and 4 the main motivating purpose of HAM. In particular, it is claimed via a Riemannian gradient flow analysis that HAM converges faster than gradient flow, but the main numerical results do not seem to focus on this. It is then claimed that HAM encourages a particular kind of implicit sparsity bias, but it is noted by the authors in Remark 4.7 that this is relatively weak alone. As such, it would be good to clarify the core motivation of HAM, and how the claimed benefits are supported by the theoretical derivations. Furthermore, neither theoretical intuition lends an explanation to why HAM seems to get better *generalization* on dense training tasks. Though the effect is good to see, does any of the theoretical motivation suggest this?\n\nSecondly, though the optimizer is presented as a drop-in adjustment, it does introduce two new hyperparameters in $\\alpha, \\beta$ without intuitive guidance of how these should be set. Indeed, the hyperparameter sweeps in the appendix seem to suggest the choice of $\\alpha$ is important, and that the optimal choice of $\\alpha$ can vary to be quite large, and the optimal corresponding choice of $\\beta$ also varies quite significantly on a logarithmic scale between $0$ and $1$. Without additional guidance or intuition, this means HAM introduces two additional independent hyperparameters to sweep, which is non-trivial to grid-search over.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjCME2ISMX", "reviewer": "Reviewer_q7Do", "summary": "This paper analyzes the training dynamics of pointwise overparametrization, where a parameter is replaced by a product $m \\odot w$.\nThe authors argue that this reparametrization, when studied as a Riemannian gradient flow, suffers from a small inverse metric near $\\theta = 0$.\nThis can cause parameters to get \"stuck\" and prevent crucial sign flips, and existing remedies like Sign-In introduce perturbations that do not clearly improve training dynamics.\nTo address this, the authors propose Hyperbolic Aware Minimization (HAM), an optimization step designed to capture the benefits of pointwise overparametrization without getting stuck near zero.\nThe main idea in HAM is to alternate a standard optimizer step with a hyperbolic mirror step.\nExperiments on vision models show that HAM improves the final generalization of state-of-the-art sparse training methods like AC/DC and RiGL.", "questions": "Q1. It's not entirely clear to me why HAM's hyperbolic steps on a \"dense\" $\\theta$ should preserve the benefits of optimizing on the product $m \\odot w$. Could the authors please give me an intuitive explanation as to why?\n\nQ2. Could the authors please expand on what the advantages of a product overparametrization are compared to the others discussed?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and clearly identifies a gap in the existing literature.\n\n2. The theoretical contribution is clear.\n\n3. The experiments appear to support the claims.", "weaknesses": "While I did not see any obvious weaknesses, I also cannot strongly endorse this paper as I am an outsider to this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PW2yOh03lZ", "reviewer": "Reviewer_FKu7", "summary": "This paper proposes a new optimizer by adding one additional hyperbolic mirror step to the original gradient descent (GD). This modification is originally inspired by the sparsity implicit bias of point-wise overparameterization. The main contribution of this paper lies in that the newly designed approach further mitigates the vanishing inverse metric issue near zero of prior approaches. The authors analyze their method in the linear regression setup, and further demonstrate the effectiveness of their method in different benchmarks.", "questions": "1. The questions raised in the Weaknesses: a. What is the connection between sign flip and performance improvements? b. How do we define the range of $\\alpha$? \n\n2. The two-step update manner makes me wonder the comparison with other two-step methods, e.g., momentum methods. Can the authors discuss the comparison regarding several aspects such as convergence speed (as the authors claim that HAP has a faster convergence rate compared to GD) with some of other two-step methods?\n\n3. Is it possible to design a one-step update optimizer inspired by HAM?\n\n4. Prior works revealed that stochasticity can boost sparsity implicit bias of point-wise overparameterization, then how does stochasticity affect HAM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well-motivated and well-organized. Specifically, \n   - The authors first state the sparsity implicit bias of point-wise overparameterization, which can be applied to achieve sparsity of training models by realizing the resulted Riemannian gradient flow. This gives the start point of this paper.\n   - Then the vanishing inverse metric issue is identified, which induce the hardness of sign flip prior approaches. This gives the motivation of this paper.  \n   - The authors then solve this issue by adding a constant 1 to $| \\theta |$ (this is in fact an approximation of $\\sqrt{ \\theta^2 + \\gamma^2}$ when $\\gamma \\to 0$) in the inverse metric. This is well-motivated in my view. \n   - Finally, the authors propose to use a two-step update manner to apply this newly designed inverse metric of  Riemannian flow. \n\n2. This paper provides a theoretical foundation of their method, which can help the readers understand some basic properties of this newly proposed optimizer in a very simple setting. Then this theoretical analysis is further supported by empirical evaluations on vision benchmarks where several methods are compared and discussed.", "weaknesses": "1. There lacks intuitive explanation on the connection between sign flip and performance gaining. While the authors motivate that the vanishing inverse metric is an issue of prior approaches, there still lacks clear evidence that solving this issue is strongly related to the performance gaining. This makes the scope of the applicability of HAM unclear.\n\n2. The theoretical analysis still has limitations. The theoretical analysis is based on very limited settings. This leaves several important questions unanswered, e.g., while the authors claimed a faster convergence rate compared to GD in the simple setting, the readers are still unclear whether this is true in more general setting. \n\n3. This HAM highly depends on the value of $\\nabla f$, hence how to choose a suitable $\\alpha$ can be very tricky. For example, the range of $\\alpha$ is not restricted so that it goes from 0 to a very large constant. This makes it very hard and non-intuitive to pick a suitable $\\alpha$. This unavoidably gives parameter tuning burden in practical training. \n\n4. Minor: The replacement of $sign(\\theta_0)$ with $sign(\\theta_k)$ is confusing: the study of object in Theorem 3.1 applies $\\theta_0$ but this is replaced by $\\theta_k$ (and further replaced by $\\theta_{k + 1 / 2}$ later). I'm a bit confused about why the authors start with analyzing the version with $\\theta_0$ instead of directly proposing HYP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6QHcWHIjp", "reviewer": "Reviewer_qGRF", "summary": "This paper introduces Hyperbolic Aware Minimization (HAM), an alternating optimization step that combines the beneficial sparsity-promoting implicit bias of Hadamard $m \\odot w$ overparameterizations in a lightweight two-step minimization routine that aims to avoid the optimization and computational difficulties of direct $m \\odot w$ parameterization. Classical $m \\odot w$ parameterizations, in which each parameter is represented as a product of two factors, are known to be able to improve generalization and induce sparsity regularization, but suffer from a small inverse metric near zero that slows down small weights and impedes often important sign changes. \nHAM aims to overcome this issue by alternating a standard optimizer update like SGD/Adam with a hyperbolic mirror step whose inverse metric is lower bounded by 1 at the origin and thus promotes flexible learning dynamics even for parameters close to zero, while maintaining a similar implicit bias to the $m \\odot w$ parameterization. This mechanism facilitates sign plasticity, allowing parameters to adapt their sign in response to changing gradients, and introduces an implicit bias that interpolates between $L_2$ and $L_1$ regularization, promoting ‘mild sparsity’. Theoretical analyses based on Riemannian gradient flow and connections to exponential gradient descent as an approximation show that HAM circumvents the small-metric problem and that it can be shown to converge using standard methods and assumptions in the literature. Empirical results on ImageNet and CIFAR benchmarks demonstrate consistent generalization improvements in both dense and sparse regimes, whereas further experiments on a range of tasks showcase the broad use of HAM from linear regression to LLMs and graph neural networks. Combined with other sparse learning approaches, HAM is demonstrated to enhance popular methods like  AC/DC, RiGL, and STR, and works complementarily with Sharpness Aware Minimization. Overall, the work positions HAM as a geometry-utilizing optimization method that reproduces the positive effects of overparameterized training, like its implicit bias, but ensures stable sign dynamics, while remaining computationally efficient and broadly applicable.", "questions": "In the following, questions (Q) and comments/suggestions (S) are given point-by-point. A dedicated response to all points, especially the suggestions, is not expected. \n\n\n- Q1: Related to W1b: What is the main purpose or application of HAM? Is it proposed mainly as a general-purpose optimization step, a modification to boost sparsification methods, a sparsification method in its own right, or a replacement of the $m \\odot w$ parameterization by the HAM step? Clearly, the method could be all of the above, but the exposition should highlight the principal use case.\n\n- Q2: Lines 17-18: The abstract states that HAM incurs less computational overhead than $m \\odot w$, but, e.g., Ziyin and Wang (2023) state that $m \\odot w$ on ResNet-18 takes less than 5% more time than without, which is almost negligible. Why or when is the claimed computational cost advantage (which is not empirically verified in the submission) of HAM over $m \\odot w$ relevant? \n\n- Q3: Why is Lemma 4.4 limited to $\\beta=0$?\n\n- Q4: Line 237: What is scaled and why by that factor? Context is missing here.\n\n- Q5: Why do the results between Tables 4 and 10 differ significantly for some of the overlapping methods, i.e.,  AC/DC and STR, as well as their sign-in/HAM variants? This seems particularly strange since the reported performances for Random(+Sign-In/HAM) are identical, suggesting identical settings.\n\n- Q6: Why does random pruning (baseline) outperform 4 out of 5 competitor methods in Table 10? This seems strange, particularly since Random performs worst across settings in Table 4.\n\n- Q7: Clearly, HAM provides an advantage over $m \\odot w$ with balanced initializations by enabling sign flips. But a simple remedy to the sign-flip problem could be to use a standard initialization for $m,w$. What additional purpose does HAM serve over $m \\odot w$ in that case?\n\n- Q8: Line 933: What justifies the use of the linearization $\\exp(x) \\approx 1+x$, which holds only around 0, specifically?\n\n- Q9: I don’t fully understand the difference between Theorem 4.2/B.3 (GF for Eqs GD;HYP) and Theorem B.6 (GF of HAM). Does Theorem B.6 correspond to Eqs GD;HYP*?\n\n- Q10: line 1252: Why does $w_i$ have an index $i$ but not $x$? And why does the setup define $z_i$ but the objective contains only $y_i$ and $x$?\n\n- Q11: Lemma E.1 is incomprehensible to me. Why a teacher-student set-up? Which four cases? The writing is also hard to understand.\n\n- Q12: Line 1435: In Fig. 5, it seems like none of the methods converge anywhere near close to the ground truth, with the lowest error still being larger than $0.01$ and already having leveled out. Why do seemingly all methods perform badly on this example?\n\nComments/suggestions:\n\n- S1: The exposition would benefit tremendously by using boldface small letters for vectors to distinguish them from scalars, and boldface capital letters for matrices.\n- S2: The re-statement of the theoretical results using a different label in the Appendix is very confusing (lines 923-927). Simply use the same names/labels in both the main text and the appendix for the same result\n- S3: Hyperbolic aware minimization implies that the proposed optimization step somehow utilizes the hyperbolicity of the loss landscape to improve generalization, but the proposed optimization step imitates increased hyperbolicity through point-wise overparameterization and then mitigates the resulting small inverse metric problem at the origin. Maybe the choice of the method's name is not optimal. Also, I think it should have been “hyperbolicity aware” and not “hyperbolic aware”.\n- S4: Lines 162-171: This paragraph misrepresents the scope of alternating or two-step optimization schemes, which have a long-standing history in optimization for ML. Important examples are proximal methods, soft thresholding, ADMM, alternating least squares/ridge, expectation maximization, etc. The paragraph should either give an appropriate overview or restrict itself to a certain scope of optimizers and modifications thereof. \n- S5: Consistently use $\\theta_{\\text{init}}$ instead of variants like $\\theta_{init}$ or $\\theta_{in}$\n- S6: Theorem 3.1 (line 211): The assumption on the initialization does not make sense to me, I think it should read $\\sqrt{|\\theta_0|}$ since theta is real-valued.  \n- S7: Clarify that Fig. 1 only holds for $\\gamma=0$ where the vanishing inverse metric problem occurs.\t\n- S8: lines 39-40: The methods by Kusupati et al. (2020) and Kolb et al. (2025) do not involve iterative parameter pruning as claimed\n- S9: line 199: Why is the condition $|w_0| \\leq m_0$ required? More explanation is needed.\n- S10: line 271: Many analyses are performed for $\\beta=0$, so shouldn’t the algorithm line read $\\geq 0$?\n- S11: line 291: Formulations like “mild sparsity” should be avoided and made more precise and rigorous.\n- S12: line 298: Why does this equation not contain the previously used weight decay parameter $\\beta$ anymore? Especially since the sentence in line 300 states that the inverse metric can change due to regularization.\n- S13: lines 315-316: The references are mixed up, Theorems A.7 and A.6 are applied, which are Theorems A.3 and 4.14 in the stated references.\n- S14: line 342: Sweeping statements like the one on convergence speed should be avoided if they are not made precise.\n- S15: lines 355-356: The target and input vectors should have $d$ not $n$ entries as defined in 353. Also the input should be a matrix, since each $z_i$ already is a vector.\n- S16: line 431: it is claimed that the optimal $\\alpha, \\beta$ are stable across the two tasks. But the best setting for CIFAR100 (Fig. 8) is $\\alpha=100, \\beta=3.2 \\times 10^{-2}$, whose closest neighbor in the ImageNet grid (Fig. 9) actually yields the worst performance. Looking at other experiments, the optimal values of $\\alpha$ seem to be unstable, ranging from as small as 1 to 200, or even -200.\n- S17: line 870: $\\lambda$ is undefined, I think this should be $\\Lambda$.\n- S18: line 954: The hyperlink in Eq. (3) actually references Eq. (HYP), not Eq. (3).\n- S19: line 1018: ‘indicate’ should be ‘indicates’\n- S20: lines 1035-1036: Sentence unclear to me, writing could be improved.\n- S21: line 162: Shouldn’t $\\delta$ be a function of $\\theta_{j+1/2}$ and $\\theta_{j}$, and the conditions in the parenthesis depend on both their signs?\n- S22: line 1044: $\\hat{\\theta}$ does not appear in Equation 8.\n- S23: lines 1088-1094: I can’t follow the writing. The exposition here could be improved.\n- S24: line 1246: ‘vain’ should be ‘vein’.\n- S25: line 1249: $\\mathcal{N}$ should be used consistently for a Gaussian. Moreover, previously, the main text used $d$ to denote the number of samples, not $k$.\n- S26: line 1252: I think there are incorrect indices in the equation (see Q10).\n- S27: lines 1260-1262: The text references an undefined balance equation and stops in the middle. \n- S28: line 1268: Text references undefined ‘balance equation’.\n- S29: line 1286-1287: I think the statement might be incorrect. If $a_{in}=w_{in}>0$, then it can’t be true that $a_{in}<-w_{in}$, which is the requirement for belonging to $\\Gamma_0$. Also $\\bar{\\Gamma}_{0}$ is undefined.\n- S30: line 1357: Why is suddenly Frobenius decay applied to the product weight instead of weight decay separately for $m$ and $w$ as in lines 192-193? This would not induce sparsity, only separately applying weight decay induces sparsity.\n\n\n**Sources**:\n\nZiyin and Wang (ICML, 2023): https://proceedings.mlr.press/v202/ziyin23a/ziyin23a.pdf\n\nKolb et al. (ICLR, 2025): https://openreview.net/pdf?id=vNdOHr7mn5\n\nKusupati et al. (ICML, 2020): https://arxiv.org/pdf/2002.03231", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The submission’s main strength lies in its conceptual originality and theoretical depth. It proposes a, to the best of my knowledge, novel and interesting idea of replacing the pointwise product parameterization with a direct modification of the training routine that reproduces a similar optimization geometry and implicit bias on the original weights. This reframing of the $m \\odot w$ parameterization as an alternating optimization scheme is both interesting and practically relevant, as it retains the sparsity-promoting benefits of overparameterized training without increasing model size, computational cost, or running into small inverse metric issues. The work also provides a clear and well-structured overview of existing sparsification methods, placing HAM within a broad literature that includes pruning at initialization, dense-to-sparse, and dynamic sparsity training. By focusing on the gradient dynamics and implicit bias of reparameterized models, the paper contributes to an important and active area of research that links optimization geometry to generalization (and sparsity), advancing the theoretical understanding of how optimizer design influences learning behavior and generalization. The analysis is mathematically rigorous, building on (Riemannian) gradient flow and mirror descent concepts, and derives the implicit regularization induced by HAM. \nThe empirical evaluation is performed on a large variety of tasks ranging from the simplest linear regression models to common vision benchmarks, LLM finetuning, and graph learning tasks. Overall, the paper nicely balances theoretical results with experimental evaluations, offering an innovative and well-substantiated contribution to the study of how geometry induces implicit bias for optimization in deep learning.", "weaknesses": "Overall, I think the submission proposes an interesting and novel method that makes a neat paper contribution. However, some major weaknesses prohibit recommending acceptance in the submission’s current state. However, I think the majority of my concerns could be addressed in the rebuttal, which would change my rating if addressed adequately. In the following, the weaknesses are presented point-by-point.\n\n- W1: The paper’s main weakness is its exposition. This pertains to several aspects.\n\n- W1a.i  quality of exposition: In parts, the submission reads rather “unfinished”, including awkward theorem referencing, using inconsistent notation, referencing undefined equations, or unfinished sentences. Some proofs were poorly written and hard to follow due to a lack of contextualization (e.g., Lemma E.1, Theorem E.2, Theorem B.6), missing definitions, or a lack of explanations of why certain steps are performed. Actionable examples are in a separate suggestions list below. \n\n- W1a.ii clarity of exposition: I got the feeling that some statements or sections implicitly only cover a certain setting (like $\\beta=0$) without making this clear enough. It was not always obvious to me, e.g., if a statement is made about discrete vs continuous time, $\\beta=0$ vs $\\beta \\neq 0$ (e.g., line 298), if weight decay on $m$ and $w$  or Frobenius (product) weight decay was used for the spred/$m \\odot w$ experiments and analyses, or whether GD+HYP/GD+HYP* was used.\n\n\n- W1a.iii precision of exposition: the submission contains many statements that come off a bit too bold or sweeping, or make imprecise claims (e.g., a potential exaggeration of the small inverse metric problem for $m \\odot w$, or claiming that Frobenius weight decay is required for $m \\odot w$ to induce sparsity (line 1360); whereas,  to my knowledge, sparsity using this parameterization can be achieved either through implicit regularization, or by using weight decay on $m$ and $w$ separately, bur not with the mentioned Frobenius decay of the product). Actionable details are in the suggestions list below.\n\n\n- W1b: The submission seems a bit “all over the place” with the use cases of HAM. It remains unclear to me if its proposed use is mainly a general-purpose optimization step improving generalization, a method to induce and control implicit biases without direct overparameterization, a way to enhance existing pruning pipelines, or a sparse learning method on its own. \n\n\n- W1c: I am not entirely convinced by one of the main motivating arguments for HAM, namely, to mitigate the small inverse metric problem caused by direct $m \\odot w$ parameterization, which causes parameters to get stuck/decelerate around 0 while crossing signs. Fig. 1 displays the authors’ argument nicely. However, the inverse metric only vanishes for $\\gamma=0$, which only happens for balanced initializations of $m,w$, i.e., $m_0^2-w_0^2=0$ (or also for $t \\to \\infty$ if $\\beta>0$ by Eq. (2)). By choosing a random/imbalanced initialization for $m,w$, as used by Ziyin and Wang (2023) or Kolb et al. (2025), sign crossings are unproblematic since $\\gamma_0>0$. Since both works report no issues of achieving competitive performances directly using $m \\odot w$ in both linear or DNN models, it makes me wonder how relevant the sign flip/inverse metric problem for the $m \\odot w$ parameterization is in practice when not using the balanced initialization that was avoided in previous works? Ziyin and Wang (2023) even report “Our initial experiments find no significant difference between making the norm balanced or not at initialization”, contradicting the small inverse metric problem, which would be intensified by using a balanced initialization. Furthermore, the submission derives the inverse metric as $1+\\alpha |\\theta|$ when $\\beta=0$, being lower bound by 1. Similarly, the inverse metric for $m \\odot w$ with $\\beta=0$  is $\\sqrt{\\theta^2+\\gamma^2}$, reducing to $|4*(m_0^2-w_0^2)|$ at the origin, also serving as a positive lower bound for imbalanced initializations. What is the advantage of HAM over imbalanced initializations then? \n\n- W2: The selection of hyperparameters is not discussed for the main experiments; they are merely listed in the Appendix. Without further explanation/justification of their selection, this significantly weakens the empirical comparisons, since the results can only be assumed to hold for some arbitrary hyperparameter setting (which could potentially be biased to favor HAM). While extensive hyperparameter tuning for each method is infeasible for large-scale experiments, typically, some established set of hyperparameters, e.g., as found in related benchmark papers, is used to establish some form of comparability. This weakness is aggravated by the observation that the performances of their comparison methods sometimes differ strongly from the reported values in the original works (e.g., spred for ResNet50 on ImageNet at 80% sparsity is reported as >77% in Fig. 5 of Ziyin and Wang (2023), while the submission reports only 72.6% in Table 4; similarly STR for ResNet50 on ImageNet at 90% sparsity is reported as 74.31% (Table 1 of Kusupati et al, 2020), while the submission reports only 68.36% for the same setting in Table 10).\n\n\n**Sources**:\n\nZiyin and Wang (ICML, 2023): https://proceedings.mlr.press/v202/ziyin23a/ziyin23a.pdf\n\nKolb et al. (ICLR, 2025): https://openreview.net/pdf?id=vNdOHr7mn5\n\nKusupati et al. (ICML, 2020): https://arxiv.org/pdf/2002.03231", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)