Hyperbolic Aware Minimization: Implicit Bias for Sparsity

## Abstract

Abstract Understanding the implicit bias of optimization algorithms has become central to explaining the generalization behavior of deep learning models. For instance, the hyperbolic implicit bias induced by the overparameterization m ⊙ w direct-product 𝑚 𝑤 m\odot w italic_m ⊙ italic_w —though effective in promoting sparsity—can result in a small effective learning rate, which slows down convergence. To overcome this obstacle, we propose HAM (Hyperbolic Aware Minimization), which alternates between an optimizer step and a new hyperbolic mirror step. We derive the Riemannian gradient flow for its combination with gradient descent, leading to improved convergence and a similar beneficial hyperbolic geometry as m ⊙ w direct-product 𝑚 𝑤 m\odot w italic_m ⊙ italic_w for feature learning. We provide an interpretation of the the algorithm by relating it to natural gradient descent, and an exact characterization of its implicit bias for underdetermined linear regression. HAM’s implicit bias consistently boosts performance —even of dense training, as we demonstrate in experiments across diverse tasks, including vision, graph and node classification, and large language model fine-tuning. HAM is especially effective in combination with different sparsification methods, improving upon the state of the art. The hyperbolic step requires minimal computational and memory overhead, it succeeds even with small batch sizes, and its implementation integrates smoothly with existing optimizers.

## 1 Introduction

The success of modern deep learning approaches relies on the highly overparameterized regime, which has led to a computationally demanding trend to increase the size of models, and thus the amount of trainable parameters by orders of magnitude (hoffmann2022training,; kaplan2020scaling,).
A common explanation for this phenomenon are implicit biases that originate from a combination of the optimizer and the overparameterization (pesme2021implicit,; gunasekar2017implicitregularizationmatrixfactorization,; woodworth2020kernel,), which regularize the training dynamics and thus boost generalization performance.

Training sparse models instead leads to suboptimal performance (li2017pruning,; Frankle2018TheLT,).
This fact has limited
pruning at initialization (PaI) approaches (synflow,; snip,; random-pruning-vita,) that aim to reduce the heavy computational and memory demands by masking the network before training the remaining parameters.
In contrast, state-of-the-art sparsification methods utilize overparameterization in some capacity, as they either iteratively prune parameters in Dense-to-Sparse (DtS) training (peste2021acdc,; kuznedelev2024cap,; kusupati20,; jacobs2024maskmirrorimplicitsparsification,; kolb2025deep,) or dynamically explore multiple sparse masks to find high-performing sparse networks with Dynamic Sparse Training (DST) (evci-rigl,; struct-dst,; chen2021chasing,).
Key observations regarding these algorithms are that a) mild amounts of sparsity and b) longer training with standard optimizers can improve generalization performance significantly. The latter indicates that sparse models are difficult to train and take longer to converge (Kuznedelev2023AccurateNN,).
Accordingly, standard optimization stands to benefit from overparameterization not only with respect to its generalization performance, but also its convergence speed.

Figure: Figure 1: The Riemannian metric magnitude $\|\mathcal{I}^{-1}(x)\|$ of HAM is above that of gradient descent (GD), while the overparameterization $m\odot w$ is below it. Therefore, HAM fixes the vanishing artificial learning rate. Note the hyperbolic geometric structure of HAM compared to the flatness of GD.
Refer to caption: x1.png

A recent development to improve sparse training is the pointwise overparameterization proposed in PILoT (jacobs2024maskmirrorimplicitsparsification,) and Sign-In (gadhikar2025signinlotteryreparameterizingsparse,).
All parameter weights $x\in\mathbb{R}^{n}$ are replaced by a pointwise product of parameters $m\odot w$, with both $m,w\in\mathbb{R}^{n}$. This changes the implicit bias of the optimization process and leads to substantial generalization benefits for sparse training.
In PILoT, a continuous sparsification method, the overparameterization is used to jointly learn the mask $m$. Meanwhile, the PaI method Sign-In uses it on the remaining network parameters to increase their plasticity and facilitate sign flips, which has been shown to be a necessary condition for sparse training (gadhikar2024masks,).
Both methods are state-of-the-art in their respective categories.
However, on their own, they fall short of the general baseline methods for sparsity, such as Ac/Dc (peste2021acdc,) and RiGL (evci-rigl,).
To understand this gap, we must investigate their training dynamics.

The resulting dynamics of the overparameterization $m~{}\odot~{}w$ can be derived within the mirror flow (Li2022ImplicitBO,) or time-dependent mirror flow framework (jacobs2025mirrormirrorflowdoes,).
It is associated with the hyperbolic mirror map (woodworth2020kernel,) and, depending on initialization, learning rate, and regularization, it changes from an implicit $L_{2}$ to $L_{1}$ bias during training.
The mirror flow induced by $m\odot w$ can also be characterized by a Riemannian gradient flow with an associated metric (Li2022ImplicitBO,).
This insight allows to establish a connection with natural gradient descent (Martens2014NewIA,), which uses either the Fisher or empirical Fisher information as the Riemannian metric.
Comparing these metrics highlights a problem: $m\odot w$ suffers from a small artificial learning rate $\mathcal{I}^{-1}(x)$ (jacobs2024maskmirrorimplicitsparsification,) (see Fig. [1](https://arxiv.org/html/2506.02630v1#S1.F1)), impeding convergence.
As an additional consequence, parameters can become stuck at 0 0, preventing the ability to effectively sign-flip.
Sign-In (gadhikar2025signinlotteryreparameterizingsparse,) partially mitigates this issue by iteratively re-initializing $m$ and $w$.
However, the remedy is unstable and introduces a hard perturbation to the training dynamics, limiting its positive effects.

In this work, we propose to capture the essential structure of these two methods, the implicit bias of pointwise overparameterization, which provably aids in finding generalizable sparse solutions. At the same time, we are able to avoid their drawbacks: their slower convergence and their need for explicit overparameterization that negatively impacts memory and compute. We do this by deriving a plug-and-play hyperbolic optimization step, which we alternate with gradient descent or any other first-order optimizer.
Our alternating method is called HAM: Hyperbolic Aware Minimization (§ [3](https://arxiv.org/html/2506.02630v1#S3),  [5](https://arxiv.org/html/2506.02630v1#S5)).
HAM mitigates the small artificial learning rate and keeps a similar but fully controllable implicit bias than $m\odot w$, as shown in § [4](https://arxiv.org/html/2506.02630v1#S4) for its combination with gradient descent.
We evaluate HAM across diverse benchmarks—including vision tasks, graph and node classification, and LLM finetuning—and find it consistently improves generalization, especially in sparse training (§ [6](https://arxiv.org/html/2506.02630v1#S6)). Remarkably, HAM tends to enhance generalization complementary to sharpness aware minimization (SAM) (foret2021sharpnessaware,), yet incurs only negligible computational overhead.
These improvements can be explained by two major mechanisms: a) It accelerates training around 0 0, thus improving sign learning.
This is facilitated by its geometry and a larger artificial learning rate.
b) The implicit bias towards mild sparsity regularizes training. Both mechanisms boost generalization of sparsification techniques, such as Ac/Dc (peste2021acdc,) and RiGL (evci-rigl,), and even of dense model training.

In summary, our contributions are:

- •
HAM step: A plug-and-play optimization update, compatible with first- or higher-order optimizers without incurring significant computational or memory overhead.
- •
We provide a characterization of the training dynamics using natural gradient descent and Riemanian gradient flow.
- •
HAM inherits the geometric benefits of the pointwise $m\odot w$ parameterization in recent sparsity methods, while mitigating their vanishing artificial learning rate problem (see Figure [1](https://arxiv.org/html/2506.02630v1#S1.F1)).
- •
HAM improves strong baseline sparsity methods, including Ac/Dc, RiGL, and STR.
- •
Moreover, it boosts dense training, as shown for vision, graph and node classification, and LLM fine tuning.

## 2 Related work

##### Sparsification

Sparse training methods can be categorized into three broad classes: Pruning at Initialization (PaI), Dense-to-Sparse training (DtS), and Dynamic Sparse Training (DTS).
PaI methods identify a sparse mask at initialization and train the remaining parameters to convergence.
They include methods like SNIP (snip,), Synflow (synflow,), NPB (pham2023paths,), PHEW (patil2021phew,), GraSP (grasp,) and random (random-pruning-vita,; er-paper,).
Their primary limitation is that standard optimizers do not find generalizable solutions on these fixed masks, as they struggle to effectively learn parameter signs (signs,; gadhikar2025signinlotteryreparameterizingsparse,).
In contrast, DtS methods learn the mask via a dense or denser phase of training, followed by any kind of pruning step and possibly more training.
This includes iterative pruning methods like IMP (frankle2019lottery,), LRR (rewindVsFinetune,; han-efficient-nns,), Ac/Dc (peste2021acdc,), CAP (kuznedelev2024cap,), and WoodFisher (singh2020woodfisher,).
Continuous sparsification methods, which start from a dense network and gradually sparsify it with a learnable mask, also fall under this category.
They include PILoT (jacobs2024maskmirrorimplicitsparsification,), STR (kusupati20,), CS (savarese2021winning,) and spred (Ziyin2022spredSL,).
The third class of methods, Dynamic Sparse Training, start from an already sparse mask but dynamically update it during training, and in this sense utilize a form of (dynamic) overparameterization (dynamicoverparam,).
Examples include RiGL (evci-rigl,), MEST (yuan2021mest,), and SET (mocanu-adaptive-sparse-train,).
While PaI methods cannot compete with the generalization performance of DtS and DST methods, Sign-In (gadhikar2025signinlotteryreparameterizingsparse,) improves on PaI by using the pointwise overparameterization $m\odot w$, which leverages a hyperbolic mirror map to facilitate sign flips.
In this work, we propose instead a simpler, more powerful hyperbolic optimization step to leverage a similar mirror map without doubling the number of parameters. Our method HAM can improve the optimization process of both dense and sparse training phases and, in contrast to PILoT and Sign-In, also DtS and DST methods like Ac/Dc and RiGL.

##### Implicit bias and mirror flow

The implicit bias of neural networks is a well studied topic that aims to explain the regularization benefits resulting from overparameterization  (woodworth2020kernel,; gunasekar2017implicit,; gunasekar2020characterizing,; Li2022ImplicitBO,).
It is primarily characterized within the mirror flow framework, a well-established concept in convex optimization (Alvarez_2004,; BECK2003167,; Rockafellar1970ConvexA,; Boyd2009ConvexO,).
A mirror flow can be interpreted as a gradient flow on a Riemannian manifold (Li2022ImplicitBO,; Alvarez_2004,), which has also been extended to cover stochastic gradient descent (SGD) (pesme2021implicit,; Even2023SGDOD,; lyu2023implicit,) and more recently to explicit regularization (jacobs2025mirrormirrorflowdoes,).
This framework allows us to characterize the implicit bias.
The main observation is that large learning rates, stochastic noise from SGD, and regularization can benefit generalization by implicitly inducing sparsity.
However, overparameterization can also lead to small artificial learning rates, slowing down convergence and potentially hampering generalization (jacobs2024maskmirrorimplicitsparsification,), which we can successfully avoid in HAM.

##### Natural gradient descent and Bayesian learning rules

The mirror flow framework also enables us to view our algorithm HAM through the lens of natural gradient descent.
Natural gradient descent uses the Fisher information matrix —or an approximation of it— to capture second order information and adapt the artificial learning rate accordingly (Martens2014NewIA,; 10.7551/mitpress/7011.003.0010,).
A more general Bayesian framework (Khan2021TheBL,) is used to gain insight into invariant distributions by using Lie groups
(Kral2023TheLB,) and for the development of a new optimizer IVON
(shen2024variational,).
In this work, we provide an interpretation of our alternating scheme HAM with gradient descent by mapping the Fisher information to a known posterior distribution.
Notably, our proposed hyperbolic update is reminiscent of exponentiated gradient descent
(10.5555/902667,).
A key distinction is that exponential gradient descent normalizes such that it stays on a probability simplex, as seen for example in Chapter 7 of (Vishnoi_2021,).
HAM, on the other hand, does not normalize, as the algorithmic step acts directly in the weight space.
Recently, exponential gradient descent has been applied to reweighting batches (Majidi2021ExponentiatedGR,).

##### Two-step and alternating schemes

Various previous works have explored alternating training schemes.
Examples include birth-death dynamics at a neuron level in two layer neural network training (Rotskoff2019GlobalCO,) or variational inference (Mielke2025HellingerKantorovichGF,; Gladin2024InteractionForceTG,; Yan2024LearningGM,).
An important difference is that we work on a weight level while other approaches work on a neuron or distribution level and serve an entirely different purpose.
Another well-known two-step approach, although not alternating, is sharpness aware minimization (SAM) (foret2021sharpnessaware,), which promotes the search for flat solutions at the expense of almost doubling the compute of one optimization step.
In contrast, HAM encourages mild sparsity and acceleration around 0 0, which are complementary mechanisms.
Our experiments (Table [2](https://arxiv.org/html/2506.02630v1#S6.T2)) demonstrate that our proposed hyperbolic step can be effectively combined with SAM to further boost generalization.

## 3 Motivation and derivation of HAM

We derive our novel optimization step by building on insights from the implicit bias of recently developed sparsification methods.
These methods have exploited a reparameterization of the neural network: They replace each weight $x$ with a product of two weights $m\odot w$, where $\odot$ is the Hadamard or pointwise product. For this reparameterization, it is known that stochastic noise and weight decay induce sparsity; that is, an implicit $L_{1}$ penalty (pesme2021implicit,; jacobs2024maskmirrorimplicitsparsification,) and thus implicit bias towards sparser solutions.
We now restate the induced gradient flow (where the learning rate $\eta\rightarrow 0$) of this reparameterization.

##### Gradient flow of m ⊙ w direct-product 𝑚 𝑤 m\odot w italic_m ⊙ italic_w

Consider a continuously differentiable and L-smooth(^1^11See Definition [C.1](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem1) in the appendix for a definition of L-smoothness.) loss function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$.
It can be trained by means of gradient descent:

$$ $x_{k+1}=x_{k}-\eta\nabla f(x_{k}),\qquad x_{0}=x_{init}$ $$

where $\eta>0$ is the learning rate.
Taking $\eta\rightarrow 0$, we obtain its gradient flow: $dx_{t}=-\nabla f(x_{t})dt$.
Its integral form is used in the mirror flow analysis and descriptions of the implicit bias:

$$ $x_{t}-x_{0}=-\int_{0}^{t}\nabla f(x_{s})ds.$ $$

(Li2022ImplicitBO,) derive a similar formulation for the reparameterization $m\odot w$ trained with gradient descent, while (jacobs2024maskmirrorimplicitsparsification,) integrate weight decay in the analysis resulting in the following hyperbolic gradient flow:

$$ $x_{t}=u_{0}^{2}\odot\exp\left(-2\int_{0}^{t}\nabla f(x_{s})ds-\beta t\right)-v _{0}^{2}\odot\exp\left(2\int_{0}^{t}\nabla f(x_{s})ds-\beta t\right)$ (1) $$

where $u_{0}$ and $v_{0}$ are chosen such that $u_{0}^{2}-v_{0}^{2}=x_{0}$ and $\beta>0$ is the strength of weight decay.

##### Exponential gradient descent

The hyperbolic gradient flow Eq. ([1](https://arxiv.org/html/2506.02630v1#S3.E1)) not only corresponds to the gradient flow of $m\odot w$, but also to exponential gradient descent. This is presented in Wu2021ImplicitRI for matrix forms in a matrix sensing task, which provides a testbed to understand the implicit bias of overparameterization (gunasekar2017implicit,).
We use this connection to derive an update without the need for the reparameterization.
The update is captured by the following theorem:

###### Theorem 3.1

If $m_{0}=\operatorname{sign}(x_{0})$ and $w_{0}=\sqrt{x_{0}}$, then

$$ $x_{k+1}=x_{k}\odot\exp\left(-\eta\left(2\ \operatorname{sign}(x_{0})\nabla f(x _{k})+\beta\right)\right)$ (2) $$

is a first order approximation of Eq. ([1](https://arxiv.org/html/2506.02630v1#S3.E1)).

Proof. See proof of Theorem [A.1](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem1) in the appendix. $\square$

Note that a more general update for a product of matrices is provided in Wu2021ImplicitRI.

###### Theorem 3.1

##### Derivation of HAM

Alternating the new hyperbolic update in Eq. ([2](https://arxiv.org/html/2506.02630v1#S3.E2)) with another optimizer forms the basis of our proposal HAM. We derive its explicit form for gradient descent as follows:

$$ $\displaystyle x_{k+\frac{1}{2}}$ $\displaystyle=x_{k}-\eta\nabla f(x_{k})$ (GD) $\displaystyle x_{k+1}$ $\displaystyle=x_{k+\frac{1}{2}}\odot\exp\left(-\eta\left(\alpha\ \operatorname {sign}(x_{k})\nabla f(x_{k})+\beta\right)\right)$ (HYP) $$

A hidden assumption in Eq. ([2](https://arxiv.org/html/2506.02630v1#S3.E2)) is that the sign of the weight stays fixed.
However, gradient descent—like most optimizers—can flip signs of weights.
Therefore, $\operatorname{sign}(x_{0})$ is replaced with $\operatorname{sign}(x_{k})$.
Moreover, it is not necessary to use the same learning rate for gradient descent and for the exponential update.
Accordingly, we have introduced an additional hyperparameter $\alpha\in\mathbb{R}$, which as it turns out, controls the strength of implicit bias towards sparsity.
The exponential update now more closely resembles the hyperbolic gradient update in Eq. ([1](https://arxiv.org/html/2506.02630v1#S3.E1)), as it can switch the sign in the exponential.
In §[4](https://arxiv.org/html/2506.02630v1#S4), we study the resulting gradient flow of this combination of steps.

###### Remark 3.2

Note that the second step ([HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)) uses both $x_{k}$ and $x_{k+\frac{1}{2}}$, which would require twice as much memory.
Therefore, in practice, we replace $x_{k}$ with $x_{k+\frac{1}{2}}$.
We restate the actually deployed second HAM step ([HYP*](https://arxiv.org/html/2506.02630v1#S5.Ex8)) in §[5](https://arxiv.org/html/2506.02630v1#S5).
The actual HAM step has benefits for the optimization itself, as we discuss in Appendix [A](https://arxiv.org/html/2506.02630v1#A1) and Figure [3](https://arxiv.org/html/2506.02630v1#A4.F3).
Building on recent work on sign flips (gadhikar2025signinlotteryreparameterizingsparse,), we argue that the sign should be aligned with the gradient evaluation to assess whether the step should be accelerated or not, in order to to promote meaningful sign flips.

###### Remark 3.2

##### Natural gradient descent interpretation

Our algorithm can be interpreted through the lens of natural gradient descent (10.7551/mitpress/7011.003.0010,; Martens2014NewIA,).
Each optimizer step corresponds to a natural gradient update
$x_{k+1}=x_{k}-\eta\mathcal{I}(x_{k})^{-1}\nabla f(x_{k})$, with $\mathcal{I}$ the Fisher information.
The key insight is that parameters follow a known parameterized distribution that is learned by the optimizer.
For gradient descent, $\mathcal{I}_{\text{GD}}(x)=1$, which corresponds to a normally distributed random variable $X$ with unit variance and learnable mean; i.e., $x=\mathbb{E}[X]$.
Thus we can interpret $x$ as the mean of a normal distribution whose position is learned.

In contrast, the exponential update ([2](https://arxiv.org/html/2506.02630v1#S3.E2)) corresponds to $\mathcal{I}_{\text{HYP}}(x)=1/|x|$, which directly follows from the first order approximation of the exponential function.
Its Fisher information may correspond to a random variable $X$ parameterized as a normal distribution with unit variance: $\mathcal{N}({2\sqrt{|x|},1})$.
In this view, weights are recovered via $x=\frac{1}{4}\operatorname{sign}(x)\mathbb{E}[X]^{2}$, meaning we learn the magnitude of the expected position. Furthermore, if the sign is not correct, the hyperbolic step ([HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)) will move the parameter exponentially fast towards zero, facilitating its sign flip.
We provide derivations for both discussed cases in Appendix [B](https://arxiv.org/html/2506.02630v1#A2).
To summarize, our combined update can be interpreted as follows:

Learn the position ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3)), and then the magnitude if the sign is correct; else move fast to zero ([HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)).

This mechanism is crucial to facilitate sparse training (gadhikar2024masks,; gadhikar2025signinlotteryreparameterizingsparse,), as portrayed in our experiments (§ [6](https://arxiv.org/html/2506.02630v1#S6)), where HAM considerably boosts its performance.

## 4 Theory: Gradient flow analysis

Our theory for HAM (Eqs. ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3);[HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4))) is based on the corresponding Riemannian gradient flow in parameter space (Thm. [4.2](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem2)).
We provide an analysis of its convergence and implicit bias (Thms. [4.3](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem3) and [4.5](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem5)).
This proves that HAM solves the artificial learning rate problem of $m\odot w$, and thus converges faster, while retaining the same asymptotic implicit bias.
In this section, we assume that the objective function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is continuously differentiable; i.e., $f\in C^{1}$, and $L-$smooth\@footnotemark.

##### Riemannian gradient flows

In order to concisely study the behavior of HAM, we consider a gradient flow formulation.
Gradient flow (flat) and $m\odot w$ (hyperbolic) flow can be described as Riemannian gradient flows, and their metrics coincide with their Fisher information from natural gradient descent.
The Riemannian gradient flow with a general metric $\mathcal{I}(x)$ is given by:

$$ $dx_{t}=-\mathcal{I}(x_{t})^{-1}\nabla f(x_{t})dt,\qquad x_{0}=x_{init}.$ $$

We refer to the quantity $\mathcal{I}^{-1}(x)$ as the artificial learning rate as it changes the convergence rate.
It is trivial for gradient flow, as the Fisher information matrix is $\mathcal{I}_{\text{GD}}(x)=1$. (jacobs2024maskmirrorimplicitsparsification,) have derived that of $m\odot w$ as $\mathcal{I}_{m\odot w}(x)=\frac{1}{\sqrt{x^{2}+\gamma^{2}}}$, where $\gamma$ depends on the initialization scale and can change due to noise and regularization.
For reference, the artificial learning rates are shown in [Table 1](https://arxiv.org/html/2506.02630v1#S4.T1).

**Table 1: Artificial learning rates of gradient descent, the overparameterization $m\odot w$, and HAM.**
|  | GD | $m\odot w$ | HAM |
| --- | --- | --- | --- |
| $\mathcal{I}^{-1}(x)$ | $1$ | $\sqrt{x^{2}+\gamma^{2}}$ | $1+\alpha|x|$ |

##### The vanishing artificial learning rate problem

For small $\gamma$ and small weights $x$ (close to 0 0), the artificial learning rate $\sqrt{x^{2}+\gamma^{2}}$ of $m\odot w$ can get much smaller than $1$ (see Figure [1](https://arxiv.org/html/2506.02630v1#S1.F1)).
This implies that learning close to 0 0 is slowed down, which makes transitions through 0 0 and thus sign flips of $x$ much harder.
This can also reduce the overall convergence rate.

###### Theorem 4.1

If $f$ is $L-$smooth (Definition [C.1](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem1)), satisfies the PL-inequality ([C.2](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem2)) or is convex and $\arg\min\{f(x):x\in\mathbb{R}^{n}\}$ is non-empty, then the iterates $x_{t}$ converge to a minimizer of $f$ for both metrics $\mathcal{I}_{GD}$ and $\mathcal{I}_{m\odot w}$.
Under the PL-inequality , the convergence rates are respectively $\Lambda$ and $\sqrt{\min x_{i,t}^{2}+\gamma^{2}}\Lambda$, where $\Lambda>0$ is the PL-inequality constant.

Proof.
It directly follows from Theorem A.3 in (jacobs2024maskmirrorimplicitsparsification,) and Theorem 4.14 in (Li2022ImplicitBO,). $\square$

###### Theorem 4.1

##### Riemannian gradient flow of HAM

In comparison, HAM speeds up learning around 0 0.
To show this, we derive its gradient flow from Eqs. ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3);[HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)) by writing out the iterates in sum notation and taking $\eta\rightarrow 0$. This gradient flow is an exact characterization of the dynamics of HAM.

###### Theorem 4.2

The gradient flow ($\eta\rightarrow 0$) of Eqs. ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3);[HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)) is given by:

$$ $dx_{t}=-\nabla f(x_{t})dt-|x_{t}|\odot\left(\alpha\nabla f(x_{t})+\text{sign}( x_{t})\beta\right)dt,\qquad x_{0}=x_{init},$ (3) $$

where $|\cdot|$ is applied pointwise. Moreover, if $\beta=0$, the associated Fisher information matrix is $\mathcal{I}_{\text{HAM}}(x)=\frac{1}{1+\alpha|x|}$.

Proof. This follows from writing out the sum update and then taking the limit to get an integral equation.
The gradient flow then follows from the Leibniz rule (differentiating under integral sign). See Theorem [A.2](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem2) in the appendix. $\square$

###### Theorem 4.2

##### Convergence of HAM

We analyze the artificial learning rates and convergence behavior of HAM when $\beta=0$. In this case, the Fisher information is given by $\mathcal{I}_{\text{HAM}}(x)=\frac{1}{1+\alpha|x|}$.
This yields an artificial learning rate of $\left(1+\alpha|x|\right)\geq 1$ ([Table 1](https://arxiv.org/html/2506.02630v1#S4.T1)), indicating that HAM can converge faster than gradient descent depending on $\alpha$ and the magnitude of the weights.
This contrasts sharply with the sparsification methods,
where a decaying $\gamma\ll 1$ reduces the learning rate and slows down convergence. We formalize this behavior again under the same conditions as Theorem [4.1](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem1):

###### Theorem 4.3

Under the same setting as Theorem [4.1](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem1), the iterates of HAM in Eq. ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3),[HYP](https://arxiv.org/html/2506.02630v1#S3.Ex4)), $x_{t}$, converge to a minimizer of $f$. Moreover, the convergence rate is $\left(1+\alpha\min|x_{i,t}|\right)\Lambda$ under the PL-inequality assumption.

Proof.
We lower bound $\mathcal{I}^{-1}_{HAM}\geq\left(1+\alpha\min|x_{i,t}|\right)$ and apply Thm. A.3 (jacobs2024maskmirrorimplicitsparsification,) or 4.14 (Li2022ImplicitBO,). $\square$

This proves that HAM avoids the vanishing learning rate from pointwise overparameterization $m\odot w$, and it can outperform gradient descent in convergence speed driven by the geometry of its update.

###### Theorem 4.3

##### Implicit bias of HAM

We now characterize the implicit bias of HAM by analyzing its associated Riemannian gradient flow or natural gradient flow.
This confirms that HAM not only affects convergence speed, but also influences the nature of the solution it converges to.
To do this, we compute the Bregman function $R_{\alpha}$ such that its Hessian yields the required Fisher information matrix $\mathcal{I}_{\text{HAM}}$; that is, $\mathcal{I}_{\text{HAM}}=\nabla^{2}R_{\alpha}$.

###### Lemma 4.4

The function $R_{\alpha}$ for $\alpha\in\mathbb{R}$ and $\beta=0$ is given by

$$ $R_{\alpha}(x)=\sum_{i}\frac{\left(\alpha\left|x_{i}\right|+1\right)\ln\left( \alpha\left|x_{i}\right|+1\right)-\alpha\left|x_{i}\right|}{\alpha^{2}}-x_{i} \frac{\operatorname{sign}(x_{0})}{\alpha}\log(1+\alpha|x_{0}|).$ $$

If $\alpha>0$, $R_{\alpha}$ is a Bregman function (Definition [C.4](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem4)).

Proof. See Lemma [A.3](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem3) in the appendix. $\square$

Concretely, we can use Lemma [4.4](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem4) to characterize the implicit bias for under-determined linear regression.
Let $\{(z_{i},y_{i})\}_{i=1}^{d}\subset\mathbb{R}^{n}\times\mathbb{R}$ be a dataset of size $d$.
The output of a linear model $x$ on the $i$-th data is $z_{i}^{T}x$. The goal therefore is
to solve the regression for the target vector $Y=(y_{1},y_{2},\ldots,y_{n})^{T}$ and input vector
$Z=(z_{1},z_{2},\ldots,z_{n})$.

###### Theorem 4.5

Consider the same setting as Theorem [4.3](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem3) with $\beta=0$. Then, if $f(x):=f(Z^{T}x-y)$, the gradient flow of $x_{t}$ in Eq. ([3](https://arxiv.org/html/2506.02630v1#S4.E3)) converges to the solution of the optimization problem:

$$ $x_{\infty}=\arg\min_{Zx=y}R_{\alpha}(x).$ $$

Proof.
Apply the mirror flow part of Theorem 4.17 (Li2022ImplicitBO,) for Bregman functions. $\square$

Theorem [4.5](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem5) can provide intuition to what type of solutions HAM converges to.
We are particularly interested in the shape of $R_{\alpha}$ when $x_{0}=0$.
Note that the gradient flow is well-defined at $x_{0}$ if $\beta=0$.

###### Theorem 4.6

Let $x_{0}=0$. Then, $R_{\alpha}\sim||x||_{L_{2}}^{2}$ if $\alpha\rightarrow 0$, and $R_{\alpha}\sim||x||_{L_{1}}$ if $\alpha\rightarrow\infty$, where $\sim$ indicates proportionality.

Proof. See proof of Theorem [A.4](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem4) in the appendix. $\square$

Theorem [4.6](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem6) is illustrated in Figure [2(b)](https://arxiv.org/html/2506.02630v1#S6.F2.sf2). Notably, the behavior of $R_{\alpha}$ in HAM interpolates between $L_{2}$ and $L_{1}$ in a similar way as the Bregman function of $m\odot w$ (jacobs2024maskmirrorimplicitsparsification,).

###### Remark 4.7

In the $\alpha\rightarrow\infty$ setting, HAM induces an $L_{1}$ bias. However, in practice, due to discretizations, this setting would require a much smaller learning rate to ensure convergence. This makes HAM less suited to fully induce sparsity on its own than the related sparsification methods.
Therefore, HAM is best used in combination with other methods to find sparse solutions, acting as a guide for sparse geometry during their training.

Remark [4.7](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem7) emphasizes that HAM needs a substantially large $\alpha$ to induce sparsity on its own. This is in line with our takeaway from §[3](https://arxiv.org/html/2506.02630v1#S3), which is that we make training aware of learning the correct magnitude of the weight with our hyperbolic step. This contrasts with the $m\odot w$ overparameterization, where sparsity emerges due to the inherently small learning rate.

###### Lemma 4.4

###### Theorem 4.5

###### Theorem 4.6

###### Remark 4.7

##### Explicit regularization β > 0 𝛽 0 \beta>0 italic_β > 0

In practice, we apply the additional regularization term $\beta>0$, which acts as weight decay directly on the parameter update.
Importantly, this does not change the artificial learning rate, as shown by Theorem [4.2](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem2).

## 5 Algorithm: Hyperbolic Aware Minimization (HAM)

We propose HAM (Algorithm [1](https://arxiv.org/html/2506.02630v1#alg1)) which alternates between any standard optimizer step and a hyperbolic (signed) mirror map to improve general trainability of neural networks.
The proposed method is inspired by recent sparsification methods, as developed in § [3](https://arxiv.org/html/2506.02630v1#S3), and theoretically justified in § [4](https://arxiv.org/html/2506.02630v1#S4).
We next state the main algorithmic innovations.

##### Hyperbolic step

Let $\eta>0$ denote the learning rate, and $\alpha,\beta\geq 0$ be positive constants. The deployed hyperbolic step for parameters $x_{k}\in\mathbb{R}^{n}$ is given by

$$ $x_{k}=x_{k+\frac{1}{2}}\odot\exp{\left(-\eta\left(\alpha\ \operatorname{sign}( x_{k+\frac{1}{2}})\nabla f(x_{k})+\beta\right)\right)},$ (HYP*) $$

where $\odot$ is the Hadamard product, and $x_{k+\frac{1}{2}}$ is the step of any other optimizer over $x_{k}$.
$\alpha$ controls the convergence speed and hyperbolic awareness of the method, and $\beta$ induces an explicit regularization similar to that of PILoT (jacobs2024maskmirrorimplicitsparsification,).
Note that we have replaced $\operatorname{sign}(x_{k})$ with $\operatorname{sign}(x_{k+\frac{1}{2}})$, as mentioned in Remark [3.2](https://arxiv.org/html/2506.02630v1#S3.Thmtheorem2). Our analysis § [4](https://arxiv.org/html/2506.02630v1#S4) still remains valid at the end of training, as we show in Appendix [A](https://arxiv.org/html/2506.02630v1#A1).

##### Memory and compute overhead

HAM does not incur any memory overhead, as it reuses the known gradient and current signs of the weights.
In contrast, the pointwise overparameterization $m\odot w$ doubles the number of parameters, which would only be negligible in case of large batch sizes (where activations dominate the memory requirements).
HAM is viable on large scale LLM tasks, where batches are small and the memory is largely consumed by model parameters.
Moreover, the amount of extra flops during training is negligible, as they are on the order of the level of parameters.

Figure: Algorithm 1 HAM

## 6 Experiments

We verify the use of our new algorithmic step in various setups.
First, we present results on dense training vision benchmarks and compare HAM’s training dynamics with $m\odot w$.
Here we also show generalization improvements and compatibility with sharpness aware minimization (SAM) (foret2021sharpnessaware,).
Second, we improve generalization for various dense-to-sparse training methods, showcasing the benefit of the changed implicit bias of HAM—which has a mild bias towards sparsity and better plasticity.
Finally, we apply HAM on a set of different tasks such as LLM finetuning, graph classification, and node classification, to emphasize HAM’s versatility.
Additionally, in Appendix [D.1](https://arxiv.org/html/2506.02630v1#A4.SS1) we verify the improved (sparse) implicit bias proven in Theorem [4.5](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem5) on underdetermined linear regression. Note, in all experiments, ([HYP*](https://arxiv.org/html/2506.02630v1#S5.Ex8)) is not applied to BatchNorm or LayerNorm layers.

Figure: (a) Training loss.
Refer to caption: x2.png

##### Improving dense training

Figure [2(a)](https://arxiv.org/html/2506.02630v1#S6.F2.sf1) compares the training loss for the best hyperparameter configuration of HAM and $m\odot w$ for ResNet50 on Imagenet.
HAM converges similarly to the dense baseline, while $m\odot w$ lacks behind and has worse generalization performance ($76.56\%$), due to the artificial small learning rate.
[Table 2](https://arxiv.org/html/2506.02630v1#S6.T2) demonstrates that HAM improves dense training for a ResNet50 on Imagenet.
Moreover, HAM works complementary to sharpeness aware minimization (SAM) (foret2021sharpnessaware,), as combining both algorithms (SAM-HAM) achieves the best performance.
[Table 2](https://arxiv.org/html/2506.02630v1#S6.T2) further highlights that training HAM longer (using a similar compute budget as SAM, whose iterations are twice expensive) achieves a similar improvement.
In Figure [5](https://arxiv.org/html/2506.02630v1#A4.F5) we track the total $L_{1}$ norm of the parameters during training to illustrate the complementary mechanisms of HAM and SAM.
In addition, [Table 7](https://arxiv.org/html/2506.02630v1#A4.T7) provides experimental validation on the smaller vision benchmark CIFAR100 (cifar,).
A grid search for HAM’s hyperparameters $\alpha$ and $\beta$ on Imagenet (imagenet,) and CIFAR100 (cifar,) is visualized in Fig. [6](https://arxiv.org/html/2506.02630v1#A4.F6) and [7](https://arxiv.org/html/2506.02630v1#A4.F7).
Note that the best values seem to be stable across different tasks.
We also show improvements for a small transformer, DeiT (touvron2021training,), on Imagenet trained with ADAMW in [Table 8](https://arxiv.org/html/2506.02630v1#A4.T8).

**Table 2: HAM improves dense training of a ResNet50 on Imagenet (imagenet,).**
| Dataset | Baseline | $\alpha=0,\beta=1e-3$ | $\alpha=200,\beta=0$ | $\alpha=200,\beta=1e-3$ |
| --- | --- | --- | --- | --- |
| HAM, 100 epochs | $76.72\pm 0.19$ | $77.01\pm 0.14$ | $76.72\pm 0.07$ | $\bf 77.51\pm 0.11$ |
| HAM, 200 epochs | $77.27\pm 0.13$ | $77.48\pm 0.09$ | $77.24\pm 0.09$ | $\bf 77.86\pm 0.05$ |
| SAM-HAM, 100 epochs | $77.10\pm 0.21$ | $77.53\pm 0.16$ | $77.21\pm 0.09$ | $\bf 77.92\pm 0.15$ |
| SAM-HAM, 200 epochs | $77.94\pm 0.16$ | $78.17\pm 0.16$ | $77.60\pm 0.03$ | $\bf 78.56\pm 0.12$ |

##### Improving Sparsification

We demonstrate that HAM improves state-of-the-art pruning methods Ac/Dc, RiGL, and random pruning at initialization with the same hyperparameter configuration used in dense training.
[Table 3](https://arxiv.org/html/2506.02630v1#S6.T3) illustrates that dense-to-sparse training becomes significantly better with HAM.
Improvements are most significant for Ac/Dc, which uses dense training phases effectively.
We attribute this also to the fact that Ac/Dc turns on parameters indiscriminately, while RiGL does so based on gradient information.
Moreover, we show that we are immediately better than Sign-In (gadhikar2025signinlotteryreparameterizingsparse,), a method designed to improve pruning at initialization training dynamics.
Note that HAM promotes sign learning with a similar artificial learning rate as Sign-In (gadhikar2025signinlotteryreparameterizingsparse,) but without hard perturbations.

**Table 3: Dense-to-sparse training and pruning at initialization with HAM on Imagenet with ResNet50.**
| Pruning type | Method | $s=0.8$ | $s=0.9$ | $s=0.95$ |
| --- | --- | --- | --- | --- |
| PaI | Random | $73.87(\pm 0.06)$ | $71.56(\pm 0.03)$ | $68.72(\pm 0.05)$ |
| Random + *Sign-In* | $74.12(\pm 0.09)$ | $72.19(\pm 0.18)$ | $69.38(\pm 0.1)$ |  |
| Random + HAM | $\mathbf{74.84(\pm 0.09)}$ | $\mathbf{72.72(\pm 0.03)}$ | $\mathbf{70.05(\pm 0.06)}$ |  |
| DtS | Ac/Dc | $75.83(\pm 0.02)$ | $74.75(\pm 0.02)$ | $72.59(\pm 0.11)$ |
| Ac/Dc + *Sign-In* | $75.9(\pm 0.14)$ | $74.74(\pm 0.12)$ | $72.88(\pm 0.13)$ |  |
| Ac/Dc + HAM | $\mathbf{77.2(\pm 0.14})$ | $\mathbf{76.66(\pm 0.12)}$ | $\mathbf{75.45(\pm 0.13)}$ |  |
| DST | RiGL | $75.02(\pm 0.1)$ | $73.7(\pm 0.2)$ | $71.89(\pm 0.07)$ |
| RiGL + HAM | $\mathbf{76.22(\pm 0.07)}$ | $\mathbf{74.83(\pm 0.08)}$ | $\mathbf{72.93(\pm 0.1)}$ |  |
| Cont. spars. | spred | $72.64$ | $71.84$ | $69.47$ |
| PILoT | $75.62$ | $74.73$ | $71.3$ |  |
| STR | $75.49(\pm 0.14)$ | $72.4(\pm 0.11)$ | $64.94(\pm 0.07)$ |  |
|  | STR + HAM | $\mathbf{76.37(\pm 0.18)}$ | $\mathbf{75.01(\pm 0.02)}$ | $\mathbf{71.41(\pm 0.1)}$ |

##### Finetuning LLMs

As we show, HAM can also boost the performance of LLM finetuning.
We evaluate on the commonsense reasoning benchmark (hu2023llm,) to finetune LlaMA 3.2 models (grattafiori2024llama,) and report accuracies across eight benchmarks in [Table 4](https://arxiv.org/html/2506.02630v1#S6.T4).
On average, HAM improves on this task, demonstrating its compatibility with the optimizer ADAM and the LoRA architecture.

**Table 4: Performance of LoRA + HAM on the commonsense reasoning (hu2023llm,) benchmark.**
| LlaMa 3.2 | Size | HS | WG | PQ | AE | AC | OB | SQ | BQ | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LoRA | 1B | $63.8$ | $65.8$ | $74.04$ | $67.63$ | $\mathbf{55.88}$ | $63.6$ | $70.98$ | $\mathbf{64.25}$ | $65.74$ |
| LoRA + HAM | 1B | $\mathbf{64.8}$ | $\mathbf{68.35}$ | $\mathbf{74.21}$ | $\mathbf{68.39}$ | $51.79$ | $\mathbf{65.8}$ | $\mathbf{71.2}$ | $62.17$ | $\mathbf{65.83}$ |
| LoRA | 3B | $88.8$ | $\mathbf{80.66}$ | $\mathbf{83.73}$ | $\mathbf{82.65}$ | $66.89$ | $76.8$ | $78.19$ | $69.44$ | $78.39$ |
| LoRA + HAM | 3B | $\mathbf{89.40}$ | $80.58$ | $82.69$ | $81.77$ | $\mathbf{68.43}$ | $\mathbf{80.2}$ | $\mathbf{78.25}$ | $\mathbf{69.48}$ | $\mathbf{78.85}$ |

##### Graph and node classification

In addition, we evaluate HAM on graph learning benchmarks. Concretely, we consider 4 graph classification tasks on the GCN architecture (gcns,) in Table [5](https://arxiv.org/html/2506.02630v1#S6.T5), and 13 node classification tasks on GCN, GATv2 (gats,), and GraphSAGE (sages,) in Tables [10](https://arxiv.org/html/2506.02630v1#A4.T10), [11](https://arxiv.org/html/2506.02630v1#A4.T11) in the appendix. Details of these experiments are reported in Appendix [D.4](https://arxiv.org/html/2506.02630v1#A4.SS4), including the hyperparameter grid search, as well as an ablation on negative $\alpha$.
The success of negative values indicates that node classification prefers a different type of implicit bias.
Nonetheless, we see consistent improvements across almost all datasets and architectures with $\alpha>0$.

**Table 5: Evaluation of HAM on 4 graph classification benchmarks from OGB (ogb-benchmark,).**
| Dataset | ogbg-ppa | ogbg-molpcba | ogbg-molhiv | ogbg-code2 |
| --- | --- | --- | --- | --- |
| Metric | Accuracy $\uparrow$ | Avg. Precision $\uparrow$ | AUROC $\uparrow$ | F1 score $\uparrow$ |
| GCN | 75.48 $\pm$ 0.15 | 27.57 $\pm$ 0.04 | 82.37 $\pm$ 0.29 | 13.89 $\pm$ 2.11 |
| GCN + HAM | 75.72 $\pm$ 0.24 | 27.81 $\pm$ 0.22 | 82.50 $\pm$ 0.69 | 13.96 $\pm$ 2.06 |

## 7 Conclusion

We have proposed a new hyperbolic update step that can be combined with any first-order optimizer and that improves generalization of dense and sparse training.
Our algorithm HAM (Hyperbolic Aware Minimization) mitigates the vanishing artificial learning rate of the pointwise overparameterization $m\odot w$ used in recent sparsification methods, while inducing a similar implicit bias.
Due to discretization, it is more suitable to control the strength and shape of the bias—and with this improve generalization, especially for dense-to-sparse training.
The main mechanisms how HAM achieves this are a mild implicit bias towards sparsity and an acceleration of learning that promotes parameter sign flips.
It remains an interesting open question if different mirror maps could create better task and optimizer-specific awareness.
For example, for some tasks one might want to take into account the data structure; for the optimizer, it might be the momentum or normalization.
This could lead to more algorithmic advances to improve generalization via implicit bias control, and to new theory for understanding the success of deep learning algorithms.
In particular, optimizers with implicit biases that emulate the positive effects of overparameterization, without explicitly requiring huge models, may represent an important leap in reducing the high computational costs associated with deep learning.

## Acknowledgements

The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). We also gratefully acknowledge funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116395 SPARSE-ML.

## References

- (1)
Felipe Alvarez, Jérôme Bolte, and Olivier Brahic.
Hessian riemannian gradient flows in convex programming.
SIAM Journal on Control and Optimization, 43(2):477–501, January 2004.
- (2)
Shun-ichi Amari.
Natural gradient works efficiently in learning.
In Unsupervised Learning: Foundations of Neural Computation. The MIT Press, 05 1999.
- (3)
Amir Beck and Marc Teboulle.
Mirror descent and nonlinear projected subgradient methods for convex optimization.
Operations Research Letters, 31(3):167–175, 2003.
- (4)
Stephen P. Boyd and Lieven Vandenberghe.
Convex optimization.
2009.
- (5)
Shaked Brody, Uri Alon, and Eran Yahav.
How attentive are graph attention networks?
In International Conference on Learning Representations, 2022.
- (6)
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang.
Chasing sparsity in vision transformers: An end-to-end exploration.
Advances in Neural Information Processing Systems, 34:19974–19988, 2021.
- (7)
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.
ImageNet: A Large-Scale Hierarchical Image Database.
In CVPR09, 2009.
- (8)
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.
Rigging the lottery: Making all tickets winners.
In International Conference on Machine Learning, pages 2943–2952. PMLR, 2020.
- (9)
Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion.
(s)gd over diagonal linear networks: Implicit bias, large stepsizes and edge of stability.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 29406–29448. Curran Associates, Inc., 2023.
- (10)
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
Sharpness-aware minimization for efficiently improving generalization.
In International Conference on Learning Representations, 2021.
- (11)
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks.
arXiv: Learning, 2018.
- (12)
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks.
In International Conference on Learning Representations, 2019.
- (13)
Advait Gadhikar and Rebekka Burkholz.
Masks, signs, and learning rate rewinding.
In Twelfth International Conference on Learning Representations, 2024.
- (14)
Advait Gadhikar, Tom Jacobs, Chao Zhou, and Rebekka Burkholz.
Sign-in to the lottery: Reparameterizing sparse training from scratch, 2025.
- (15)
Advait Harshal Gadhikar and Rebekka Burkholz.
Masks, signs, and learning rate rewinding.
In International Conference on Learning Representations, 2024.
- (16)
Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz.
Why random pruning is all we need to start sparse.
In International Conference on Machine Learning, 2023.
- (17)
Egor Gladin, Pavel Dvurechensky, Alexander Mielke, and Jia-Jie Zhu.
Interaction-force transport gradient flows.
ArXiv, abs/2405.17075, 2024.
- (18)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- (19)
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro.
Characterizing implicit bias in terms of optimization geometry.
In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1832–1841. PMLR, 10–15 Jul 2018.
- (20)
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro.
Implicit regularization in matrix factorization, 2017.
- (21)
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro.
Implicit regularization in matrix factorization.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- (22)
Will Hamilton, Zhitao Ying, and Jure Leskovec.
Inductive representation learning on large graphs.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- (23)
Song Han, Jeff Pool, John Tran, and William Dally.
Learning both weights and connections for efficient neural network.
Advances in neural information processing systems, 28, 2015.
- (24)
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.
Training compute-optimal large language models.
arXiv preprint arXiv:2203.15556, 2022.
- (25)
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.
Open graph benchmark: datasets for machine learning on graphs.
In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
- (26)
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria.
Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.
arXiv preprint arXiv:2304.01933, 2023.
- (27)
Tom Jacobs and Rebekka Burkholz.
Mask in the mirror: Implicit sparsification.
In The Thirteenth International Conference on Learning Representations, 2025.
- (28)
Tom Jacobs, Chao Zhou, and Rebekka Burkholz.
Mirror, mirror of the flow: How does regularization shape implicit bias?
In Forty-second International Conference on Machine Learning, 2025.
- (29)
Adarsh Jamadandi, Celia Rubio-Madrigal, and Rebekka Burkholz.
Spectral graph pruning against over-squashing and over-smoothing.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
- (30)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.
- (31)
Mohammad Emtiyaz Khan and H. Rue.
The bayesian learning rule.
J. Mach. Learn. Res., 24:281:1–281:46, 2021.
- (32)
Thomas N. Kipf and Max Welling.
Semi-supervised classification with graph convolutional networks.
In International Conference on Learning Representations, 2017.
- (33)
Jyrki Kivinen and Manfred Warmuth.
Exponentiated gradient versus gradient descent for linear predictors.
Technical report, USA, 1994.
- (34)
Chris Kolb, Tobias Weber, Bernd Bischl, and David Rügamer.
Deep weight factorization: Sparse learning through the lens of artificial symmetries.
In The Thirteenth International Conference on Learning Representations, 2025.
- (35)
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
- (36)
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi.
Soft threshold weight reparameterization for learnable sparsity.
In Proceedings of the International Conference on Machine Learning, July 2020.
- (37)
Denis Kuznedelev, Eldar Kurtić, Elias Frantar, and Dan Alistarh.
Cap: Correlation-aware pruning for highly-accurate sparse vision models.
36, 2024.
- (38)
Denis Kuznedelev, Eldar Kurtic, Eugenia Iofinova, Elias Frantar, Alexandra Peste, and Dan Alistarh.
Accurate neural network pruning requires rethinking sparse optimization.
Trans. Mach. Learn. Res., 2024, 2023.
- (39)
Eren Mehmet Kıral, Thomas Möllenhoff, and Mohammad Emtiyaz Khan.
The lie-group bayesian learning rule.
In International Conference on Artificial Intelligence and Statistics, 2023.
- (40)
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou.
Dynamic sparse training with structured sparsity.
arXiv preprint arXiv:2305.02299, 2023.
- (41)
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr.
Snip: single-shot network pruning based on connection sensitivity.
In International Conference on Learning Representations, 2019.
- (42)
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.
Pruning filters for efficient convnets.
In International Conference on Learning Representations, 2017.
- (43)
Zhiyuan Li, Tianhao Wang, Jason D. Lee, and Sanjeev Arora.
Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent.
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
- (44)
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.
Dora: Weight-decomposed low-rank adaptation.
In Forty-first International Conference on Machine Learning, 2024.
- (45)
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy.
The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.
In International Conference on Learning Representations, 2021.
- (46)
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy.
Do we actually need dense over-parameterization? in-time over-parameterization in sparse training.
In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6989–7000. PMLR, 18–24 Jul 2021.
- (47)
Yuankai Luo, Lei Shi, and Xiao-Ming Wu.
Classic GNNs are strong baselines: Reassessing GNNs for node classification.
In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
- (48)
Yuankai Luo, Lei Shi, and Xiao-Ming Wu.
GNN+: Can classic GNNs be strong baselines for graph-level tasks?
In International Conference on Machine Learning (ICML), 2025.
- (49)
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman, and Eric-Jan Wagenmakers.
A tutorial on fisher information, 2017.
- (50)
Bochen Lyu and Zhanxing Zhu.
Implicit bias of (stochastic) gradient descent for rank-1 linear neural network.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- (51)
Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K. Warmuth.
Exponentiated gradient reweighting for robust training under label noise and beyond.
ArXiv, abs/2104.01493, 2021.
- (52)
James Martens.
New insights and perspectives on the natural gradient method.
J. Mach. Learn. Res., 21:146:1–146:76, 2014.
- (53)
Péter Mernyei and Cătălina Cangea.
Wiki-cs: A wikipedia-based benchmark for graph neural networks, 2022.
- (54)
Alexander Mielke and Jia-Jie Zhu.
Hellinger-kantorovich gradient flows: Global exponential decay of entropy functionals.
2025.
- (55)
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta.
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.
Nature Communications, 9(1), jun 2018.
- (56)
Shreyas Malakarjun Patil and Constantine Dovrolis.
Phew: Constructing sparse networks that learn fast and generalize well without training data.
In International Conference on Machine Learning, pages 8432–8442. PMLR, 2021.
- (57)
Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion.
Implicit bias of SGD for diagonal linear networks: a provable benefit of stochasticity.
In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
- (58)
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh.
Ac/dc: Alternating compressed/decompressed training of deep neural networks, 2021.
- (59)
Hoang Pham, The-Anh Ta, Shiwei Liu, Lichuan Xiang, Dung D. Le, Hongkai Wen, and Long Tran-Thanh.
Towards data-agnostic pruning at initialization: What makes a good sparse mask?
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- (60)
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova.
A critical look at the evaluation of GNNs under heterophily: Are we really making progress?
In The Eleventh International Conference on Learning Representations, 2023.
- (61)
Chendi Qian, Andrei Manolache, Kareem Ahmed, Zhe Zeng, Guy Van den Broeck, Mathias Niepert, and Christopher Morris.
Probabilistically rewired message-passing neural networks.
In The Twelfth International Conference on Learning Representations, 2024.
- (62)
Alex Renda, Jonathan Frankle, and Michael Carbin.
Comparing rewinding and fine-tuning in neural network pruning.
In International Conference on Learning Representations, 2020.
- (63)
Tyrrel R Rockafellar and Werner Fenchel.
Convex Analysis.
1970.
- (64)
Grant M. Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden-Eijnden.
Global convergence of neuron birth-death dynamics.
ArXiv, abs/1902.01843, 2019.
- (65)
Celia Rubio-Madrigal, Adarsh Jamadandi, and Rebekka Burkholz.
GNNs getting comfy: Community and feature similarity guided rewiring.
In The Thirteenth International Conference on Learning Representations, 2025.
- (66)
Pedro Savarese, Hugo Silva, and Michael Maire.
Winning the lottery with continuous sparsification, 2021.
- (67)
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann.
Pitfalls of graph neural network evaluation, 2019.
- (68)
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff.
Variational learning is effective for large deep networks.
In International Conference on Machine Learning (ICML), 2024.
- (69)
Sidak Pal Singh and Dan Alistarh.
Woodfisher: Efficient second-order approximation for neural network compression.
Advances in Neural Information Processing Systems, 33:18098–18109, 2020.
- (70)
Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli.
Pruning neural networks without any data by iteratively conserving synaptic flow.
In Advances in Neural Information Processing Systems, 2020.
- (71)
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through attention.
In International conference on machine learning, pages 10347–10357. PMLR, 2021.
- (72)
Nisheeth K. Vishnoi.
Algorithms for Convex Optimization.
Cambridge University Press, 2021.
- (73)
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse.
Picking winning tickets before training by preserving gradient flow.
In International Conference on Learning Representations, 2020.
- (74)
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro.
Kernel and rich regimes in overparametrized models.
In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 3635–3673. PMLR, 09–12 Jul 2020.
- (75)
Fan Wu and Patrick Rebeschini.
Implicit regularization in matrix sensing via mirror descent.
In Neural Information Processing Systems, 2021.
- (76)
Yuling Yan, Kaizheng Wang, and Philippe Rigollet.
Learning gaussian mixtures using the wasserstein–fisher–rao gradient flow.
The Annals of Statistics, 2024.
- (77)
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al.
Mest: Accurate and fast memory-economic sparse training framework on the edge.
Advances in Neural Information Processing Systems, 34:20838–20850, 2021.
- (78)
Liu Ziyin and Zihao Wang.
spred: Solving l1 penalty with sgd.
In International Conference on Machine Learning, 2022.

## Appendix A Proofs of theoretical statements

Here we provide detailed proofs of the main statements in the paper.
The theorems correspondence is:

- •
The proof of Theorem [3.1](https://arxiv.org/html/2506.02630v1#S3.Thmtheorem1) is in [A.1](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem1).
- •
The proof of Theorem [4.2](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem2) is in [A.2](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem2) .
- •
The proof of Lemma [4.4](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem4) is in [A.3](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem3).
- •
The proof of Theorem [4.6](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem6) is in [A.4](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem4).

###### Theorem A.1

If $m_{0}=\operatorname{sign}(x_{0})w_{0}=\sqrt{x_{0}}$ then

$$ $x_{k+1}=x_{k}\exp\left(-\eta\left(2\operatorname{sign}(x_{0})\nabla f(x_{k})+ \beta\right)\right).$ (4) $$

is a first order approximation of Eq. ([1](https://arxiv.org/html/2506.02630v1#S3.E1)).

First we use the Taylor approximation of the exponential function $\exp z\simeq 1+z$ to get the update:

$$ $x_{k+1}\simeq x_{k}-2\eta|x_{k}|\nabla f(x_{k})-\eta\beta x_{k}+\mathcal{O}( \eta^{2}).$ $$

We show this is equivalent up to first order to the gradient descent of the overparameterization:

$$ $x_{k+1}=m_{k+1}\odot w_{k+1}\simeq x_{k}-\eta\left(m_{k}^{2}+w_{k}^{2}\right) \nabla f(x_{k})-\eta\beta x_{k}$ $$

It remains to be shown that $m_{k}^{2}+w_{k}^{2}=2|x_{k}|$ for all $k\in[T]$ up to zero order approximation, we can do this with induction.
For $k=0$ the statement follows by the assumption on the initialization.
Now the induction step:

$$ $m_{k+1}^{2}+w_{k+1}^{2}=m_{k}^{2}+w_{k}^{2}-4x_{k}\eta\nabla f(x_{k})-4\eta \beta x_{k}+\eta^{2}\left(m_{k}^{2}+w_{k}^{2}\right)\nabla f(x_{k})^{2}\simeq 2 |x_{k}|+\mathcal{O}(\eta),$ $$

this concludes the induction and the proof. $\square$

###### Theorem A.2

The gradient flow ($\eta\rightarrow 0$) of Eq. ([GD](https://arxiv.org/html/2506.02630v1#S3.Ex3)) is given by:

$$ $dx_{t}=-\nabla f(x_{t})dt-|x_{t}|\left(\alpha\nabla f(x_{t})+\operatorname{ sign}(x_{t})\beta\right)dt\qquad x_{0}=x_{init}.$ $$

Writing out the computation of iterates $x_{k}$ give us:

$$ $x_{k}=x_{0}\exp{\left(\sum_{j=0}^{k-1}\eta\alpha\operatorname{sign}(x_{j}) \nabla f(x_{j})-\eta\beta\right)}-\sum_{j=0}^{k-1}\eta\nabla f(x_{j})\exp{ \left(-\sum_{l=j}^{k-1}\eta\alpha\operatorname{sign}(x_{l})\nabla f(x_{l})- \eta\beta\right)}.$ $$

This allows use to take $\eta\rightarrow 0$ and get an integral equation for the dynamics:

$$ $x_{t}=x_{0}\exp{\left(-\int_{0}^{t}\alpha\operatorname{sign}(x_{s})\nabla f(x_ {s})+\beta ds\right)}-\int_{0}^{t}\nabla f(x_{s})\exp{\left(-\int_{s}^{t} \alpha\operatorname{sign}(x_{c})\nabla f(x_{c})+\beta dc\right)ds}.$ (5) $$

Differentiating the first term under the Leibniz rule gives:

$$ $\displaystyle\frac{d}{dt}x_{0}\exp{\left(-\int_{0}^{t}\alpha\operatorname{sign }(x_{s})\nabla f(x_{s})+\beta ds\right)}=$ $\displaystyle x_{0}\exp{\left(-\int_{0}^{t}\alpha\operatorname{sign}(x_{s}) \nabla f(x_{s})-\beta ds\right)}\left(-\alpha\operatorname{sign}(x_{t})\nabla f (x_{t})+\beta\right)$ $$

Next, differentiate the second term under the Leibniz rule:

$$ $\displaystyle\frac{d}{dt}\left(-\int_{0}^{t}\nabla f(x_{s})\exp{\left(-\int_{s }^{t}\alpha\operatorname{sign}(x_{c})\nabla f(x_{c})+\beta dc\right)ds}\right)$ $\displaystyle=$ $\displaystyle-\nabla f(x_{t})-\int_{0}^{t}\frac{d}{dt}\nabla f(x_{s})\exp{ \left(-\int_{s}^{t}\alpha\operatorname{sign}(x_{c})\nabla f(x_{c})+\beta dc \right)ds}$ $\displaystyle=$ $\displaystyle-\nabla f(x_{t})dt-\int_{0}^{t}\nabla f(x_{s})\exp{\left(-\int_{s }^{t}\alpha\operatorname{sign}(x_{c})\nabla f(x_{c})+\beta dc\right)ds}\left(- \alpha\operatorname{sign}(x_{t})\nabla f(x_{t})-\beta\right)$ $$

Combining gives by noticing the form of $x_{t}$:

$$ $\displaystyle dx_{t}$ $\displaystyle=-\nabla f(x_{t})dt-x_{t}\left(\alpha\operatorname{sign}(x_{t}) \nabla f(x_{t})+\beta\right)dt$ $\displaystyle=-\nabla f(x_{t})dt-|x_{t}|\left(\alpha\nabla f(x_{t})+ \operatorname{sign}(x_{t})\beta\right)dt$ $$

$\square$

###### Lemma A.3

The Bregman function $R_{\alpha}$ for $\alpha>0$ and $\beta=0$ is given by:

$$ $R_{\alpha}(x)=\sum_{i}\frac{\left(\alpha\left|x_{i}\right|+1\right)\ln\left( \alpha\left|x_{i}\right|+1\right)-\alpha\left|x_{i}\right|}{\alpha^{2}}-x_{i} \frac{\operatorname{sign}(x_{0})}{\alpha}\log(1+\alpha|x_{0}|)$ $$

Proof.

We first construct the mirror map $R_{\alpha}$ by using the corresponding Hessian $\mathcal{I}_{HAM}$. Next we check that $R_{\alpha}$ is a Bregman function.
In the case $\beta=0$, the Hessian of the mirror map $R_{\alpha}(x)$ is:

$$ $\nabla^{2}R_{\alpha}(x)=\frac{1}{1+\alpha|x|}$ $$

Moreover we need $\nabla R_{\alpha}(x_{0})=0$.
Therefore by integrating twice, $R_{\alpha}$ for $\alpha>0$ is given by:

$$ $R_{\alpha}(x)=\sum_{i}\frac{\left(\alpha\left|x_{i}\right|+1\right)\ln\left( \alpha\left|x_{i}\right|+1\right)-\alpha\left|x_{i}\right|}{\alpha^{2}}-x\frac {\operatorname{sign}(x_{0})}{\alpha}\log(1+\alpha|x_{0}|)$ $$

It remains to be checked if $R_{\alpha}$ is Bregman.
For this we use a relationship between Legendre and Bregman functions.
We first show that $R_{\alpha}$ is Legendre and its convex conjugate as well.
Then it follows from Theorem [C.5](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem5) that $R_{\alpha}$ is Bregman for $\alpha>0$.

Note that we have $domR_{\alpha}=int(domR_{\alpha})=\mathbb{R}^{n}$.
$R_{\alpha}$ is strictly convex as for all $x\in\mathbb{R}^{n}$ the Hessian is positive definite.
This shows the first statement.
Next, since $R_{\alpha}$ is separable we can show the second statement for each parameter separately.
Take a sequence $\{x_{i,j}\}_{i=1}^{\infty}$ for coordinate $j\in[n]$ such that $|x_{i,j}|\rightarrow\infty$ then by construction of $R_{\alpha}$ we have

$$ $\lim_{i\rightarrow\infty}\partial_{j}R_{\alpha}(x_{i,j})^{2}=\infty$ $$

as $|\cdot|$ and $\log(\cdot)$ are increasing functions.
Therefore $R_{\alpha}$ is Legendre.

The convex conjugate gradient $dom\nabla R^{*}_{\alpha}=(range\nabla R_{\alpha})^{-1}=\mathbb{R}^{n}$.
Therefore since $\mathbb{R}^{n}=dom\nabla R^{*}_{\alpha}\subset domR^{*}_{\alpha}$ we can apply Theorem [C.5](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem5).
This concludes the result. $\square$

###### Theorem A.4

Let $x_{0}=0$, then $R_{\alpha}\sim||x||_{L_{2}}^{2}$ if $\alpha\rightarrow 0$ and $R_{\alpha}\sim||x||_{L_{1}}$ if $\alpha\rightarrow\infty$, where $\sim$ indicate proportionality.

Proof.
The first statement follows from the Taylor approximation:

$$ $R_{\alpha}\simeq\sum_{i}x_{i}^{2}+\frac{1}{\alpha}|x_{i}|-\frac{1}{\alpha}|x_{ i}|\simeq||x||^{2}_{L_{2}}$ $$

which is valid if $|x_{i}|<<\frac{1}{\alpha}$.

For large $\alpha>0$ we have that $R_{\alpha}\simeq\sum_{i}\frac{|x_{i}|}{\alpha}\log(\alpha|x_{i}|)$.
Therefore, we have

$$ $\displaystyle\frac{\alpha}{\log(\alpha)}R_{\alpha}(x)\simeq\sum_{i}|x_{i}|=||x ||_{L_{1}}.$ $$

This concludes the proof. $\square$

The proof of Theorem [A.4](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem4) follows similar steps as that of [74].

###### Theorem A.1

###### Theorem A.2

###### Lemma A.3

###### Theorem A.4

##### Extra statement: sign stability

In our actual algorithm (HAM) we use $\operatorname{sign}(x_{k+\frac{1}{2}})$ instead of $\operatorname{sign}(x_{k})$.
Based on [13, 14]
we can make assumptions about the sign stability during training.
In the end of training sign flips become stable, i.e, they stop flipping. Therefore, we can make the following assumption:

###### Assumption A.5

(Sign stability)
There is a $t_{0}$ such that for all $t\geq t_{0}$, $\operatorname{sign}(x_{t})=\operatorname{sign}(x_{t_{0}})$.

Under this assumption, the gradient flow analysis for the idealized version still holds in the later training part for our actual implemented algorithm HAM.

###### Theorem A.6

Under Assumption [A.5](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem5), the gradient flow of HAM for $t\geq t_{0}$ given by:

$$ $dx_{t}=-\nabla f(x_{t})dt-|x_{t}|\left(\alpha\nabla f(x_{t})+\operatorname{ sign}(x_{t})\beta\right)dt\qquad x_{t_{0}}=\hat{x},$ (6) $$

where $\hat{x}$ is the HAM iterate at $t_{0}$.

Proof.
The statement follows from making two observations:

- •
$f$ is $L-$smooth, this implies $x_{t}$ stays bounded for $t\leq t_{0}$, so $\hat{x}$ is well-defined.
- •
For $t\geq t_{0}$ by assumption $\operatorname{sign}(x_{t})=\operatorname{sign}(x_{0})$, so we can directly apply Theorem [A.2](https://arxiv.org/html/2506.02630v1#A1.Thmtheorem2). $\square$

###### Assumption A.5

###### Theorem A.6

## Appendix B Fisher information derivations

We provide here the Fisher information $\mathcal{I}$ calculations for random variables $\mathcal{N}(x,1)$ and $\mathcal{N}(\sqrt{x},1)$.
The Fisher information is defined as see for example Definition 1.1 in [49]:

$$ $\mathcal{I}(x)=\mathbb{E}\left[\log f(X)\right]$ (7) $$

where $f$ is the probability density function of the random variable $X$ and $\mathbb{E}$ is the expectation with respect to the random variable.

Let $X\sim\mathcal{N}(x,1)$, where $x\in\mathbb{R}$ is the mean parameter. The likelihood function is:

$$ $f(X;x)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(X-x)^{2}\right)$ $$

Taking the natural logarithm:

$$ $\ell(x)=\log f(X;x)=-\frac{1}{2}\log(2\pi)-\frac{1}{2}(X-x)^{2}$ $$

The score function is:

$$ $\frac{d\ell}{dx}=(X-x)$ $$

Then the Fisher Information is given by:

$$ $\mathcal{I}_{\text{GD}}(x)=\mathbb{E}\left[\left(\frac{d\ell}{dx}\right)^{2} \right]=\mathbb{E}[(X-x)^{2}]=\operatorname{Var}(X)=1$ $$

Let $X\sim\mathcal{N}(2\sqrt{|x|},1)$. The likelihood function is:

$$ $f(X;x)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(X-2\sqrt{|x|})^{2}\right)$ $$

The log-likelihood function is:

$$ $\ell(x)=-\frac{1}{2}\log(2\pi)-\frac{1}{2}(X-2\sqrt{|x|})^{2}$ $$

Differentiate to get the score function:

$$ $\frac{d\ell}{dx}=(X-2\sqrt{|x|})\cdot\left(-\frac{\text{sign}(x)}{\sqrt{|x|}} \right)=-\text{sign}(x)\frac{X-2\sqrt{x}}{\sqrt{x}}$ $$

Now square the score and take the expectation:

$$ $\mathcal{I}_{\text{HYP}}(x)=\mathbb{E}\left[\left(\frac{d\ell}{dx}\right)^{2} \right]=\mathbb{E}\left[\left(\frac{X-2\sqrt{|x|}}{\sqrt{|x|}}\right)^{2} \right]=\frac{1}{|x|}\mathbb{E}[(X-2\sqrt{|x|})^{2}]=\frac{1}{|x|}Var(X)=\frac {1}{|x|}$ $$

Note that instead of being part of the exponential family of distributions this distribution is part of the curved exponential family distributions.

## Appendix C Optimization definitions and results

In this section we recall some basic definitions from convex and non-convex optimization.

###### Definition C.1

($L-$smooth)
A differentiable function $f:\mathbb{R}^{n}\to\mathbb{R}$ is said to be *L-smooth* if its gradient is Lipschitz continuous with constant $L>0$. That is, for all $x,y\in\mathbb{R}^{n}$,

$$ $\|\nabla f(x)-\nabla f(y)\|\leq L\|x-y\|,$ $$

or equivalently,

$$ $f(y)\leq f(x)+\langle\nabla f(x),y-x\rangle+\frac{L}{2}\|y-x\|^{2}.$ $$

or equivalently,

$$ $\frac{1}{2}\|\nabla f(x)\|^{2}\leq L\left(f(x)-f^{*}\right),$ $$

where $f^{*}=\min_{x\in\mathbb{R}^{n}}f(x)$.

###### Definition C.2

(PL-inequality)
A differentiable function $f:\mathbb{R}^{n}\to\mathbb{R}$ satisfies the *PL inequality* with parameter $\Lambda>0$ if for all $x\in\mathbb{R}^{n}$,

$$ $\frac{1}{2}\|\nabla f(x)\|^{2}\geq\Lambda\left(f(x)-f^{*}\right),$ $$

where $f^{*}=\min_{x\in\mathbb{R}^{n}}f(x)$.

###### Definition C.3

(Legendre function Definition 3.8 ([43]))
Let $R:\mathbb{R}^{d}\rightarrow\mathbb{R}\cup\{\infty\}$ be a differentiable
convex function. We say $R$ is a Legendre function when the following holds:

- •
$R$ is strictly convex on $\operatorname{int}(\operatorname{dom}R)$.
- •
For any sequence $\{x_{i}\}^{\infty}_{i=1}$ going to the boundary of $\operatorname{dom}R$, $\lim_{i\rightarrow\infty}||\nabla R(x_{i})||_{L_{2}}^{2}=\infty$.

In order to recover the convergence result in Theorem 4.14 in [43] the function $R$ also needs to be a Bregman function, which we define in Definition [C.4](https://arxiv.org/html/2506.02630v1#A3.Thmtheorem4).

###### Definition C.4

(Bregman function Definition 4.1 [1])
A function $R$ is called a
Bregman function if it satisfies the following properties:

- •
$\operatorname{dom}R$ is closed. $R$ is strictly convex and continuous on $\operatorname{dom}R$. $R$ is $C^{1}$ on $\operatorname{int}(\operatorname{dom}R))$.
- •
For any $x\in\operatorname{dom}R$ and $\gamma\in\mathbb{R}$, $\{y\in\operatorname{dom}R|D_{R}(x,y)\leq\gamma\}$ is bounded.
- •
For any $x\in\operatorname{dom}R$ and sequence $\{x_{i}\}^{\infty}_{i=1}\subset\operatorname{int}(\operatorname{dom}R)$ such that $\lim_{i\rightarrow\infty}x_{i}=x$, it holds that $\lim_{i\rightarrow\infty}D_{R}(x,x_{i})\rightarrow 0$.

###### Theorem C.5

(Theorem 4.7 [1])
If $R$ is a Legendre function with $\operatorname{dom}R=\mathbb{R}^{n}$,
then if the domain of the convex conjugate $\operatorname{dom}R^{*}=\mathbb{R}^{n}$ implies that $R$ is a Bregman function

###### Definition C.1

###### Definition C.2

###### Definition C.3

###### Definition C.4

###### Theorem C.5

## Appendix D Experimental details

We present the additional hyper parameters and other details of the experimental setup. Furthermore we provide ablations on various setups. In general HAM is applied to all layers except the batchnorm or layernorm layers.

### D.1 Overparameterized linear regression

We illustrate Theorem [4.5](https://arxiv.org/html/2506.02630v1#S4.Thmtheorem5) by considering a under-determined linear regression setup, similar to that of [57, 27]. We consider a sparse groundtruth $x^{*}$ and initialize at $x_{0}=0$.
Moreover, we use the mean squared error loss function.
We generate data by sampling $z_{i}\sim N(0,\mathbf{I})$ i.i.d. for $i\in[d]$, with $d=40$ and $n=100$.
We compare gradient descent with and without HAM.
Moreover we also show what happens if we replace $\text{sign}(x_{k+\frac{1}{2}})$ with $\text{sign}(x_{k})$ denoted with HAM signed.
The learning rate is set $\eta=10e-4$, and both algorithms are run for $10e+6$ steps.
We track the distance to the ground truth during training.
In Figure [3](https://arxiv.org/html/2506.02630v1#A4.F3) we observe that HAM gets closer to the ground truth and converges faster then both gradient descent and HAM signed.

Figure: Figure 3: Gradient flow simulation of HAM, HAM with corrected sign and gradient descent.
Refer to caption: x4.png

### D.2 Dense and sparse training on vision tasks and ablation

We provide additional results on CIFAR100 [35] in Table [7](https://arxiv.org/html/2506.02630v1#A4.T7).
Furthermore, we train a small DeiT [71] with AdamW in Table [8](https://arxiv.org/html/2506.02630v1#A4.T8).
All results are for $3$ seeds.
We provide the hyperparameter grid search for CIFAR 100 and Imagenet [7] in Figures [6](https://arxiv.org/html/2506.02630v1#A4.F6) and [7](https://arxiv.org/html/2506.02630v1#A4.F7).
We find that the grid search is consistent i.e. there is a global best configuration. This implies it is easy to tune for specific tasks.
We illustrate the convergence and implicit bias behaviour by tracking the training loss and $L_{1}$ norm in Figure [4](https://arxiv.org/html/2506.02630v1#A4.F4).
We also track the $L_{1}$ norm when comparing with SAM to show that SAM and HAM exploit different principles in Figure [5](https://arxiv.org/html/2506.02630v1#A4.F5).
The additional hyper-parameters of the experiments can be found in Table [6](https://arxiv.org/html/2506.02630v1#A4.T6).
The same parameters are used for the sparse training setup.
Each experiment was run on 4 A100 GPUS.

##### Parameters for m ⊙ w direct-product 𝑚 𝑤 m\odot w italic_m ⊙ italic_w

The
$m\odot w$ parameterization is not regularized with weight decay for the dense scenario, as this induces sparsity.
Instead, weight decay is applied on the product $(m\odot w)^{2}$ with strength $5e-5$, the same strength as for dense training [14].

**Table 6: Training Details for the dense vision experiments presented in the paper.**
| Dataset | Model | LR | Weight Decay | Epochs | Batch Size | Optim | LR Schedule |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR100 | ResNet18 | $0.2$ | $1e-4$ | $150$ | $512$ | SGD, $m=0.9$, SAM | Triangular |
| ImageNet | ResNet50 | $0.25$ | $5e-5$ | $100,200$ | $1024$ | SGD, $m=0.9$, SAM | Triangular |
| DeIT Small | $0.001$ | $1e-1$ | $300$ | $1024$ | AdamW | Triangular |  |

Figure: Figure 4: Training dynamics of HAM compared to the baseline for a ResNet 50 on Imagenet. Observe in the first figure (left) that the average training loss converges faster with higher $\alpha$ given the same $\beta$. This illustrates the convergence speed up predicted by our developed theory. Moreover, in the second figure (right) the average $L_{1}$ norm decays more due the regularization constant $\beta$, whereas larger $\alpha$ leads to a larger initial increase in the average $L_{1}$ norm it decays faster in the end. This is in line with being more uncertain about the sign of the weights in the beginning of training.
Refer to caption: x5.png

Figure: Figure 5: Training dynamics of HAM with and without SAM for a ResNet50 on Imagenet. Observe that the choice of our hyperparameters $\alpha$ and $\beta$ determine the general trend of the average $L_{1}$ norm while the choice between SGD and SAM make less of a difference. This provides additional evidence for their complementary working.
Refer to caption: x6.png

Figure: Figure 6: One seed hyperparameter search for a ResNet18 on CIFAR100.
Refer to caption: x7.png

Figure: Figure 7: One seed hyperparameter search for a Resnet 50 on Imagenet.
Refer to caption: x8.png

**Table 7: Dense training with HAM on the CIFAR100 vision benchmarks.**
| Dataset | Baseline | $\alpha=0,\beta=16e-3$ | $\alpha=200,\beta=0$ | $\alpha=200,\beta=16e-3$ |
| --- | --- | --- | --- | --- |
| HAM | $75.25\pm 0.24$ | $75.36\pm 0.04$ | $75.31\pm 0.30$ | $\bf 76.12\pm 0.27$ |
| SAM-HAM | $75.12\pm 0.68$ | $76.30\pm 0.11$ | $75.25\pm 0.20$ | $\bf 76.65\pm 0.23$ |

**Table 8: Pre-training a vision transformer from scratch for $300$ epochs on ImageNet.**
| Setup | AdamW + HAM | AdamW |
| --- | --- | --- |
| ImageNet+ DeiT Small | $\mathbf{72.65(\pm 0.09)}$ | $72.31(\pm 0.09)$ |

### D.3 LLM finetuning

Each experiment was run on 4 A100 GPUs. We use th experimental setup of DoRA [44] to finetune LlaMA 3.2 models of size 1B and 3B with $\alpha=200,\beta=1e-3$ and $\alpha=100,\beta=1e-4$ respectively.
We find that larger models benefit from less strong regularization which is consistent with our different regularization strengths $\beta$ for ResNet18 and ResNet50.

### D.4 Graph and node classification

We report the experimental details of the 4 graph classification benchmarks from [Table 5](https://arxiv.org/html/2506.02630v1#S6.T5). We also include results on 13 node classification benchmarks, and an ablation on $\alpha<0$. Each experiment was run on an A100.

#### D.4.1 Graph classification

We report the performance of HAM on four graph classification datasets from Open Graph Benchmark (OGB) [25].
The code to run these benchmarks is based on [48], using their choice of hyperparameters and ADAM as the optimizer. We use their GCN+ architecture, which is a GCN equipped with edge features, normalization, dropout, residual connections, feed-forward networks, and positional encodings. In order to implement HAM in combination with dropout, we mask the regularization term $\beta$ with $(\text{grad}\neq 0)$. We only apply HAM on the weights and biases associated with the convolutional layers. The results, shown in [Table 5](https://arxiv.org/html/2506.02630v1#S6.T5), are averaged over 3 seeds. We report the best validation metric for the best values of $\alpha$ and $\beta$ for HAM, the selection of which is displayed in [Table 9](https://arxiv.org/html/2506.02630v1#A4.T9). The tuning range is $\alpha\in\{1,10,100,200\}$, and $\beta\in\{0,0.01,0.1\}$. [Table 9](https://arxiv.org/html/2506.02630v1#A4.T9) also includes the size of the datasets in terms of number of graphs. Note that for three datasets, $\alpha$ is the same as for the vision tasks.

**Table 9: $\alpha$ and $\beta$ best values for the graph classification tasks.**
| Dataset | ogbg-ppa | ogbg-molpcba | ogbg-molhiv | ogbg-code2 |
| --- | --- | --- | --- | --- |
| # graphs | 158,100 | 437,929 | 452,741 | 41,127 |
| $\alpha$ | 200 | 200 | 200 | 1 |
| $\beta$ | 0.1 | 0.1 | 0.1 | 0.01 |

#### D.4.2 Node classification

We furthermore report the performance of HAM on thirteen node classification datasets: Cora, CiteSeer, and PubMed [32], Wiki-CS [53], Coauthor-CS, Coauthor-Physics, Amazon-Computers, and Amazon-Photo [67] (homophilic); Amazon-Ratings, Squirrel, Chameleon, Minesweeper, and Roman-Empire [60] (heterophilic). We evaluate over GCN, GATv2, and GraphSAGE. We only apply HAM on the weights and biases associated with the convolutional layers and on the attention parameters. The code to run these benchmarks is based on [47], using their choice of hyperparameters and ADAM as the optimizer. The results, shown in Tables [10](https://arxiv.org/html/2506.02630v1#A4.T10) (homophilic) and [11](https://arxiv.org/html/2506.02630v1#A4.T11) (heterophilic), are averaged over different runs according to the original setup. We report the best validation accuracy for the best values of $\alpha$ and $\beta$ for HAM, the selection of which is displayed in [Table 12](https://arxiv.org/html/2506.02630v1#A4.T12). The tuning range is $\alpha\in\{1,10,100,200\}$, and $\beta\in\{0,0.01,0.1\}$. The best value per architecture is in bold, and ties are underlined. Tables [10](https://arxiv.org/html/2506.02630v1#A4.T10) and [11](https://arxiv.org/html/2506.02630v1#A4.T11) also include the size of the datasets in terms of number of nodes and edges.

**Table 10: Evaluation of HAM on 8 homophilic node classification benchmarks.**
| Dataset | cora | citeseer | pubmed | wikics | coauthor-cs | coauthor-physics | amazon-computer | amazon-photo |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # nodes | 2,708 | 3,327 | 19,717 | 11,701 | 18,333 | 34,493 | 13,752 | 7,650 |
| # edges | 5,278 | 4,522 | 44,324 | 216,123 | 81,894 | 247,962 | 245,861 | 119,081 |
| GCN | 81.32 $\pm$ 0.30 | 68.60 $\pm$ 0.94 | 77.96 $\pm$ 0.46 | 80.71 $\pm$ 0.29 | 95.32 $\pm$ 0.12 | 97.17 $\pm$ 0.01 | 83.47 $\pm$ 0.70 | 94.33 $\pm$ 0.61 |
| GCN+HAM | 81.44 $\pm$ 0.43 | 68.64 $\pm$ 0.97 | 78.16 $\pm$ 0.65 | 80.84 $\pm$ 0.27 | 95.32 $\pm$ 0.04 | 97.18 $\pm$ 0.02 | 83.93 $\pm$ 0.58 | 95.33 $\pm$ 0.23 |
| GAT | 81.24 $\pm$ 0.68 | 68.68 $\pm$ 0.30 | 79.00 $\pm$ 0.95 | 82.35 $\pm$ 0.39 | 95.33 $\pm$ 0.07 | 97.15 $\pm$ 0.01 | 83.67 $\pm$ 0.23 | 94.47 $\pm$ 0.31 |
| GAT+HAM | 81.56 $\pm$ 0.67 | 68.92 $\pm$ 0.27 | 79.08 $\pm$ 0.88 | 82.47 $\pm$ 0.38 | 95.33 $\pm$ 0.07 | 97.16 $\pm$ 0.01 | 84.40 $\pm$ 0.92 | 94.60 $\pm$ 0.69 |
| SAGE | 80.44 $\pm$ 1.03 | 67.44 $\pm$ 0.26 | 79.36 $\pm$ 0.67 | 81.72 $\pm$ 0.46 | 95.50 $\pm$ 0.08 | 97.01 $\pm$ 0.10 | 83.07 $\pm$ 0.90 | 95.33 $\pm$ 0.31 |
| SAGE+HAM | 80.60 $\pm$ 0.63 | 67.48 $\pm$ 0.18 | 79.96 $\pm$ 0.62 | 81.78 $\pm$ 0.41 | 95.58 $\pm$ 0.09 | 97.03 $\pm$ 0.08 | 83.60 $\pm$ 1.11 | 95.40 $\pm$ 0.40 |

**Table 11: Evaluation of HAM on 5 heterophilic node classification benchmarks.**
| Dataset | amazon-ratings | squirrel | chameleon | minesweeper | roman-empire |
| --- | --- | --- | --- | --- | --- |
| # nodes | 24,492 | 2,223 | 890 | 10,000 | 22,662 |
| # edges | 93,050 | 46,998 | 8,854 | 39,402 | 32,927 |
| GCN | 53.23 $\pm$ 0.54 | 44.52 $\pm$ 1.12 | 46.12 $\pm$ 2.38 | 97.46 $\pm$ 0.24 | 90.96 $\pm$ 0.33 |
| GCN+HAM | 53.43 $\pm$ 0.44 | 44.55 $\pm$ 1.28 | 46.78 $\pm$ 2.22 | 97.78 $\pm$ 0.53 | 91.22 $\pm$ 0.40 |
| GAT | 55.47 $\pm$ 0.20 | 42.22 $\pm$ 1.73 | 45.84 $\pm$ 3.02 | 97.98 $\pm$ 0.21 | 90.58 $\pm$ 0.91 |
| GAT+HAM | 55.58 $\pm$ 0.47 | 43.17 $\pm$ 1.37 | 46.37 $\pm$ 3.32 | 98.37 $\pm$ 0.46 | 90.83 $\pm$ 0.89 |
| SAGE | 55.05 $\pm$ 0.50 | 40.91 $\pm$ 1.27 | 42.80 $\pm$ 2.90 | 97.02 $\pm$ 0.59 | 90.51 $\pm$ 0.33 |
| SAGE+HAM | 55.50 $\pm$ 0.55 | 40.85 $\pm$ 1.16 | 43.07 $\pm$ 2.84 | 97.77 $\pm$ 0.16 | 90.57 $\pm$ 0.44 |

**Table 12: $\alpha$ and $\beta$ best values for the node classification tasks.**
|  | GCN | GAT | SAGE |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset $\downarrow$ | $\alpha$ | $\beta$ | $\alpha$ | $\beta$ | $\alpha$ | $\beta$ |
| cora | 200 | 0.1 | 200 | 0.1 | 200 | 0.1 |
| citeseer | 1 | 0.01 | 10 | 0 | 200 | 0.01 |
| pubmed | 200 | 0.1 | 100 | 0 | 10 | 0.01 |
| wikics | 10 | 0.01 | 1 | 0.1 | 1 | 0.1 |
| coauthor-cs | 1 | 0.1 | 10 | 0.1 | 10 | 0.1 |
| coauthor-physics | 10 | 0 | 10 | 0.01 | 1 | 0 |
| amazon-computer | 10 | 0.01 | 10 | 0 | 200 | 0 |
| amazon-photo | 10 | 0.01 | 10 | 0 | 1 | 0.01 |
| amazon-ratings | 10 | 0.01 | 200 | 0.01 | 200 | 0 |
| squirrel | 10 | 0.01 | 200 | 0 | 200 | 0.1 |
| chameleon | 200 | 0 | 200 | 0 | 200 | 0.1 |
| minesweeper | 200 | 0.1 | 100 | 0.1 | 1 | 0.01 |
| roman-empire | 10 | 0.1 | 10 | 0.1 | 10 | 0.1 |

#### D.4.3 Node classification with α < 0 𝛼 0 \alpha<0 italic_α < 0

We perform an ablation on the node classification tasks by assigning negative values to $\alpha$, denoted as nHAM. Tables [13](https://arxiv.org/html/2506.02630v1#A4.T13) (homophilic) and [14](https://arxiv.org/html/2506.02630v1#A4.T14) (heterophilic) show the results of this ablation, while [Table 15](https://arxiv.org/html/2506.02630v1#A4.T15) displays the optimal pair of ($\alpha<0$, $\beta$) hyperparameters. Note that the baselines never perform better than HAM or nHAM. Surprisingly, heterophilic datasets appear to be able to benefit more consistently from nHAM, especially for GCN and GraphSAGE. In homophilic datasets and GATs, it still provides consistent but smaller improvements, or matches the best performance of $\alpha>0$. This intriguing phenomenon requires further investigation, as it may indicate the need for a different kind of implicit bias in certain graph-based architectures. Other methods particularly effective in heterophilic settings, such as modifying the adjacency matrix [61, 29, 65], may offer insight into these results.

**Table 13: HAM ($\alpha\!>\!0$) compared to nHAM ($\alpha\!<\!0$) on 8 homophilic node classification benchmarks.**
| Dataset | cora | citeseer | pubmed | wikics | coauthor-cs | coauthor-physics | amazon-computer | amazon-photo |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCN | 81.32 $\pm$ 0.30 | 68.60 $\pm$ 0.94 | 77.96 $\pm$ 0.46 | 80.71 $\pm$ 0.29 | 95.32 $\pm$ 0.12 | 97.17 $\pm$ 0.01 | 83.47 $\pm$ 0.70 | 94.33 $\pm$ 0.61 |
| GCN+HAM | 81.44 $\pm$ 0.43 | 68.64 $\pm$ 0.97 | 78.16 $\pm$ 0.65 | 80.84 $\pm$ 0.27 | 95.32 $\pm$ 0.04 | 97.18 $\pm$ 0.02 | 83.93 $\pm$ 0.58 | 95.33 $\pm$ 0.23 |
| GCN+nHAM | 81.44 $\pm$ 0.26 | 68.68 $\pm$ 0.99 | 78.08 $\pm$ 0.58 | 80.89 $\pm$ 0.30 | 95.34 $\pm$ 0.09 | 97.17 $\pm$ 0.00 | 84.20 $\pm$ 0.40 | 95.27 $\pm$ 0.50 |
| GAT | 81.24 $\pm$ 0.68 | 68.68 $\pm$ 0.30 | 79.00 $\pm$ 0.95 | 82.35 $\pm$ 0.39 | 95.33 $\pm$ 0.07 | 97.15 $\pm$ 0.01 | 83.67 $\pm$ 0.23 | 94.47 $\pm$ 0.31 |
| GAT+HAM | 81.56 $\pm$ 0.67 | 68.92 $\pm$ 0.27 | 79.08 $\pm$ 0.88 | 82.47 $\pm$ 0.38 | 95.33 $\pm$ 0.07 | 97.16 $\pm$ 0.01 | 84.40 $\pm$ 0.92 | 94.60 $\pm$ 0.69 |
| GAT+nHAM | 81.56 $\pm$ 0.43 | 68.92 $\pm$ 0.18 | 79.08 $\pm$ 0.88 | 82.45 $\pm$ 0.41 | 95.33 $\pm$ 0.07 | 97.16 $\pm$ 0.01 | 83.93 $\pm$ 0.31 | 94.80 $\pm$ 0.72 |
| SAGE | 80.44 $\pm$ 1.03 | 67.44 $\pm$ 0.26 | 79.36 $\pm$ 0.67 | 81.72 $\pm$ 0.46 | 95.50 $\pm$ 0.08 | 97.01 $\pm$ 0.10 | 83.07 $\pm$ 0.90 | 95.33 $\pm$ 0.31 |
| SAGE+HAM | 80.60 $\pm$ 0.63 | 67.48 $\pm$ 0.18 | 79.96 $\pm$ 0.62 | 81.78 $\pm$ 0.41 | 95.58 $\pm$ 0.09 | 97.03 $\pm$ 0.08 | 83.60 $\pm$ 1.11 | 95.40 $\pm$ 0.40 |
| SAGE+nHAM | 81.00 $\pm$ 0.35 | 67.48 $\pm$ 0.23 | 79.84 $\pm$ 0.93 | 81.82 $\pm$ 0.45 | 95.56 $\pm$ 0.12 | 97.02 $\pm$ 0.10 | 83.60 $\pm$ 1.04 | 95.67 $\pm$ 0.61 |

**Table 14: HAM ($\alpha\!>\!0$) compared to nHAM ($\alpha\!<\!0$) on 5 heterophilic node classification benchmarks.**
| Dataset | amazon-ratings | squirrel | chameleon | minesweeper | roman-empire |
| --- | --- | --- | --- | --- | --- |
| GCN | 53.23 $\pm$ 0.54 | 44.52 $\pm$ 1.12 | 46.12 $\pm$ 2.38 | 97.46 $\pm$ 0.24 | 90.96 $\pm$ 0.33 |
| GCN+HAM | 53.43 $\pm$ 0.44 | 44.55 $\pm$ 1.28 | 46.78 $\pm$ 2.22 | 97.78 $\pm$ 0.53 | 91.22 $\pm$ 0.40 |
| GCN+nHAM | 53.43 $\pm$ 0.25 | 44.58 $\pm$ 1.09 | 46.78 $\pm$ 1.85 | 97.85 $\pm$ 0.10 | 91.23 $\pm$ 0.36 |
| GAT | 55.47 $\pm$ 0.20 | 42.22 $\pm$ 1.73 | 45.84 $\pm$ 3.02 | 97.98 $\pm$ 0.21 | 90.58 $\pm$ 0.91 |
| GAT+HAM | 55.58 $\pm$ 0.47 | 43.17 $\pm$ 1.37 | 46.37 $\pm$ 3.32 | 98.37 $\pm$ 0.46 | 90.83 $\pm$ 0.89 |
| GAT+nHAM | 55.76 $\pm$ 0.55 | 42.84 $\pm$ 1.25 | 46.23 $\pm$ 2.91 | 98.53 $\pm$ 0.25 | 90.74 $\pm$ 0.82 |
| SAGE | 55.05 $\pm$ 0.50 | 40.91 $\pm$ 1.27 | 42.80 $\pm$ 2.90 | 97.02 $\pm$ 0.59 | 90.51 $\pm$ 0.33 |
| SAGE+HAM | 55.50 $\pm$ 0.55 | 40.85 $\pm$ 1.16 | 43.07 $\pm$ 2.84 | 97.77 $\pm$ 0.16 | 90.57 $\pm$ 0.44 |
| SAGE+nHAM | 55.25 $\pm$ 1.00 | 41.11 $\pm$ 1.61 | 43.32 $\pm$ 2.92 | 97.81 $\pm$ 0.14 | 90.64 $\pm$ 0.56 |

**Table 15: $\alpha$ and $\beta$ best values for the node classification tasks with $\alpha<0$.**
|  | GCN | GAT | SAGE |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset $\downarrow$ | $\alpha$ | $\beta$ | $\alpha$ | $\beta$ | $\alpha$ | $\beta$ |
| cora | -10 | 0.01 | -10 | 0 | -200 | 0.01 |
| citeseer | -1 | 0.01 | -10 | 0 | -100 | 0 |
| pubmed | -100 | 0.01 | -100 | 0 | -1 | 0.1 |
| wikics | -100 | 0.1 | -10 | 0.01 | -10 | 0.01 |
| coauthor-cs | -10 | 0.01 | -100 | 0.01 | -100 | 0.01 |
| coauthor-physics | -1 | 0.1 | -1 | 0.01 | -100 | 0.1 |
| amazon-computer | -200 | 0 | -1 | 0.01 | -1 | 0.01 |
| amazon-photo | -200 | 0 | -200 | 0 | -1 | 0.01 |
| amazon-ratings | -10 | 0 | -200 | 0.1 | -1 | 0 |
| squirrel | -100 | 0 | -200 | 0.01 | -200 | 0.1 |
| chameleon | -200 | 0 | -100 | 0 | -200 | 0.1 |
| minesweeper | -100 | 0.1 | -10 | 0.1 | -200 | 0.1 |
| roman-empire | -200 | 0.1 | -10 | 0.1 | -1 | 0.1 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - I'm generally interested in evaluating whether the regular optimization step is strictly to facilitate sign flips, or offers more performance beyond that. To this end, I was wondering, (1) what is the effect of only the hyperbolic step, and (2) what is the effect of only applying the regular gradient step for small magnitude weights (i.e. ones whose signs could be flipped).
- What kind of sparsity does HAM trained models get (without combining with any sparse training). I.e. is direct magnitude pruning at the end of initialization more successful with HAM than with standard training due to HAM's implicit bias to sparse networks?
- Can the authors offer any intuition why HAM improves performance in dense training? Given that the motivation of the paper was directed towards sparse training, the improvement in dense training is surprising.
soundness: 3
strengths: - Clear motivation and presentation
- The method is compatible with existing optimizers (such as SAM), and requires little computational overhead
- Strong empirical results - with the method surprisingly also improving dense training
- Thorough theoretical analysis
summary: - This paper studies the training of sparse networks, specifically those which learn the mask as $\theta = m \odot w$
- This paper builds on previous work which identify a limitation of these models in learning sign flips due to the small inverse Riemannian metric as $m$ or $w$ approaches zero
- Motivated by the reparameterized gradient flow of these masked models, the authors propose interleaving hyperbolic updates between regular gradient steps.
- The hyperbolic updates inherit the favourable sparsity inducing dynamics from training masked models, while the regular gradient steps handle sign flips
- Empirically, HAM outperforms methods which only use the base optimizer on ImageNet and CIFAR-100 on both dense and sparse training
weaknesses: - The method additionally adds two hyperparameters ($\alpha$, $\beta$). Based on figure 8 and 9 in the appendix, the performance of HAM is quite sensitive of these hyperparameters, and optimal hyperparameters seems to differ between models, making it difficult to tune
- It seems that although HAM is motivated from the perspective of sparse training, it doesn't actually improve sparse training directly without pairing it with another method (remark 4.7 and 4.8). Given this, the motivation behind HAM seems less clear

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: On the note of the hyperparameter scale, in line with recent lines of literature studying the effective learning rate induced by various hyperparameters, is it possible that the "correct" scale of $\alpha$ (and $\beta$) actually has an implicit dependence on the scale of the parameter block (e.g. width)? For example, the Maximum Update Parameterization framework (see e.g. [1]) suggests that layer-wise descent methods require a per-layer adjustment based on the dimensionality in order to guarantee stable evolution across model scales. It seems that an entrywise modification like HAM should also naturally admit some layerwise dimensionality scaling. In particular, did the authors experience the optimal choice of $\alpha, \beta$ changing across different model sizes?

The HAM step compares the sign of the gradient with the iterate. When using a different descent method than GD, is the hyperbolic step adjusted accordingly, or does it still compare to the gradient?

[1] Yang et al. "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"
soundness: 2
strengths: This paper proposes an interesting optimizer modification that attempts to reap the benefits of "point-wise" overparameterization via parameter-wise multiplication without introducing a memory overhead. The sign-adjustment using the previous iterate in the hyperbolic step seems to be novel, and has an interesting interpretation via Riemannian gradient flow. The experiments seem to demonstrate that HAM provides modest improvements across many scenarios as a drop-in adjustment, which seems to imply its general-purpose nature. Furthermore, it seems that the "sparsity-friendly" nature of HAM synergizes well with pruning or sparse training methods.
summary: This paper proposes a new optimizer step: Hyperbolic-Aware Minimization (HAM), where first a nominal step is taking by the base optimizer (e.g. Gradient Descent), then a "hyperbolic" step analogous to exponential gradient descent. By taking the sign of the previous iterate into account, this method reduces the updated iterate's magnitude for the components of the previous iterate and the gradient that agree, otherwise the exponential step blows up the updated iterate's magnitude to exaggerate the descent step on disagreeing components. The optimizer behavior is supported by deriving the corresponding Riemannian gradient flow, and applying it to the toy problem of underdetermined linear regression. Therein, it is found that HAM induces an in-between of $L^1$ and $L^2$-flavored regularization, with the behavior approaching $L^1$ when the hyperparameter $\alpha$ is large. It is noted that this implicit bias toward sparsity is rather weak. The efficacy of HAM is then demonstrated on a wide range of training tasks.
weaknesses: Overall, to me it is not made clear in Sections 3 and 4 the main motivating purpose of HAM. In particular, it is claimed via a Riemannian gradient flow analysis that HAM converges faster than gradient flow, but the main numerical results do not seem to focus on this. It is then claimed that HAM encourages a particular kind of implicit sparsity bias, but it is noted by the authors in Remark 4.7 that this is relatively weak alone. As such, it would be good to clarify the core motivation of HAM, and how the claimed benefits are supported by the theoretical derivations. Furthermore, neither theoretical intuition lends an explanation to why HAM seems to get better *generalization* on dense training tasks. Though the effect is good to see, does any of the theoretical motivation suggest this?

Secondly, though the optimizer is presented as a drop-in adjustment, it does introduce two new hyperparameters in $\alpha, \beta$ without intuitive guidance of how these should be set. Indeed, the hyperparameter sweeps in the appendix seem to suggest the choice of $\alpha$ is important, and that the optimal choice of $\alpha$ can vary to be quite large, and the optimal corresponding choice of $\beta$ also varies quite significantly on a logarithmic scale between $0$ and $1$. Without additional guidance or intuition, this means HAM introduces two additional independent hyperparameters to sweep, which is non-trivial to grid-search over.

---

confidence: 1
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1. It's not entirely clear to me why HAM's hyperbolic steps on a "dense" $\theta$ should preserve the benefits of optimizing on the product $m \odot w$. Could the authors please give me an intuitive explanation as to why?

Q2. Could the authors please expand on what the advantages of a product overparametrization are compared to the others discussed?
soundness: 3
strengths: 1. The paper is well-written and clearly identifies a gap in the existing literature.

2. The theoretical contribution is clear.

3. The experiments appear to support the claims.
summary: This paper analyzes the training dynamics of pointwise overparametrization, where a parameter is replaced by a product $m \odot w$.
The authors argue that this reparametrization, when studied as a Riemannian gradient flow, suffers from a small inverse metric near $\theta = 0$.
This can cause parameters to get "stuck" and prevent crucial sign flips, and existing remedies like Sign-In introduce perturbations that do not clearly improve training dynamics.
To address this, the authors propose Hyperbolic Aware Minimization (HAM), an optimization step designed to capture the benefits of pointwise overparametrization without getting stuck near zero.
The main idea in HAM is to alternate a standard optimizer step with a hyperbolic mirror step.
Experiments on vision models show that HAM improves the final generalization of state-of-the-art sparse training methods like AC/DC and RiGL.
weaknesses: While I did not see any obvious weaknesses, I also cannot strongly endorse this paper as I am an outsider to this area.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. The questions raised in the Weaknesses: a. What is the connection between sign flip and performance improvements? b. How do we define the range of $\alpha$? 

2. The two-step update manner makes me wonder the comparison with other two-step methods, e.g., momentum methods. Can the authors discuss the comparison regarding several aspects such as convergence speed (as the authors claim that HAP has a faster convergence rate compared to GD) with some of other two-step methods?

3. Is it possible to design a one-step update optimizer inspired by HAM?

4. Prior works revealed that stochasticity can boost sparsity implicit bias of point-wise overparameterization, then how does stochasticity affect HAM?
soundness: 3
strengths: 1. This paper is well-motivated and well-organized. Specifically, 
   - The authors first state the sparsity implicit bias of point-wise overparameterization, which can be applied to achieve sparsity of training models by realizing the resulted Riemannian gradient flow. This gives the start point of this paper.
   - Then the vanishing inverse metric issue is identified, which induce the hardness of sign flip prior approaches. This gives the motivation of this paper.  
   - The authors then solve this issue by adding a constant 1 to $| \theta |$ (this is in fact an approximation of $\sqrt{ \theta^2 + \gamma^2}$ when $\gamma \to 0$) in the inverse metric. This is well-motivated in my view. 
   - Finally, the authors propose to use a two-step update manner to apply this newly designed inverse metric of  Riemannian flow. 

2. This paper provides a theoretical foundation of their method, which can help the readers understand some basic properties of this newly proposed optimizer in a very simple setting. Then this theoretical analysis is further supported by empirical evaluations on vision benchmarks where several methods are compared and discussed.
summary: This paper proposes a new optimizer by adding one additional hyperbolic mirror step to the original gradient descent (GD). This modification is originally inspired by the sparsity implicit bias of point-wise overparameterization. The main contribution of this paper lies in that the newly designed approach further mitigates the vanishing inverse metric issue near zero of prior approaches. The authors analyze their method in the linear regression setup, and further demonstrate the effectiveness of their method in different benchmarks.
weaknesses: 1. There lacks intuitive explanation on the connection between sign flip and performance gaining. While the authors motivate that the vanishing inverse metric is an issue of prior approaches, there still lacks clear evidence that solving this issue is strongly related to the performance gaining. This makes the scope of the applicability of HAM unclear.

2. The theoretical analysis still has limitations. The theoretical analysis is based on very limited settings. This leaves several important questions unanswered, e.g., while the authors claimed a faster convergence rate compared to GD in the simple setting, the readers are still unclear whether this is true in more general setting. 

3. This HAM highly depends on the value of $\nabla f$, hence how to choose a suitable $\alpha$ can be very tricky. For example, the range of $\alpha$ is not restricted so that it goes from 0 to a very large constant. This makes it very hard and non-intuitive to pick a suitable $\alpha$. This unavoidably gives parameter tuning burden in practical training. 

4. Minor: The replacement of $sign(\theta_0)$ with $sign(\theta_k)$ is confusing: the study of object in Theorem 3.1 applies $\theta_0$ but this is replaced by $\theta_k$ (and further replaced by $\theta_{k + 1 / 2}$ later). I'm a bit confused about why the authors start with analyzing the version with $\theta_0$ instead of directly proposing HYP.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: In the following, questions (Q) and comments/suggestions (S) are given point-by-point. A dedicated response to all points, especially the suggestions, is not expected. 


- Q1: Related to W1b: What is the main purpose or application of HAM? Is it proposed mainly as a general-purpose optimization step, a modification to boost sparsification methods, a sparsification method in its own right, or a replacement of the $m \odot w$ parameterization by the HAM step? Clearly, the method could be all of the above, but the exposition should highlight the principal use case.

- Q2: Lines 17-18: The abstract states that HAM incurs less computational overhead than $m \odot w$, but, e.g., Ziyin and Wang (2023) state that $m \odot w$ on ResNet-18 takes less than 5% more time than without, which is almost negligible. Why or when is the claimed computational cost advantage (which is not empirically verified in the submission) of HAM over $m \odot w$ relevant? 

- Q3: Why is Lemma 4.4 limited to $\beta=0$?

- Q4: Line 237: What is scaled and why by that factor? Context is missing here.

- Q5: Why do the results between Tables 4 and 10 differ significantly for some of the overlapping methods, i.e.,  AC/DC and STR, as well as their sign-in/HAM variants? This seems particularly strange since the reported performances for Random(+Sign-In/HAM) are identical, suggesting identical settings.

- Q6: Why does random pruning (baseline) outperform 4 out of 5 competitor methods in Table 10? This seems strange, particularly since Random performs worst across settings in Table 4.

- Q7: Clearly, HAM provides an advantage over $m \odot w$ with balanced initializations by enabling sign flips. But a simple remedy to the sign-flip problem could be to use a standard initialization for $m,w$. What additional purpose does HAM serve over $m \odot w$ in that case?

- Q8: Line 933: What justifies the use of the linearization $\exp(x) \approx 1+x$, which holds only around 0, specifically?

- Q9: I don’t fully understand the difference between Theorem 4.2/B.3 (GF for Eqs GD;HYP) and Theorem B.6 (GF of HAM). Does Theorem B.6 correspond to Eqs GD;HYP*?

- Q10: line 1252: Why does $w_i$ have an index $i$ but not $x$? And why does the setup define $z_i$ but the objective contains only $y_i$ and $x$?

- Q11: Lemma E.1 is incomprehensible to me. Why a teacher-student set-up? Which four cases? The writing is also hard to understand.

- Q12: Line 1435: In Fig. 5, it seems like none of the methods converge anywhere near close to the ground truth, with the lowest error still being larger than $0.01$ and already having leveled out. Why do seemingly all methods perform badly on this example?

Comments/suggestions:

- S1: The exposition would benefit tremendously by using boldface small letters for vectors to distinguish them from scalars, and boldface capital letters for matrices.
- S2: The re-statement of the theoretical results using a different label in the Appendix is very confusing (lines 923-927). Simply use the same names/labels in both the main text and the appendix for the same result
- S3: Hyperbolic aware minimization implies that the proposed optimization step somehow utilizes the hyperbolicity of the loss landscape to improve generalization, but the proposed optimization step imitates increased hyperbolicity through point-wise overparameterization and then mitigates the resulting small inverse metric problem at the origin. Maybe the choice of the method's name is not optimal. Also, I think it should have been “hyperbolicity aware” and not “hyperbolic aware”.
- S4: Lines 162-171: This paragraph misrepresents the scope of alternating or two-step optimization schemes, which have a long-standing history in optimization for ML. Important examples are proximal methods, soft thresholding, ADMM, alternating least squares/ridge, expectation maximization, etc. The paragraph should either give an appropriate overview or restrict itself to a certain scope of optimizers and modifications thereof. 
- S5: Consistently use $\theta_{\text{init}}$ instead of variants like $\theta_{init}$ or $\theta_{in}$
- S6: Theorem 3.1 (line 211): The assumption on the initialization does not make sense to me, I think it should read $\sqrt{|\theta_0|}$ since theta is real-valued.  
- S7: Clarify that Fig. 1 only holds for $\gamma=0$ where the vanishing inverse metric problem occurs.	
- S8: lines 39-40: The methods by Kusupati et al. (2020) and Kolb et al. (2025) do not involve iterative parameter pruning as claimed
- S9: line 199: Why is the condition $|w_0| \leq m_0$ required? More explanation is needed.
- S10: line 271: Many analyses are performed for $\beta=0$, so shouldn’t the algorithm line read $\geq 0$?
- S11: line 291: Formulations like “mild sparsity” should be avoided and made more precise and rigorous.
- S12: line 298: Why does this equation not contain the previously used weight decay parameter $\beta$ anymore? Especially since the sentence in line 300 states that the inverse metric can change due to regularization.
- S13: lines 315-316: The references are mixed up, Theorems A.7 and A.6 are applied, which are Theorems A.3 and 4.14 in the stated references.
- S14: line 342: Sweeping statements like the one on convergence speed should be avoided if they are not made precise.
- S15: lines 355-356: The target and input vectors should have $d$ not $n$ entries as defined in 353. Also the input should be a matrix, since each $z_i$ already is a vector.
- S16: line 431: it is claimed that the optimal $\alpha, \beta$ are stable across the two tasks. But the best setting for CIFAR100 (Fig. 8) is $\alpha=100, \beta=3.2 \times 10^{-2}$, whose closest neighbor in the ImageNet grid (Fig. 9) actually yields the worst performance. Looking at other experiments, the optimal values of $\alpha$ seem to be unstable, ranging from as small as 1 to 200, or even -200.
- S17: line 870: $\lambda$ is undefined, I think this should be $\Lambda$.
- S18: line 954: The hyperlink in Eq. (3) actually references Eq. (HYP), not Eq. (3).
- S19: line 1018: ‘indicate’ should be ‘indicates’
- S20: lines 1035-1036: Sentence unclear to me, writing could be improved.
- S21: line 162: Shouldn’t $\delta$ be a function of $\theta_{j+1/2}$ and $\theta_{j}$, and the conditions in the parenthesis depend on both their signs?
- S22: line 1044: $\hat{\theta}$ does not appear in Equation 8.
- S23: lines 1088-1094: I can’t follow the writing. The exposition here could be improved.
- S24: line 1246: ‘vain’ should be ‘vein’.
- S25: line 1249: $\mathcal{N}$ should be used consistently for a Gaussian. Moreover, previously, the main text used $d$ to denote the number of samples, not $k$.
- S26: line 1252: I think there are incorrect indices in the equation (see Q10).
- S27: lines 1260-1262: The text references an undefined balance equation and stops in the middle. 
- S28: line 1268: Text references undefined ‘balance equation’.
- S29: line 1286-1287: I think the statement might be incorrect. If $a_{in}=w_{in}>0$, then it can’t be true that $a_{in}<-w_{in}$, which is the requirement for belonging to $\Gamma_0$. Also $\bar{\Gamma}_{0}$ is undefined.
- S30: line 1357: Why is suddenly Frobenius decay applied to the product weight instead of weight decay separately for $m$ and $w$ as in lines 192-193? This would not induce sparsity, only separately applying weight decay induces sparsity.


**Sources**:

Ziyin and Wang (ICML, 2023): https://proceedings.mlr.press/v202/ziyin23a/ziyin23a.pdf

Kolb et al. (ICLR, 2025): https://openreview.net/pdf?id=vNdOHr7mn5

Kusupati et al. (ICML, 2020): https://arxiv.org/pdf/2002.03231
soundness: 3
strengths: The submission’s main strength lies in its conceptual originality and theoretical depth. It proposes a, to the best of my knowledge, novel and interesting idea of replacing the pointwise product parameterization with a direct modification of the training routine that reproduces a similar optimization geometry and implicit bias on the original weights. This reframing of the $m \odot w$ parameterization as an alternating optimization scheme is both interesting and practically relevant, as it retains the sparsity-promoting benefits of overparameterized training without increasing model size, computational cost, or running into small inverse metric issues. The work also provides a clear and well-structured overview of existing sparsification methods, placing HAM within a broad literature that includes pruning at initialization, dense-to-sparse, and dynamic sparsity training. By focusing on the gradient dynamics and implicit bias of reparameterized models, the paper contributes to an important and active area of research that links optimization geometry to generalization (and sparsity), advancing the theoretical understanding of how optimizer design influences learning behavior and generalization. The analysis is mathematically rigorous, building on (Riemannian) gradient flow and mirror descent concepts, and derives the implicit regularization induced by HAM. 
The empirical evaluation is performed on a large variety of tasks ranging from the simplest linear regression models to common vision benchmarks, LLM finetuning, and graph learning tasks. Overall, the paper nicely balances theoretical results with experimental evaluations, offering an innovative and well-substantiated contribution to the study of how geometry induces implicit bias for optimization in deep learning.
summary: This paper introduces Hyperbolic Aware Minimization (HAM), an alternating optimization step that combines the beneficial sparsity-promoting implicit bias of Hadamard $m \odot w$ overparameterizations in a lightweight two-step minimization routine that aims to avoid the optimization and computational difficulties of direct $m \odot w$ parameterization. Classical $m \odot w$ parameterizations, in which each parameter is represented as a product of two factors, are known to be able to improve generalization and induce sparsity regularization, but suffer from a small inverse metric near zero that slows down small weights and impedes often important sign changes. 
HAM aims to overcome this issue by alternating a standard optimizer update like SGD/Adam with a hyperbolic mirror step whose inverse metric is lower bounded by 1 at the origin and thus promotes flexible learning dynamics even for parameters close to zero, while maintaining a similar implicit bias to the $m \odot w$ parameterization. This mechanism facilitates sign plasticity, allowing parameters to adapt their sign in response to changing gradients, and introduces an implicit bias that interpolates between $L_2$ and $L_1$ regularization, promoting ‘mild sparsity’. Theoretical analyses based on Riemannian gradient flow and connections to exponential gradient descent as an approximation show that HAM circumvents the small-metric problem and that it can be shown to converge using standard methods and assumptions in the literature. Empirical results on ImageNet and CIFAR benchmarks demonstrate consistent generalization improvements in both dense and sparse regimes, whereas further experiments on a range of tasks showcase the broad use of HAM from linear regression to LLMs and graph neural networks. Combined with other sparse learning approaches, HAM is demonstrated to enhance popular methods like  AC/DC, RiGL, and STR, and works complementarily with Sharpness Aware Minimization. Overall, the work positions HAM as a geometry-utilizing optimization method that reproduces the positive effects of overparameterized training, like its implicit bias, but ensures stable sign dynamics, while remaining computationally efficient and broadly applicable.
weaknesses: Overall, I think the submission proposes an interesting and novel method that makes a neat paper contribution. However, some major weaknesses prohibit recommending acceptance in the submission’s current state. However, I think the majority of my concerns could be addressed in the rebuttal, which would change my rating if addressed adequately. In the following, the weaknesses are presented point-by-point.

- W1: The paper’s main weakness is its exposition. This pertains to several aspects.

- W1a.i  quality of exposition: In parts, the submission reads rather “unfinished”, including awkward theorem referencing, using inconsistent notation, referencing undefined equations, or unfinished sentences. Some proofs were poorly written and hard to follow due to a lack of contextualization (e.g., Lemma E.1, Theorem E.2, Theorem B.6), missing definitions, or a lack of explanations of why certain steps are performed. Actionable examples are in a separate suggestions list below. 

- W1a.ii clarity of exposition: I got the feeling that some statements or sections implicitly only cover a certain setting (like $\beta=0$) without making this clear enough. It was not always obvious to me, e.g., if a statement is made about discrete vs continuous time, $\beta=0$ vs $\beta \neq 0$ (e.g., line 298), if weight decay on $m$ and $w$  or Frobenius (product) weight decay was used for the spred/$m \odot w$ experiments and analyses, or whether GD+HYP/GD+HYP* was used.


- W1a.iii precision of exposition: the submission contains many statements that come off a bit too bold or sweeping, or make imprecise claims (e.g., a potential exaggeration of the small inverse metric problem for $m \odot w$, or claiming that Frobenius weight decay is required for $m \odot w$ to induce sparsity (line 1360); whereas,  to my knowledge, sparsity using this parameterization can be achieved either through implicit regularization, or by using weight decay on $m$ and $w$ separately, bur not with the mentioned Frobenius decay of the product). Actionable details are in the suggestions list below.


- W1b: The submission seems a bit “all over the place” with the use cases of HAM. It remains unclear to me if its proposed use is mainly a general-purpose optimization step improving generalization, a method to induce and control implicit biases without direct overparameterization, a way to enhance existing pruning pipelines, or a sparse learning method on its own. 


- W1c: I am not entirely convinced by one of the main motivating arguments for HAM, namely, to mitigate the small inverse metric problem caused by direct $m \odot w$ parameterization, which causes parameters to get stuck/decelerate around 0 while crossing signs. Fig. 1 displays the authors’ argument nicely. However, the inverse metric only vanishes for $\gamma=0$, which only happens for balanced initializations of $m,w$, i.e., $m_0^2-w_0^2=0$ (or also for $t \to \infty$ if $\beta>0$ by Eq. (2)). By choosing a random/imbalanced initialization for $m,w$, as used by Ziyin and Wang (2023) or Kolb et al. (2025), sign crossings are unproblematic since $\gamma_0>0$. Since both works report no issues of achieving competitive performances directly using $m \odot w$ in both linear or DNN models, it makes me wonder how relevant the sign flip/inverse metric problem for the $m \odot w$ parameterization is in practice when not using the balanced initialization that was avoided in previous works? Ziyin and Wang (2023) even report “Our initial experiments find no significant difference between making the norm balanced or not at initialization”, contradicting the small inverse metric problem, which would be intensified by using a balanced initialization. Furthermore, the submission derives the inverse metric as $1+\alpha |\theta|$ when $\beta=0$, being lower bound by 1. Similarly, the inverse metric for $m \odot w$ with $\beta=0$  is $\sqrt{\theta^2+\gamma^2}$, reducing to $|4*(m_0^2-w_0^2)|$ at the origin, also serving as a positive lower bound for imbalanced initializations. What is the advantage of HAM over imbalanced initializations then? 

- W2: The selection of hyperparameters is not discussed for the main experiments; they are merely listed in the Appendix. Without further explanation/justification of their selection, this significantly weakens the empirical comparisons, since the results can only be assumed to hold for some arbitrary hyperparameter setting (which could potentially be biased to favor HAM). While extensive hyperparameter tuning for each method is infeasible for large-scale experiments, typically, some established set of hyperparameters, e.g., as found in related benchmark papers, is used to establish some form of comparability. This weakness is aggravated by the observation that the performances of their comparison methods sometimes differ strongly from the reported values in the original works (e.g., spred for ResNet50 on ImageNet at 80% sparsity is reported as >77% in Fig. 5 of Ziyin and Wang (2023), while the submission reports only 72.6% in Table 4; similarly STR for ResNet50 on ImageNet at 90% sparsity is reported as 74.31% (Table 1 of Kusupati et al, 2020), while the submission reports only 68.36% for the same setting in Table 10).


**Sources**:

Ziyin and Wang (ICML, 2023): https://proceedings.mlr.press/v202/ziyin23a/ziyin23a.pdf

Kolb et al. (ICLR, 2025): https://openreview.net/pdf?id=vNdOHr7mn5

Kusupati et al. (ICML, 2020): https://arxiv.org/pdf/2002.03231


Decision: Accept (Poster)