Understanding and Improving Hyperbolic Deep Reinforcement Learning
Authors: Timo Klein
Thomas Lang
Andrii Shkabrii
Alexander Sturm
Kevin Sidak
Lukas Miklautz
Claudia Plant
Yllka Velaj
Sebastian Tschiatschek
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
representation learning
hyperbolic space
hyperbolic deep learning
Primary area: reinforcement learning
TL;DR: We analyze training issues in hyperbolic deep reinforcement learning and address them, leading to improved performance.

Abstract: The exponential volume growth of hyperbolic geometry can embed the hierarchical relationships between states in reinforcement learning (RL) with far less distortion than Euclidean space. However, hyperbolic deep RL faces severe optimization challenges, and formal analysis of why optimization fails is lacking. We identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincaré Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce Hyper++, a new hyperbolic deep RL agent that consists of three components: (1) feature regularization guaranteeing bounded norms while avoiding the curse of dimensionality from clipping; (2) a categorical value loss for stable critic training; and (3) a more optimization-friendly formulation of hyperbolic network layers. On ProcGen, we show that Hyper++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, Hyper++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at https://github.com/Probabilistic-and-Interactive-ML/hyper-rl.

Published as a conference paper at ICLR 2026
UNDERSTANDING AND IMPROVING HYPERBOLIC DEEP REINFORCEMENT LEARNING
Timo Klein*.1.2 Thomas Lang*.1., Andrii Shkabrii'.2 Alexander Sturm Kevin Sidak Lukas Miklautz', Claudia Plant Yilka Velajt.1, Sebastian Vebastin Tschiatschek 1.4
Faculty of Computer Science University of Vienna, Vienna, Austria
2 UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria
3 Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry,
Martinsried, Germany
4 ds:Uniiie, University of Vienna, Vienna, Austria
* Joint first authors
fi irstnname
ABSTRACT
The exponential volume growth of hyperbolic geometry can embed the hierarchical relationships between states reinforcement learning (RL) with far less distortion than Euclidean space. However, hyperbolic deep RL faces severe optimization challenges, and formal analysis of why optimization fails is lacking. We identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincaré Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce HYPER++, a new hyperbolic deep RL agent that consists of three components: (i) feature regulariza- tion guarantecing bounded norms while avoiding the curse of dimensionality from clipping; (ii) categorical value loss for stable critic training; and (iii) more opti- mizzation-freendly formulation of hyperbolic network layers. On ProcGen, we show that +YPER++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, HYPER++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at
INTRODUCTION
Consider a chess agent evaluating next move: each action branches into exponentially many future states, cre- ating a vast tree of possibilities. This same structure de- fines common reinforcement learning (RL) benchmarks like
Rel Improvment (N)
+52.3%
ProcGen BIGFISH (Cobbe et al. 2020), where an agent grows by eating smaller fish following inreversible hierar- chy. More generally, sequential decision-making produces inherently hierarchical data: each state branches into mul- tiple potential next states, forming tree-like structures that grow exponentially with depth. In contrast, Euclidean vol- ume grows only polynomially relative to its radius, resulting in fundamental geometric mismatch between the expo- nential branching of decision processes and the polynomial capacity of Euclidean embedding spaces. This forces an agent's 's representation to severely distort hierarchical rela- tionships, a structural limitation that may contribute to deep RL's notorious data inefficiency (Sarkar 2011 Gromov 1987)
-6.0%
Normalized Proccen test ScorD
Hyper+S-RYM
19jor
Euclisean
Hyper++ (Ours)
Figure 1: Baseline improvement ProcGen. We compare mean test re- wards for agent (HYPER++), Fu- clidean agent, and an unregularized hy- perbolic agent (Hyper) with Cetin et al. (2023) 's agent (Hyper+S-RYM).
Hyperbolic geometry offers natural solution to these limitations. Its exponential volume growth aligns with hierarchical structures, enabling efficient, low-distortion embeddings of trees. While

---

Published as a conference paper at ICLR 2026
applications of hyperbolic geometry in deep learning have achieved strong results in classifica- tion Gannaa et al. 20188) unsupervised representation learning (Mathieu et al., 2019), deep metric
learning (Ermolov et al. 2022). and image-text aligmment (Pal al. [025], optimization instabilitiess
have limited broader adoption (Guo et al. 2022 Mishne et al.]| 2023). This is particularly evident in
RI., where nonstationarity amplifies gradient instability (Cetin al. 2023).
We study these optimization failures in proximal policy optimization (PPO) (Schulman et al. 2017)
agents using hybrid Euclidean- auclidean-hyperbolie encoders, commonly used architecture in deep RL (Cetin et al. 2023; Salemohamed et al. 2023). Through formal gradient analysis, we find that growing embedding norms destabilize training in both the Poincaré Ball and Hyperboloid models, causing trust-region violations despite PPO's clipping mechanism. Existing stabilization techniques, such as SpoctralNorm, are insufficient as they cannot mitigate gradient pathologies without severely limiting network capacity.
HYPER++ addresses these failures with three targeted components. RMSNorm (Zhang Sennrich 2019) combined with novel Iearned scaling layer bounds embedding norms without sacrificing capacity - eliminating SpectralNorm's stability-capacciy trade-off. Switching to the Hyperboloid model removes instabilities inherent to the Poincaré ball model at their source, preventing large gradients from propagating via the chain rule. Finally, we replace MSE regression with a categorical value loss, aligning the critic's output with the hyperplane-distance geometry of hyperbolic multi- nomial logistic regression. This stabilizes critic learning under nonstationary targets. Compared to prior hyperbolic agents, HYPER++ achieves better performance, is faster, and more general: improves test return by 52% (PPO+ProcGen), reduces forward pass time by 30%. and performance gains transfer to Double DQN (Atari-5) and Phasic Policy Gradient (Cobbe al. 2021) (ProcGen).
Our Key Contributions
1. Characterization of training issues. For both the Poincaré Ball and Hyperboloid, we formally analyze key operations and link them to training instability in deep RL.
2. Principled regularization. We study the weaknesses of current approaches and propose improvements rooted in our insights into hyperbolic deep RL training.
3. HYPER++, a strong and general hyperbolic agent. We combine RMSNorm with a novel scaling layer, the hyperboloid model, and a categorical value loss.
BACKGROUND
This section first reviews Markov decision processes (MDPs) and the PPO optimization procedure (Section 2.1) then presents the mathematical foundations of hyperbolic representation learning in Section 2.2 A more thorough overview of the Poincaré Ball and Hyperboloid models can be found in Ganea et al. (2018); (Shimizu et al. [202]] Bdeir et al. (2024).
2.1 REINFORCHMENT LEARNING
We formalize RL discrete MDP M (S.A,P,R,.)) with state space S and action space A. At each time step t. the agent observes a state s E S and selects an action a E AA with its policy
:: S + [0, 1]1-1. The environment generates a reward via its reward function R: S x A + R
and transitions to the next state according to the transition kernel P: S S x A [0, 1]. The
agent maximizes discounted future rewards [T() = Ez 1E 7'r( a4) T. where E [0. 1) is
discount factor determining how much the agent values future rewards (Sutton & Barto 2018).
PPO Proximal Policy Optimization (PPO) (Schulman et al. 2017 is an actor-critic algorithm directly maximizing cumulative reward via gradient ascent on surrogate objective. It replaces the hard trust-region constraint of Rrgsto Policy Optimization (TRPO) (Schulman et al. 2015) with the clipped objective
CLIP (0) = E,, min(r.(0)A.. clamp(rr(0),
(I)
ro(a)]ss) where r(()) Tega (8) are importance sampling ratios policies parameterized by 0, and É, is the empirical mean with respect the samples generated in episode The min-clamping in Equation
2

---

Published as a conference paper at ICLR 2026
truncates the incentive to move probability ratios beyond [1 - c, + c]. acting as unconstrained proxy for TRPO's s KL-divergence trust region (see Appendix B.1.
2.2 HYPERBOLIC REPRESENTATION LEARNING
Hyperbolic Geometry
In this work, we employ two common models of hyperbolic space: the Poincaré Ball and the Hyperboloid. The two isometrically equivalent cistance-preservingg models d-dimensional comply-connected Riemannian submanifolds (M, g) with constant negative sectional curvature -c (see Figure 4. with с Rzo.
Poincaré Ball The d-dimensional Poincaré Ball is defined as the Riemannian submanifold (P% Pe:) with P ( E RR: |a 1 ] Its Riemannian metric Dp. is given by the collection of inner products (u, v) T.Pe x T pd + R, (u, v) + X (u, v) that smoothly varies between tangent spaces TaIR with base points E IPd That is, the Poincaré Ball is conformal (angle-
preserving) to the Euclidean space with conformal factor X; = --<||=]F-
Hyperboloid The d-dimensional Hyperboloid, often called Lorentz manifold, is de- fined as the forward sheet (rag) of a two-sheeted Hyperboloid, where Hi (so...., ..)) E IRd+1; (x. x) Pn > 0} and (x. x) = -=0 + xf +... + z is the Minkowski inner product. It is endowed with the Riemannian metric Iug that arises when restrict-
ing the Minkowski inner product to the tangent spaces Hg ic. (u, v) : TH x Ty + R. (u, v) + (u, v) In this work, we frequently refer to the first component zo of H as time component and to the other components Iid as space component.
Hyperbolic Encoding In experiments, mapping Euclidean vectors v E Rd to the tangent space at the manifold's origin 0 followed by applying the exponential map at the origin exPo- to project it onto the manifold M.. This process can
retrieve hyperbolic latent representations by first
be summarized Rdd S ToM expo, .M. The exponential map at the origin expo ToM M maps vectors v E ToM to the manifold M such that the curve L E [o, 1] expo(Iv) is geodesic (shortest path) joining the manifold's origin Ô and expo(r). The specific mapping functions are:
Poincaré Ball: The origin Ô is the Euclidean origin O, i.c. Φ is the identity function and the tanb( (ellll))
exponential map at the origin is expo -
Vellol
Hyperboloid: The origin
(4/ve, 0, ...., 0). The map d projects a Euclidean vector v Fd onto the tangent space Toll! = {o gd+1, (v, (),0)c- 0} by setting its first coordinate to zero, i.e. v + (0, The exponential map the origin is expp cosh (Vc(v,u)) Ô sinh (Vo(u, Velu,ote
Hyperbolic Multinomial Logistic Regression For the policy and value function of our PPO agent,
we compute the Multinomial Logistic Regression (MLR) (Lebanon & Lafferty 2004 Shimizu et al. 2021 Bdeir al. 2024) in hyperbolic space. The method computes the probability p k x) of an input E M ~ Rdd belonging to specific class i E (1.....K))
p ls x) x exp(vz..ra
(x)), (x))), UzL.re (x) || ( du(æ H...).)
(2)
Here, exp(u (z)) is the logit for class l: and Uza.ra r.. (a)) the signed distance to the margin hyper- plane Henrr with learnable parameters 2µ Rd. r& E R specifying the normal and shift vector pk. respectively. The specific definitions for these parameters and the hyperplane itself depend on the hyperbolic model. We expand this further in Appendix B.3
B.3
DIAGNOSING ISSUES WITH HYPERBOLIC PPO AGENTS
In this section, we analyze training issues of hyperbolic PPO agents (Section We link these issues to the gradients of common hybrid neural network architectures as used in Cetin et al. (2023) in Sections 3.2 3.3 and 3.4. These networks consist of shared Fuclidean encoder with only the
3

---

Published as a conference paper at ICLR 2026
Hyper++ (Ours)
Hyper S-RYM
Hyper
Euclidean
2.0
Update KL Diverr
9:20
0.00
0.18
0.25
0.00
80.00
(a)
(b)
(c)
(d)
Conformal Pactor
1.50
Grad. Norm
0.35
Lastrc
Tast
0.00
(c)
(f)
(h)
Figure 2: PPO training metrics. Unregularized agents (Hyper, Euclidean) lose entropy and show unstable updates (higher update KI. and clip fraction), with lower returns and larger gradients (BigFish). Hyper's conformal factor explodes. In contrast, HYPER++ uses the Hyperboloid, which has no conformal factor. Metrics are means over six seeds with one standard deviation.
last layers for the actor and the critic being hyperbolic (ef. Figure in in the Appendix). Appen- dices B.3 B.3. B.3.2 contain additional background on the MI. formulations of the Poincaré Ball and the Hyperboloid.
3.1 PPO OPTIMIZATION
PPO's clipped surrogate objective (Eq. restricts the per-sample importance sampling ratios and acts as heuristic trust region (Schulman et al. 2017). A high clipping fraction indicates many samples are the trust region boundary. Crucially, PPO constrains ratios only on the sampled states in batch. Gradient steps leading to large policy changes on unseen states remain unconstrained, so the heuristic trust region can fail. This cross-state interference can produce large unintended policy shifts beyond the sampled states in the batch (Moalla et al.) (2024).
Figure [] key training metrics for hyperbolic PPO training in the BIGFISH environment (top row). As noted by Cetin al. (2023), unregularized hyperbolic PPO is prone early entropy collapse in Figure 2a| This coincides with a rapid rise in entropy variance across batch states, producing large policy updates that potentially interfere (Figure 2hp. Figures [2] and [d confirm: unregularized agents experience larger update KL-divergence and more trust-region violations. Cetin et al. (2023) propose to mitigate this with S-RYM, a combination of Euclidean embeddings scaled by !/vd and SpectralNorm to bound the encoder's 's Lipschitz constant (Hyper+S-RYM in Fig. ( In comparison, our method (Section 4 achieves lower update KL and markedly less clipping while avoiding the overhead of SpectralNorm and instabilities from the conformal factor.
3.2 GRADIENT ANALYSIS PRELIMINARIES
To explain the trust-region instability of the hyperbolic agent, we follow Cetin et al. (2023) and analyze the gradients of the last encoder layer (Fig. 2g) 2howwss that the conformal factor of the Poincaré Ball X I-e क| is a key driver for inducing instability. In the following, we derive closed-form, curvature-
2
gradients for core hyperbolic layers and maps to study optimization failure points, extending Guo et al. (2022); Mishne et al. (002) with new expressions for PPO. Below, (
present the gradient with respect to the last Euclidean layer weights E for generic loss L.
ØL
aL (2H) OwH
dah
dxg
OWE (vH,r(HH)
Oog
OWE
where Usrr denotes the score function of any hyperbolic multinomial regression (MLR) layer, my are the Euclidean embeddings from the encoder, and zи expo (zg) are the embeddings represented as tangent vectors mapped to hyperbolic space. For the Poincaré MLR layer used by Cetin et al. (2023), Guo et al. (002)) have shown that backpropagating through the exponential map yields vanishing
4

---

Published as a conference paper at ICLR 2026
gradients the boundary because the Riemannian gradient scales with the inverse conformal factor gradient:
4CHH
Væn
(1 (1-68 cl<
3.3 GRADIENT ANALYSIS FOR HYPERBOLIC NETWORK++ MIR
The derivative (Appendix A.2) of the HNN++ MLR formulation (Shimizu et al. 2021) with respect to its input 2h is:
2
1
dan 2.
HNNH (an)
Ve
v +++ (()1)*
dah F(æn)
where
(5)
4 4C0H (- sinh(22/r) + vecosh(22/rr
( xn))
0 F(mn)= Orh
2 (2coosl(2Vor)
2
c|
(1- |2)2
where = =/11|1| the (normalized) Euclidean weight vector of the layer and is scalar bias term. The problematic term is the denominator (1- c|æ 3)) in (F(z)) (0z))/0z stemming from Equation 4 it causes gradient explosion near the Poincaré Ball boundary as | - 1/ Clipping Xa is undesirable because HNN++ MLR logits depend on X and alter the hyperbolic geometry by shifting decision boundaries, leading to performance plateaus. Hence, while HNN++ removes vver-parameterization (Shimizu et al. 2021), it does not, by itself, resolve PPO training instabilities.
Next, we analyze the Jacobian of the Poincaré Ball exponential map Sen similar to Guo et al. (Appendix A.I):
2022
deg
Oæn expo(xx)= tanh(vel 0æg dog
(sech" (
(Vellae/)) |
tanh +Ell)
Tg
I+
Vol=g
|
Although the exponential map Jacobian decays like O[
the directional term (second sum-
mand) is highly sensitive growing ||> Figures 2g and 2h show how volatile layer-wise gradients can get during training without proper handling. Cettn et al. (2023'ss 's S-RYM scaling factor LE + 2p/v keeps | moderate, preventing expi (Zx) /0z from destabilizing the learning signal fed back the encoder (Eq, (3 while reducing directional variability. Hence, regularizing Euclidean embeddings before the hyperbolic layers is a necessity for stable hyperbolic PPO agents.
3.4 GRADIENT ANALYSIS FOR HYPERBOLOID MLR
Prior work establishes that the Hyperboloid trains more stably than the Poincaré Ball (Mettes et al. 2024; Mishne et al., 2023 Bdeir et al. 2024) for two reasons. First, the Hyperboloid MLR score (Eq. 266 contains no conformal factor as it is not conformal Euclidean space. Second, neither multiplies nor divides by the Euclidean feature norm. As result, its gradients avoid the instabilities of the Poincaré Ball. However, we will show in the following that the Jacobian dæu 8 of the Hyperboloid's exponential may still destabilize training. We denote
Zv (v)
op æg]
I To as
the Euclidean embeddings mapped into the tangent space of the Hyperboloid (cf. Section 2):
dah Ov
0
sinh( (ell |
mg
sinb (i(bellll Ia Vellawll coshl (Gllaalll Villag]
sinnlll)ssmmhyyyi (ella))))))))
aga
Equation is (1 + ())(1 + d) matrix, where the first column is zero. For large |æ sinh( (e [ellee)) and cosh c [xll grow exponentially, i..., faster rate than /cll=ll Thus, the Hyperboloid expo- nential map can destabilize gradients when Euclidean Suctideanfeetuuee norms grow, requiring regularization of ||æ
Summarizing the findings in this section, we arrive at more nuanced understanding of the training issues of hyperbolic deep RL agents: Policy breakdown and large-norm gradients in the encoder are function of the hyperbolic layers used in the actor and the critic. The conformal factor, in particular, a source of numerical instability in Riemannian optimization methods (Guo et al. 2022 Mishne et al.) 2023). This numerical instability gets exacerbated by noisy gradients in actor-critie training, particularly from the critic's ' side (Sutton & Barto] 2018 Nauman et al. 2024a). In the next section, will show how method HYPER++ deals with these issues.
5

---

Published as a conference paper at ICLR 2026
4 STABILIZING HYPERBOLIC DEEP RL
In this part, we establish the components of our agent HYPER++: Section 4.1 proposes RMM-
SNorm (Zhang & Sennrich, 2019 as an alternative to SpectralNorm. Section 4.2 introduces novel feature scaling layer. Section
discusses how these components relate to the Hyperboloid. Beyond these design choices, we use a categorical loss stabilize critic gradients (Imani & White 2018; Farebrother et al.) 2024) and to resolve an architectural mismatch in hyperbolic value learning. While Euclidean linear layers naturally support MSE regression over continuous values, hyperbolic MLR layers output classfication-oriented hyperplane distances, making the categorical loss over dis- crete bins a better geometrical fit. Collectively, our components target complementary sources of instability in Equation 3 the categorical loss stabilizes the loss derivative (first term), Hyperboloid MLR stabilizes the hyperbolic layer Jacobian (second term), and RMSNorm with feature scaling stabilizes the Jacobian of the exponential map (third term). Figure illustrates the underlying hybrid network architecture (Guo et al. 2022 Cetin et al. 2023) analyzed in the following.
4.1 REGULARIZATION
Here, we study how SpectralNorm (Miyato et al. 2018) affects the Euclidean embeddings produced by the encoder (cf. Figure IT). To this end, consider Lemma 4.1 which provides a bound on the norm of the embeddings computed by a single layer, depending on the input norm:
Lemma 4.1. Let x R", W E Rdx= and b E Rd. Then, for any function f Rd gd with Lipschitz constant L, it holds that
Ilf(W + b)ll & Ilf(0))la + L
+ L/1b122
In particular, for ReLU activation functions and any normalized weight matrix W, we have
|RLUU(W + b)]l2 = ||ح||| 116112
(8)
shows that for multi-layer encoders such the used by Cetin al. (2023), applying SpeetralNorm only to the last (linear) layer of the encoder is not sufficient to prevent the Euclidean embedding norms from growing via the preceding layers. To tangibly affect these norms, Spec- tralNorm must be applied to every layer of the encoder (Cetin et al.] 2023). This constrains the Lipschitz constant of all layers and reduces expressivity by globally enforeing smoothness (Rosca) et al. 2020 Cetin et al. 2023). Additionally, SpectralNorm incurs computational overhead from the power-iteration steps needed at each forward pass.
Lemma 4.1
Ideally, we want to use regularization via spectral normalization only where needed and such that we can guarantee stable training, without limiting the expressivity of the entire Euclidean encoder. Proposition 4.2 shows that applying RMSNorm (Zhang & Sennrich] 2019) before the activation of the encoder's last linear layer achieves stability without overly restricting its representational capacity (if the other layers not regularized, their expressivity is not limited).
Proposition 4.2. Let æ E Id and f : Rd + Rd with Lipschitz constant L. Then, for ā = Va (RMS(())) holds that:
\expo(s) < <2 2cohh (ve (1100)1)++)) +
||*||2 |f(0)]l2+ + L,
(9)
Proposition ensures stable hyperbolic operations for broad class of activation functions. For common 1- LLpischitz activations such as TanH and ReLU, the bounds reduce to ||ě|| < 1 and ||expa(*)]| Ve tanh(v/). Unlike SpectralNorm, which constrains every encoder layer, we only require applying RMSNorm to the pre-activation output embeddings the final linear layer. This retains the expressivity of each encoder layer. We use RMSNorm (Zhang & Sennrich 2019) rather than LayerNorm (Ba et al. 2016) because we do not want the meen-centeringg in LayerNorm to distort the hierarchical structure of the hyperbolic embeddings. Additionally, RMSNorm brings three further advantages: it smoothes gradients, prevents dead ReLU or saturated TanH units (Zhang & Sennrich 22099] Xu al. 2019 Lyle al. 2024), , and supports arbitrary embedding dimensions d, since the bound in Proposition 4.2 dimension--ndeeenddee for activation functions with fixed point 0.

---

Published as a conference paper at ICLR 2026
4.2 LEARNED EUCLIDEAN FEATURE SCALING
Poincaré Ball
Proposition 4.2 guarantees stability by bounding both Eu- clidean embedding norms and the conformal factor. How- ever, this may still affect representational capacity in the hyperbolic layers of the agent. For example, with ReLU as the last encoder layer's activation function and curva- ture e = 1, the bound restricts the Poincaré Ball radius to |æ v 0.76 (see the proof of Proposition 4.2 Since the volume of d-ball scales Va(r) -(=-1) r rd,
RMS+
Learned
Figure 3: Learned scaling effect.
even a modest restriction of the radius causes an exponen-
tial loss of available volume in d. To mitigate this, we rescale the Euclidean tangent embeddings obtained after application of Proposition 4.2 dg by a learnable scalar Ee:
atanh(a) VC
rescale F
Pmax (Ex) dE,
Pinxx
(10)
where denotes the sigmoid function. By choosing this particular form for Pmax: we have that Setting
0.95 (and = 1) expands the usable ball radius from 0.76 to 0.95, i.e., volume gain of (0.95 (0.95/0.76) For d = 32, this is approximately 1.2 x 10 more volume while still preventing the explosion of the conformal factor according to Proposition 4.2 Figure illustrates the effect in 2D.
||expo (# )ll S a/ a/ve since tanh(ye
4.3 HYPERBOLOID MODEL
Section (.4tioon..44s0 3.4 shows that the Hyperboloid avoids con- formal factor instabilities and is therefore more ro- bust against large norms. Yet. operations can be- come ill-conditioned far from the origin, i.e., when the sheet approaches the asymptotic null cone, and the Jacobian of the exponential map in Equation 6 gets more sensitive to large Fuelidean norms. Since the Poincaré Ball and the Hyperboloid are isomet- ric (Fig. 4 models, our stabilization strategy trans- fers: instead of capping the Poincaré Ball radius, we propose to apply RMSNorm and feature scaling before the last Euclidean activation bound the Hy- perboloid through its time component To- Corollary [43 formalizes this insight by combining Proposition 4.2 with the Poincaré Ball-Hyypebolood isometry (Chami et al. 2021 Mishne al. 2023).
H
Isometry =
Learned boundary
ppn
Figure 4: Isometry between Poincaré Ball and Hyperboloid.
Corollary 4.3. Let äg E 1n be point regularized by RMSNorm with learnable scaling, and = expo (xpn($E) E I=. Then, the maximum value of the time component zo of that point on the Hyperboloid
1 c|| 12 20 Ve(1 - <|=
1 1+ tanh (vel(g)))
Ve (1 tanh" 2 (1alaananVellaaaaaaa (vel@nl))
Since the time and space components are dependent (cf. Section 2.2 bounding the maximum norm of amjax also ensures that the space component Fg remains bounded. Therefore, we also apply regularization with RMSNorm and learned scaling when training agents using the Hyperboloid. In Section 5.2 we show that this approach works well empirically. Our proposed HYPER++ architecture visualized in Figure 17 17 (Appendix) and consists of the following components:
7

---

Published as a conference paper at ICLR 2026
Median
IQM
Optimality Gap
Hyper++ (Ours)-
Hyper S-RYM-
Euclidean-
Hyper-
Figure 5: Normalized test rewards on ProcGen for PPO. HYPER+ outperforms baselines for all aggregation methods without increasing variance (as measured by the bootstrap confidence interval). We report median, interquartile mean (IQM), mean, and optimality gap, which is 1 IQM.
+YPER++
HYPER++ tackles optimization issues in hyperbolic deep RL through formal and empirical analysis of training dynamics:
1. RL nonstationarity
Categorical value function.
2. Growing Euclidean feature norms
= RMSNorm + Feature scaling.
3. Conformal factor instability
Hyperboloid model.
EXPERIMENTS
We evaluate HYPER++ on ProcGen (Cobbe et al. 2020) with PPO (Schulman et al., 2017) and PPG (Cobbe al. 2021) Section 5.1 Section 5.2 provides ablation studies for PPO. We test performance with the off-policy algorithm DDQN (van Hasselt et al. (016)) on a subset of Atari games (Bellemare et al. 2015 Towers et al., 2024 Aitchison et al. al.] 2023). Unless stated otherwise,
error bands show one standard deviation. Wall-clock times are reported in Appendix D.
5.1 PROCGEN
Figure 5 shows normalized aggregate test rewards for 25M time steps on all 16 ProcGen environments with PPO. We normalize using random performance as the minimum and either a theoretical or empirically determined maximum (Cobbe et al. (2020). We use the rliable library (Agarwal et al.) 2021 to compute aggregate metries such as the interquartile-mean (IQM) and optimality gap with bootstrap confidence intervals.
Optimality Gap
Hyper++ (Ours).
S-RYM-
Euclidean
0.32
0.66
Figure 6: Normalized test rewards ProcGen for PPG. HYPER++ outperforms both Euclidean and existing hyperbolic agents. Hyper+S-RV S--H----RRR-- is substantially worse than the Euclidean baseline, performing even worse compared to HYPER++ with PPO.
With PPO, HYPER++ outperforms Poincaré agents with and without S-RYM, as well as the Euclidean baseline. Tables 7 and 8 show HYPER++ winning head-to-head vs. Hyper+S-RYM in 8/16 games on the train set and 11 T6 games the test set. Training curves for all runs are in Appendix E.1 We further test the performance of HYPER++ by evaluating with more recent baseline, PPG (Cobbe al. 2021 Figure 6 shows the results: Our method outperforms strong Euclidean baseline in all metrics, whereas the Hyper+S-RYy
agent is substantially worse than Euclidean. Notably, HYPER+ with PPO achieves higher test IQM than Hyper+S-RYN R-M with PPG. We provide full PPG results in Appendix E.3 E.8. summary, our ProcGen experiments with PPO and PPG highlight the strong performance and generality of HYPER++ compared to previous hyperbolic approaches.
8

---

Published as a conference paper at ICLR 2026
5.2 ABLATION STUDIES
Figure 7 presents ablations of HYPER++'s components using test IQM with bootstrapped confidence intervals. We begin with the most critical component: normalization. Removing RMSNorm (Zhang Sennrich 2019) and 1/vd feature scaling causes complete learning failure (-RMSNorm), confirming
the predictions of Proposition 4.2
This failure manifests as large embedding norms and near-zero gradients in the en- coder's final layer (Figure 14). providing empirical support for the theoretical analy- sis in Section 3 and Proposition 4.2 The next most important architectural choice is learned scaling (-Scaling), which we attribute to its synergy with RMSNorm. Among the loss function variants, replacing the categorical HL-Gauss loss (Imani & White, (018) with MSE (+MSE) degrades performance, though not uniformly across all games. This aligns with the findings of Fare- brother et al. (2024), who similarly observe that HI-Gaass does not con- sistently improve performance on all environments. Interestingly, substitut- ing the C51 (Bellemare et al. 2017) distributional loss for HL-Gauss per- forms even worse than MSE. Using Poincaré ball instead of the hyper- boloid model (+Poincaré) leads modest drop in performance, which is expected given their isometry.
0.4
0.3
0.2
0.1
0.0
Hyper+++ (Ours)
+Poincarg
+MSE
-Scaling
Hyper++ (C51)
Hyper++ (SN Penultimate)
RRMSNorm
Hyper++ (SN Full]
Ablation studies ProcGen with Hyperbolic geometry. We report the test interquar- tile mean (IQM) across six seeds with bootstrap confidence intervals. indicates that component is removed from HYPER++, + indicates a component replacing its analog.
Figure 7:
We further validate Lemma by testing SpectralNorm (Miyato al.) 2018) as an alternative to RMSNorm two configurations: applying Spec- tralNorm to the complete Euclidean encoder (HYPER+ (SN Full)) and applying it only to the penultimate layer (HYPER++ (SN Penultimate)). In both cases, the agent fails to learn entirely. This underscores the critical importance of RMSNorm for obtaining the bounded feature norms guaranteed by Proposition 4.2
Finally, Figure 8 isolates the contribution of hy- perbolic representations by evaluating Euclidean agents equipped with HL-Gauss, R.MSNrrm and our full regularization combination. For Fuclidean representations, the HI.-Gauss loss (EuclideanCCutggrrcaa) performs worse than MSE. Adding RMSNorm Euclidean agents improves performance, and equipping Euclidean agents with our full method yields IQM of 0.35, which is slightly better than HYPER++ with the Poincaré ball (IQM=0.34). However, HYPER++ with the Hyperboloid achieves the best overall performance (IQM=0.40). The un- rela-
0.4
0.3
0.1
0.0-
Eudidean (Ours)
Euckdean RMSMorm
Euclidean
Euclidean Cateporical
Figure Ablation studies on ProcGen with Euclidean geometry. We report the test interquar- tile mean (IQM) across six seeds with bootstrap confidence intervals.
derperformance of Euclidean+HL-Gar tive to Fuelidean+MSE indicates that categori- cal losses are particularly well-suited for hyper- bolic agents due to the geometric alignment between the loss and the critic's architecture. Overall, our results demonstrate that hyperbolic representations can benefit deep RL agents, but require an optimization -rrinndlfrryyyy approach to hyperbolic geometry to realize these benefits. We present complete ablation results in Tables and I0 In summary, every ablation underperforms HYPER++ with the Hy Hyperolloid confirming the synergistic interactions between hyperbolic geometry and our method's components.

---

Published as a conference paper at ICLR 2026
Median
IQM
Optimality Gap
Hyper++ (Ours)-
Hyper S-RYM-
Euclidean
1.2
1.2
1.6
Figure 9: Human-normalized performance for DDQN on Atari-S. All agents are trained for 10M steps and five seeds. HYPER++ strongly improves over the basclines.
5.3
ATARI DDQN
We evaluate our algorithm using the value-based off- algorithm DDQN (van Hasselt et al. 2016) (Appendix B.2). We focus on the Atari-5 subset of games (Aitchison et al. (023). which consists of PHOENIX, BATTLEZONE, DOUBLE DUNK, and Q* BERT. This subset has been shown to be the most predictive of overall performance across all Atari environments (Bellemare al. 2015 Towers al., 2024 We train each agent for 10M steps and five random seeds. Figure shows the final episode rewards achieved by each method. HYPER++ substantially outperforms the baselines across all five games in all metries. Appendix E.
provides the full learning curves for each individual game. We find that performance varies across games: our method achieves its strongest gains on NAMETHISGAME and BERT. On PHOENIX, +YPER++ exhibits strong initial performance but subsequently plateaus, mirroring the behavior of the baseline agents. This plateauing consistent with plasticity loss being a confounding factor on this particular game (Klein et al., 2024 We ablate the choice of Polyak vs. hard target updates on NAMETHISGAME the most representative Atari-5 game (Aitchison et al. 2023). Figure 17 (Appendix) shows only minor performance differences, suggesting robustness this design choice.
6 RELATED WORK
Hyperbolic deep learning has progressed quickly from early hyperbolic neural networks and Riemannian optimization (Ganea et al. 2018 [ééiignul Ganea 2019). which Cetin et (2023) adapt to RL. Parameter redundancy in Poincare Ball MLR was reduced by Shimizu et al. (2021) Fully
hyperbolic architectures on the Hyperboloid now include transformers and convolutional networks, as well as Hyperboloid MLR layer (Chen et al.) 2022: Bdeir et al. 2024), Mettes et al. (0244 survey this literature from vision perspective. Optimization and numerical stability have been analyzed independently (Mishne et al. 2023 Guo et al. 2022). We study optimization problems of hyperbolic networks within RL and propose a principled solution.
Reinforcement learning. We focus on PPO (Schulman al. 2017) which remains under active study (Andrychowicz et al. [021] Moalla et al.
2024) because of its strong performance. Several works show that regularization can improve deep RL training: with LayerNorm being widely adopted
2016 Lyle et al. 2023 Nauman et al. 2024b Lee al., 2025 Gallici al., 2025). Instead, we regularize our agent using RMSNorm (Zhang
in the deep RL (Henderson et al. 2018 Ba et al.
Sennrich 2019). preventing interference with hyperbolic representations. A separate line of research
stabilizing value function learning via categorical objectives (Bellemare et al.) [007 [Schrittwieser
et al. 2020 Imani & White 2018 Farebrother et al. 2024), which we extend to hyperbolic PPO.
7 LIMITATIONS AND CONCLUSION
Limitations and Future Work. Our analysis takes an optimization-centric view, focusing on training dynamics and the question of how hyperbolic deep RL learns, rather than what structures their representations capture. We also do not address which environments are most suited to hyperbolic representations. Moreover, the interaction between geometric choices and the design of different deep RL. algorithms, remains unexplored. Each of these directions is an exciting avenue for future work. Conclusion. Our work analyzes gradients in the Poincaré Ball and Hyperboloid, linking large-norm embeddings to PPO trust-region breakdowns. Based on these insights, we introduce HYPER++, which combines RMSNorm with learned feature scaling and a categorical value loss to stabilize hyperbolic deep RL. On ProcGen, HYPER+ improves performance and substantially reduces wall-clock time compared to existing hyperbolic PPO agents. Our findings transfer to Atari and DDQN with strong gains, indicating broader applicability beyond PPO.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
Our code is publicly available at https://github.comm proobablistic-an--ntrraaii pendix A contains the derivations for our gradient analysis and proofs for our theoretical results. In Appendix D we state agent architecture, hyperparameters, relevant implementation details, and hardware used. In Appendix C.3 we discuss differences in results to existing works.
Ap-
USAGE OF LARGE LANGUAGE MODELS (LLMS)
During this project, we used LLMs as an assistive tool. In the early stages of our project, we used LLMs for literature search and paper summarization. During the implementation phase, used code assistants to support repetitive coding tasks such as Matplotlib figure generation. For the paper, LLMs were used tool to iterate "Shorten this paragraph." All LLM outputs used in this paper were thoroughly reviewed to ensure accuracy. LLMs were not used for idea generation, experimental design, for proofs. Mathematical expressions were derived independently.
writing. An example case paragraph shortening with
ETHICS STATEMENT
Our work advances the fundamental capabilities hyperbolic deep RL agents and has no direct ethical implications by itself. We cannot rule out that unethical uses could occur in downstream applications because RI. and PPO, particular, are used to train L.L,s, and hyperbolic embeddings are well-suited for text data. However, such uses would require significant extensions and modifications beyond the work submitted here.
ACKNOWLEDGMENTS
This work has been funded in parts by the Vienna Science and Technology Fund (WWTF) (1040037//C We acknowledge EuroHPC JU for awarding the project ID EHPC-DEV- 2025D08-024 access the Luxembourg national supercomputer, MeluXina, and the Spanish su- percomputer, MareNostrum. The authors gratefully acknowledge the LuxProvide and Barcelona Supercomputing Center for their expert support. Without Nikolaus Süß' tireless work maintaining our research group's 's computing infrastructure, this work would not have been possible. We are deeply grateful. Lastly, we want to thank the open source community, particularly the developers of Py TyToch (Paszke et al., 2019) and Matplotlib (Hunter (007).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 29304-29320 2021. URL https:ttp/occm http:tps:://pooceednnge neuriis CC cC/ http:/pappr/221111ep/1 hhml
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5:
Distilling the arcade learning environment down five games. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learming. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of mlr
Machine Learning Research, pp. 421 438. PMLR, 2023. URL ppess/vv02/eet 2tt.02/aaaichcc soccisson3ss html
Marcin Andrychowicz. Anton Raichuk. Piotr Stanczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marein Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
n tatee/foriie forum? iddnnnn wedddnnnnjssaao lAxjsniDzg
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, Abs/1600.0645M 2016. URL hTR..ttp:://// http:://axiv..orrtt./
607. 06450

---

Published as a conference paper at ICLR 2026
Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neu- ral networks for computer vision. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL ew. net forum?id=ede 1hN5@Nh
Gary Bécigneul and Octavian- Eugen Ganea. Riemannian adaptive optimization methods. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019, URL
net/ forum?id
rleig109K7
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents (extended abstract). In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the Twenty-Fourne International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 4148 4152. AAAI Press, 2015. URL htttt:://jjai..oot wtt../////..s.att//////5////////
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 449-458. PMLR, 2017. URL
http::/proceedi.n mlr. wwwps/esssssseee 70/ bewe////lllee www.eemmeeeea. 7a.htm.
Edoardo Cetin, Benjamin Paul Chamberlain, Michael M. Bronstein, and Jonathan J. Hunt. Hy- perbolic deep reinforcement learning. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
ltttss:/ net
Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. International Conference on Muchine Learning, pp.
1419-1429. PMLR, 2021.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 5672-5686. Association for Computational Linguistics, 2022. doi: N011863//1/22222ALL--LLO https://doi www.o/i.rgg/../ 1653 / awl.l22.cllloom 389
URL
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. volume 119 of Proceedings of Machine Learning Research, pp. 4048-2056. PMLR, 2020. URL http:::/pocceeeaaeee
m1 r. wtr.press/vll cobbe html
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2020-2027. PMLR, 2021. URL http:http/pproceeeiiiiiie cobbe2la.html
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan V. Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 24, 2022, pp.
URL IEEE, 2022. doi: 10. 1199/CVPR5268..2220077777777 https: / /ww.//oi.ooo// worg//g/11 /10. 109 CWWR 222.. 00726
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA:
scalable distributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1415. PMLR, 2018. URL http: 7/proceed ngs. ml prwss /ttp://00//eee/a//a/ 88 hhmml
12

---

Published as a conference paper at ICLR 2026
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Forty-first International Conference on Machine Learning, ICML, 2024, Vienna, Austria, July 2 1-27, 2024. OpenReview.net, 2024. URL /ttt:s:://pennevvvee
net. wtt.tffooum// /=tp://rmmmmmmmmmo/o/moom dw
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja. Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference Learning Representattins, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: http///eeeeeeieeeeeee//e/ee / net. wttp:eeneettooommmdddddeeeeeee
Octavian-Bugen Ganea, Gary Béeigneul, and Thomas Hofmann. Hyperbolic neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NewrlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 5350-5360, 2018. URL URL/httpss//pprcceeiinng neur cc/ .html
M Gromov. Hyperbolic groups. Essays in Group Theory, pages/Springgr--eelaa. 1987.
Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X. Yu. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In IEEE/CVF Conference on Computer Vision and Pattenn Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1-10. IEEE, 2022. doi: 10.1 2WW.9CVVR2268.222222ooo 2022.00010. URL https::/doo org/ 10. htt.///...11///////// 1109//1099/2268888
00010
Peter Henderson, Riashat Islam. Philip Bachman, Joelle Pineau. Doina Precup, and David Meger. Deep reinforcement learning that matters. Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Teiody-Second AAAI Conference Artificial Intelligence, (AAAI-18) the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3207-3214. AAAI Press, 2018, doi: 10. (169//AAALV22I1 1694. URL 1609 aaa0 v3211. 11694
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. J. Mach. Learn. Res., 23:274:1-274:1 2022. URL
hattps://imlr.r /mlr. http://wwww..og////rrreee//ee/// v233221 1342. html
D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9()):
95-95, 2007. doi: 10.1 109/MCSE.2007.55.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2162 1662-1171. PMLR, 2018. URL
http://proceed// ngs ml prwss v80/ mani 8a. html
Isay Katsman and Anna Gilbert. Shedding light on problems with hyperbolic graph learning. wttp:/ewww.eee.ooeoeeeeee f
Trans. Mach. Learn. Res., 2025, 2025. URL
rKAkp1f3R7
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: 7/arxiv. otg /aalt 6980
Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss in deep reinforcement learning: A survey. CoRR, abs/241 1.0433,, 2024. doi: 10.48550/ARXIV, 2411 1.44332. URL httpttttt///t //doi wrg/10........ 48550/arxivv 2411 04832
13

---

Published as a conference paper at ICLR 2026
Guy Lebanon and John D. Lafferty. Hyperplane margin classifiers the multinomial manifold. In
Carla E. Brodley (ed.), Machine Learning, Proceedings the Twenty-first International Conference (ICML 2004), Banff. Alberta, Canada, July 4-8, 2004, volume 69 of ACM International Conference Proceeding Series. ACM, 2004. doi: 10.1 10.1145/111333000 1514110333330.1113333 URL wttp::ttp////t...... //doi. org/
1o. 08.1145/10153331 1015335
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL net forumriccoooo htwpr//wwwie..eeeeoooroo--e----. i DKsuDo
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference
on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 13190-23211, PMLR, 2023. html
URL https
//proceedings mlr www.s www.oe/ss////////////2
Clare Lyle. Zeyu Zheng, Khimya Khetarpal, James Martens, Hado Philip van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. URL http::ttt:/pppppp wnpperrs.pppp...mmmm cc cwtpacer.pppppecceeccc fiwerffieel/elee /wwpss/ppper//00000 44 9d2d5 / Abstatt Confecetcooooeeeeeeeeoeeooee htm_
bash/c04d37ben
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc. Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. nwurips......m pcc/paper/2/ /201 9/hash/ Abst
12544-12555, 2019. URL /tt.tpp:/pppocceeeiiiie Dec04cb3912c4E0BB5 a html
Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hy- perbolic deep learning in computer vision: A survey. Int. J. Comput. Vis., 1324))3384- 3508, 2024. doi:
URL https: http:///oooooo//.1100/0/
s11263- 022-6-3-24-2203---1
Gal Mishne, Zhengchao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic representation learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 29925-24949, PMLR, 2023. URL https: / http://pooccecs..ceeeeee mlr ptw.ppssssss.s.p.s../...e....sss press/ v202 2
mi www.iiheeee..mmmmmm 23..neee3a....... html
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpt::///// http///penrrevieeeeeeeeeeee /ttppp/://eeeeeeeii.e....
net./ forum? d= QRgziT-
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human- control through deep rein- forcement learning. Nat., 518(7540):529-55 533, 2015. doi: 10.103/NATUREE4225 14236. URL htttps://ooi.... . www.orgglll.. 10. 1038 ww..///natrelll /ntt
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representa- tion, no trust: Connecting representation, collapse, and trust issues in PPO. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng
14

---

Published as a conference paper at ICLR 2026
Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference Neu- ral Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10- 15. 2024, 2024. URL http://tttppsss.../// 811 66d 9cc5dd
httppppppp_rllellaaeeaaeeeeeeeee Files/paper/202 hash/
Michal Nauman, Michal Bortkiewicz, Piotr Milos, Tomasz Trzcinski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL net/Eozum?ic 5vžzmCeTYu
Michal Nauman, Matcusz Ostaszewski, Krzysztof Jankowski, Piotr Milos, and Marek Cy- gan. Bigger, regularized, optimistic: scaling for compute and sample efficient con- tinuous control. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neu- ral Information Processing Systems 38: Annual Conference on Neural Information Pro- cessing Systems 2024, NeurIPS 2024. Vancouver. BC. Canada, December 10 15, www.iisss...pppaaoom
wwwes//apeeeeeee//////// haahh/ 2024, 2024b. URL cabaaaaess10000000333356aaaaaaa2
Avik Pal, Max van Spengler, Guido Maria D' Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In The Thirteenth International Conference on Learning Representationns ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL. hnttps::/oppnreeeeee Forum?id=3) 3Gev2hV
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library.
Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Aluch-BBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019. Vancouver, BC, Canada. pp. neurips wtt.riss..cccpppee cctppper/200 9/hash/
8035, 2019. URL https: htttp:://ppoceeeeeeeeeeee 1odbca288feee 7F2F2eeee2222bffrbe www.2//fff997022222 -bst2277777---btaaaae html
Mihacla Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. A case for new neural network smoothness constraints. In Jessica Zosa Forde, Francisco J. R. Ruiz, Melanie F. Pradier, and Aaron Schein (eds.), "I Can't it Believe It's I's Not Better!" at NeurlPS Workshops, Virtual, December 12, 2020, volume 137 of Proceedings of Machine Learning Research, pp. 21-32. PMLR, 2020. URL htattp::://poocedinggg mlr
Omar Salemohamed, Edoardo Cetin, Sai Rajeswar, and Amab Kumar Mondal. Hyperbolic deep reinforcement learning for continuous control. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023. OpenReview.net, 2023. URL id=Mrz9PgP3sT
Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International symposium on graph drawing, pp. 355-366. Springer, 2011.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap. and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nat., 588(7839):604 609, 2020. doi: 10. T038/ s 0088/s41586-12 030--
URL htt.htttss////
org/10 10
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015,
15

---

Published as a conference paper at ICLR 2026
volume 37 of JMLR Workshop and Conference Proceedings, pp- 1889-1897 JMLR.org, 2015. URL
html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, /0//1777.06347 2017. URL htctp://arxiv.orgg.
0107063347
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. In 9th Interna- tional Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction,, 2nd Edition. MIT Press, 2018. URL httpp:/www..nccmmleeee..eeeeeeee
html
Mark Towers, Ariel Kwiatkowski, Jordan K. Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: standard interface for reinforcement learning environments. CoRR, abs/2407, 17032, 2024. doi: 10.48550/ ARXIV.2407. 1732.. URL TRI..httss.. hst http:://oooooo///////// 48550/ 2407 17032
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 2094- 2100. AAAI Press, 2016. doi: 11609/AAALLV30I.. 1609/aaai. v3011.10000000 10295
URL http:st://oo..org////
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Anmual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp. 4383 4393, 2019. URL htttpss:/pprcsediiiig neurridddinnnnnninpp.......... nwwripu....... hctccc/ceeeeeee wctpappr/20111..p/ 9/hashi Sf4fe03a7772444
Abst html
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp- neurips, Cc/ wcc/paper/2/ /201 9/hash, Abstrace html
12360-12371 2019. URL htttpps:/ppooceediiiiii
[eea19926222c3883333
Appendix
Table summarizes the contents our Appendix:
Table I: Structure of our appendix.
Appendix Section
Content
Appendix A
Proois & Derivations
Appendix
Additional Background for RL and hyperbolic MLR Layers
Appendix
Environment descriptions
Appendix
Compute details and hyperparameters
Appendix
Complete results and additional metrics
16

---

Published as a conference paper at ICLR 2026
A DERIVATIONS AND PROOFS
This section contains derivations and proofs for the results
A. POINCARÉ EXPONENTIAL MAP GRADIENTS
We build on the analysis by Guo et al. (2022 and derive the Jacobian of the Poincaré Ball exponential map at the origin.
Dæy
expo (mgg)
Ezg
tanh( vcl
dæg
lanh(ve|l
lanh(vellan Vellægll
Ia +
(11)
Vellgg
Özg
where Ia denotes the d x d identity matrix.
Deriving the second term yields:
tanh( (Cll())
tanh (anb(ve/aEll)
Vellgg (vcl
8am
Sem
Vcl m
(vel me
sech² (vcl (
mell tanh Vedc
c |
1
x sech² (ol((ve
cmp
tanh vcl
c||=g|
sech (vcll //ll=elll ag (anh(V/llazgl)) mp
| 11
sech (vol
FE
(12)
[gg]
vcl a
where we use &g
aree -
in (i).
Jee
Putting Equation
and 12
together yields:
tanh (ccc)=l))
(vecl((Vellze))
tanh( (
Tg
exPo (xz)
(13)
dzp
Vcl [F
Fr
vel
We that the Jacobian of the exponential map decays with O (1I==l1->)) although the important directional term (second summand) can vanish faster with 01
A.2 HYPERBOLIC NETWORKS+ GRADIENTS
Let us first re-state the forward pass for the hyperbolic networks++ formulation (Shimizu et al. 2021 of the Poincaré multinomial logistic regression (MLR) layer:
2 2||=||
HNN+ (zH)
sinh -1 ((1 X²n) sinh(2 Ver) + VoXe,, cosh Ver) (2, mn))
(14)
Vc
Here 2h expo (2E), Na i-el æu is the conformal factor of the Poincaré Ball, is the weight
vector the layer, 2 |||| are the weights normalized to unit length, and r scalar bias term. We omit the class index I as in the main paper to simplify the notation.
=
We can re-state Equation
2||2||
IINN+ (xH)
sinh (F(#n)),
(15)
vcc
17

---

Published as a conference paper at ICLR 2026
with
F(mn)= - Xea) sinh Ver) + VeXen cosh (er) (2,
(16)
We first calculate the outer derivative (Equation 15
2 1
my 2z"""
HNN++
(201)
Vag P(=H)
(17)
v v1 + F(x
The gradient of Equation 16 V mu F(x11) is:
V Fa
Non + Ven Velg cosh(2
Vexa cosh cosh (oor)))==(sssinn) (inhh((2vor) + V Van Nen
4CæH (-sinh(2Vor) + /a (18)
2 (2 2
c 2
(1 - c 11
We plug in the definition the conformal factor X I-c and derivative the last step.
The term
V1+F() < 1 in Equation 17 cannot blow up. However, the gradients in Equation 18
grow with ((1 c||a ? for samples zи close to the boundary of the Poincaré Ball.
A.3 HYPERBOLOID EXPONENTIAL MAP GRADIENTS
The exponential map the Hyperboloid at the origin 0 [0, æg] T to the Hyperboloid (Bdeir et al. 2024):
maps a tangent vector
expo(v)
1 Vc
cosh(vel(=ell) sinh sinh(Vel(ag)))
ag
(19)
|
where the first element is a scalar time component and the remaining elements constitute the space component.
The derivative of the time component is d 1-dimensional vector whose first element is zero:
i
cosh( (clx(vc(
0, sinh(((
TE
(20)
de
Ve
Zr.
For the derivative of the space component, we start by reformulating as:
i sinh( /Vl||
i
f'([ x
0,
f ([|==|])
f(
Tp
(21)
dv Vell
where f[[[=]]) = sinh( /dllall) Vellaall
Ia is the d x d identity matrix, and 0 E Rdd.
For f'(()=))),, have:
/ cosh( ([
sinh /C||ægl))
f' f(([x
(22)
Plugging Equation 22 into Equation 21 yieldss:
sinh Vc Øp
/c @g| cosh /cl Øg ll)
1
(23)
Vell
We arrive the Jacobian of the exponential map by putting Equation 20 and Equation 23 together:
dxh i
/cllana sinh( ( p ||)
expo(u)=
Tep
(24)
Do
Oo
0
sinh sinh(y/l[mg I +
V mg posh J mg sinhh v mg
T
Vell
/el /
18

---

Published as a conference paper at ICLR 2026
A.4 PROOFS
Lemma 4.1
[I + b]]l2 - (| < [ x + b) - f(0)l2
L W + bll2
(49
L. W |2||2||22 + T.11b||2
where (t) uses the Lipschitz property of f and (ii) follows from the definition of the induced matrix norm. The special case follows directly by observing that ReLU is 1-Lipschitz with RcLU(()
Proposition 4.2 First, we bound the norm of the normalized feature vector. Recall that RMS(2)
2/ w/u(w) with p(æ)) V || Then,
|
a 13
|RMSSx))12
d,
+ ||10113
||
and
||
1
f( (MM(()))
(l) - ( L|RMMS())12)
<
vd [f(()]]2 + L,
where (i) follows from Lemma conclude the first part. Second, we bound the conformal factor. Let t = expo (.)) [Vel(Ve v|
#
We have:
Lanh(Vellæell) Velløl12
tanh /cl(ll2)
| tanh(V/cl@(la)
||=||2
V (clll22
ve
Applying the previous equality gives
x 1 c/|00| tanh( vc
which can be further bounded by combining it with the bound on || and using the fact that cosh is a monotonically increasing function on Ryo:
Vi (1110011 +1))
X tanh Vcll cosh 2 (Vella|) s 2 cosh?
B ADDITIONAL BACKGROUND
B.1 TRUST-REGION POLICY OPTIMIZATION (TRPO)
TRPO maximizes cumulative reward through gradient ascent surrogate objective (Equation equa-
tion 25
max E, To (a: (8))
Teau (az st) A,
(25)
9
subject to Eg DaL(To(aL (t) || daul] 5.
Additionally, it enforces an average KL-divergence constraint to keep the new policy close to the data- getta-gnerratigg policy. Theoretically, optimizing this objective guarantees monotonic improve- ment (Schulman et al. 2015). In practice, several approximations are used for deep neural networks. Nevertheless, TRPO tends retain the monotonic improvement of its theory.
19

---

Published as a conference paper at ICLR 2026
B.2 DOUBLE DEEP Q-NETWORK
Deep Q Network (DQN) (Mnih et al. 2015) learns the optimal Q-function for discrete action spaces by minimizing a mean-squared error loss against an off-policy bootstrap target while reusing replayed transitions. The standard target is (s, a) + ? max, ("'s', (sss'''') ,a'), which, together with experience replay and periodically updated target network, stabilizes training. However, because the same function approximator effectively selects both the maximizing action and evaluates its value under noise and approximation error, the max operator induces systematic overestimation bias (Sutton & Barto 2018).
Double DQN (DDQN) (van Hasselt et al., 2016) reduces the overestimation bias that arises when the same network both selects and evaluates the maximized next-state value. DDQN uses the online network with parameters to select the greedy next action, and a target network with parameters to evaluate that action when calculating the TD target: Q (D, a)
r + Q5(s' arg max,' (s', (ss,aa)) a')). This decorrelates action selection from evaluation, effectively mitigating overestimation bias.
B.3 HYPERBOLIC MULTINOMIAL LOGISTIC REGRESSION
Multinomial Logistic Regression (MLR) hyperbolic space M is defined as the log-linear model with parameters 2 RR, тж E R that predicts the probability p(y k a:) of an input a E M = Rd belonging to a specific class & E {1....,K}: K]:
p k x) x exp(Us (z)), (x)),
Usr.rr (x) || Tp dM(w Hzr.r))
Here, exp( exp(Uxu.rr (x) ) is the logit for class & and Usc.ry (x) the signed distance to the margin hy- perplane Hza.ru- To prevent wver-parametrization,, each hyperplane is characterized by aligning its normal vector ag and shift ps. requiring only d + 1-parameters per hyperplane (Shimizu et al.) 2021 Bdeir et al. 2024). To leverage established Euclidean optimization algorithms, all parameters maintained in Euclidean space and mapped to their hyperbolic counterparts. The normal vector ag M is obtained by parallel transporting the Euclidean parameter 2₁ I& the origin Ō: a₃ = (=)) with 2₁ E M. The hyperplane's 's shift рı E .M is defined as scalar multiple of
the same unit tangent vector ps= expo (T 2k: with E R.
B.3.1 POINCARÉ BALL MLR
For the Poincaré Ball we have:
tanh (Vers)
Tk [|
Pk =exp
=
Vc |
Zk.
a PTo-p((zk) (1 tanh? (Verk)) 2k:
Heur = =Har Dr = fx pdd
(ax: Spa
+ x) = =0]
1
2/e | (ak: Pp e x) (1 - θ pa
dea (z, Her.re) = dg(x Har.p.)
sinh
Vc
+ x as
where @ and @ denote the Mobius addition and subtraction MLR layer (Shimizu et al. 2021) can then be summarized as
(Ganea et al. 2018). The Poincaré Ball
HNN++ (x)) U SE.Tk
2 ||| sinh (1- sinh (Crr) + Vere (err() Ve
Zk |
x
B.3.2 HYPERBOLOID MLR
For the Hyperboloid, the tangent vectors 2; represented in Rd rather than the full d space that would typically characterize tangent vectors at the origin of the Hyperboloid. However, to preserve the correct number of degrees of freedom, we omit the time component, which is constrained to be
20

---

Published as a conference paper at ICLR 2026
That said, we have:
Pk
1
cosh(Verk), sinh(vers 2/
Ps = expo
||>*|||
2k
Vc
|
PTo-pp22
sinh (verk) zxll cosh( veru)=x
Hznrr "Har.p
{ar E Hy: (ak. = 0)
VC
dug (a, Hze.)r)
sinh -1
(-xo sinh(Vervl) ||*1|| + cosh (eab(((er)))sss
Vc
||
The Hyperboloid MLR layer (Bdeir al. 2024) can then be summarized as
Zz Vc
VC
HB
(x)
sinh-!
(-xo sinh(verx))
||= cosh cocr(Vccr)
(26)
#k.7k
||
where Vx ( . Td) denotes the spuce component and ag the time component.
21

---

Published as a conference paper at ICLR 2026
Figure 10: Visualization of all ProcGen environments.
C ENVIRONMENTS
In this Section, we review the environments used in our paper and discuss evaluation differences to existing methods.
C.I PROCGEN
ProeGen (Cobbe et al. 2020) uses RGB frames of size 64 x 64 x 3 as observations. We visualize the 16 games in Figure 10 The action space is discrete with 15 actions. For training, follow the protocol by (Cetin et al. 2023): fix difficulty to "easy" and train on the first 200 levels (seeds 0-199). For testing, evaluate all levels of the easy distribution. For the table, end-of-training evaluation on 100 parallel environments sampled from the train and test distribution, respectively. We then normalize the scores for each individual run before aggregating.
single
C.2 ATARI
The Arcade Learning Environment (Bellemare et al. 2015 Towers et al., 2024) provides standard- ized interface for deep RL research based on dozens of Atari 2600 games. Of these, 57 are commonly used in evaluation. The observations are RGB frames 210 160 x 3. which are preprocessed via grayscaling, downsampling to x 84, and frame stacking. The action space consists up to 18 discrete joystick/button combinations, with most games using subset and action repeat (frame skipping) to help with jittering. The rewards are clipped to (-1, 0, +1}. As the game dynamics are naturally deterministic, the benchmark uses randomized no-op resets outlined in the original DQN paper (Miih et af.) 2015) and cleanRI. (Huang et al. 2022).
C.3 EVALUATION DIFFERENCES WITH EXISTING WORKS
Our paper builds on the seminal work by Cetin (0023). However, we struggled to reproduce their results, which we believe is mainly due to three reasons. First, their source code does not use seeding. As deep RI. notoriously seed-dependent (Henderson al.) 2018 Agarwal al. 2021), we find exact reproduction impossible. Second, we use different implementation for the mathematical operations of hyperbolic geometry, which possibly affects the results. This issue is known within the hyperbolic deep learning community (Katsman & Gilbert 2025). Third, we use different versions Py Torch and Python. Additionally, our evaluation follows a slightly different protocol (see Appendix C.I). and use Pytorch's evaluation mode before generating results for agents. We hope that by releasing our complete code, we can take step towards more reproducible research in hyperbolic deep RI.
22

---

Published as a conference paper at ICLR 2026
Table 2: Wall-clock results.
ProcGen forward
NameThisGame
Euclidean
14ms
17h52m
Hyper+S-RYM (Cetin et al. 2023
19.3ms
m
HYPER++
14.7ms
35h25m
D EXPERIMENTAL DETAILS
D.1 HARDWARE & RUNTIME
For our experiments, used Nvidia A100 GPUs. For ProcGen, we can train up four agents in parallel on a single GPU with 40GB. We report wall-clock times for forward passes on ProcGen and for full runs on NameThisGame factor for results when timing on full experiments because agent performance can either be positively or negatively correlated with episode length. For NameThisGGmme, e-g., better agents generate longer episodes. We average over 100 passes for the forward pass results and five seeds for the NameThisGame results.
Table Note that agent performance can be confounding
D.2 NETWORK ARCHITECTURE
For ProcGen, the same Impala-ResNet (Espeholt et al.) 2018) (Cetin al. 2023 which we visualize in Figure II
Our modifications are shaded in purple. They consist of
using RMSNorm (Zhang & Sennrich, 2019) before scaling the Euclidean features,
2. using TanH instead of ReLU as penultimate activation,
3. applying learned feature scaling before the exponential map, and
4. using the Hyperboloid instead of the Poincaré Ball.
For Atari, we use the NatureCNN (Mnih et al. 2015) architecture with the same modifications applied for ProcGen.
23

---

Published as a conference paper at ICLR 2026
fn a(a an)
HI. Hau-Saus)(
Hyperboloid ML.
Hyperholoid MLR
expo(at)
Learned Scaling
1/Vd
tanh
RMS(#g)
Fully connected W
ReLU
Impala residual black
Impala residual block
x3 12, 32] channels
Max pooling 3 x 3, stride 2
Convolution 3 x
Figure 11: Hybrid neural network architecture, Sn denotes hyperbolic layers, f Fuclidean layers. Components that are specific to HYPER++ shaded in purple.
24

---

Published as a conference paper at ICLR 2026
Table 4: PPO hyperparameters for ProcGen.
PPO hyperparameters
Parallel environments
64
Stacked input frames
1
Steps per rollout
16384
Training epochs per rollout
3
Batch size
2048
Normalize rewards
True
Discount Y
0.999
GAE A (Schulman al., 2015)
0.95
PPO clipping
0.2
Entropy coefficient
0.01
Value coefficient
0.5
Shared network
True
Impala stack filter sizes
16, 32, 32
Default latent representation size
32
Optimizer
Adam (Kngmma & Ba 2015
Optimizer learning rate
x 10 4
Optimizer stabilization constant (
1 x 10
Maximum gradient norm.
0.5
Table 5: DDQN hyperparameters for Atari.
Atari hyperparameters
Environment steps
10M
Discount 7
0,99
start
1
end
0.01
Exploration fraction
10% of steps
Replay buffer size
1M
Target network update frequency
1000
Default latent representation size
512
Batch size
32
Training frequency
4
Optimizer
Adam (Kingma Ba 2015
Optimizer learning rate
x 10
Optimizer stabilization constant (c)
2.5 x 10
D.3 HYPERPARAMETERS
We specify the hyperparameters for all PPO agents in Table 4 for the DDQN agents in Ta- 5 and for PPG in Table For DDQN and PPG, we use the hyperparameters and prepro- cessing steps from cleanRL (Huang et al. 2022). Additional parameters for our method are Ta-
Table HYPER++ hyperparameters.
HYPER++
Loss: Number of bins
51
ble On ProcGenn our agent uses a latent dimension d = 64 compared to d = 32 for Hyper+S-RYM. For HL-Gauss (Imani & White 2018), we use the default parameters specified by Farebrother et al. (2024). We set the learned scaling hyperparameter to tuning for all experiments. When using RMSNorm or LayerNorm, we do not use affine parameters, because they can overfit (Xu et al. 2019) the training set.
Loss: Min clip
-10.0
Loss: Max clip
+10.0
Last Euclidean Activation
TanH
Learned scaling a
0.95
0.95 without
25

---

Published as a conference paper at ICLR 2026
Table 6: PPG hyperparameters for ProcGen.
PPG hyperparameters
Policy iterations (N+)
32
Policy phase epochs (Fx))
1
Value phase epochs (Ev)
Auxiliary phase epochs (Eaxx)
6
Behavior cloning coefficient (Sclone)
1.0
Auxiliary phase minibatches
4
Gradient accumulation steps
E ADDITIONAL RESULTS
E.1 FULL PPO RESULTS
Hyper++ (Ours)
Hyper + S-RYM
Hyper
Euclidean
Bigfish
Bossfight
Caveflyer
Chaser
Train Reward
10
10
20
5
0
0
0
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Climber
Coinrun
Dodgeball
Fruitbot
Train Reward
10
10
10
20
0
OM
10M Heist
20M
0M
10M
20M
OM
10M
20M
0M
10M
20M
Jumper
Leaper
Maze
Train Reward
10
10
10
5
OM
10M
20M
0M
10M
20M
0M
10M
20M
10M
20M
Miner
Ninja
Plunder
Starpilot
Train Reward
15
10
50
5
10
25
0
0
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Figure 12: PPO ProcGen Train Learning curves.
26

---

Published as a conference paper at ICLR 2026
+yper++ (Ours)
Hyper + S-RYM
Hyper
Euclidean
Bigfish
Bossfight
Caveflyer
Chaser
Test Reward
10
20
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Climber
Coinrun
Dodgeball
Fruitbot
Test Reward
10
10
10
5
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Heist
Jumper
Leaper
Maze
Test Reward
7.5
5
0
OM
10M
20M
0M
10M
20M
OM
10M
20M
0M
10M
20M
Miner
Ninja
Plunder
Starpilot
15
Test Reward
10
5
5
0
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Figure 13: PPO ProcGen Test Learning curves.
Table 7: ProcGen Train Results (mean + std).
Hyper
Euclidean
Hyper + S-RYM
Hyper++ (Ours)
bigfish
7.12+1.7
7.81
17.38113.3
25.66:12.8
starpilot
10.43+
31.1414.4
38.92+1.9
43.43=3.7
dodgeball
5,42+0.9
3.28+0.7
6.26=10.7
10.28=1.0
coinrun
6.25++1.4
7.37+1.2
5.53+11.0
9.67-10.2
leaper
1.65+
3.35+1.0
3.23+220
5.25++0.9
ninja
4.63+0..8
5.65+0.9
4.37+0.55
6.67++0.6
fruitbot
27.09+0.9
29.4040.8
29.36110.7
29.89-0.4
jumper
8.2740.3
8.1720.6
8.1740.7
8.53+0.3
bossfight
8.49+0.7
9.40+0.5
9.38++0.7
9.27+00.7
miner
6.8610.5
8.52+0.8
8.53+1.0
8.10+11.4
chaser
3.85+0.5
5.04
7.60
7.12-00.9
climber
6.72+00.7
7.79-0.5
7.43-L0.8
7.27+00.7
caveflyer
5.8110.5
6.41-0.3
6.1310.6
5.04
maze
8,88
8.88+0.7
7,80
6.90+1.4
plunder
4.9910.9
5.30
8.81+1.5
6.23
heist
7.57++0.6
6.78+1122
6.68
3.9
IQM (normalized)
0.37
0.45
0.46
0.55
27

---

Published as a conference paper at ICLR 2026
Table 8: ProcGen Test Results (mean + std).
Hyper
Euclidean
Hyper S-RYM
Hyper++ (Ours)
bigfish
4.8
3.60+3.44
11.88
20.01+
starpilot
22.
30.4443.1
35.59
42.49-2.66
dodgeball
2.3810.2
1.79-1.33
3.95
9.4111.2
coinrun
5.07+11.3
6.13+1.2
5.57
8.72++0.4
leaper
1.38+114
3.58=1.3
3.33+2.00
5.13+11.0
ninja
4.0
4.65+0.5
4.27=0.4
5.681004
fruitbot
26.37.1..0
27.88
28.44.1..5
28.28
jumper
5.28
5.08
5.28+10.4
5.30-10.4
bossfight
8,50+0.7
9.5010.8
9.03+0.8
9.40+0.7
miner
5.78
7.15
1.07
7.21
chaser
3.34+0.6
4.5
6.67+0.7
7.27+0.9
climber
5.41
5.89+0.7
5.52+0.99
5.86+1122
caveflyer
4.91
5.2210.3
4.83+00.5
4.12
maze
6.38
6.4
5.85
5.17
plunder
4.58
4.71
7.45
5.82
heist
3.58+0.5
3.68-0.5
3.38+0.5
2.28+1.0
IQM (normalized)
0.19
0.26
0.27
0.41
28

---

Published as a conference paper at ICLR 2026
Ourse Eac
24.8741.66
38.7843.3
9,72+
9.55=00.2
4,38+
5.071-044
27.49.0099
8.37=006
8.29
6.68
4.87=
6.58=006
4.81
6.95
7.68
4.12
0.45
Fac. N.C..HI.. G.aass
1..99-3.8
32.95-7.4
4.63
9,27
4.13--099
4.32-10.3
29. 19.70-1.7
7.55+114
6.46
4.65
4.70=11.4
6. 6.111:14
4.44-10.3
8.52-11.3
5.26-#1.1
2.88
0.32
Fac. Gacc+RMS
5.95=4.7
25.22+3.5
5.08 1.0
8.45 1.4
3333=
5.55
08.08=111
8.57-0.3
9.14
6.09
4.73=2.0
7.64=
5.71
9.38-033
5.46=
7.62-117
0.45
Ours+SNI (Pemult.)
1.04=0.2
2.86-0.33
1.73
6.07
3.35=0.3
3.32-0.5
-1.54--022
3.10=0.3
0.43
1.27
0.66=
2.16#0.4
3.97-0.5
5.12-0.3
4.58=0.5
3.33
0.01
Ours+ (Full)
Table 9: ProcGen ablation Train results (mean + std).
0.94++0.1
2.78++0.4
1.641:0.3
5,78:
3.57+04
3.174:06
-1882-0..3
3.05
0.48:10.2
1.43+0.3
0.69
1.87::02
3.874004
5.42+0.7
4,741:0.2
3.15#:0.4
0.01
Ours RMS
1.04=0.2
2.71+0.4
0.3
5. 5.68=0.22
3.57=0.6
3.43=0.4
-149-003
3.08=0.6
0.48=0.2
1.54
0.63=0.0
2.33=0.2
3.45-0.5
5.17=0.3
0.3
3.73
0.01
Ours+C51
19.93+3.5
37.71++4.0
7.951114
9.55
4.22+1.2
5,13199
27.23409
7.75+0.4
8.6210066
4.53+15
4.02+066
3,394088
4,27400
5.60+-.2
6.37++0.6
3.25
0.33
Ours+MSE
+-66
9.06 1.5
0.3
2 2.80 0.5
5.27-06
29 1.3
8.87 0.3
9.94 0.4
7.32-1.2
6.23=099
7.65 0.2
-
9.72-0.
8.47 1.1
8.38-08
0.55
Ours Scaling
42.97+2.0
8.1210.55
9.8
388+008
5.2210777
30.0410.8
8.57+0.3
8.86110.7
7,22+112
8.03+088
5.00
3.294-033
9.28++0.5
6.89+.0.5
4.22+1.1
0.50
Durs+Poine
17.35
60.22+-2.0
8. 15.100
9.52+0.1
3,4740.9
6,49
8.68+00.4
9.36.11.2
7,931088
7.51+0.9
6.221114
3.05+04
8.48+1.4
7.77+1.7
3. 3.70:10.5
0.50
29.
27.02 102
11-80--3.5 3.8
1.0
9.681:0.2
4.87:10.77
3.90--0.7
R.20++0.3
9.55110.9
7.49:0.6
6.66+-10
6.524:16
4.85+0055
0.784-2.1
6. 101:1.7
3.65::0.5
0.51
IQM (normalized)
bigfish
starpilet
Bodgeball
coinrun
leaper
ninja
fruitbot
jumper
bossfight
miner
chaser
elibber
cuveffyee
plunder
beist
29

---

Published as a conference paper at ICLR 2026
Ourse Ours+Euc. Euc.
19.31+22.7
35.35
9.11
8.38=0.3
4,67
4,77-06
27.04+1.0
5.22
7.73
6.18=0077
4,84
5.88-099
4.32=
5.55
6.39
2.68=004
0.35
Euc, Mic.++IL-aamm GauGG
7.60==3.6
32.12-7.1
2.92=1.3
8.05=
3.88-10.8
3.20+1.1
28. 18.17=0.9
5.03-10.6
6.10
4.10=2.11
4.24
4.23
3.99
5.35
4.33
2.28:
0.20
Euc. -RC.
3.71=2.9
25.61-299
2.74=09
766=
337--111
5.13-04
26.82-1.3
5.37-
9.19
525=
4.67-
6.43-04
4.45
6.92
4.83
3.92=099
0.28
Ours+ (Penult.)
0.96=0.J
2.58
1.61
5. 5.40=0.5
3.27=0.3
3,40
1.65+-033
2. 2.60
0. 0.59=0.3
1.33=022
0.64
2.39-088
3.95
5.27
4.45
3.27=0.3
0.00
Ours SN (Full)
Table 10: ProcGen ablation Test results (mean + std).
1.01
2. 7.801...3
1.54+00.2
5,35+0.3
3.18:005
3
-1.74100.3
2. 2.4010.4
0.62+:0.2
1.48+0.3
0.651:00
2.441003
3.574.0.5
5.08
4.42::0.3
2.95#:0.4
0.00
Ours RMS
1.05=0.1
2.761:0.3
1.59=-0.3
5.25=0.6
3.30
3.28-0.5
-169
2.50=0.3
0.78=0.3
1.62=0.4
0.65
2.47=0.4
3.62=0.77
5.22.1..4
4.44=0.3
2.80=0.4
0.00
14.77+4.7
33 33.68
5. 5.70+1.3
8.67+0.3
4,20111
5.35+04
27.21
5.23
8. 3.69
4.40+144
3.991:05
3.404005
3.874033
5.2....9
5. 5.94-00.6
2.55#00.9
0.27
Ours
16.47:44.66
40.14
5.42=1.3
2.47=022
4.78-022
28 1.2
5.52-04
9.50=0.7
6.23=1.0
08
5.410077
6.18-0.8
4.23=0,7
7.75=00
7.59-1.
0.33
Ours Scaling
15.02+--.3
39 .6.6661335
5.28+1.0 1.0
8.75+005
4,221.11
4,70+10.5
27.76+1.3
5.3810.5
8.46
6.44144
6.941066
4.13+1.3
4.85+
5.8710.6
6.16
2.00+00.6
0.33
Poinc.
12.52+33.8
37.34
5.611111
8,58+04
3,581099
5.334088
28.04+1.3
5.73100,2
8.6011.0
7.02+:055
110
5,1411.2
4.0540,7
5. 5.40.00.7
6. 6.8611.6
2.35+0.7
0.34
21.56=35
29.5522
3.91-0.7
8.75+0..4
4,621100
5.57::0.6
28.62
5.37.10.3
9.5110.77
7.42+:1.2
6.75-10.8
4.951111
1.91-0022
3.58.1..5
5.088110.7
2.43:00.4
0.40
IQM (normalized)
bigfish
starpilot
dodgeball
coinrun
leaper
fruithot
jumper
bossfight
miner
chaser
climber
caveftyer
plunder
ninja
heist
30

---

Published as a conference paper at ICLR 2026
E.2
ADDITIONAL ABLATION METRICS
Hyper++
+MSE
RMSNorm
Scaling
Poincan
Last FC Gradient Norm
Embedding Norm
PPO Clipping Fraction
Last FC Gradient Norm
4.0
0.3
Starpilot
0.2
2.0
0.1
15
1.0
0.3
0.0
Last FC Gradient Norm
0.35
0.30
Dodgeball
0.25
0.15
2.0
0.19
1.5
0.0
1.0
0.5
Last FC Gradient Norm
0.5
0.4
Bigfish
0.2
0.1
1.5
0.2
0.5
10
10 15
20
25
15 Number Frames millions!
Number Frames millions)
Number of Frames (in millions)
Additional training metrics hyperbolic deep RL agents. Agents w/o RM- SNorm (Zhang & Sennrich (019) (-RMSNorm) suffer from growing embedding norms and vanish- ing gradients in the encoder. Using MSE instead of Hl.-Gauss (Imani White 2018) (+MSE) leads to larger initial encoder gradients due gradients scaling proportional to the loss for MSE. Not using learned feature scaling -Saaling) has the largest embedding norms and gradients, which are quickly compensated by RMSNorm's gradient variance normalization (Zhang & Sennrich 2019.
Figure 14:
E.3 GRADIENT AND LOss VARIANCE ANALYSIS
Figure 15
shows the evolution of the loss and the loss variance over the course of the training, averaged over all runs. Compared to MSE, the categorical loss is both higher and has higher variance. However, our paper argues that not the loss values matter, but the gradients instead. Table (T shows the LI and L.2 gradient norms of the penultimate layer (last FC layer in the encoder) using the final 25% of training steps. Despite higher loss values and variance, using the HL-Gauss loss yields smaller, lower-variance gradients.
Table 11: Penultimate layer layer gradient statistics.
Agent
L2 Grad Norm
L1 Grad Norm
N runs
Euclidean
0.0788 = 0.0276
0.0506 + 0.0256
96
CattideanCCatogorical
0.0695 = 0.0228
0.0460 = 0.0240
96
Hyper++
0.0258 + 0.0099
0.0294 + 0.0155
96
Hyper++mmse
0.0327 = 0.0102
0.0346 + 0.0134
96
31

---

Published as a conference paper at ICLR 2026
+yper++ (Ours)
+ MSE
Rolling Variance
0.15
Critic Loss
0.10
0.05
0.00
OM
10M
20M
OM
10M
20M
Figure 15: Critic loss and variance. We plot the critic loss and variance for method when using MSE and the categorical loss, averaged over all runs and environments. The categorical HL-Gauss loss (Imani & White 2018) has higher loss values and variance than MSE.
E.4
LEARNING CURVES FOR ATARI GAMES
+yper++ (Ours)
Hyper S-RYM
Euclidean
NameThisGame
Phoenix
BattleZone
Episode Return
15K
40K
10K
6K
4K
20K
2K
0
0
0
0M
2M
4M
6M
8M
10M
DM
2M
4M
6M
8M
10M
0M
2M
4M
6M
8M
10M
DoubleDunk
Qbert
Episode Return
20K
-10
-15
10K
-20
.25
0
0M
2M
4M
6M
8M
10M
0M
2M
4M
6M
8M
10M
Figure 16: Atari-5 learning curves. HYPER+ outperforms the baselines on all environments with particularly strong gains in NAMETHISGAME and QBERT. RESULTS ARE AVERAGED FIVE SEEDS.
32

---

Published as a conference paper at ICLR 2026
E.5
POLYAK AVERAGING EXPERIMENT
Hyper++ (Ours)
Hyper + S-RYM
Euclidean
NameTnisGame
Base
Polyak
Episode Return
20K
20K
15K
10K
10K
5K
5K
OM
2M
4M
6M
8M
10M
0M
2M
4M
6M
8M
10M
Figure 17: Polyak averaging (NAMETHISGAMEE Polyak averaging refers to exponential moving average updates for the target network instead of hard replacement updates. Algorithm performance not meaningfully affected by the form of the target network update. Runs are averaged over five seeds, with one standard deviation error.
E.6
OFF-BATCH PPO METRICS
Hyper+ (Ours)
Hyper S-RYM
Euclidean
0.035
Off-Batch Update KL Diver.
0.04
Update KL Diver.
0.030
0.03
0.025
0.020
0.02
0.015
0.010
0.005
0.000
0.00
0M
5M
10M
15M
20M
25M
0M
5M
10M
15M
20M
25M
Off-Batch PPO Clip. Fraction
0.20
PPO Clip. Fraction
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0M
5M
10M
15M
20M
25M
0M
5M
10M
15M
20M
25M
Figure 18: Off-batch PPO stability metrics. We track the update KL divergence and PPO clipping fraction for the batch that has currently been updated (left column) and for a batch randomly sampled on-policy states (right column). The figures show high level of similarity for the evolution of both metrics. For off--batch data, the update KL divergence has a noticeably higher variance.
33

---

Published as a conference paper at ICLR 2026
E.7 ARCHITECTURE ABLATIONS
Table 12: Latent dimension and learned scaling a ablation studies. Test results with PPO on ProcGen, averaged over six seeds.
Environment
64
512
64,
0.9
d 64,
0.85
bigfish
20.77 2.0
20.01
+ 4.8
20.96
19.84 + 4.2
19.55
2.9
20.89 3.3
starpilot
40.07
42.49
39.07 2.9
39.55
2.7
dodgeball
9,02
9.41 1.2
9.15 1.9
9.62 1.4
1.5
coinrun
8.60 + 0.4
0.7
8.68
8.82 0.3
8.67
8.67 0.4
Ieaper
1.
5.13 1.0
4.27
4.35
1.7
4.90 1.0
ninja
5.07 -07
5.68
=.4
5.40
5.45
5.68 - 0.9
fruitbot
1.8
28.28
28.81
2..75
27
28.01
jumper
0.1
0.3
5.22 0.6
5.9
5.7
5,98
bossfight
9.23
9.4
8.94
9.7
9.00
0.9
miner
7.76 + 0.6
7.71
6.63 0.9
6.7
7.11
7.44
chaser
6.42
0.2
7.30 0.7
6,63
6.74
6.40 1.2
climber
5.10
1.26
6,54 1.0
6.42
6,20
5.70
caveflyer
4.33 0.9
4.12
3.9
4.08
3,93
3,96
maze
1.4
5.17
5,70 1.2
5.
5.4
5.48
plunder
6.40
5.82
6.17
1.2
5.85 0.7
6.71 0.6
heist
2.67 =
2.28
2.42 = 0.5
2.60 = 0.6
2.73 0.7
2.20 0.4
IQM (normalized)
0.38
0.41
0.42
0.39
0.40
0.41
34

---

Published as a conference paper at ICLR 2026
E.8
PHASIC POLICY GRADIENT
Tables 13 and 14 show ProcGen results using Phasic Policy Gradient (PPG) (Cobbe et al. 2021 averaged over seeds. We the default cleanRL. hyperparameters for all (Huang al. 2022) HYPER+ outperforms both baselines in terms of train and test performance. HYPER++ outperforms Hyper+S-RYM by 53% terms of IQM test reward. Notably, Hyper+S-RY3 fails outperform the Euclidean baseline, highlighting the generality of our method where previous hyperbolic agents fail.
Table 13: PPG ProcGen Train Results (mean + std).
Euclidean
Hyper + S-RYM
Ours
bigfish
26.20:4.9
20.8444.4
27.42:4.0
starpilot
43.39++.7
43.23++38
45.33++..4
dodgeball
9.19+t1.2
5.27+11.7
13.17+2.3
coinrun
9.87++0.1
9.28++0.3
9.80+1022
leaper
5.45
3.3042.88
6.82-1211
ninja
9.07+10.
5.73+0..8
9.13-10.3
fruitbot
10.58
31.55+0..8
31.63=0.7
jumper
8.70=1044
8.32+0.3
8.62=0.4
bossfight
10.5
8.91+2.0
11.31
miner
9,15+
7.68+0.6
6.77+
chaser
6.9413.1
7.94
8.86
climber
8.71-1..6
6,33
8.02
caveflyer
7.24
5.12:00.6
6.03
maze
9.07
6.67+0.5
9.0
plunder
13.78+1.2
10.27+3.2
13.67
heist
0.95
5.50+L0.9
5.37
IQM (normalized)
0.67
0.47
0.69
Table 14: PPG ProcGen Test Results (mean + std).
Euclidean
Hyper + S-RYM
Ours
bigfish
21.35+3.7
15.42+22.9
23.15::2.1
starpilot
40.06:2.9
41.84+14.7
43.70+13.4
dodgeball
5.56-0.9
3.14
11.63
coinrun
8.78=0.1
8.15+0.3
9.02-10.2
leaper
4.93=2,2
3.13+2.5
6.33
ninja
7.38=0.3
4.82+0.4
7.45
fruithot
29.46
29.50+1.0
29.72.10.9
jumper
5.93=0.6
5.25
5.22+10.4
bossfight
10.20+0.6
8.10+2.4
10,89-10.6
miner
7.35-0.6
7.25
6.62+11.9
chaser
6.6
7.57-0.8
8.31-10.4
climber
6.57=0.5
5.30
6.881008
caveflyer
5.70-0.6
4.50
5,43+009
maze
5.87=0.3
5,8310.8
6.03-10.9
plunder
12.29+2.9
7.89+2.2
12.58+1.4
heist
3.38=0.8
3.73-10.6
3.55-L0.7
IQM (normalized)
0.47
0.34
0.52
35

---

Published as a conference paper at ICLR 2026
+yper++ (Ours)
Hyper + S-RYM
Euclidean
Bigfish
Bossfight
Caveflyer
Chaser
Train Reward
10-
10
20
10M
20M
10M
20M
0M
20M
10M Fruitbot
20M
Climber
Coinrun
Dodgeball
Train Reward
10
10
10
20
5
0-
0-
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
20M
Heist
Jumper
Leaper
Maze
Train Reward
10
10
10
5
5
0M
10M
20M
0M
10M
20M
0M
10M Plunder
20M
OM
10M Starpilot
20M
Miner
Ninja
Train Reward
10
20
50
5-
5
10
25
0-
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
16M
20M
Figure 19: PPG ProcGen Train Learning curves.
Hyper++ (Ours)
Hyper + S-RYM
Euclidean
Bigfish
Bossfight
Caveflyer
Chaser
Test Reward
10.
0M
10M
20M
0M
10M
26M
0M
10M Dodgeball
20M
OM
16M
20M
Climber
Coinrun
Fruitbot
Test Reward
10
10
20
5
0M
10M
20M
0M
10M
20M
0M
10M
20M
OM
10M Maze
20M
Heist
Jumper
Leaper
Test Reward
10
10
5
5
0M
10M
20M
0M
10M
20M
0M
10M
20M
OM
10M Starpilot
20M
Miner
Ninja
Plunder
Test Reward
10
10
50
5
10
0-
0M
10M
20M
0M
10M
20M
0M
10M
20M
0M
10M
20M
Figure 20: PPG ProcGen Test Learning curves.
36


Reviews:
[{"review_id": "a9q9gOAcOk", "reviewer": "Reviewer_Hm1V", "summary": "This paper studies why deep RL agents with hyperbolic latent spaces are hard to optimize and proposes HYPER++, a PPO-based agent that stabilizes training and improves performance on ProcGen and a small Atari subset. The analysis traces instability to (i) exploding/vanishing gradients induced by the conformal factor and exponential map in the Poincaré ball, (ii) exponential growth in the Hyperboloid exponential Jacobian for large Euclidean feature norms, and (iii) non-stationary critics. The method has three ingredients: (1) replace value regression with a categorical value loss; (2) add RMSNorm at the last Euclidean layer plus a learned feature-scaling that caps norms in an optimization-friendly way; and (3) prefer the Hyperboloid model for the final layers. Empirically, HYPER++ improves median/IQM/mean normalized test rewards on ProcGen with better update-KL/clip-fraction metrics and ~30% wall-clock reductions versus prior hyperbolic agents, and transfers to DDQN on Atari-3.", "questions": "1. **Categorical critic details.** Which distributional parameterization is used (e.g., HL-Gauss vs. C51-style fixed supports)? How are supports chosen and are they shared across tasks? Could you report critic loss variance to corroborate stability claims? \n\n2. **RMSNorm placement.** Did you try RMSNorm earlier in the encoder, or combining with SpectralNorm only on the last layer? A small table contrasting capacity vs. stability would help.\n\n3. **Trust-region proxies.** Beyond update-KL and clip-fraction, do you observe changes in policy-entropy across states not in the batch (to quantify unseen-state interference)?\n\n4. **Off-policy breadth.** Any results with SAC/DrQ (image-based), and do your findings about feature-norm control translate when using target networks and Polyak averaging?\n\n5. **Comparisons to Euclidean + categorical.** To isolate hyperbolic benefits, could you add Euclidean + categorical critic and Euclidean + RMSNorm baselines?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear diagnosis of instability.** The chain-rule decomposition (Eq. 3) and analysis of the conformal-factor gradient (Eq. 4) and HNN++ MLR derivative (Eq. 5) convincingly explain Poincaré instability; the Hyperboloid exponential Jacobian (Eq. 6) highlights a distinct failure mode—large Euclidean norms—justifying feature-norm control. RMSNorm placed only at the last Euclidean layer, plus a learnable radius cap, maintains capacity without per-layer SpectralNorm overhead; the bound in Prop. 4.2 ties directly to conformal-factor control.\n\n**Well-motivated critic choice.** Switching to a categorical value loss is compatible with distributional RL’s stability theory and practice, and the ablation (+MSE) shows degradation on noisy-reward games. \n \n**Comprehensive PPO metrics.** Reporting entropy, update-KL, clip-fraction, and gradient norms is great and aligns with best-practice PPO diagnostics. \n\n**Empirical evidence.** Consistent ProcGen gains across aggregation metrics and competitive Atari-3 results demonstrate utility; reductions in wall-clock (~30%) are practically meaningful.", "weaknesses": "**Breadth of algorithms.** Off-policy validation is limited to DDQN. I think modern strong baselines (e.g., SAC, DrQ-v2) would better test generality. Ablations on Euclidean + categorical critic and Euclidean + RMSNorm controls would isolate which pieces help independent of hyperbolic geometry.\n\n**Scope of environments.** ProcGen is appropriate, but analysis claims about hierarchy/tree-like structure would be stronger with goal-conditioned or hierarchical RL tasks (MiniGrid, Crafter, options) and long-horizon domains.\n\n**Theory tightness and assumptions.** Some proofs rely on Lipschitz constants and dimension-independent bounds; clarity on tightness and behavior under non-1-Lipschitz activations would help. A formal link from stability bounds to PPO trust-region behavior (e.g., expected update-KL control) would strengthen the story but is not necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfuqfIw04B", "reviewer": "Reviewer_ZmLs", "summary": "The paper aims to stabilize the learning process of PPO agents with hyperbolic representation. Authors provide empirical (Figure 2) and theoretical evidence (Section 3.2-3.4) that the instabilities arise in unstable gradients induced by nonstationarities in RL, large euclidean feature norms and exploding conformal factors in Poincare ball formulation of the hyperbolic space.\n\nTo this end, authors make three improvements to the naive Hyper method:\n1. Categorical value function (HL-Gauss) to stabilize gradients\n2. RMSNorm to prevent feature norm growth\n3. Hyperboloid model (in replacement of Poincare model) to remove the conformal factor\n\nThis results in a superior performance in 4 Procgen and 3 Atari environments, compared to prior work (Hyper + S-RYM) and PPO with default Euclidean space.", "questions": "- Have you tried C51-style categorical distribution instead of HL-Gauss, as a lot of works in RL tend to adapt the former [1,2,3] instead of HL-Gauss?\n- Line 60-63: I think claiming that \"PPO with a hybrid Euclidean–hyperbolic encoder is the prevalent architecture in deep RL\" is an overstatement unless it is supported by a number of citations.\n\n[1] FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control., Seo et al., ArXiv'25.\n\n[2] Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners., Nauman et al., ArXiv'25.\n\n[3] Hyperspherical Normalization for Scalable Deep Reinforcement Learning., Lee et al., ICML'25.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Rich theoretical results**. Thorough analysis of the feature norms and gradients for hyperboloid MLR seems to be a contribution by itself. It also gives enough justification to the components such as RMSNorm and Hyperboloid model, which are also verified in their ablation studies.\n- **Careful implementation details**. When replacing SpectralNorm with RMSNorm, authors also point out the exponential shrinkage of the hyperbolic space after normalization and proposes to rescale the features accordingly.", "weaknesses": "- **Lack of motivation for HL-Gauss**. Although all other components seem to be well-motivated, there are no sufficient justifications or observations that lead to the addition of categorical loss. Indeed, the benefits of categorical loss is not limited to hyperbolic DRL but for most DRL methods in general [1,2,3].\n- **Limited evaluation setup**. The proposed method is verified in 4 ProcGen and 3 Atari environments, which is significantly smaller compared to the prior work by [4]. I suggest expanding the benchmark size, or at least incorporate environments where prior work (PPO + S-RYM) has degraded performance compared to PPO, such as `heist`, `caveflyer` and `ninja`.\n\nOverall, I think the results are a bit weak to confidently accept this paper. However, I may be overlooking the theoretical value of this paper, as I do not have the sufficient theoretical backgrounds for it. If the authors are able to provide more empirical results, I will positively consider raising my score.\n\n[1] FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control., Seo et al., ArXiv'25.\n\n[2] Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners., Nauman et al., ArXiv'25.\n\n[3] Hyperspherical Normalization for Scalable Deep Reinforcement Learning., Lee et al., ICML'25.\n\n[4] Hyperbolic Deep Reinforcement Learning., Cetin et al., ArXiv'22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QpxmCgrMwT", "reviewer": "Reviewer_sPEy", "summary": "The paper proposes HYPER++, a modified hyperbolic PPO agent with: (i) a stable critic using categorical value loss, (ii) feature regularization to control embedding norms, and (iii) improved hyperbolic layer formulations. On ProcGen benchmarks, HYPER++ achieves more stable learning, better performance, and 30% faster training than previous hyperbolic RL methods.", "questions": "- Since each component of HYPER++ (RMSNorm, scaling, categorical loss, hyperboloid model) are not fully novel, what new insight does their combination provide beyond improved stability? In other words, is there theoretical justification for why these specific components interact synergistically, or was the design largely empirical?\n- Atari-3 and only 5M training steps is a very limited setup for the Atari benchmark. How do you know that it is representative of a larger set of environments (e.g. all ATARI games) and that the first 5M steps are representative of a longer training?\n- How sensitive are the results to hyperparameter choices (e.g. NN architecture, etc.)\n- The ablation results for HYPER++ in Table 1 are only reported on the 3 games with the highest improvements with the proposed architecture.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well-written and combines interesting ideas\n- There is a quite strong theoretical section", "weaknesses": "None of the idea alone is truly novel and only the combination of several ideas. I therefore see the paper as largely empirical in scope. There is a 30% improvement on procgen as well as small improvements in 3 Atari games. However, the experimental setup is not fully convincing:\n- relatively limited scope\n- the interpretation of the improvements in practice are not fully clear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pcTdXThExO", "reviewer": "Reviewer_Pwnf", "summary": "The paper proposes an extension of the Hyperbolic Deep RL (HyperRL) framework from [1]. In particular, the paper analyzes the instabilities in prior hyperRL methods and proposes several improvements to counteract them. These improvements come together in the  Hyper++ algorithm, which extends the implementation from [1] with a categorical value function, a feature-scaled RMSNorm layer, and switching from the Poincaré to the Hyperboloid model as the space for the latent features. Empirically, the proposed method is validated on the Procgen benchmark [2], and a subset of three environments from the Atari benchmark [3].\n\n\n[1] Cetin, Edoardo, et al. \"Hyperbolic deep reinforcement learning.\" arXiv preprint arXiv:2210.01542 (2022).\n\n[2] Cobbe, Karl, et al. \"Leveraging procedural generation to benchmark reinforcement learning.\" International conference on machine learning. PMLR, 2020.\n\n[3] Bellemare, Marc G., et al. \"The arcade learning environment: An evaluation platform for general agents.\" Journal of artificial intelligence research 47 (2013): 253-279.", "questions": "I have raised what I believe to be the main areas for improvement of the paper in the Weaknesses Section above. I would encourage the authors to address these aspects, and have no further questions at this stage.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- I found the overall exposition of the paper to be clear and intuitive. Clear text boxes are used to highlight the main methodology aspects, which are introduced naturally within the text.  While hyperbolic geometry for ML could be a challenging topic for some readers, I believe the paper does a good job at summarizing past results and providing analysis without overdoing the theoretical complexity.\n- Each component of the proposed extensions seems logical and introduced from grounded considerations on the challenges of applying hyporbolic ML to reinforcement learning.\n- The paper does a good job of introducing itself within the surrounding literature of RL and hyperbolic deep learning.", "weaknesses": "**Main:**\nMy main area of concern is the empirical validation of the paper. The proposed Hyper++ algorithm has only been properly examined on Procgen on top of vanilla PPO. The generality of the method is yet to be validated:\n- Since its inception, several improvements upon PPO have been proposed for Procgen. It would have been nice to test if Hyper++ could also be applied to improve the performance of any more recent baselines from the last couple of years.\n- The evaluation for DDQN seems to be constrained to only three handpicked environments from the whole suite. Not only is this too limited to provide evidence of generality, but it is also using a very outdated baseline in DDQN (9+ years old), which is far from state-of-the-art in sample efficiency.\nWithout empirical validation to potentially establish the methodology, I am unsure how much the proposed simple extension to the existing hyperRL framework would be relevant to the broader ICLR RL community.\n\n**Other:**\n1) The ablation study is only focused on three Procgen environments, rather than looking at average normalized performance across the Procgen suite, providing limited insights. Moreover, I think the ablation for RMSNorm should have added back the SpectralNorm introduced in the original HyperRL work. \n2) The authors recollect their own numbers for the HyperRL baseline, which appear consistent but slightly inferior to the original results reported in this prior work. Yet, even with these baseline numbers, performance improvements of the proposed method on Procgen appear far from consistent (improving in only 8/16 and 11/16 train/test settings, respectively).\n3) The paper lacks any analysis on what is the actual effect of each of their proposed addition and their effect on the learned latent representations, which is the critical main aspect of adding hyperbolic space to RL.\n4) While it is good that some code was shared with the submission, it is currently far from allowing/facilitating reproducibility for the reviewers and the public. For instance, the README is empty without installation/execution instructions. I quickly did some checks trying to install the dependencies with uv and launching the .\"sh\" files, but was not able to get experiments started. I think for the code to be actually useful to reviewers, a simple README with instructions should have been provided at the time of submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)